Jul 10 02:48:01.590124 ip-10-0-130-139 systemd[1]: Starting Kubernetes Kubelet... Jul 10 02:48:02.185431 ip-10-0-130-139 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 02:48:02.185431 ip-10-0-130-139 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 10 02:48:02.185431 ip-10-0-130-139 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 02:48:02.185431 ip-10-0-130-139 kubenswrapper[2575]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 10 02:48:02.185431 ip-10-0-130-139 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 02:48:02.185431 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.083888 2575 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 10 02:48:02.185431 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.090590 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 02:48:02.185431 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.090605 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 02:48:02.185431 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.090609 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 02:48:02.185431 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.090613 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 02:48:02.185431 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.090616 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 02:48:02.185431 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.090620 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 02:48:02.185431 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.090622 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 02:48:02.185431 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.090625 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 02:48:02.161639 ip-10-0-130-139 systemd[1]: Started Kubernetes Kubelet. Jul 10 02:48:02.187361 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.090630 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 02:48:02.187361 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.090634 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 02:48:02.187361 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.090636 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 02:48:02.187361 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.090639 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 02:48:02.187361 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.090642 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 02:48:02.187361 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.090645 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 02:48:02.187361 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.090648 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 02:48:02.187361 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.090650 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 02:48:02.187361 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.090653 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 02:48:02.187361 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.090657 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 02:48:02.187361 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.090661 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 10 02:48:02.187361 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.090666 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 02:48:02.187361 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.090669 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 02:48:02.187361 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.090672 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 02:48:02.187361 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.090675 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 02:48:02.187361 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.090677 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 02:48:02.187361 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.090680 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 02:48:02.187361 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.090682 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 02:48:02.187361 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.090685 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 02:48:02.189918 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.090688 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 02:48:02.189918 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.090691 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 02:48:02.189918 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.091799 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 02:48:02.189918 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.091827 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 02:48:02.189918 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.091831 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 02:48:02.189918 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.091835 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 02:48:02.189918 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.091838 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 02:48:02.189918 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.091841 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 02:48:02.189918 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.091844 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 02:48:02.189918 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.091847 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 02:48:02.189918 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.091851 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 02:48:02.189918 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.091854 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 02:48:02.189918 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.091857 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 02:48:02.189918 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.091859 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 02:48:02.189918 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.091862 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 02:48:02.189918 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.091869 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 02:48:02.189918 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.091871 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 02:48:02.189918 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.091874 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 02:48:02.189918 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.091877 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 02:48:02.189918 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.091880 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 02:48:02.191660 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.091883 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 02:48:02.191660 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.091886 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 02:48:02.191660 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.091889 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 02:48:02.191660 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.091892 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 02:48:02.191660 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.091895 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 02:48:02.191660 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.091909 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 02:48:02.191660 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.091911 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 02:48:02.191660 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.091917 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 02:48:02.191660 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.091920 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 02:48:02.191660 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.091923 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 02:48:02.191660 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.091926 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 02:48:02.191660 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.091929 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 02:48:02.191660 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.091932 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 02:48:02.191660 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.091935 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 02:48:02.191660 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.091941 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 02:48:02.191660 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.091944 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 02:48:02.191660 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.091947 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 02:48:02.191660 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.091950 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 02:48:02.191660 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.091953 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 02:48:02.191660 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.091960 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 02:48:02.193733 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.091963 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 02:48:02.193733 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.091966 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 02:48:02.193733 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.091971 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 02:48:02.193733 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.091975 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 02:48:02.193733 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.091979 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 02:48:02.193733 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.091982 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 02:48:02.193733 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.091985 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 02:48:02.193733 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.091988 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 02:48:02.193733 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.091991 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 02:48:02.193733 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.091994 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 02:48:02.193733 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.091997 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 02:48:02.193733 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.092003 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 02:48:02.193733 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.092006 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 02:48:02.193733 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.092008 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 02:48:02.193733 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.092012 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 02:48:02.193733 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.092015 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 02:48:02.193733 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.092019 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 02:48:02.193733 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.092022 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 02:48:02.193733 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.092025 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 02:48:02.193733 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.092029 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 02:48:02.195454 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.092032 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 02:48:02.195454 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.092035 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 02:48:02.195454 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.092038 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 02:48:02.195454 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.092043 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 02:48:02.195454 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.092046 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 02:48:02.195454 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.092049 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 02:48:02.195454 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.092052 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 02:48:02.195454 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.092055 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 02:48:02.195454 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.092058 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 02:48:02.195454 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.092061 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 02:48:02.195454 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.092063 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 02:48:02.195454 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.092066 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 02:48:02.195454 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093236 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 02:48:02.195454 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093243 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 02:48:02.195454 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093245 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 02:48:02.195454 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093249 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 02:48:02.195454 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093252 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 02:48:02.195454 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093256 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 02:48:02.195454 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093261 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 02:48:02.195454 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093282 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 02:48:02.196301 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093287 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 02:48:02.196301 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093290 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 02:48:02.196301 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093293 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 02:48:02.196301 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093296 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 02:48:02.196301 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093299 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 02:48:02.196301 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093303 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 02:48:02.196301 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093307 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 02:48:02.196301 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093310 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 02:48:02.196301 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093313 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 02:48:02.196301 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093316 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 02:48:02.196301 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093322 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 02:48:02.196301 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093325 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 02:48:02.196301 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093328 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 02:48:02.196301 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093331 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 02:48:02.196301 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093333 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 02:48:02.196301 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093336 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 02:48:02.196301 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093340 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 02:48:02.196301 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093343 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 02:48:02.196301 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093346 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 02:48:02.196301 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093349 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 02:48:02.196934 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093352 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 02:48:02.196934 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093355 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 02:48:02.196934 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093360 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 02:48:02.196934 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093363 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 02:48:02.196934 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093366 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 02:48:02.196934 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093368 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 02:48:02.196934 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093371 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 02:48:02.196934 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093374 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 02:48:02.196934 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093376 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 02:48:02.196934 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093379 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 02:48:02.196934 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093383 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 02:48:02.196934 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093386 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 02:48:02.196934 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093389 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 02:48:02.196934 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093391 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 02:48:02.196934 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093397 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 02:48:02.196934 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093399 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 02:48:02.196934 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093402 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 02:48:02.196934 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093404 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 02:48:02.196934 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093407 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 02:48:02.197448 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093410 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 02:48:02.197448 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093412 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 02:48:02.197448 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093415 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 02:48:02.197448 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093418 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 02:48:02.197448 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093421 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 10 02:48:02.197448 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093424 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 02:48:02.197448 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093427 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 02:48:02.197448 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093430 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 02:48:02.197448 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093435 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 02:48:02.197448 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093438 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 02:48:02.197448 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093440 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 02:48:02.197448 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093443 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 02:48:02.197448 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093447 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 02:48:02.197448 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093449 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 02:48:02.197448 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093452 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 02:48:02.197448 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093455 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 02:48:02.197448 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093458 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 02:48:02.197448 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093460 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 02:48:02.197448 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093464 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 02:48:02.198136 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093468 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 02:48:02.198136 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093473 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 02:48:02.198136 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093477 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 02:48:02.198136 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093480 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 02:48:02.198136 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093483 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 02:48:02.198136 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093486 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 02:48:02.198136 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093489 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 02:48:02.198136 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093494 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 02:48:02.198136 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093498 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 02:48:02.198136 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093501 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 02:48:02.198136 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093504 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 02:48:02.198136 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093507 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 02:48:02.198136 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093510 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 02:48:02.198136 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093516 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 02:48:02.198136 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093519 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 02:48:02.198136 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093522 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 02:48:02.198136 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093525 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 02:48:02.198136 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093528 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 02:48:02.198136 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093531 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 02:48:02.198599 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093534 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 02:48:02.198599 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093537 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 02:48:02.198599 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093539 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 02:48:02.198599 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093542 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 02:48:02.198599 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093545 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 02:48:02.198599 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093548 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 02:48:02.198599 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093553 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 02:48:02.198599 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093555 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 02:48:02.198599 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093558 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 02:48:02.198599 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093561 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 02:48:02.198599 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093564 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 02:48:02.198599 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093566 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 02:48:02.198599 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093569 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 02:48:02.198599 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.093572 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 02:48:02.198599 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094270 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 02:48:02.198599 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094278 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 02:48:02.198599 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094281 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 02:48:02.198599 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094284 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 02:48:02.198599 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094288 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 02:48:02.198599 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094291 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 02:48:02.199211 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094294 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 02:48:02.199211 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094297 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 02:48:02.199211 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094301 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 02:48:02.199211 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094304 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 02:48:02.199211 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094307 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 02:48:02.199211 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094310 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 02:48:02.199211 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094317 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 02:48:02.199211 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094319 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 02:48:02.199211 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094323 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 02:48:02.199211 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094326 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 02:48:02.199211 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094329 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 02:48:02.199211 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094331 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 02:48:02.199211 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094335 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 02:48:02.199211 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094337 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 02:48:02.199211 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094340 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 02:48:02.199211 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094343 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 02:48:02.199211 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094345 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 02:48:02.199211 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094348 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 02:48:02.199211 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094351 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 02:48:02.199988 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094356 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 02:48:02.199988 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094358 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 02:48:02.199988 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094361 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 02:48:02.199988 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094364 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 02:48:02.199988 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094368 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 02:48:02.199988 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094371 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 02:48:02.199988 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094377 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 02:48:02.199988 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094380 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 02:48:02.199988 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094383 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 02:48:02.199988 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094385 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 02:48:02.199988 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094388 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 02:48:02.199988 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094391 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 02:48:02.199988 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094396 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 02:48:02.199988 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094399 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 02:48:02.199988 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094401 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 02:48:02.199988 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094406 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 02:48:02.199988 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094409 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 02:48:02.199988 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094412 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 02:48:02.199988 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094414 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 02:48:02.199988 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094417 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 02:48:02.200537 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094420 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 02:48:02.200537 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094422 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 02:48:02.200537 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094426 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 02:48:02.200537 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094429 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 02:48:02.200537 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094431 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 02:48:02.200537 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094436 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 02:48:02.200537 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094439 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 02:48:02.200537 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094442 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 02:48:02.200537 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094445 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 02:48:02.200537 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094447 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 02:48:02.200537 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094450 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 02:48:02.200537 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094453 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 02:48:02.200537 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094455 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 02:48:02.200537 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094458 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 02:48:02.200537 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094460 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 02:48:02.200537 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094463 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 02:48:02.200537 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094466 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 02:48:02.200537 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094471 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 02:48:02.200537 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094475 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 02:48:02.200537 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094478 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 02:48:02.201162 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094481 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 02:48:02.201162 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094484 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 02:48:02.201162 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094486 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 02:48:02.201162 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094489 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 02:48:02.201162 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094492 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 02:48:02.201162 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094494 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 02:48:02.201162 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094497 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 02:48:02.201162 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094501 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 02:48:02.201162 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094504 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 02:48:02.201162 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094507 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 02:48:02.201162 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094512 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 02:48:02.201162 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094514 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 02:48:02.201162 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094517 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 02:48:02.201162 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094520 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 02:48:02.201162 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094522 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 02:48:02.201162 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094526 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 10 02:48:02.201162 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094529 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 02:48:02.201162 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094531 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 02:48:02.201162 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094534 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 02:48:02.201162 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094541 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 02:48:02.201775 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094544 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 02:48:02.201775 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094549 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 02:48:02.201775 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094551 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 02:48:02.201775 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094554 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 02:48:02.201775 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094557 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 02:48:02.201775 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094561 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 02:48:02.201775 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094565 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 02:48:02.201775 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094567 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 02:48:02.201775 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094570 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 02:48:02.201775 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094573 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 02:48:02.201775 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094577 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 02:48:02.201775 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094580 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 02:48:02.201775 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094583 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 02:48:02.201775 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.094585 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 02:48:02.201775 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.094956 2575 flags.go:64] FLAG: --address="0.0.0.0" Jul 10 02:48:02.201775 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.094982 2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 10 02:48:02.201775 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.094991 2575 flags.go:64] FLAG: --anonymous-auth="true" Jul 10 02:48:02.201775 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.094996 2575 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 10 02:48:02.201775 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095000 2575 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 10 02:48:02.201775 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095003 2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 10 02:48:02.202347 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095008 2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 10 02:48:02.202347 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095013 2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 10 02:48:02.202347 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095016 2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 10 02:48:02.202347 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095020 2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 10 02:48:02.202347 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095023 2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 10 02:48:02.202347 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095027 2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 10 02:48:02.202347 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095030 2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 10 02:48:02.202347 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095033 2575 flags.go:64] FLAG: --cgroup-root="" Jul 10 02:48:02.202347 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095036 2575 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 10 02:48:02.202347 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095039 2575 flags.go:64] FLAG: --client-ca-file="" Jul 10 02:48:02.202347 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095042 2575 flags.go:64] FLAG: --cloud-provider="external" Jul 10 02:48:02.202347 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095045 2575 flags.go:64] FLAG: --cluster-dns="[]" Jul 10 02:48:02.202347 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095050 2575 flags.go:64] FLAG: --cluster-domain="" Jul 10 02:48:02.202347 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095054 2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 10 02:48:02.202347 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095057 2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 10 02:48:02.202347 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095060 2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 10 02:48:02.202347 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095063 2575 flags.go:64] FLAG: --container-log-max-files="5" Jul 10 02:48:02.202347 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095067 2575 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 10 02:48:02.202347 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095070 2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 10 02:48:02.202347 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095073 2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 10 02:48:02.202347 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095077 2575 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 10 02:48:02.202347 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095080 2575 flags.go:64] FLAG: --contention-profiling="false" Jul 10 02:48:02.202347 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095082 2575 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 10 02:48:02.202347 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095087 2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 10 02:48:02.202974 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095093 2575 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 10 02:48:02.202974 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095098 2575 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 10 02:48:02.202974 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095102 2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 10 02:48:02.202974 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095107 2575 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 10 02:48:02.202974 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095110 2575 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 10 02:48:02.202974 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095113 2575 flags.go:64] FLAG: --enable-load-reader="false" Jul 10 02:48:02.202974 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095117 2575 flags.go:64] FLAG: --enable-server="true" Jul 10 02:48:02.202974 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095120 2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 10 02:48:02.202974 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095129 2575 flags.go:64] FLAG: --event-burst="100" Jul 10 02:48:02.202974 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095132 2575 flags.go:64] FLAG: --event-qps="50" Jul 10 02:48:02.202974 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095135 2575 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 10 02:48:02.202974 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095138 2575 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 10 02:48:02.202974 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095141 2575 flags.go:64] FLAG: --eviction-hard="" Jul 10 02:48:02.202974 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095145 2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 10 02:48:02.202974 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095148 2575 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 10 02:48:02.202974 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095151 2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 10 02:48:02.202974 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095154 2575 flags.go:64] FLAG: --eviction-soft="" Jul 10 02:48:02.202974 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095157 2575 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 10 02:48:02.202974 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095160 2575 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 10 02:48:02.202974 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095163 2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 10 02:48:02.202974 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095166 2575 flags.go:64] FLAG: --experimental-mounter-path="" Jul 10 02:48:02.202974 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095169 2575 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 10 02:48:02.202974 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095172 2575 flags.go:64] FLAG: --fail-swap-on="true" Jul 10 02:48:02.202974 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095175 2575 flags.go:64] FLAG: --feature-gates="" Jul 10 02:48:02.202974 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095179 2575 flags.go:64] FLAG: --file-check-frequency="20s" Jul 10 02:48:02.203789 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095182 2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 10 02:48:02.203789 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095185 2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 10 02:48:02.203789 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095188 2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 10 02:48:02.203789 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095191 2575 flags.go:64] FLAG: --healthz-port="10248" Jul 10 02:48:02.203789 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095194 2575 flags.go:64] FLAG: --help="false" Jul 10 02:48:02.203789 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095197 2575 flags.go:64] FLAG: --hostname-override="ip-10-0-130-139.ec2.internal" Jul 10 02:48:02.203789 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095200 2575 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 10 02:48:02.203789 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095204 2575 flags.go:64] FLAG: --http-check-frequency="20s" Jul 10 02:48:02.203789 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095207 2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 10 02:48:02.203789 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095211 2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 10 02:48:02.203789 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095215 2575 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 10 02:48:02.203789 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095218 2575 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 10 02:48:02.203789 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095220 2575 flags.go:64] FLAG: --image-service-endpoint="" Jul 10 02:48:02.203789 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095223 2575 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 10 02:48:02.203789 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095226 2575 flags.go:64] FLAG: --kube-api-burst="100" Jul 10 02:48:02.203789 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095229 2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 10 02:48:02.203789 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095233 2575 flags.go:64] FLAG: --kube-api-qps="50" Jul 10 02:48:02.203789 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095236 2575 flags.go:64] FLAG: --kube-reserved="" Jul 10 02:48:02.203789 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095239 2575 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 10 02:48:02.203789 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095242 2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 10 02:48:02.203789 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095245 2575 flags.go:64] FLAG: --kubelet-cgroups="" Jul 10 02:48:02.203789 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095248 2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 10 02:48:02.203789 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095250 2575 flags.go:64] FLAG: --lock-file="" Jul 10 02:48:02.203789 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095253 2575 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 10 02:48:02.204394 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095256 2575 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 10 02:48:02.204394 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095259 2575 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 10 02:48:02.204394 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095264 2575 flags.go:64] FLAG: --log-json-split-stream="false" Jul 10 02:48:02.204394 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095267 2575 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 10 02:48:02.204394 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095270 2575 flags.go:64] FLAG: --log-text-split-stream="false" Jul 10 02:48:02.204394 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095273 2575 flags.go:64] FLAG: --logging-format="text" Jul 10 02:48:02.204394 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095276 2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 10 02:48:02.204394 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095279 2575 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 10 02:48:02.204394 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095282 2575 flags.go:64] FLAG: --manifest-url="" Jul 10 02:48:02.204394 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095285 2575 flags.go:64] FLAG: --manifest-url-header="" Jul 10 02:48:02.204394 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095290 2575 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 10 02:48:02.204394 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095293 2575 flags.go:64] FLAG: --max-open-files="1000000" Jul 10 02:48:02.204394 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095297 2575 flags.go:64] FLAG: --max-pods="110" Jul 10 02:48:02.204394 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095300 2575 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 10 02:48:02.204394 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095303 2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 10 02:48:02.204394 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095306 2575 flags.go:64] FLAG: --memory-manager-policy="None" Jul 10 02:48:02.204394 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095310 2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 10 02:48:02.204394 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095313 2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 10 02:48:02.204394 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095316 2575 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 10 02:48:02.204394 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095319 2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 10 02:48:02.204394 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095327 2575 flags.go:64] FLAG: --node-status-max-images="50" Jul 10 02:48:02.204394 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095330 2575 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 10 02:48:02.204394 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095333 2575 flags.go:64] FLAG: --oom-score-adj="-999" Jul 10 02:48:02.204394 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095337 2575 flags.go:64] FLAG: --pod-cidr="" Jul 10 02:48:02.204993 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095340 2575 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 10 02:48:02.204993 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095346 2575 flags.go:64] FLAG: --pod-manifest-path="" Jul 10 02:48:02.204993 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095348 2575 flags.go:64] FLAG: --pod-max-pids="-1" Jul 10 02:48:02.204993 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095352 2575 flags.go:64] FLAG: --pods-per-core="0" Jul 10 02:48:02.204993 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095355 2575 flags.go:64] FLAG: --port="10250" Jul 10 02:48:02.204993 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095358 2575 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 10 02:48:02.204993 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095361 2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02a2e9996669fd689" Jul 10 02:48:02.204993 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095364 2575 flags.go:64] FLAG: --qos-reserved="" Jul 10 02:48:02.204993 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095368 2575 flags.go:64] FLAG: --read-only-port="10255" Jul 10 02:48:02.204993 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095371 2575 flags.go:64] FLAG: --register-node="true" Jul 10 02:48:02.204993 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095374 2575 flags.go:64] FLAG: --register-with-taints="" Jul 10 02:48:02.204993 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095377 2575 flags.go:64] FLAG: --registry-burst="10" Jul 10 02:48:02.204993 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095380 2575 flags.go:64] FLAG: --registry-qps="5" Jul 10 02:48:02.204993 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095383 2575 flags.go:64] FLAG: --reserved-cpus="" Jul 10 02:48:02.204993 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095386 2575 flags.go:64] FLAG: --reserved-memory="" Jul 10 02:48:02.204993 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095389 2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 10 02:48:02.204993 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095392 2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 10 02:48:02.204993 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095395 2575 flags.go:64] FLAG: --rotate-certificates="false" Jul 10 02:48:02.204993 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095398 2575 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 10 02:48:02.204993 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095401 2575 flags.go:64] FLAG: --runonce="false" Jul 10 02:48:02.204993 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095404 2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 10 02:48:02.204993 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095407 2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 10 02:48:02.204993 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095410 2575 flags.go:64] FLAG: --seccomp-default="false" Jul 10 02:48:02.204993 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095416 2575 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 10 02:48:02.204993 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095422 2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 10 02:48:02.205614 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095425 2575 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 10 02:48:02.205614 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095429 2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 10 02:48:02.205614 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095432 2575 flags.go:64] FLAG: --storage-driver-password="root" Jul 10 02:48:02.205614 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095435 2575 flags.go:64] FLAG: --storage-driver-secure="false" Jul 10 02:48:02.205614 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095438 2575 flags.go:64] FLAG: --storage-driver-table="stats" Jul 10 02:48:02.205614 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095440 2575 flags.go:64] FLAG: --storage-driver-user="root" Jul 10 02:48:02.205614 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095443 2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 10 02:48:02.205614 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095446 2575 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 10 02:48:02.205614 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095449 2575 flags.go:64] FLAG: --system-cgroups="" Jul 10 02:48:02.205614 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095452 2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 10 02:48:02.205614 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095458 2575 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 10 02:48:02.205614 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095461 2575 flags.go:64] FLAG: --tls-cert-file="" Jul 10 02:48:02.205614 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095464 2575 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 10 02:48:02.205614 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095469 2575 flags.go:64] FLAG: --tls-min-version="" Jul 10 02:48:02.205614 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095472 2575 flags.go:64] FLAG: --tls-private-key-file="" Jul 10 02:48:02.205614 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095474 2575 flags.go:64] FLAG: --topology-manager-policy="none" Jul 10 02:48:02.205614 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095478 2575 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 10 02:48:02.205614 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095481 2575 flags.go:64] FLAG: --topology-manager-scope="container" Jul 10 02:48:02.205614 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095484 2575 flags.go:64] FLAG: --v="2" Jul 10 02:48:02.205614 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095488 2575 flags.go:64] FLAG: --version="false" Jul 10 02:48:02.205614 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095492 2575 flags.go:64] FLAG: --vmodule="" Jul 10 02:48:02.205614 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095496 2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 10 02:48:02.205614 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095499 2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 10 02:48:02.205614 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095638 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 02:48:02.205614 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095643 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 02:48:02.206385 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095646 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 02:48:02.206385 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095649 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 02:48:02.206385 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095652 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 02:48:02.206385 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095656 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 02:48:02.206385 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095660 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 02:48:02.206385 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095668 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 02:48:02.206385 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095672 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 02:48:02.206385 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095675 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 02:48:02.206385 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095678 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 02:48:02.206385 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095682 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 02:48:02.206385 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095684 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 02:48:02.206385 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095687 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 02:48:02.206385 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095690 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 02:48:02.206385 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095692 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 02:48:02.206385 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095695 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 02:48:02.206385 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095699 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 02:48:02.206385 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095703 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 02:48:02.206385 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095706 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 02:48:02.206385 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095708 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 02:48:02.207014 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095711 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 02:48:02.207014 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095714 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 02:48:02.207014 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095716 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 02:48:02.207014 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095719 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 02:48:02.207014 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095721 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 02:48:02.207014 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095725 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 02:48:02.207014 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095727 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 02:48:02.207014 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095730 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 02:48:02.207014 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095733 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 02:48:02.207014 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095735 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 02:48:02.207014 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095738 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 02:48:02.207014 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095741 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 02:48:02.207014 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095743 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 02:48:02.207014 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095746 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 02:48:02.207014 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095749 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 02:48:02.207014 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095751 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 02:48:02.207014 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095754 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 02:48:02.207014 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095757 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 02:48:02.207014 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095761 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 02:48:02.207014 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095765 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 02:48:02.207485 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095767 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 02:48:02.207485 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095770 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 02:48:02.207485 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095773 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 02:48:02.207485 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095776 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 02:48:02.207485 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095778 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 02:48:02.207485 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095781 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 02:48:02.207485 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095783 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 02:48:02.207485 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095786 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 02:48:02.207485 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095788 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 02:48:02.207485 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095791 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 02:48:02.207485 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095793 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 02:48:02.207485 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095796 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 02:48:02.207485 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095799 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 02:48:02.207485 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095801 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 02:48:02.207485 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095804 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 02:48:02.207485 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095821 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 02:48:02.207485 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095823 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 02:48:02.207485 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095826 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 02:48:02.207485 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095829 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 02:48:02.207485 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095832 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 02:48:02.208014 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095834 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 02:48:02.208014 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095837 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 02:48:02.208014 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095840 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 02:48:02.208014 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095842 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 02:48:02.208014 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095845 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 02:48:02.208014 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095847 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 02:48:02.208014 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095850 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 02:48:02.208014 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095853 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 10 02:48:02.208014 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095855 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 02:48:02.208014 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095858 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 02:48:02.208014 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095861 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 02:48:02.208014 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095867 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 02:48:02.208014 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095870 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 02:48:02.208014 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095873 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 02:48:02.208014 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095876 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 02:48:02.208014 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095880 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 02:48:02.208014 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095883 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 02:48:02.208014 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095886 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 02:48:02.208014 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095889 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 02:48:02.208552 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095892 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 02:48:02.208552 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095895 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 02:48:02.208552 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095898 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 02:48:02.208552 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095901 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 02:48:02.208552 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095904 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 02:48:02.208552 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095907 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 02:48:02.208552 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095909 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 02:48:02.208552 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095912 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 02:48:02.208552 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095914 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 02:48:02.208552 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095917 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 02:48:02.208552 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095919 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 02:48:02.208552 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095922 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 02:48:02.208552 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095925 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 02:48:02.208552 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095927 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 02:48:02.208552 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095930 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 02:48:02.208552 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095933 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 02:48:02.208552 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095935 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 02:48:02.208552 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095937 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 02:48:02.208552 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.095940 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 02:48:02.209048 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.095946 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 02:48:02.209048 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.103104 2575 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 10 02:48:02.209048 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.103121 2575 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 10 02:48:02.209048 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.103143 2575 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 10 02:48:02.209048 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.103149 2575 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 10 02:48:02.209048 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103204 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 02:48:02.209048 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103212 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 02:48:02.209048 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103216 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 02:48:02.209048 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103219 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 02:48:02.209048 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103222 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 02:48:02.209048 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103225 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 02:48:02.209048 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103228 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 02:48:02.209048 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103230 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 02:48:02.209048 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103234 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 02:48:02.209048 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103237 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 02:48:02.209416 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103239 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 02:48:02.209416 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103242 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 02:48:02.209416 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103245 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 02:48:02.209416 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103248 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 02:48:02.209416 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103250 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 02:48:02.209416 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103253 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 02:48:02.209416 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103256 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 02:48:02.209416 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103259 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 02:48:02.209416 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103262 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 02:48:02.209416 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103264 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 02:48:02.209416 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103267 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 02:48:02.209416 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103269 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 02:48:02.209416 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103272 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 02:48:02.209416 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103274 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 02:48:02.209416 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103279 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 02:48:02.209416 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103281 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 02:48:02.209416 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103284 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 02:48:02.209416 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103286 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 02:48:02.209416 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103289 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 02:48:02.209950 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103293 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 02:48:02.209950 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103297 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 02:48:02.209950 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103300 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 02:48:02.209950 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103303 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 02:48:02.209950 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103307 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 02:48:02.209950 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103310 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 02:48:02.209950 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103313 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 02:48:02.209950 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103316 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 02:48:02.209950 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103318 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 02:48:02.209950 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103321 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 02:48:02.209950 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103323 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 02:48:02.209950 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103326 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 02:48:02.209950 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103328 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 02:48:02.209950 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103331 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 02:48:02.209950 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103333 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 02:48:02.209950 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103336 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 02:48:02.209950 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103339 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 02:48:02.209950 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103342 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 02:48:02.209950 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103344 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 02:48:02.210412 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103347 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 02:48:02.210412 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103349 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 02:48:02.210412 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103352 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 02:48:02.210412 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103354 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 02:48:02.210412 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103357 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 02:48:02.210412 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103359 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 02:48:02.210412 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103362 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 02:48:02.210412 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103364 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 02:48:02.210412 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103368 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 02:48:02.210412 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103370 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 02:48:02.210412 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103373 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 02:48:02.210412 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103375 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 02:48:02.210412 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103378 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 02:48:02.210412 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103381 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 02:48:02.210412 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103384 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 02:48:02.210412 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103387 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 02:48:02.210412 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103390 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 02:48:02.210412 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103396 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 02:48:02.210412 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103398 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 02:48:02.210412 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103401 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 02:48:02.211018 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103403 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 02:48:02.211018 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103406 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 02:48:02.211018 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103408 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 02:48:02.211018 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103411 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 02:48:02.211018 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103413 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 02:48:02.211018 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103416 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 02:48:02.211018 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103418 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 02:48:02.211018 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103421 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 02:48:02.211018 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103423 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 02:48:02.211018 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103426 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 02:48:02.211018 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103429 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 02:48:02.211018 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103431 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 02:48:02.211018 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103434 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 02:48:02.211018 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103437 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 02:48:02.211018 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103440 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 02:48:02.211018 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103444 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 02:48:02.211018 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103447 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 02:48:02.211018 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103450 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 02:48:02.211018 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103452 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 02:48:02.211018 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103455 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 10 02:48:02.211499 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103459 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 02:48:02.211499 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103461 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 02:48:02.211499 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103464 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 02:48:02.211499 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103467 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 02:48:02.211499 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103469 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 02:48:02.211499 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103471 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 02:48:02.211499 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103474 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 02:48:02.211499 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103477 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 02:48:02.211499 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103479 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 02:48:02.211499 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103484 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 02:48:02.211499 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103488 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 02:48:02.211499 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.103493 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 02:48:02.211499 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103619 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 02:48:02.211499 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103625 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 02:48:02.211958 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103628 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 02:48:02.211958 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103631 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 02:48:02.211958 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103634 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 02:48:02.211958 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103637 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 02:48:02.211958 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103640 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 02:48:02.211958 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103642 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 02:48:02.211958 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103645 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 02:48:02.211958 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103648 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 02:48:02.211958 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103650 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 02:48:02.211958 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103654 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 02:48:02.211958 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103658 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 02:48:02.211958 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103661 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 02:48:02.211958 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103664 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 02:48:02.211958 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103667 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 02:48:02.211958 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103669 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 02:48:02.211958 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103672 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 02:48:02.211958 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103675 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 02:48:02.211958 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103677 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 02:48:02.211958 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103681 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 02:48:02.212736 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103684 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 02:48:02.212736 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103686 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 02:48:02.212736 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103689 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 02:48:02.212736 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103692 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 02:48:02.212736 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103696 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 02:48:02.212736 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103699 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 02:48:02.212736 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103702 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 02:48:02.212736 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103705 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 02:48:02.212736 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103708 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 02:48:02.212736 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103712 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 02:48:02.212736 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103714 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 02:48:02.212736 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103717 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 02:48:02.212736 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103720 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 02:48:02.212736 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103722 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 02:48:02.212736 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103725 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 02:48:02.212736 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103728 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 02:48:02.212736 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103730 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 02:48:02.212736 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103733 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 02:48:02.212736 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103735 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 02:48:02.212736 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103738 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 02:48:02.213517 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103740 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 02:48:02.213517 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103743 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 02:48:02.213517 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103745 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 02:48:02.213517 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103748 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 02:48:02.213517 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103750 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 02:48:02.213517 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103753 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 02:48:02.213517 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103755 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 10 02:48:02.213517 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103758 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 02:48:02.213517 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103761 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 02:48:02.213517 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103763 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 02:48:02.213517 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103766 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 02:48:02.213517 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103770 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 02:48:02.213517 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103772 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 02:48:02.213517 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103775 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 02:48:02.213517 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103777 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 02:48:02.213517 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103780 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 02:48:02.213517 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103782 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 02:48:02.213517 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103785 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 02:48:02.213517 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103787 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 02:48:02.213517 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103790 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 02:48:02.214062 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103793 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 02:48:02.214062 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103797 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 02:48:02.214062 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103799 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 02:48:02.214062 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103802 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 02:48:02.214062 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103820 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 02:48:02.214062 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103823 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 02:48:02.214062 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103825 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 02:48:02.214062 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103828 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 02:48:02.214062 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103830 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 02:48:02.214062 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103833 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 02:48:02.214062 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103836 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 02:48:02.214062 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103838 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 02:48:02.214062 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103841 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 02:48:02.214062 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103844 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 02:48:02.214062 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103847 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 02:48:02.214062 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103849 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 02:48:02.214062 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103852 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 02:48:02.214062 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103854 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 02:48:02.214062 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103857 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 02:48:02.214062 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103860 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 02:48:02.214544 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103863 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 02:48:02.214544 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103865 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 02:48:02.214544 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103868 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 02:48:02.214544 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103872 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 02:48:02.214544 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103874 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 02:48:02.214544 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103877 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 02:48:02.214544 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103880 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 02:48:02.214544 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103882 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 02:48:02.214544 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103885 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 02:48:02.214544 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103887 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 02:48:02.214544 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103891 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 02:48:02.214544 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103895 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 02:48:02.214544 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103897 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 02:48:02.214544 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103902 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 02:48:02.214544 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103905 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 02:48:02.214544 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103908 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 02:48:02.214544 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103911 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 02:48:02.214544 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:02.103914 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 02:48:02.215049 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.103920 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 02:48:02.215049 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.104033 2575 server.go:962] "Client rotation is on, will bootstrap in background" Jul 10 02:48:02.215049 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.106407 2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 10 02:48:02.215049 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.108028 2575 server.go:1019] "Starting client certificate rotation" Jul 10 02:48:02.215049 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.108134 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 02:48:02.215049 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.108176 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 02:48:02.215049 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.110863 2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 02:48:02.215049 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.112044 2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 02:48:02.215049 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.123365 2575 log.go:25] "Validated CRI v1 runtime API" Jul 10 02:48:02.215049 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.128187 2575 log.go:25] "Validated CRI v1 image API" Jul 10 02:48:02.215049 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.129941 2575 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 10 02:48:02.215049 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.132503 2575 fs.go:135] Filesystem UUIDs: map[57eb9c8f-4779-4bdc-a98c-2d31355a279a:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 e3d7d63d-47a8-4507-9d78-f844f6396e79:/dev/nvme0n1p4] Jul 10 02:48:02.215359 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.132518 2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 10 02:48:02.215359 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.138281 2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 02:48:02.215416 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.138561 2575 manager.go:216] Machine: {Timestamp:2026-07-10 02:48:02.137193114 +0000 UTC m=+0.418309391 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101331 MemoryCapacity:33164505088 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec285a5e66d87a39363cdb1e85124c87 SystemUUID:ec285a5e-66d8-7a39-363c-db1e85124c87 BootID:3fa3d4a0-922b-4ee8-831b-0845c4c8cabd Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582254592 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:87:f9:a4:73:17 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:87:f9:a4:73:17 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2a:d4:1e:b8:5a:b3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164505088 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 10 02:48:02.215416 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.138662 2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 10 02:48:02.215416 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.138790 2575 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 10 02:48:02.215416 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.142235 2575 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 10 02:48:02.215416 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.142272 2575 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-139.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 10 02:48:02.215416 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.143173 2575 topology_manager.go:138] "Creating topology manager with none policy" Jul 10 02:48:02.215416 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.143186 2575 container_manager_linux.go:309] "Creating device plugin manager" Jul 10 02:48:02.215416 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.143196 2575 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 02:48:02.215416 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.143214 2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 02:48:02.215416 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.145118 2575 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 10 02:48:02.215416 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.149788 2575 state_mem.go:36] "Initialized new in-memory state store" Jul 10 02:48:02.215416 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.149950 2575 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 10 02:48:02.215416 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.149986 2575 kubelet.go:486] "Attempting to sync node with API server" Jul 10 02:48:02.215416 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.149998 2575 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 10 02:48:02.215416 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.150015 2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 10 02:48:02.215416 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.150028 2575 kubelet.go:398] "Adding apiserver pod source" Jul 10 02:48:02.215416 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.150041 2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 10 02:48:02.215416 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.150456 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 02:48:02.215916 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.150474 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 02:48:02.215916 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.152974 2575 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 10 02:48:02.215916 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.155548 2575 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 10 02:48:02.215916 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.155594 2575 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 10 02:48:02.215916 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.155789 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 10 02:48:02.215916 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.155817 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 10 02:48:02.215916 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.155826 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 10 02:48:02.215916 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.155834 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 10 02:48:02.215916 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.155843 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 10 02:48:02.215916 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.155852 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 10 02:48:02.215916 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.156459 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 10 02:48:02.215916 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.156466 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 10 02:48:02.215916 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.156473 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 10 02:48:02.215916 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.156478 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 10 02:48:02.215916 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.156487 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 10 02:48:02.215916 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.156494 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 10 02:48:02.215916 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.157420 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 10 02:48:02.215916 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.157429 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 10 02:48:02.215916 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.160321 2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-139.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 10 02:48:02.215916 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.160680 2575 server.go:1268] "Started kubelet" Jul 10 02:48:02.215916 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.160767 2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 10 02:48:02.215916 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.160847 2575 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 10 02:48:02.215916 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.160920 2575 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 10 02:48:02.215916 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:02.161078 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 10 02:48:02.215916 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:02.161218 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-139.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 10 02:48:02.215916 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.164219 2575 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 10 02:48:02.215916 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.165922 2575 server.go:310] "Adding debug handlers to kubelet server" Jul 10 02:48:02.215916 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:02.170209 2575 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 10 02:48:02.215916 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:02.170314 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-139.ec2.internal.18c0cdc13636b7bf default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-139.ec2.internal,UID:ip-10-0-130-139.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-139.ec2.internal,},FirstTimestamp:2026-07-10 02:48:02.160654271 +0000 UTC m=+0.441770548,LastTimestamp:2026-07-10 02:48:02.160654271 +0000 UTC m=+0.441770548,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-139.ec2.internal,}" Jul 10 02:48:02.215916 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.173213 2575 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 10 02:48:02.215916 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.173245 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 10 02:48:02.215916 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:02.173315 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-139.ec2.internal\" not found" Jul 10 02:48:02.215916 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.174044 2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 10 02:48:02.215916 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.174065 2575 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 10 02:48:02.215916 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.174081 2575 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 10 02:48:02.215916 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.174199 2575 reconstruct.go:97] "Volume reconstruction finished" Jul 10 02:48:02.215916 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.174211 2575 reconciler.go:29] "Reconciler: start to sync state" Jul 10 02:48:02.215916 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.174342 2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 10 02:48:02.215916 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.174358 2575 factory.go:55] Registering systemd factory Jul 10 02:48:02.215916 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.174367 2575 factory.go:223] Registration of the systemd container factory successfully Jul 10 02:48:02.216963 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.174589 2575 factory.go:153] Registering CRI-O factory Jul 10 02:48:02.216963 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.174597 2575 factory.go:223] Registration of the crio container factory successfully Jul 10 02:48:02.216963 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.174618 2575 factory.go:103] Registering Raw factory Jul 10 02:48:02.216963 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.174633 2575 manager.go:1196] Started watching for new ooms in manager Jul 10 02:48:02.216963 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.175247 2575 manager.go:349] Starting recovery of all containers Jul 10 02:48:02.216963 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.180335 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-29jd4" Jul 10 02:48:02.216963 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.182696 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 10 02:48:02.216963 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.185524 2575 manager.go:354] Recovery completed Jul 10 02:48:02.216963 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.190929 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-29jd4" Jul 10 02:48:02.216963 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.191092 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 02:48:02.216963 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:02.191528 2575 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-139.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 10 02:48:02.216963 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:02.191540 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 10 02:48:02.216963 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.192964 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-139.ec2.internal" event="NodeHasSufficientMemory" Jul 10 02:48:02.216963 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.192996 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-139.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 02:48:02.216963 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.193010 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-139.ec2.internal" event="NodeHasSufficientPID" Jul 10 02:48:02.216963 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.193557 2575 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 10 02:48:02.216963 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.193833 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 02:48:02.216963 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.193565 2575 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 10 02:48:02.216963 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.193910 2575 state_mem.go:36] "Initialized new in-memory state store" Jul 10 02:48:02.216963 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:02.194897 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-139.ec2.internal.18c0cdc13823f390 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-139.ec2.internal,UID:ip-10-0-130-139.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-139.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-139.ec2.internal,},FirstTimestamp:2026-07-10 02:48:02.192978832 +0000 UTC m=+0.474095111,LastTimestamp:2026-07-10 02:48:02.192978832 +0000 UTC m=+0.474095111,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-139.ec2.internal,}" Jul 10 02:48:02.216963 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.195205 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-139.ec2.internal" event="NodeHasSufficientMemory" Jul 10 02:48:02.216963 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.195244 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-139.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 02:48:02.216963 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.195262 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-139.ec2.internal" event="NodeHasSufficientPID" Jul 10 02:48:02.216963 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.196938 2575 policy_none.go:49] "None policy: Start" Jul 10 02:48:02.216963 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.196954 2575 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 10 02:48:02.216963 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.196965 2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 10 02:48:02.216963 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.199013 2575 policy_none.go:47] "Start" Jul 10 02:48:02.244438 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.244420 2575 manager.go:337] "Starting Device Plugin manager" Jul 10 02:48:02.260212 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:02.244466 2575 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 10 02:48:02.260212 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.244477 2575 server.go:87] "Starting device plugin registration server" Jul 10 02:48:02.260212 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.244760 2575 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 10 02:48:02.260212 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.244773 2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 10 02:48:02.260212 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.244916 2575 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 10 02:48:02.260212 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.244989 2575 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 10 02:48:02.260212 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.244998 2575 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 10 02:48:02.260212 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:02.246583 2575 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 10 02:48:02.260212 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:02.246627 2575 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-139.ec2.internal\" not found" Jul 10 02:48:02.281948 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.281913 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 10 02:48:02.282072 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.281973 2575 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 10 02:48:02.282072 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.282000 2575 kubelet.go:2448] "Starting kubelet main sync loop" Jul 10 02:48:02.282072 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:02.282041 2575 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 10 02:48:02.284303 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.284283 2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 02:48:02.345281 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.345238 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 02:48:02.346274 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.346251 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-139.ec2.internal" event="NodeHasSufficientMemory" Jul 10 02:48:02.346403 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.346285 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-139.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 02:48:02.346403 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.346300 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-139.ec2.internal" event="NodeHasSufficientPID" Jul 10 02:48:02.346403 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.346327 2575 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-139.ec2.internal" Jul 10 02:48:02.357688 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.357668 2575 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-139.ec2.internal" Jul 10 02:48:02.357743 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:02.357692 2575 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-139.ec2.internal\": node \"ip-10-0-130-139.ec2.internal\" not found" Jul 10 02:48:02.381352 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:02.381326 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-139.ec2.internal\" not found" Jul 10 02:48:02.382301 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.382283 2575 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-139.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-139.ec2.internal"] Jul 10 02:48:02.382350 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.382340 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 02:48:02.383732 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.383715 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-139.ec2.internal" event="NodeHasSufficientMemory" Jul 10 02:48:02.383856 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.383746 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-139.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 02:48:02.383856 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.383761 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-139.ec2.internal" event="NodeHasSufficientPID" Jul 10 02:48:02.386077 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.386062 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 02:48:02.386206 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.386188 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-139.ec2.internal" Jul 10 02:48:02.386257 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.386225 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 02:48:02.386730 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.386717 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-139.ec2.internal" event="NodeHasSufficientMemory" Jul 10 02:48:02.386791 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.386737 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-139.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 02:48:02.386791 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.386746 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-139.ec2.internal" event="NodeHasSufficientPID" Jul 10 02:48:02.386791 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.386771 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-139.ec2.internal" event="NodeHasSufficientMemory" Jul 10 02:48:02.386791 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.386789 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-139.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 02:48:02.386938 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.386799 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-139.ec2.internal" event="NodeHasSufficientPID" Jul 10 02:48:02.388971 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.388957 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-139.ec2.internal" Jul 10 02:48:02.389042 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.388980 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 02:48:02.389624 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.389611 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-139.ec2.internal" event="NodeHasSufficientMemory" Jul 10 02:48:02.389705 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.389636 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-139.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 02:48:02.389705 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.389650 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-139.ec2.internal" event="NodeHasSufficientPID" Jul 10 02:48:02.415445 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:02.415421 2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-139.ec2.internal\" not found" node="ip-10-0-130-139.ec2.internal" Jul 10 02:48:02.419710 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:02.419695 2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-139.ec2.internal\" not found" node="ip-10-0-130-139.ec2.internal" Jul 10 02:48:02.476823 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.476775 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f238b60151ef7c6e38d6cb1b9cdf59c0-config\") pod \"kube-apiserver-proxy-ip-10-0-130-139.ec2.internal\" (UID: \"f238b60151ef7c6e38d6cb1b9cdf59c0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-139.ec2.internal" Jul 10 02:48:02.476954 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.476803 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e1e6758aab956feebe21df5c8cc1fa0a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-139.ec2.internal\" (UID: \"e1e6758aab956feebe21df5c8cc1fa0a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-139.ec2.internal" Jul 10 02:48:02.476954 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.476846 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1e6758aab956feebe21df5c8cc1fa0a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-139.ec2.internal\" (UID: \"e1e6758aab956feebe21df5c8cc1fa0a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-139.ec2.internal" Jul 10 02:48:02.481885 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:02.481871 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-139.ec2.internal\" not found" Jul 10 02:48:02.577172 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.577144 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f238b60151ef7c6e38d6cb1b9cdf59c0-config\") pod \"kube-apiserver-proxy-ip-10-0-130-139.ec2.internal\" (UID: \"f238b60151ef7c6e38d6cb1b9cdf59c0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-139.ec2.internal" Jul 10 02:48:02.577257 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.577178 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e1e6758aab956feebe21df5c8cc1fa0a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-139.ec2.internal\" (UID: \"e1e6758aab956feebe21df5c8cc1fa0a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-139.ec2.internal" Jul 10 02:48:02.577257 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.577198 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1e6758aab956feebe21df5c8cc1fa0a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-139.ec2.internal\" (UID: \"e1e6758aab956feebe21df5c8cc1fa0a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-139.ec2.internal" Jul 10 02:48:02.577335 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.577256 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f238b60151ef7c6e38d6cb1b9cdf59c0-config\") pod \"kube-apiserver-proxy-ip-10-0-130-139.ec2.internal\" (UID: \"f238b60151ef7c6e38d6cb1b9cdf59c0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-139.ec2.internal" Jul 10 02:48:02.577335 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.577311 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1e6758aab956feebe21df5c8cc1fa0a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-139.ec2.internal\" (UID: \"e1e6758aab956feebe21df5c8cc1fa0a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-139.ec2.internal" Jul 10 02:48:02.577413 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.577344 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e1e6758aab956feebe21df5c8cc1fa0a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-139.ec2.internal\" (UID: \"e1e6758aab956feebe21df5c8cc1fa0a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-139.ec2.internal" Jul 10 02:48:02.582678 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:02.582653 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-139.ec2.internal\" not found" Jul 10 02:48:02.683554 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:02.683522 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-139.ec2.internal\" not found" Jul 10 02:48:02.718599 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.718556 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-139.ec2.internal" Jul 10 02:48:02.722180 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:02.722163 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-139.ec2.internal" Jul 10 02:48:02.784310 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:02.784280 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-139.ec2.internal\" not found" Jul 10 02:48:02.885479 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:02.885412 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-139.ec2.internal\" not found" Jul 10 02:48:02.986521 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:02.986483 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-139.ec2.internal\" not found" Jul 10 02:48:03.028124 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.028107 2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 02:48:03.075373 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.075331 2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-139.ec2.internal" Jul 10 02:48:03.088784 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.088760 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 02:48:03.090495 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.090481 2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-139.ec2.internal" Jul 10 02:48:03.100187 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.100170 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 02:48:03.107495 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.107465 2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 10 02:48:03.107625 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.107607 2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 02:48:03.107678 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.107643 2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 02:48:03.151169 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.151106 2575 apiserver.go:52] "Watching apiserver" Jul 10 02:48:03.169869 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.169844 2575 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 10 02:48:03.171048 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.171024 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-spcmm","openshift-dns/node-resolver-shf9m","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-139.ec2.internal","openshift-multus/multus-additional-cni-plugins-tfrkp","openshift-multus/network-metrics-daemon-bgbpr","kube-system/konnectivity-agent-kbfmm","openshift-image-registry/node-ca-gp2gc","openshift-multus/multus-9hgr6","openshift-network-diagnostics/network-check-target-nf7q8","openshift-network-operator/iptables-alerter-xdwmm","openshift-ovn-kubernetes/ovnkube-node-25tvc","kube-system/global-pull-secret-syncer-q28nr","kube-system/kube-apiserver-proxy-ip-10-0-130-139.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7fp4"] Jul 10 02:48:03.173700 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.173679 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 10 02:48:03.173846 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.173826 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-spcmm" Jul 10 02:48:03.176150 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.176130 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-shf9m" Jul 10 02:48:03.176269 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.176250 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 10 02:48:03.176768 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.176753 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 10 02:48:03.176843 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.176829 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8l456\"" Jul 10 02:48:03.178185 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.178166 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jgnl5\"" Jul 10 02:48:03.178271 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.178243 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 10 02:48:03.178383 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.178364 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 10 02:48:03.178523 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.178509 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tfrkp" Jul 10 02:48:03.180506 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.180487 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-626dv\"" Jul 10 02:48:03.180602 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.180534 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 10 02:48:03.180602 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.180567 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 10 02:48:03.180779 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.180601 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 10 02:48:03.180779 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.180689 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 10 02:48:03.180897 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.180872 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bgbpr" Jul 10 02:48:03.180949 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.180930 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 10 02:48:03.180949 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:03.180930 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bgbpr" podUID="7d037586-6a56-4354-ba36-a40c2cc2ab48" Jul 10 02:48:03.182386 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.182370 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7a694fd5-981d-4946-8d5c-e8a62f67dff6-etc-systemd\") pod \"tuned-spcmm\" (UID: \"7a694fd5-981d-4946-8d5c-e8a62f67dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-spcmm" Jul 10 02:48:03.182426 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.182394 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7a694fd5-981d-4946-8d5c-e8a62f67dff6-run\") pod \"tuned-spcmm\" (UID: \"7a694fd5-981d-4946-8d5c-e8a62f67dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-spcmm" Jul 10 02:48:03.182426 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.182413 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/731666e1-c05f-4169-98f4-df0efa01817a-hosts-file\") pod \"node-resolver-shf9m\" (UID: \"731666e1-c05f-4169-98f4-df0efa01817a\") " pod="openshift-dns/node-resolver-shf9m" Jul 10 02:48:03.182517 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.182429 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/731666e1-c05f-4169-98f4-df0efa01817a-tmp-dir\") pod \"node-resolver-shf9m\" (UID: \"731666e1-c05f-4169-98f4-df0efa01817a\") " pod="openshift-dns/node-resolver-shf9m" Jul 10 02:48:03.182517 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.182448 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a694fd5-981d-4946-8d5c-e8a62f67dff6-var-lib-kubelet\") pod \"tuned-spcmm\" (UID: \"7a694fd5-981d-4946-8d5c-e8a62f67dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-spcmm" Jul 10 02:48:03.182517 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.182464 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll2rs\" (UniqueName: \"kubernetes.io/projected/731666e1-c05f-4169-98f4-df0efa01817a-kube-api-access-ll2rs\") pod \"node-resolver-shf9m\" (UID: \"731666e1-c05f-4169-98f4-df0efa01817a\") " pod="openshift-dns/node-resolver-shf9m" Jul 10 02:48:03.182517 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.182506 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a694fd5-981d-4946-8d5c-e8a62f67dff6-sys\") pod \"tuned-spcmm\" (UID: \"7a694fd5-981d-4946-8d5c-e8a62f67dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-spcmm" Jul 10 02:48:03.182690 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.182536 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7a694fd5-981d-4946-8d5c-e8a62f67dff6-lib-modules\") pod \"tuned-spcmm\" (UID: \"7a694fd5-981d-4946-8d5c-e8a62f67dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-spcmm" Jul 10 02:48:03.182690 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.182562 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a694fd5-981d-4946-8d5c-e8a62f67dff6-tmp\") pod \"tuned-spcmm\" (UID: \"7a694fd5-981d-4946-8d5c-e8a62f67dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-spcmm" Jul 10 02:48:03.182690 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.182577 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hrpn\" (UniqueName: \"kubernetes.io/projected/7a694fd5-981d-4946-8d5c-e8a62f67dff6-kube-api-access-2hrpn\") pod \"tuned-spcmm\" (UID: \"7a694fd5-981d-4946-8d5c-e8a62f67dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-spcmm" Jul 10 02:48:03.182690 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.182596 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a694fd5-981d-4946-8d5c-e8a62f67dff6-etc-kubernetes\") pod \"tuned-spcmm\" (UID: \"7a694fd5-981d-4946-8d5c-e8a62f67dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-spcmm" Jul 10 02:48:03.182690 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.182627 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7a694fd5-981d-4946-8d5c-e8a62f67dff6-host\") pod \"tuned-spcmm\" (UID: \"7a694fd5-981d-4946-8d5c-e8a62f67dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-spcmm" Jul 10 02:48:03.182690 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.182663 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7a694fd5-981d-4946-8d5c-e8a62f67dff6-etc-tuned\") pod \"tuned-spcmm\" (UID: \"7a694fd5-981d-4946-8d5c-e8a62f67dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-spcmm" Jul 10 02:48:03.182690 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.182689 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7a694fd5-981d-4946-8d5c-e8a62f67dff6-etc-modprobe-d\") pod \"tuned-spcmm\" (UID: \"7a694fd5-981d-4946-8d5c-e8a62f67dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-spcmm" Jul 10 02:48:03.182950 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.182715 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7a694fd5-981d-4946-8d5c-e8a62f67dff6-etc-sysconfig\") pod \"tuned-spcmm\" (UID: \"7a694fd5-981d-4946-8d5c-e8a62f67dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-spcmm" Jul 10 02:48:03.182950 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.182744 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7a694fd5-981d-4946-8d5c-e8a62f67dff6-etc-sysctl-d\") pod \"tuned-spcmm\" (UID: \"7a694fd5-981d-4946-8d5c-e8a62f67dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-spcmm" Jul 10 02:48:03.182950 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.182769 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7a694fd5-981d-4946-8d5c-e8a62f67dff6-etc-sysctl-conf\") pod \"tuned-spcmm\" (UID: \"7a694fd5-981d-4946-8d5c-e8a62f67dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-spcmm" Jul 10 02:48:03.183336 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.183319 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kbfmm" Jul 10 02:48:03.185299 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.185280 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-q5m54\"" Jul 10 02:48:03.185393 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.185378 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 10 02:48:03.185550 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.185532 2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 02:48:03.186202 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.185533 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 10 02:48:03.186202 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.185621 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gp2gc" Jul 10 02:48:03.187787 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.187765 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 10 02:48:03.187897 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.187797 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 10 02:48:03.188041 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.187988 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-64qm7\"" Jul 10 02:48:03.188190 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.188151 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9hgr6" Jul 10 02:48:03.188252 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.188192 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 10 02:48:03.190166 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.190148 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 10 02:48:03.190166 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.190161 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-x288b\"" Jul 10 02:48:03.190624 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.190610 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nf7q8" Jul 10 02:48:03.190699 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:03.190677 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nf7q8" podUID="25d9393a-d380-4ef0-a976-9fd998deb429" Jul 10 02:48:03.197056 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.197007 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-09 02:43:02 +0000 UTC" deadline="2028-01-22 15:15:10.034391683 +0000 UTC" Jul 10 02:48:03.197056 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.197058 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13476h27m6.837340074s" Jul 10 02:48:03.197556 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.197541 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xdwmm" Jul 10 02:48:03.200592 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.200567 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.203220 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.203204 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q28nr" Jul 10 02:48:03.203296 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:03.203262 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q28nr" podUID="dc1f4b26-669e-42a0-abd4-f858ac6b1543" Jul 10 02:48:03.205794 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.205776 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7fp4" Jul 10 02:48:03.208884 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.208866 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 10 02:48:03.209133 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.209118 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 10 02:48:03.209284 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.209270 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vdwhm\"" Jul 10 02:48:03.209637 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.209621 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 10 02:48:03.209678 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.209640 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 10 02:48:03.209746 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.209727 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hd2gq\"" Jul 10 02:48:03.209808 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.209773 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 10 02:48:03.209872 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.209857 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 10 02:48:03.209947 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.209922 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 10 02:48:03.210115 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.210100 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 10 02:48:03.210494 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.210468 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 10 02:48:03.210568 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.210512 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 10 02:48:03.231189 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.231165 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-d8brx\"" Jul 10 02:48:03.231912 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.231891 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 10 02:48:03.232136 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.232121 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 10 02:48:03.249783 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.249761 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-rdf9m" Jul 10 02:48:03.259748 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.259730 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rdf9m" Jul 10 02:48:03.275559 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.275532 2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 10 02:48:03.283018 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.282991 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ff323ccb-9efc-43fd-8ce7-9365fe9af996-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tfrkp\" (UID: \"ff323ccb-9efc-43fd-8ce7-9365fe9af996\") " pod="openshift-multus/multus-additional-cni-plugins-tfrkp" Jul 10 02:48:03.283160 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.283041 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a694fd5-981d-4946-8d5c-e8a62f67dff6-var-lib-kubelet\") pod \"tuned-spcmm\" (UID: \"7a694fd5-981d-4946-8d5c-e8a62f67dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-spcmm" Jul 10 02:48:03.283160 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.283069 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll2rs\" (UniqueName: \"kubernetes.io/projected/731666e1-c05f-4169-98f4-df0efa01817a-kube-api-access-ll2rs\") pod \"node-resolver-shf9m\" (UID: \"731666e1-c05f-4169-98f4-df0efa01817a\") " pod="openshift-dns/node-resolver-shf9m" Jul 10 02:48:03.283160 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.283095 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f9bfec96-adaf-4cfd-9789-cce3b7276803-iptables-alerter-script\") pod \"iptables-alerter-xdwmm\" (UID: \"f9bfec96-adaf-4cfd-9789-cce3b7276803\") " pod="openshift-network-operator/iptables-alerter-xdwmm" Jul 10 02:48:03.283160 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.283119 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f9bfec96-adaf-4cfd-9789-cce3b7276803-host-slash\") pod \"iptables-alerter-xdwmm\" (UID: \"f9bfec96-adaf-4cfd-9789-cce3b7276803\") " pod="openshift-network-operator/iptables-alerter-xdwmm" Jul 10 02:48:03.283160 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.283120 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a694fd5-981d-4946-8d5c-e8a62f67dff6-var-lib-kubelet\") pod \"tuned-spcmm\" (UID: \"7a694fd5-981d-4946-8d5c-e8a62f67dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-spcmm" Jul 10 02:48:03.283515 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.283167 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps682\" (UniqueName: \"kubernetes.io/projected/f9bfec96-adaf-4cfd-9789-cce3b7276803-kube-api-access-ps682\") pod \"iptables-alerter-xdwmm\" (UID: \"f9bfec96-adaf-4cfd-9789-cce3b7276803\") " pod="openshift-network-operator/iptables-alerter-xdwmm" Jul 10 02:48:03.283515 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.283197 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7a694fd5-981d-4946-8d5c-e8a62f67dff6-lib-modules\") pod \"tuned-spcmm\" (UID: \"7a694fd5-981d-4946-8d5c-e8a62f67dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-spcmm" Jul 10 02:48:03.283515 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.283259 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cc6bf0c3-5e10-4619-9748-a2e9e46754ed-konnectivity-ca\") pod \"konnectivity-agent-kbfmm\" (UID: \"cc6bf0c3-5e10-4619-9748-a2e9e46754ed\") " pod="kube-system/konnectivity-agent-kbfmm" Jul 10 02:48:03.283515 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.283292 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/11f18b36-f920-47ba-8b04-10483f768ebf-serviceca\") pod \"node-ca-gp2gc\" (UID: \"11f18b36-f920-47ba-8b04-10483f768ebf\") " pod="openshift-image-registry/node-ca-gp2gc" Jul 10 02:48:03.283515 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.283323 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b93349d9-10f0-430c-b00f-74f141f508de-env-overrides\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.283515 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.283367 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b93349d9-10f0-430c-b00f-74f141f508de-host-run-ovn-kubernetes\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.283515 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.283413 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7a694fd5-981d-4946-8d5c-e8a62f67dff6-etc-modprobe-d\") pod \"tuned-spcmm\" (UID: \"7a694fd5-981d-4946-8d5c-e8a62f67dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-spcmm" Jul 10 02:48:03.283515 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.283373 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7a694fd5-981d-4946-8d5c-e8a62f67dff6-lib-modules\") pod \"tuned-spcmm\" (UID: \"7a694fd5-981d-4946-8d5c-e8a62f67dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-spcmm" Jul 10 02:48:03.283515 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.283439 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/29d54ad7-c565-4c22-a182-0e9fe8c8b981-host-run-netns\") pod \"multus-9hgr6\" (UID: \"29d54ad7-c565-4c22-a182-0e9fe8c8b981\") " pod="openshift-multus/multus-9hgr6" Jul 10 02:48:03.283515 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.283465 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/29d54ad7-c565-4c22-a182-0e9fe8c8b981-host-run-multus-certs\") pod \"multus-9hgr6\" (UID: \"29d54ad7-c565-4c22-a182-0e9fe8c8b981\") " pod="openshift-multus/multus-9hgr6" Jul 10 02:48:03.283515 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.283490 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b93349d9-10f0-430c-b00f-74f141f508de-ovnkube-script-lib\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.283954 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.283526 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btvh4\" (UniqueName: \"kubernetes.io/projected/b93349d9-10f0-430c-b00f-74f141f508de-kube-api-access-btvh4\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.283954 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.283541 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7a694fd5-981d-4946-8d5c-e8a62f67dff6-etc-modprobe-d\") pod \"tuned-spcmm\" (UID: \"7a694fd5-981d-4946-8d5c-e8a62f67dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-spcmm" Jul 10 02:48:03.283954 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.283557 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ff323ccb-9efc-43fd-8ce7-9365fe9af996-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tfrkp\" (UID: \"ff323ccb-9efc-43fd-8ce7-9365fe9af996\") " pod="openshift-multus/multus-additional-cni-plugins-tfrkp" Jul 10 02:48:03.283954 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.283583 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnxs7\" (UniqueName: \"kubernetes.io/projected/ff323ccb-9efc-43fd-8ce7-9365fe9af996-kube-api-access-qnxs7\") pod \"multus-additional-cni-plugins-tfrkp\" (UID: \"ff323ccb-9efc-43fd-8ce7-9365fe9af996\") " pod="openshift-multus/multus-additional-cni-plugins-tfrkp" Jul 10 02:48:03.283954 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.283601 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/29d54ad7-c565-4c22-a182-0e9fe8c8b981-hostroot\") pod \"multus-9hgr6\" (UID: \"29d54ad7-c565-4c22-a182-0e9fe8c8b981\") " pod="openshift-multus/multus-9hgr6" Jul 10 02:48:03.283954 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.283628 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b93349d9-10f0-430c-b00f-74f141f508de-host-slash\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.283954 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.283650 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b93349d9-10f0-430c-b00f-74f141f508de-host-cni-bin\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.283954 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.283686 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzkpc\" (UniqueName: \"kubernetes.io/projected/25d9393a-d380-4ef0-a976-9fd998deb429-kube-api-access-jzkpc\") pod \"network-check-target-nf7q8\" (UID: \"25d9393a-d380-4ef0-a976-9fd998deb429\") " pod="openshift-network-diagnostics/network-check-target-nf7q8" Jul 10 02:48:03.283954 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.283711 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a96aece5-40cd-4431-a0e9-1c54d4c6f066-socket-dir\") pod \"aws-ebs-csi-driver-node-n7fp4\" (UID: \"a96aece5-40cd-4431-a0e9-1c54d4c6f066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7fp4" Jul 10 02:48:03.283954 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.283737 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hrpn\" (UniqueName: \"kubernetes.io/projected/7a694fd5-981d-4946-8d5c-e8a62f67dff6-kube-api-access-2hrpn\") pod \"tuned-spcmm\" (UID: \"7a694fd5-981d-4946-8d5c-e8a62f67dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-spcmm" Jul 10 02:48:03.283954 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.283760 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d037586-6a56-4354-ba36-a40c2cc2ab48-metrics-certs\") pod \"network-metrics-daemon-bgbpr\" (UID: \"7d037586-6a56-4354-ba36-a40c2cc2ab48\") " pod="openshift-multus/network-metrics-daemon-bgbpr" Jul 10 02:48:03.283954 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.283782 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29d54ad7-c565-4c22-a182-0e9fe8c8b981-etc-kubernetes\") pod \"multus-9hgr6\" (UID: \"29d54ad7-c565-4c22-a182-0e9fe8c8b981\") " pod="openshift-multus/multus-9hgr6" Jul 10 02:48:03.283954 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.283807 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7a694fd5-981d-4946-8d5c-e8a62f67dff6-host\") pod \"tuned-spcmm\" (UID: \"7a694fd5-981d-4946-8d5c-e8a62f67dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-spcmm" Jul 10 02:48:03.283954 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.283844 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7a694fd5-981d-4946-8d5c-e8a62f67dff6-etc-tuned\") pod \"tuned-spcmm\" (UID: \"7a694fd5-981d-4946-8d5c-e8a62f67dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-spcmm" Jul 10 02:48:03.283954 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.283877 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a96aece5-40cd-4431-a0e9-1c54d4c6f066-device-dir\") pod \"aws-ebs-csi-driver-node-n7fp4\" (UID: \"a96aece5-40cd-4431-a0e9-1c54d4c6f066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7fp4" Jul 10 02:48:03.283954 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.283901 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86sq4\" (UniqueName: \"kubernetes.io/projected/7d037586-6a56-4354-ba36-a40c2cc2ab48-kube-api-access-86sq4\") pod \"network-metrics-daemon-bgbpr\" (UID: \"7d037586-6a56-4354-ba36-a40c2cc2ab48\") " pod="openshift-multus/network-metrics-daemon-bgbpr" Jul 10 02:48:03.284732 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.283905 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7a694fd5-981d-4946-8d5c-e8a62f67dff6-host\") pod \"tuned-spcmm\" (UID: \"7a694fd5-981d-4946-8d5c-e8a62f67dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-spcmm" Jul 10 02:48:03.284732 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.283974 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/731666e1-c05f-4169-98f4-df0efa01817a-tmp-dir\") pod \"node-resolver-shf9m\" (UID: \"731666e1-c05f-4169-98f4-df0efa01817a\") " pod="openshift-dns/node-resolver-shf9m" Jul 10 02:48:03.284732 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.284001 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/29d54ad7-c565-4c22-a182-0e9fe8c8b981-host-run-k8s-cni-cncf-io\") pod \"multus-9hgr6\" (UID: \"29d54ad7-c565-4c22-a182-0e9fe8c8b981\") " pod="openshift-multus/multus-9hgr6" Jul 10 02:48:03.284732 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.284036 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/29d54ad7-c565-4c22-a182-0e9fe8c8b981-multus-conf-dir\") pod \"multus-9hgr6\" (UID: \"29d54ad7-c565-4c22-a182-0e9fe8c8b981\") " pod="openshift-multus/multus-9hgr6" Jul 10 02:48:03.284732 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.284060 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b93349d9-10f0-430c-b00f-74f141f508de-run-systemd\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.284732 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.284092 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b93349d9-10f0-430c-b00f-74f141f508de-node-log\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.284732 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.284130 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b93349d9-10f0-430c-b00f-74f141f508de-log-socket\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.284732 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.284160 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ff323ccb-9efc-43fd-8ce7-9365fe9af996-cnibin\") pod \"multus-additional-cni-plugins-tfrkp\" (UID: \"ff323ccb-9efc-43fd-8ce7-9365fe9af996\") " pod="openshift-multus/multus-additional-cni-plugins-tfrkp" Jul 10 02:48:03.284732 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.284188 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmcjk\" (UniqueName: \"kubernetes.io/projected/11f18b36-f920-47ba-8b04-10483f768ebf-kube-api-access-nmcjk\") pod \"node-ca-gp2gc\" (UID: \"11f18b36-f920-47ba-8b04-10483f768ebf\") " pod="openshift-image-registry/node-ca-gp2gc" Jul 10 02:48:03.284732 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.284193 2575 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 10 02:48:03.284732 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.284207 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/29d54ad7-c565-4c22-a182-0e9fe8c8b981-host-var-lib-cni-bin\") pod \"multus-9hgr6\" (UID: \"29d54ad7-c565-4c22-a182-0e9fe8c8b981\") " pod="openshift-multus/multus-9hgr6" Jul 10 02:48:03.284732 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.284230 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b93349d9-10f0-430c-b00f-74f141f508de-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.284732 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.284238 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/731666e1-c05f-4169-98f4-df0efa01817a-tmp-dir\") pod \"node-resolver-shf9m\" (UID: \"731666e1-c05f-4169-98f4-df0efa01817a\") " pod="openshift-dns/node-resolver-shf9m" Jul 10 02:48:03.284732 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.284258 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ff323ccb-9efc-43fd-8ce7-9365fe9af996-os-release\") pod \"multus-additional-cni-plugins-tfrkp\" (UID: \"ff323ccb-9efc-43fd-8ce7-9365fe9af996\") " pod="openshift-multus/multus-additional-cni-plugins-tfrkp" Jul 10 02:48:03.284732 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.284286 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dc1f4b26-669e-42a0-abd4-f858ac6b1543-kubelet-config\") pod \"global-pull-secret-syncer-q28nr\" (UID: \"dc1f4b26-669e-42a0-abd4-f858ac6b1543\") " pod="kube-system/global-pull-secret-syncer-q28nr" Jul 10 02:48:03.284732 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.284340 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc1f4b26-669e-42a0-abd4-f858ac6b1543-original-pull-secret\") pod \"global-pull-secret-syncer-q28nr\" (UID: \"dc1f4b26-669e-42a0-abd4-f858ac6b1543\") " pod="kube-system/global-pull-secret-syncer-q28nr" Jul 10 02:48:03.284732 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.284379 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a96aece5-40cd-4431-a0e9-1c54d4c6f066-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n7fp4\" (UID: \"a96aece5-40cd-4431-a0e9-1c54d4c6f066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7fp4" Jul 10 02:48:03.285576 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.284411 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpp5d\" (UniqueName: \"kubernetes.io/projected/a96aece5-40cd-4431-a0e9-1c54d4c6f066-kube-api-access-gpp5d\") pod \"aws-ebs-csi-driver-node-n7fp4\" (UID: \"a96aece5-40cd-4431-a0e9-1c54d4c6f066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7fp4" Jul 10 02:48:03.285576 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.284442 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a694fd5-981d-4946-8d5c-e8a62f67dff6-sys\") pod \"tuned-spcmm\" (UID: \"7a694fd5-981d-4946-8d5c-e8a62f67dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-spcmm" Jul 10 02:48:03.285576 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.284469 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/29d54ad7-c565-4c22-a182-0e9fe8c8b981-system-cni-dir\") pod \"multus-9hgr6\" (UID: \"29d54ad7-c565-4c22-a182-0e9fe8c8b981\") " pod="openshift-multus/multus-9hgr6" Jul 10 02:48:03.285576 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.284494 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/29d54ad7-c565-4c22-a182-0e9fe8c8b981-os-release\") pod \"multus-9hgr6\" (UID: \"29d54ad7-c565-4c22-a182-0e9fe8c8b981\") " pod="openshift-multus/multus-9hgr6" Jul 10 02:48:03.285576 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.284519 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/29d54ad7-c565-4c22-a182-0e9fe8c8b981-multus-socket-dir-parent\") pod \"multus-9hgr6\" (UID: \"29d54ad7-c565-4c22-a182-0e9fe8c8b981\") " pod="openshift-multus/multus-9hgr6" Jul 10 02:48:03.285576 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.284524 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a694fd5-981d-4946-8d5c-e8a62f67dff6-sys\") pod \"tuned-spcmm\" (UID: \"7a694fd5-981d-4946-8d5c-e8a62f67dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-spcmm" Jul 10 02:48:03.285576 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.284544 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29d54ad7-c565-4c22-a182-0e9fe8c8b981-host-var-lib-kubelet\") pod \"multus-9hgr6\" (UID: \"29d54ad7-c565-4c22-a182-0e9fe8c8b981\") " pod="openshift-multus/multus-9hgr6" Jul 10 02:48:03.285576 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.284570 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/29d54ad7-c565-4c22-a182-0e9fe8c8b981-multus-daemon-config\") pod \"multus-9hgr6\" (UID: \"29d54ad7-c565-4c22-a182-0e9fe8c8b981\") " pod="openshift-multus/multus-9hgr6" Jul 10 02:48:03.285576 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.284593 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b93349d9-10f0-430c-b00f-74f141f508de-var-lib-openvswitch\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.285576 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.284616 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b93349d9-10f0-430c-b00f-74f141f508de-etc-openvswitch\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.285576 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.284661 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b93349d9-10f0-430c-b00f-74f141f508de-run-openvswitch\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.285576 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.284687 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlznj\" (UniqueName: \"kubernetes.io/projected/29d54ad7-c565-4c22-a182-0e9fe8c8b981-kube-api-access-qlznj\") pod \"multus-9hgr6\" (UID: \"29d54ad7-c565-4c22-a182-0e9fe8c8b981\") " pod="openshift-multus/multus-9hgr6" Jul 10 02:48:03.285576 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.284711 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b93349d9-10f0-430c-b00f-74f141f508de-run-ovn\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.285576 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.284743 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7a694fd5-981d-4946-8d5c-e8a62f67dff6-etc-sysctl-conf\") pod \"tuned-spcmm\" (UID: \"7a694fd5-981d-4946-8d5c-e8a62f67dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-spcmm" Jul 10 02:48:03.285576 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.284776 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7a694fd5-981d-4946-8d5c-e8a62f67dff6-run\") pod \"tuned-spcmm\" (UID: \"7a694fd5-981d-4946-8d5c-e8a62f67dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-spcmm" Jul 10 02:48:03.285576 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.284848 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7a694fd5-981d-4946-8d5c-e8a62f67dff6-run\") pod \"tuned-spcmm\" (UID: \"7a694fd5-981d-4946-8d5c-e8a62f67dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-spcmm" Jul 10 02:48:03.285576 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.284859 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/731666e1-c05f-4169-98f4-df0efa01817a-hosts-file\") pod \"node-resolver-shf9m\" (UID: \"731666e1-c05f-4169-98f4-df0efa01817a\") " pod="openshift-dns/node-resolver-shf9m" Jul 10 02:48:03.286233 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.284883 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/29d54ad7-c565-4c22-a182-0e9fe8c8b981-cnibin\") pod \"multus-9hgr6\" (UID: \"29d54ad7-c565-4c22-a182-0e9fe8c8b981\") " pod="openshift-multus/multus-9hgr6" Jul 10 02:48:03.286233 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.284899 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/29d54ad7-c565-4c22-a182-0e9fe8c8b981-cni-binary-copy\") pod \"multus-9hgr6\" (UID: \"29d54ad7-c565-4c22-a182-0e9fe8c8b981\") " pod="openshift-multus/multus-9hgr6" Jul 10 02:48:03.286233 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.284915 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ff323ccb-9efc-43fd-8ce7-9365fe9af996-system-cni-dir\") pod \"multus-additional-cni-plugins-tfrkp\" (UID: \"ff323ccb-9efc-43fd-8ce7-9365fe9af996\") " pod="openshift-multus/multus-additional-cni-plugins-tfrkp" Jul 10 02:48:03.286233 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.284927 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7a694fd5-981d-4946-8d5c-e8a62f67dff6-etc-sysctl-conf\") pod \"tuned-spcmm\" (UID: \"7a694fd5-981d-4946-8d5c-e8a62f67dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-spcmm" Jul 10 02:48:03.286233 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.284938 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/29d54ad7-c565-4c22-a182-0e9fe8c8b981-host-var-lib-cni-multus\") pod \"multus-9hgr6\" (UID: \"29d54ad7-c565-4c22-a182-0e9fe8c8b981\") " pod="openshift-multus/multus-9hgr6" Jul 10 02:48:03.286233 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.284968 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/731666e1-c05f-4169-98f4-df0efa01817a-hosts-file\") pod \"node-resolver-shf9m\" (UID: \"731666e1-c05f-4169-98f4-df0efa01817a\") " pod="openshift-dns/node-resolver-shf9m" Jul 10 02:48:03.286233 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.284974 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b93349d9-10f0-430c-b00f-74f141f508de-ovnkube-config\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.286233 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.285007 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ff323ccb-9efc-43fd-8ce7-9365fe9af996-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tfrkp\" (UID: \"ff323ccb-9efc-43fd-8ce7-9365fe9af996\") " pod="openshift-multus/multus-additional-cni-plugins-tfrkp" Jul 10 02:48:03.286233 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.285036 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a96aece5-40cd-4431-a0e9-1c54d4c6f066-etc-selinux\") pod \"aws-ebs-csi-driver-node-n7fp4\" (UID: \"a96aece5-40cd-4431-a0e9-1c54d4c6f066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7fp4" Jul 10 02:48:03.286233 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.285063 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a694fd5-981d-4946-8d5c-e8a62f67dff6-tmp\") pod \"tuned-spcmm\" (UID: \"7a694fd5-981d-4946-8d5c-e8a62f67dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-spcmm" Jul 10 02:48:03.286233 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.285089 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b93349d9-10f0-430c-b00f-74f141f508de-host-kubelet\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.286233 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.285115 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b93349d9-10f0-430c-b00f-74f141f508de-ovn-node-metrics-cert\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.286233 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.285140 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dc1f4b26-669e-42a0-abd4-f858ac6b1543-dbus\") pod \"global-pull-secret-syncer-q28nr\" (UID: \"dc1f4b26-669e-42a0-abd4-f858ac6b1543\") " pod="kube-system/global-pull-secret-syncer-q28nr" Jul 10 02:48:03.286233 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.285166 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a96aece5-40cd-4431-a0e9-1c54d4c6f066-registration-dir\") pod \"aws-ebs-csi-driver-node-n7fp4\" (UID: \"a96aece5-40cd-4431-a0e9-1c54d4c6f066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7fp4" Jul 10 02:48:03.286233 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.285193 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a694fd5-981d-4946-8d5c-e8a62f67dff6-etc-kubernetes\") pod \"tuned-spcmm\" (UID: \"7a694fd5-981d-4946-8d5c-e8a62f67dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-spcmm" Jul 10 02:48:03.286233 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.285215 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/11f18b36-f920-47ba-8b04-10483f768ebf-host\") pod \"node-ca-gp2gc\" (UID: \"11f18b36-f920-47ba-8b04-10483f768ebf\") " pod="openshift-image-registry/node-ca-gp2gc" Jul 10 02:48:03.286233 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.285258 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a694fd5-981d-4946-8d5c-e8a62f67dff6-etc-kubernetes\") pod \"tuned-spcmm\" (UID: \"7a694fd5-981d-4946-8d5c-e8a62f67dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-spcmm" Jul 10 02:48:03.286738 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.285260 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/29d54ad7-c565-4c22-a182-0e9fe8c8b981-multus-cni-dir\") pod \"multus-9hgr6\" (UID: \"29d54ad7-c565-4c22-a182-0e9fe8c8b981\") " pod="openshift-multus/multus-9hgr6" Jul 10 02:48:03.286738 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.285308 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b93349d9-10f0-430c-b00f-74f141f508de-systemd-units\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.286738 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.285335 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a96aece5-40cd-4431-a0e9-1c54d4c6f066-sys-fs\") pod \"aws-ebs-csi-driver-node-n7fp4\" (UID: \"a96aece5-40cd-4431-a0e9-1c54d4c6f066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7fp4" Jul 10 02:48:03.286738 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.285359 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cc6bf0c3-5e10-4619-9748-a2e9e46754ed-agent-certs\") pod \"konnectivity-agent-kbfmm\" (UID: \"cc6bf0c3-5e10-4619-9748-a2e9e46754ed\") " pod="kube-system/konnectivity-agent-kbfmm" Jul 10 02:48:03.286738 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.285387 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7a694fd5-981d-4946-8d5c-e8a62f67dff6-etc-sysconfig\") pod \"tuned-spcmm\" (UID: \"7a694fd5-981d-4946-8d5c-e8a62f67dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-spcmm" Jul 10 02:48:03.286738 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.285414 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7a694fd5-981d-4946-8d5c-e8a62f67dff6-etc-sysctl-d\") pod \"tuned-spcmm\" (UID: \"7a694fd5-981d-4946-8d5c-e8a62f67dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-spcmm" Jul 10 02:48:03.286738 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.285437 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7a694fd5-981d-4946-8d5c-e8a62f67dff6-etc-systemd\") pod \"tuned-spcmm\" (UID: \"7a694fd5-981d-4946-8d5c-e8a62f67dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-spcmm" Jul 10 02:48:03.286738 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.285463 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b93349d9-10f0-430c-b00f-74f141f508de-host-run-netns\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.286738 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.285472 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7a694fd5-981d-4946-8d5c-e8a62f67dff6-etc-sysconfig\") pod \"tuned-spcmm\" (UID: \"7a694fd5-981d-4946-8d5c-e8a62f67dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-spcmm" Jul 10 02:48:03.286738 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.285488 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b93349d9-10f0-430c-b00f-74f141f508de-host-cni-netd\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.286738 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.285519 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ff323ccb-9efc-43fd-8ce7-9365fe9af996-cni-binary-copy\") pod \"multus-additional-cni-plugins-tfrkp\" (UID: \"ff323ccb-9efc-43fd-8ce7-9365fe9af996\") " pod="openshift-multus/multus-additional-cni-plugins-tfrkp" Jul 10 02:48:03.286738 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.285533 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7a694fd5-981d-4946-8d5c-e8a62f67dff6-etc-sysctl-d\") pod \"tuned-spcmm\" (UID: \"7a694fd5-981d-4946-8d5c-e8a62f67dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-spcmm" Jul 10 02:48:03.286738 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.285522 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7a694fd5-981d-4946-8d5c-e8a62f67dff6-etc-systemd\") pod \"tuned-spcmm\" (UID: \"7a694fd5-981d-4946-8d5c-e8a62f67dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-spcmm" Jul 10 02:48:03.287610 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.287593 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7a694fd5-981d-4946-8d5c-e8a62f67dff6-etc-tuned\") pod \"tuned-spcmm\" (UID: \"7a694fd5-981d-4946-8d5c-e8a62f67dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-spcmm" Jul 10 02:48:03.287699 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.287645 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a694fd5-981d-4946-8d5c-e8a62f67dff6-tmp\") pod \"tuned-spcmm\" (UID: \"7a694fd5-981d-4946-8d5c-e8a62f67dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-spcmm" Jul 10 02:48:03.296172 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.296152 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hrpn\" (UniqueName: \"kubernetes.io/projected/7a694fd5-981d-4946-8d5c-e8a62f67dff6-kube-api-access-2hrpn\") pod \"tuned-spcmm\" (UID: \"7a694fd5-981d-4946-8d5c-e8a62f67dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-spcmm" Jul 10 02:48:03.296280 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.296210 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll2rs\" (UniqueName: \"kubernetes.io/projected/731666e1-c05f-4169-98f4-df0efa01817a-kube-api-access-ll2rs\") pod \"node-resolver-shf9m\" (UID: \"731666e1-c05f-4169-98f4-df0efa01817a\") " pod="openshift-dns/node-resolver-shf9m" Jul 10 02:48:03.385969 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.385935 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b93349d9-10f0-430c-b00f-74f141f508de-host-cni-netd\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.386150 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.385976 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ff323ccb-9efc-43fd-8ce7-9365fe9af996-cni-binary-copy\") pod \"multus-additional-cni-plugins-tfrkp\" (UID: \"ff323ccb-9efc-43fd-8ce7-9365fe9af996\") " pod="openshift-multus/multus-additional-cni-plugins-tfrkp" Jul 10 02:48:03.386150 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.386002 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ff323ccb-9efc-43fd-8ce7-9365fe9af996-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tfrkp\" (UID: \"ff323ccb-9efc-43fd-8ce7-9365fe9af996\") " pod="openshift-multus/multus-additional-cni-plugins-tfrkp" Jul 10 02:48:03.386150 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.386026 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f9bfec96-adaf-4cfd-9789-cce3b7276803-iptables-alerter-script\") pod \"iptables-alerter-xdwmm\" (UID: \"f9bfec96-adaf-4cfd-9789-cce3b7276803\") " pod="openshift-network-operator/iptables-alerter-xdwmm" Jul 10 02:48:03.386150 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.386048 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f9bfec96-adaf-4cfd-9789-cce3b7276803-host-slash\") pod \"iptables-alerter-xdwmm\" (UID: \"f9bfec96-adaf-4cfd-9789-cce3b7276803\") " pod="openshift-network-operator/iptables-alerter-xdwmm" Jul 10 02:48:03.386150 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.386055 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b93349d9-10f0-430c-b00f-74f141f508de-host-cni-netd\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.386150 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.386070 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps682\" (UniqueName: \"kubernetes.io/projected/f9bfec96-adaf-4cfd-9789-cce3b7276803-kube-api-access-ps682\") pod \"iptables-alerter-xdwmm\" (UID: \"f9bfec96-adaf-4cfd-9789-cce3b7276803\") " pod="openshift-network-operator/iptables-alerter-xdwmm" Jul 10 02:48:03.386150 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.386112 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cc6bf0c3-5e10-4619-9748-a2e9e46754ed-konnectivity-ca\") pod \"konnectivity-agent-kbfmm\" (UID: \"cc6bf0c3-5e10-4619-9748-a2e9e46754ed\") " pod="kube-system/konnectivity-agent-kbfmm" Jul 10 02:48:03.386150 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.386142 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/11f18b36-f920-47ba-8b04-10483f768ebf-serviceca\") pod \"node-ca-gp2gc\" (UID: \"11f18b36-f920-47ba-8b04-10483f768ebf\") " pod="openshift-image-registry/node-ca-gp2gc" Jul 10 02:48:03.386585 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.386168 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b93349d9-10f0-430c-b00f-74f141f508de-env-overrides\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.386585 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.386167 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ff323ccb-9efc-43fd-8ce7-9365fe9af996-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tfrkp\" (UID: \"ff323ccb-9efc-43fd-8ce7-9365fe9af996\") " pod="openshift-multus/multus-additional-cni-plugins-tfrkp" Jul 10 02:48:03.386585 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.386186 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f9bfec96-adaf-4cfd-9789-cce3b7276803-host-slash\") pod \"iptables-alerter-xdwmm\" (UID: \"f9bfec96-adaf-4cfd-9789-cce3b7276803\") " pod="openshift-network-operator/iptables-alerter-xdwmm" Jul 10 02:48:03.386585 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.386195 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b93349d9-10f0-430c-b00f-74f141f508de-host-run-ovn-kubernetes\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.386585 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.386234 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b93349d9-10f0-430c-b00f-74f141f508de-host-run-ovn-kubernetes\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.386585 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.386232 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/29d54ad7-c565-4c22-a182-0e9fe8c8b981-host-run-netns\") pod \"multus-9hgr6\" (UID: \"29d54ad7-c565-4c22-a182-0e9fe8c8b981\") " pod="openshift-multus/multus-9hgr6" Jul 10 02:48:03.386585 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.386271 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/29d54ad7-c565-4c22-a182-0e9fe8c8b981-host-run-netns\") pod \"multus-9hgr6\" (UID: \"29d54ad7-c565-4c22-a182-0e9fe8c8b981\") " pod="openshift-multus/multus-9hgr6" Jul 10 02:48:03.386585 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.386271 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/29d54ad7-c565-4c22-a182-0e9fe8c8b981-host-run-multus-certs\") pod \"multus-9hgr6\" (UID: \"29d54ad7-c565-4c22-a182-0e9fe8c8b981\") " pod="openshift-multus/multus-9hgr6" Jul 10 02:48:03.386585 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.386305 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/29d54ad7-c565-4c22-a182-0e9fe8c8b981-host-run-multus-certs\") pod \"multus-9hgr6\" (UID: \"29d54ad7-c565-4c22-a182-0e9fe8c8b981\") " pod="openshift-multus/multus-9hgr6" Jul 10 02:48:03.386585 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.386307 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b93349d9-10f0-430c-b00f-74f141f508de-ovnkube-script-lib\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.386585 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.386342 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btvh4\" (UniqueName: \"kubernetes.io/projected/b93349d9-10f0-430c-b00f-74f141f508de-kube-api-access-btvh4\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.386585 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.386374 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ff323ccb-9efc-43fd-8ce7-9365fe9af996-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tfrkp\" (UID: \"ff323ccb-9efc-43fd-8ce7-9365fe9af996\") " pod="openshift-multus/multus-additional-cni-plugins-tfrkp" Jul 10 02:48:03.386585 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.386401 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnxs7\" (UniqueName: \"kubernetes.io/projected/ff323ccb-9efc-43fd-8ce7-9365fe9af996-kube-api-access-qnxs7\") pod \"multus-additional-cni-plugins-tfrkp\" (UID: \"ff323ccb-9efc-43fd-8ce7-9365fe9af996\") " pod="openshift-multus/multus-additional-cni-plugins-tfrkp" Jul 10 02:48:03.386585 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.386424 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/29d54ad7-c565-4c22-a182-0e9fe8c8b981-hostroot\") pod \"multus-9hgr6\" (UID: \"29d54ad7-c565-4c22-a182-0e9fe8c8b981\") " pod="openshift-multus/multus-9hgr6" Jul 10 02:48:03.386585 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.386445 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b93349d9-10f0-430c-b00f-74f141f508de-host-slash\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.386585 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.386465 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b93349d9-10f0-430c-b00f-74f141f508de-host-cni-bin\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.386585 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.386489 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzkpc\" (UniqueName: \"kubernetes.io/projected/25d9393a-d380-4ef0-a976-9fd998deb429-kube-api-access-jzkpc\") pod \"network-check-target-nf7q8\" (UID: \"25d9393a-d380-4ef0-a976-9fd998deb429\") " pod="openshift-network-diagnostics/network-check-target-nf7q8" Jul 10 02:48:03.387519 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.386634 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a96aece5-40cd-4431-a0e9-1c54d4c6f066-socket-dir\") pod \"aws-ebs-csi-driver-node-n7fp4\" (UID: \"a96aece5-40cd-4431-a0e9-1c54d4c6f066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7fp4" Jul 10 02:48:03.387519 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.386662 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d037586-6a56-4354-ba36-a40c2cc2ab48-metrics-certs\") pod \"network-metrics-daemon-bgbpr\" (UID: \"7d037586-6a56-4354-ba36-a40c2cc2ab48\") " pod="openshift-multus/network-metrics-daemon-bgbpr" Jul 10 02:48:03.387519 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.386703 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29d54ad7-c565-4c22-a182-0e9fe8c8b981-etc-kubernetes\") pod \"multus-9hgr6\" (UID: \"29d54ad7-c565-4c22-a182-0e9fe8c8b981\") " pod="openshift-multus/multus-9hgr6" Jul 10 02:48:03.387519 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.386762 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a96aece5-40cd-4431-a0e9-1c54d4c6f066-device-dir\") pod \"aws-ebs-csi-driver-node-n7fp4\" (UID: \"a96aece5-40cd-4431-a0e9-1c54d4c6f066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7fp4" Jul 10 02:48:03.387519 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.386760 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ff323ccb-9efc-43fd-8ce7-9365fe9af996-cni-binary-copy\") pod \"multus-additional-cni-plugins-tfrkp\" (UID: \"ff323ccb-9efc-43fd-8ce7-9365fe9af996\") " pod="openshift-multus/multus-additional-cni-plugins-tfrkp" Jul 10 02:48:03.387519 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.386776 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f9bfec96-adaf-4cfd-9789-cce3b7276803-iptables-alerter-script\") pod \"iptables-alerter-xdwmm\" (UID: \"f9bfec96-adaf-4cfd-9789-cce3b7276803\") " pod="openshift-network-operator/iptables-alerter-xdwmm" Jul 10 02:48:03.387519 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.386787 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86sq4\" (UniqueName: \"kubernetes.io/projected/7d037586-6a56-4354-ba36-a40c2cc2ab48-kube-api-access-86sq4\") pod \"network-metrics-daemon-bgbpr\" (UID: \"7d037586-6a56-4354-ba36-a40c2cc2ab48\") " pod="openshift-multus/network-metrics-daemon-bgbpr" Jul 10 02:48:03.387519 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.386813 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/29d54ad7-c565-4c22-a182-0e9fe8c8b981-host-run-k8s-cni-cncf-io\") pod \"multus-9hgr6\" (UID: \"29d54ad7-c565-4c22-a182-0e9fe8c8b981\") " pod="openshift-multus/multus-9hgr6" Jul 10 02:48:03.387519 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.386839 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b93349d9-10f0-430c-b00f-74f141f508de-env-overrides\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.387519 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.386894 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cc6bf0c3-5e10-4619-9748-a2e9e46754ed-konnectivity-ca\") pod \"konnectivity-agent-kbfmm\" (UID: \"cc6bf0c3-5e10-4619-9748-a2e9e46754ed\") " pod="kube-system/konnectivity-agent-kbfmm" Jul 10 02:48:03.387519 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.386948 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b93349d9-10f0-430c-b00f-74f141f508de-host-slash\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.387519 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.386971 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/11f18b36-f920-47ba-8b04-10483f768ebf-serviceca\") pod \"node-ca-gp2gc\" (UID: \"11f18b36-f920-47ba-8b04-10483f768ebf\") " pod="openshift-image-registry/node-ca-gp2gc" Jul 10 02:48:03.387519 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.387036 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b93349d9-10f0-430c-b00f-74f141f508de-ovnkube-script-lib\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.387519 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.387141 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/29d54ad7-c565-4c22-a182-0e9fe8c8b981-multus-conf-dir\") pod \"multus-9hgr6\" (UID: \"29d54ad7-c565-4c22-a182-0e9fe8c8b981\") " pod="openshift-multus/multus-9hgr6" Jul 10 02:48:03.387519 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.387177 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b93349d9-10f0-430c-b00f-74f141f508de-run-systemd\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.387519 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:03.387202 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 02:48:03.387519 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.387226 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/29d54ad7-c565-4c22-a182-0e9fe8c8b981-multus-conf-dir\") pod \"multus-9hgr6\" (UID: \"29d54ad7-c565-4c22-a182-0e9fe8c8b981\") " pod="openshift-multus/multus-9hgr6" Jul 10 02:48:03.387519 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.387242 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b93349d9-10f0-430c-b00f-74f141f508de-node-log\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.388442 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.387203 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b93349d9-10f0-430c-b00f-74f141f508de-node-log\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.388442 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.387290 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b93349d9-10f0-430c-b00f-74f141f508de-run-systemd\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.388442 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.387287 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/29d54ad7-c565-4c22-a182-0e9fe8c8b981-host-run-k8s-cni-cncf-io\") pod \"multus-9hgr6\" (UID: \"29d54ad7-c565-4c22-a182-0e9fe8c8b981\") " pod="openshift-multus/multus-9hgr6" Jul 10 02:48:03.388442 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:03.387309 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d037586-6a56-4354-ba36-a40c2cc2ab48-metrics-certs podName:7d037586-6a56-4354-ba36-a40c2cc2ab48 nodeName:}" failed. No retries permitted until 2026-07-10 02:48:03.887264168 +0000 UTC m=+2.168380660 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d037586-6a56-4354-ba36-a40c2cc2ab48-metrics-certs") pod "network-metrics-daemon-bgbpr" (UID: "7d037586-6a56-4354-ba36-a40c2cc2ab48") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 02:48:03.388442 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.387334 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b93349d9-10f0-430c-b00f-74f141f508de-log-socket\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.388442 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.387355 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b93349d9-10f0-430c-b00f-74f141f508de-host-cni-bin\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.388442 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.387364 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ff323ccb-9efc-43fd-8ce7-9365fe9af996-cnibin\") pod \"multus-additional-cni-plugins-tfrkp\" (UID: \"ff323ccb-9efc-43fd-8ce7-9365fe9af996\") " pod="openshift-multus/multus-additional-cni-plugins-tfrkp" Jul 10 02:48:03.388442 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.387391 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmcjk\" (UniqueName: \"kubernetes.io/projected/11f18b36-f920-47ba-8b04-10483f768ebf-kube-api-access-nmcjk\") pod \"node-ca-gp2gc\" (UID: \"11f18b36-f920-47ba-8b04-10483f768ebf\") " pod="openshift-image-registry/node-ca-gp2gc" Jul 10 02:48:03.388442 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.387398 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29d54ad7-c565-4c22-a182-0e9fe8c8b981-etc-kubernetes\") pod \"multus-9hgr6\" (UID: \"29d54ad7-c565-4c22-a182-0e9fe8c8b981\") " pod="openshift-multus/multus-9hgr6" Jul 10 02:48:03.388442 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.387417 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/29d54ad7-c565-4c22-a182-0e9fe8c8b981-host-var-lib-cni-bin\") pod \"multus-9hgr6\" (UID: \"29d54ad7-c565-4c22-a182-0e9fe8c8b981\") " pod="openshift-multus/multus-9hgr6" Jul 10 02:48:03.388442 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.387443 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b93349d9-10f0-430c-b00f-74f141f508de-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.388442 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.387469 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ff323ccb-9efc-43fd-8ce7-9365fe9af996-os-release\") pod \"multus-additional-cni-plugins-tfrkp\" (UID: \"ff323ccb-9efc-43fd-8ce7-9365fe9af996\") " pod="openshift-multus/multus-additional-cni-plugins-tfrkp" Jul 10 02:48:03.388442 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.387476 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ff323ccb-9efc-43fd-8ce7-9365fe9af996-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tfrkp\" (UID: \"ff323ccb-9efc-43fd-8ce7-9365fe9af996\") " pod="openshift-multus/multus-additional-cni-plugins-tfrkp" Jul 10 02:48:03.388442 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.387469 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b93349d9-10f0-430c-b00f-74f141f508de-log-socket\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.388442 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.387497 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dc1f4b26-669e-42a0-abd4-f858ac6b1543-kubelet-config\") pod \"global-pull-secret-syncer-q28nr\" (UID: \"dc1f4b26-669e-42a0-abd4-f858ac6b1543\") " pod="kube-system/global-pull-secret-syncer-q28nr" Jul 10 02:48:03.388442 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.387521 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc1f4b26-669e-42a0-abd4-f858ac6b1543-original-pull-secret\") pod \"global-pull-secret-syncer-q28nr\" (UID: \"dc1f4b26-669e-42a0-abd4-f858ac6b1543\") " pod="kube-system/global-pull-secret-syncer-q28nr" Jul 10 02:48:03.389129 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.387546 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a96aece5-40cd-4431-a0e9-1c54d4c6f066-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n7fp4\" (UID: \"a96aece5-40cd-4431-a0e9-1c54d4c6f066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7fp4" Jul 10 02:48:03.389129 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.387556 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/29d54ad7-c565-4c22-a182-0e9fe8c8b981-host-var-lib-cni-bin\") pod \"multus-9hgr6\" (UID: \"29d54ad7-c565-4c22-a182-0e9fe8c8b981\") " pod="openshift-multus/multus-9hgr6" Jul 10 02:48:03.389129 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.387569 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpp5d\" (UniqueName: \"kubernetes.io/projected/a96aece5-40cd-4431-a0e9-1c54d4c6f066-kube-api-access-gpp5d\") pod \"aws-ebs-csi-driver-node-n7fp4\" (UID: \"a96aece5-40cd-4431-a0e9-1c54d4c6f066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7fp4" Jul 10 02:48:03.389129 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.387589 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/29d54ad7-c565-4c22-a182-0e9fe8c8b981-system-cni-dir\") pod \"multus-9hgr6\" (UID: \"29d54ad7-c565-4c22-a182-0e9fe8c8b981\") " pod="openshift-multus/multus-9hgr6" Jul 10 02:48:03.389129 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.387556 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ff323ccb-9efc-43fd-8ce7-9365fe9af996-cnibin\") pod \"multus-additional-cni-plugins-tfrkp\" (UID: \"ff323ccb-9efc-43fd-8ce7-9365fe9af996\") " pod="openshift-multus/multus-additional-cni-plugins-tfrkp" Jul 10 02:48:03.389129 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.387606 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/29d54ad7-c565-4c22-a182-0e9fe8c8b981-os-release\") pod \"multus-9hgr6\" (UID: \"29d54ad7-c565-4c22-a182-0e9fe8c8b981\") " pod="openshift-multus/multus-9hgr6" Jul 10 02:48:03.389129 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.387602 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a96aece5-40cd-4431-a0e9-1c54d4c6f066-device-dir\") pod \"aws-ebs-csi-driver-node-n7fp4\" (UID: \"a96aece5-40cd-4431-a0e9-1c54d4c6f066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7fp4" Jul 10 02:48:03.389129 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.387623 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/29d54ad7-c565-4c22-a182-0e9fe8c8b981-multus-socket-dir-parent\") pod \"multus-9hgr6\" (UID: \"29d54ad7-c565-4c22-a182-0e9fe8c8b981\") " pod="openshift-multus/multus-9hgr6" Jul 10 02:48:03.389129 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.387640 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29d54ad7-c565-4c22-a182-0e9fe8c8b981-host-var-lib-kubelet\") pod \"multus-9hgr6\" (UID: \"29d54ad7-c565-4c22-a182-0e9fe8c8b981\") " pod="openshift-multus/multus-9hgr6" Jul 10 02:48:03.389129 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.387646 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/29d54ad7-c565-4c22-a182-0e9fe8c8b981-hostroot\") pod \"multus-9hgr6\" (UID: \"29d54ad7-c565-4c22-a182-0e9fe8c8b981\") " pod="openshift-multus/multus-9hgr6" Jul 10 02:48:03.389129 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.387656 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/29d54ad7-c565-4c22-a182-0e9fe8c8b981-multus-daemon-config\") pod \"multus-9hgr6\" (UID: \"29d54ad7-c565-4c22-a182-0e9fe8c8b981\") " pod="openshift-multus/multus-9hgr6" Jul 10 02:48:03.389129 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.387654 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b93349d9-10f0-430c-b00f-74f141f508de-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.389129 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.387673 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b93349d9-10f0-430c-b00f-74f141f508de-var-lib-openvswitch\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.389129 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.387690 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b93349d9-10f0-430c-b00f-74f141f508de-etc-openvswitch\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.389129 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.387703 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a96aece5-40cd-4431-a0e9-1c54d4c6f066-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n7fp4\" (UID: \"a96aece5-40cd-4431-a0e9-1c54d4c6f066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7fp4" Jul 10 02:48:03.389129 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.387708 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b93349d9-10f0-430c-b00f-74f141f508de-run-openvswitch\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.389129 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:03.387764 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 02:48:03.389129 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.387770 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/29d54ad7-c565-4c22-a182-0e9fe8c8b981-os-release\") pod \"multus-9hgr6\" (UID: \"29d54ad7-c565-4c22-a182-0e9fe8c8b981\") " pod="openshift-multus/multus-9hgr6" Jul 10 02:48:03.389698 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.387769 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a96aece5-40cd-4431-a0e9-1c54d4c6f066-socket-dir\") pod \"aws-ebs-csi-driver-node-n7fp4\" (UID: \"a96aece5-40cd-4431-a0e9-1c54d4c6f066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7fp4" Jul 10 02:48:03.389698 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.387673 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dc1f4b26-669e-42a0-abd4-f858ac6b1543-kubelet-config\") pod \"global-pull-secret-syncer-q28nr\" (UID: \"dc1f4b26-669e-42a0-abd4-f858ac6b1543\") " pod="kube-system/global-pull-secret-syncer-q28nr" Jul 10 02:48:03.389698 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.387800 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ff323ccb-9efc-43fd-8ce7-9365fe9af996-os-release\") pod \"multus-additional-cni-plugins-tfrkp\" (UID: \"ff323ccb-9efc-43fd-8ce7-9365fe9af996\") " pod="openshift-multus/multus-additional-cni-plugins-tfrkp" Jul 10 02:48:03.389698 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.387809 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/29d54ad7-c565-4c22-a182-0e9fe8c8b981-multus-socket-dir-parent\") pod \"multus-9hgr6\" (UID: \"29d54ad7-c565-4c22-a182-0e9fe8c8b981\") " pod="openshift-multus/multus-9hgr6" Jul 10 02:48:03.389698 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:03.387809 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc1f4b26-669e-42a0-abd4-f858ac6b1543-original-pull-secret podName:dc1f4b26-669e-42a0-abd4-f858ac6b1543 nodeName:}" failed. No retries permitted until 2026-07-10 02:48:03.887797022 +0000 UTC m=+2.168913287 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dc1f4b26-669e-42a0-abd4-f858ac6b1543-original-pull-secret") pod "global-pull-secret-syncer-q28nr" (UID: "dc1f4b26-669e-42a0-abd4-f858ac6b1543") : object "kube-system"/"original-pull-secret" not registered Jul 10 02:48:03.389698 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.387841 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlznj\" (UniqueName: \"kubernetes.io/projected/29d54ad7-c565-4c22-a182-0e9fe8c8b981-kube-api-access-qlznj\") pod \"multus-9hgr6\" (UID: \"29d54ad7-c565-4c22-a182-0e9fe8c8b981\") " pod="openshift-multus/multus-9hgr6" Jul 10 02:48:03.389698 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.387842 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b93349d9-10f0-430c-b00f-74f141f508de-run-openvswitch\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.389698 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.387864 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29d54ad7-c565-4c22-a182-0e9fe8c8b981-host-var-lib-kubelet\") pod \"multus-9hgr6\" (UID: \"29d54ad7-c565-4c22-a182-0e9fe8c8b981\") " pod="openshift-multus/multus-9hgr6" Jul 10 02:48:03.389698 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.387707 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/29d54ad7-c565-4c22-a182-0e9fe8c8b981-system-cni-dir\") pod \"multus-9hgr6\" (UID: \"29d54ad7-c565-4c22-a182-0e9fe8c8b981\") " pod="openshift-multus/multus-9hgr6" Jul 10 02:48:03.389698 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.387921 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b93349d9-10f0-430c-b00f-74f141f508de-run-ovn\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.389698 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.387956 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/29d54ad7-c565-4c22-a182-0e9fe8c8b981-cnibin\") pod \"multus-9hgr6\" (UID: \"29d54ad7-c565-4c22-a182-0e9fe8c8b981\") " pod="openshift-multus/multus-9hgr6" Jul 10 02:48:03.389698 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.387988 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b93349d9-10f0-430c-b00f-74f141f508de-etc-openvswitch\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.389698 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.387996 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/29d54ad7-c565-4c22-a182-0e9fe8c8b981-cnibin\") pod \"multus-9hgr6\" (UID: \"29d54ad7-c565-4c22-a182-0e9fe8c8b981\") " pod="openshift-multus/multus-9hgr6" Jul 10 02:48:03.389698 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.387997 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b93349d9-10f0-430c-b00f-74f141f508de-run-ovn\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.389698 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.388036 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b93349d9-10f0-430c-b00f-74f141f508de-var-lib-openvswitch\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.389698 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.388062 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/29d54ad7-c565-4c22-a182-0e9fe8c8b981-cni-binary-copy\") pod \"multus-9hgr6\" (UID: \"29d54ad7-c565-4c22-a182-0e9fe8c8b981\") " pod="openshift-multus/multus-9hgr6" Jul 10 02:48:03.389698 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.388115 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ff323ccb-9efc-43fd-8ce7-9365fe9af996-system-cni-dir\") pod \"multus-additional-cni-plugins-tfrkp\" (UID: \"ff323ccb-9efc-43fd-8ce7-9365fe9af996\") " pod="openshift-multus/multus-additional-cni-plugins-tfrkp" Jul 10 02:48:03.390294 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.388137 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/29d54ad7-c565-4c22-a182-0e9fe8c8b981-host-var-lib-cni-multus\") pod \"multus-9hgr6\" (UID: \"29d54ad7-c565-4c22-a182-0e9fe8c8b981\") " pod="openshift-multus/multus-9hgr6" Jul 10 02:48:03.390294 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.388153 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b93349d9-10f0-430c-b00f-74f141f508de-ovnkube-config\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.390294 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.388174 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ff323ccb-9efc-43fd-8ce7-9365fe9af996-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tfrkp\" (UID: \"ff323ccb-9efc-43fd-8ce7-9365fe9af996\") " pod="openshift-multus/multus-additional-cni-plugins-tfrkp" Jul 10 02:48:03.390294 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.388199 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a96aece5-40cd-4431-a0e9-1c54d4c6f066-etc-selinux\") pod \"aws-ebs-csi-driver-node-n7fp4\" (UID: \"a96aece5-40cd-4431-a0e9-1c54d4c6f066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7fp4" Jul 10 02:48:03.390294 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.388216 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/29d54ad7-c565-4c22-a182-0e9fe8c8b981-host-var-lib-cni-multus\") pod \"multus-9hgr6\" (UID: \"29d54ad7-c565-4c22-a182-0e9fe8c8b981\") " pod="openshift-multus/multus-9hgr6" Jul 10 02:48:03.390294 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.388226 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b93349d9-10f0-430c-b00f-74f141f508de-host-kubelet\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.390294 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.388257 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b93349d9-10f0-430c-b00f-74f141f508de-ovn-node-metrics-cert\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.390294 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.388272 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b93349d9-10f0-430c-b00f-74f141f508de-host-kubelet\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.390294 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.388281 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dc1f4b26-669e-42a0-abd4-f858ac6b1543-dbus\") pod \"global-pull-secret-syncer-q28nr\" (UID: \"dc1f4b26-669e-42a0-abd4-f858ac6b1543\") " pod="kube-system/global-pull-secret-syncer-q28nr" Jul 10 02:48:03.390294 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.388306 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a96aece5-40cd-4431-a0e9-1c54d4c6f066-registration-dir\") pod \"aws-ebs-csi-driver-node-n7fp4\" (UID: \"a96aece5-40cd-4431-a0e9-1c54d4c6f066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7fp4" Jul 10 02:48:03.390294 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.388324 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a96aece5-40cd-4431-a0e9-1c54d4c6f066-etc-selinux\") pod \"aws-ebs-csi-driver-node-n7fp4\" (UID: \"a96aece5-40cd-4431-a0e9-1c54d4c6f066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7fp4" Jul 10 02:48:03.390294 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.388333 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/11f18b36-f920-47ba-8b04-10483f768ebf-host\") pod \"node-ca-gp2gc\" (UID: \"11f18b36-f920-47ba-8b04-10483f768ebf\") " pod="openshift-image-registry/node-ca-gp2gc" Jul 10 02:48:03.390294 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.388356 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/29d54ad7-c565-4c22-a182-0e9fe8c8b981-multus-cni-dir\") pod \"multus-9hgr6\" (UID: \"29d54ad7-c565-4c22-a182-0e9fe8c8b981\") " pod="openshift-multus/multus-9hgr6" Jul 10 02:48:03.390294 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.388378 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b93349d9-10f0-430c-b00f-74f141f508de-systemd-units\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.390294 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.388385 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/29d54ad7-c565-4c22-a182-0e9fe8c8b981-multus-daemon-config\") pod \"multus-9hgr6\" (UID: \"29d54ad7-c565-4c22-a182-0e9fe8c8b981\") " pod="openshift-multus/multus-9hgr6" Jul 10 02:48:03.390294 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.388399 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a96aece5-40cd-4431-a0e9-1c54d4c6f066-sys-fs\") pod \"aws-ebs-csi-driver-node-n7fp4\" (UID: \"a96aece5-40cd-4431-a0e9-1c54d4c6f066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7fp4" Jul 10 02:48:03.390294 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.388438 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cc6bf0c3-5e10-4619-9748-a2e9e46754ed-agent-certs\") pod \"konnectivity-agent-kbfmm\" (UID: \"cc6bf0c3-5e10-4619-9748-a2e9e46754ed\") " pod="kube-system/konnectivity-agent-kbfmm" Jul 10 02:48:03.390849 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.388445 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a96aece5-40cd-4431-a0e9-1c54d4c6f066-sys-fs\") pod \"aws-ebs-csi-driver-node-n7fp4\" (UID: \"a96aece5-40cd-4431-a0e9-1c54d4c6f066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7fp4" Jul 10 02:48:03.390849 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.388472 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b93349d9-10f0-430c-b00f-74f141f508de-host-run-netns\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.390849 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.388480 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/11f18b36-f920-47ba-8b04-10483f768ebf-host\") pod \"node-ca-gp2gc\" (UID: \"11f18b36-f920-47ba-8b04-10483f768ebf\") " pod="openshift-image-registry/node-ca-gp2gc" Jul 10 02:48:03.390849 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.388503 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dc1f4b26-669e-42a0-abd4-f858ac6b1543-dbus\") pod \"global-pull-secret-syncer-q28nr\" (UID: \"dc1f4b26-669e-42a0-abd4-f858ac6b1543\") " pod="kube-system/global-pull-secret-syncer-q28nr" Jul 10 02:48:03.390849 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.388529 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/29d54ad7-c565-4c22-a182-0e9fe8c8b981-cni-binary-copy\") pod \"multus-9hgr6\" (UID: \"29d54ad7-c565-4c22-a182-0e9fe8c8b981\") " pod="openshift-multus/multus-9hgr6" Jul 10 02:48:03.390849 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.388541 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b93349d9-10f0-430c-b00f-74f141f508de-host-run-netns\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.390849 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.388554 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a96aece5-40cd-4431-a0e9-1c54d4c6f066-registration-dir\") pod \"aws-ebs-csi-driver-node-n7fp4\" (UID: \"a96aece5-40cd-4431-a0e9-1c54d4c6f066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7fp4" Jul 10 02:48:03.390849 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.388576 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b93349d9-10f0-430c-b00f-74f141f508de-systemd-units\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.390849 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.388606 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/29d54ad7-c565-4c22-a182-0e9fe8c8b981-multus-cni-dir\") pod \"multus-9hgr6\" (UID: \"29d54ad7-c565-4c22-a182-0e9fe8c8b981\") " pod="openshift-multus/multus-9hgr6" Jul 10 02:48:03.390849 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.388635 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ff323ccb-9efc-43fd-8ce7-9365fe9af996-system-cni-dir\") pod \"multus-additional-cni-plugins-tfrkp\" (UID: \"ff323ccb-9efc-43fd-8ce7-9365fe9af996\") " pod="openshift-multus/multus-additional-cni-plugins-tfrkp" Jul 10 02:48:03.390849 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.388662 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b93349d9-10f0-430c-b00f-74f141f508de-ovnkube-config\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.390849 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.388737 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ff323ccb-9efc-43fd-8ce7-9365fe9af996-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tfrkp\" (UID: \"ff323ccb-9efc-43fd-8ce7-9365fe9af996\") " pod="openshift-multus/multus-additional-cni-plugins-tfrkp" Jul 10 02:48:03.390849 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.390780 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cc6bf0c3-5e10-4619-9748-a2e9e46754ed-agent-certs\") pod \"konnectivity-agent-kbfmm\" (UID: \"cc6bf0c3-5e10-4619-9748-a2e9e46754ed\") " pod="kube-system/konnectivity-agent-kbfmm" Jul 10 02:48:03.391326 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.391290 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b93349d9-10f0-430c-b00f-74f141f508de-ovn-node-metrics-cert\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.394244 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.394220 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps682\" (UniqueName: \"kubernetes.io/projected/f9bfec96-adaf-4cfd-9789-cce3b7276803-kube-api-access-ps682\") pod \"iptables-alerter-xdwmm\" (UID: \"f9bfec96-adaf-4cfd-9789-cce3b7276803\") " pod="openshift-network-operator/iptables-alerter-xdwmm" Jul 10 02:48:03.399313 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:03.399296 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 02:48:03.399393 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:03.399316 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 02:48:03.399393 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:03.399328 2575 projected.go:196] Error preparing data for projected volume kube-api-access-jzkpc for pod openshift-network-diagnostics/network-check-target-nf7q8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 02:48:03.399393 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:03.399384 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25d9393a-d380-4ef0-a976-9fd998deb429-kube-api-access-jzkpc podName:25d9393a-d380-4ef0-a976-9fd998deb429 nodeName:}" failed. No retries permitted until 2026-07-10 02:48:03.899368278 +0000 UTC m=+2.180484552 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jzkpc" (UniqueName: "kubernetes.io/projected/25d9393a-d380-4ef0-a976-9fd998deb429-kube-api-access-jzkpc") pod "network-check-target-nf7q8" (UID: "25d9393a-d380-4ef0-a976-9fd998deb429") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 02:48:03.401923 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.401849 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnxs7\" (UniqueName: \"kubernetes.io/projected/ff323ccb-9efc-43fd-8ce7-9365fe9af996-kube-api-access-qnxs7\") pod \"multus-additional-cni-plugins-tfrkp\" (UID: \"ff323ccb-9efc-43fd-8ce7-9365fe9af996\") " pod="openshift-multus/multus-additional-cni-plugins-tfrkp" Jul 10 02:48:03.402088 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.402059 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmcjk\" (UniqueName: \"kubernetes.io/projected/11f18b36-f920-47ba-8b04-10483f768ebf-kube-api-access-nmcjk\") pod \"node-ca-gp2gc\" (UID: \"11f18b36-f920-47ba-8b04-10483f768ebf\") " pod="openshift-image-registry/node-ca-gp2gc" Jul 10 02:48:03.402395 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.402371 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btvh4\" (UniqueName: \"kubernetes.io/projected/b93349d9-10f0-430c-b00f-74f141f508de-kube-api-access-btvh4\") pod \"ovnkube-node-25tvc\" (UID: \"b93349d9-10f0-430c-b00f-74f141f508de\") " pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.403237 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.403214 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86sq4\" (UniqueName: \"kubernetes.io/projected/7d037586-6a56-4354-ba36-a40c2cc2ab48-kube-api-access-86sq4\") pod \"network-metrics-daemon-bgbpr\" (UID: \"7d037586-6a56-4354-ba36-a40c2cc2ab48\") " pod="openshift-multus/network-metrics-daemon-bgbpr" Jul 10 02:48:03.403455 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.403437 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpp5d\" (UniqueName: \"kubernetes.io/projected/a96aece5-40cd-4431-a0e9-1c54d4c6f066-kube-api-access-gpp5d\") pod \"aws-ebs-csi-driver-node-n7fp4\" (UID: \"a96aece5-40cd-4431-a0e9-1c54d4c6f066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7fp4" Jul 10 02:48:03.403570 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.403555 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlznj\" (UniqueName: \"kubernetes.io/projected/29d54ad7-c565-4c22-a182-0e9fe8c8b981-kube-api-access-qlznj\") pod \"multus-9hgr6\" (UID: \"29d54ad7-c565-4c22-a182-0e9fe8c8b981\") " pod="openshift-multus/multus-9hgr6" Jul 10 02:48:03.409848 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:03.409825 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf238b60151ef7c6e38d6cb1b9cdf59c0.slice/crio-ef40b33d7eec85d3ebede19152bae4a419167abf00d3bef8967f7d6c7d34a240 WatchSource:0}: Error finding container ef40b33d7eec85d3ebede19152bae4a419167abf00d3bef8967f7d6c7d34a240: Status 404 returned error can't find the container with id ef40b33d7eec85d3ebede19152bae4a419167abf00d3bef8967f7d6c7d34a240 Jul 10 02:48:03.415936 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.415919 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 02:48:03.449611 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.449588 2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 02:48:03.504020 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.503991 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-spcmm" Jul 10 02:48:03.512680 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:03.512657 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a694fd5_981d_4946_8d5c_e8a62f67dff6.slice/crio-32cbe1a1618bf6b3a61768374b5a1d6e85934a9c0d155b02d526634b66a0379a WatchSource:0}: Error finding container 32cbe1a1618bf6b3a61768374b5a1d6e85934a9c0d155b02d526634b66a0379a: Status 404 returned error can't find the container with id 32cbe1a1618bf6b3a61768374b5a1d6e85934a9c0d155b02d526634b66a0379a Jul 10 02:48:03.521962 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.521944 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-shf9m" Jul 10 02:48:03.530491 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:03.530465 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod731666e1_c05f_4169_98f4_df0efa01817a.slice/crio-7d18fd1d87e2ca84887017a3bb9b2b33ddf49c6734e96dd9115d69219e1044d5 WatchSource:0}: Error finding container 7d18fd1d87e2ca84887017a3bb9b2b33ddf49c6734e96dd9115d69219e1044d5: Status 404 returned error can't find the container with id 7d18fd1d87e2ca84887017a3bb9b2b33ddf49c6734e96dd9115d69219e1044d5 Jul 10 02:48:03.540922 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.540901 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tfrkp" Jul 10 02:48:03.543060 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.543043 2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 02:48:03.549862 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:03.549842 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff323ccb_9efc_43fd_8ce7_9365fe9af996.slice/crio-975fbc9c64bbda0472f67fc40871d1af1131118ccc8f067bbd68c0b956e387ee WatchSource:0}: Error finding container 975fbc9c64bbda0472f67fc40871d1af1131118ccc8f067bbd68c0b956e387ee: Status 404 returned error can't find the container with id 975fbc9c64bbda0472f67fc40871d1af1131118ccc8f067bbd68c0b956e387ee Jul 10 02:48:03.555976 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.555956 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kbfmm" Jul 10 02:48:03.560964 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.560944 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gp2gc" Jul 10 02:48:03.564373 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:03.564354 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc6bf0c3_5e10_4619_9748_a2e9e46754ed.slice/crio-bf072e2ea6df1628fd4470fa0674e1dfd708dd05da26748e891b766980af0eef WatchSource:0}: Error finding container bf072e2ea6df1628fd4470fa0674e1dfd708dd05da26748e891b766980af0eef: Status 404 returned error can't find the container with id bf072e2ea6df1628fd4470fa0674e1dfd708dd05da26748e891b766980af0eef Jul 10 02:48:03.570831 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:03.570788 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11f18b36_f920_47ba_8b04_10483f768ebf.slice/crio-3e74fd7ecd78fdc3def86127648200d853f94ad85241db84a973dd22ed95a3f7 WatchSource:0}: Error finding container 3e74fd7ecd78fdc3def86127648200d853f94ad85241db84a973dd22ed95a3f7: Status 404 returned error can't find the container with id 3e74fd7ecd78fdc3def86127648200d853f94ad85241db84a973dd22ed95a3f7 Jul 10 02:48:03.577049 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.577029 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9hgr6" Jul 10 02:48:03.586083 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:03.586051 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29d54ad7_c565_4c22_a182_0e9fe8c8b981.slice/crio-be1c62d8d00263d6575a2ceb974c0573ee9fb9161c3973ade511827777f90814 WatchSource:0}: Error finding container be1c62d8d00263d6575a2ceb974c0573ee9fb9161c3973ade511827777f90814: Status 404 returned error can't find the container with id be1c62d8d00263d6575a2ceb974c0573ee9fb9161c3973ade511827777f90814 Jul 10 02:48:03.595262 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.595243 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xdwmm" Jul 10 02:48:03.603200 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:03.603175 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9bfec96_adaf_4cfd_9789_cce3b7276803.slice/crio-e56bea9294bd2345e4eb9b1265667d71d0ffcbd051b5541c03e2072cde2f7660 WatchSource:0}: Error finding container e56bea9294bd2345e4eb9b1265667d71d0ffcbd051b5541c03e2072cde2f7660: Status 404 returned error can't find the container with id e56bea9294bd2345e4eb9b1265667d71d0ffcbd051b5541c03e2072cde2f7660 Jul 10 02:48:03.619405 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.619385 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:03.625399 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.625381 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7fp4" Jul 10 02:48:03.628243 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:03.628221 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb93349d9_10f0_430c_b00f_74f141f508de.slice/crio-d5abf9244d6b2657b24f124c0b4e4dc63c12ef6d128c96e461a92093c743e3fc WatchSource:0}: Error finding container d5abf9244d6b2657b24f124c0b4e4dc63c12ef6d128c96e461a92093c743e3fc: Status 404 returned error can't find the container with id d5abf9244d6b2657b24f124c0b4e4dc63c12ef6d128c96e461a92093c743e3fc Jul 10 02:48:03.633744 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:03.633718 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda96aece5_40cd_4431_a0e9_1c54d4c6f066.slice/crio-78cb0e06838f7ad70c1c59e41a3fd5d3df707e07ccb7cbbd8692e2d0ec05c541 WatchSource:0}: Error finding container 78cb0e06838f7ad70c1c59e41a3fd5d3df707e07ccb7cbbd8692e2d0ec05c541: Status 404 returned error can't find the container with id 78cb0e06838f7ad70c1c59e41a3fd5d3df707e07ccb7cbbd8692e2d0ec05c541 Jul 10 02:48:03.893059 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.892972 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d037586-6a56-4354-ba36-a40c2cc2ab48-metrics-certs\") pod \"network-metrics-daemon-bgbpr\" (UID: \"7d037586-6a56-4354-ba36-a40c2cc2ab48\") " pod="openshift-multus/network-metrics-daemon-bgbpr" Jul 10 02:48:03.893059 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.893030 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc1f4b26-669e-42a0-abd4-f858ac6b1543-original-pull-secret\") pod \"global-pull-secret-syncer-q28nr\" (UID: \"dc1f4b26-669e-42a0-abd4-f858ac6b1543\") " pod="kube-system/global-pull-secret-syncer-q28nr" Jul 10 02:48:03.893283 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:03.893154 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 02:48:03.893283 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:03.893159 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 02:48:03.893283 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:03.893218 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc1f4b26-669e-42a0-abd4-f858ac6b1543-original-pull-secret podName:dc1f4b26-669e-42a0-abd4-f858ac6b1543 nodeName:}" failed. No retries permitted until 2026-07-10 02:48:04.893198111 +0000 UTC m=+3.174314378 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dc1f4b26-669e-42a0-abd4-f858ac6b1543-original-pull-secret") pod "global-pull-secret-syncer-q28nr" (UID: "dc1f4b26-669e-42a0-abd4-f858ac6b1543") : object "kube-system"/"original-pull-secret" not registered Jul 10 02:48:03.893283 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:03.893237 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d037586-6a56-4354-ba36-a40c2cc2ab48-metrics-certs podName:7d037586-6a56-4354-ba36-a40c2cc2ab48 nodeName:}" failed. No retries permitted until 2026-07-10 02:48:04.893228107 +0000 UTC m=+3.174344382 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d037586-6a56-4354-ba36-a40c2cc2ab48-metrics-certs") pod "network-metrics-daemon-bgbpr" (UID: "7d037586-6a56-4354-ba36-a40c2cc2ab48") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 02:48:03.993655 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:03.993602 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzkpc\" (UniqueName: \"kubernetes.io/projected/25d9393a-d380-4ef0-a976-9fd998deb429-kube-api-access-jzkpc\") pod \"network-check-target-nf7q8\" (UID: \"25d9393a-d380-4ef0-a976-9fd998deb429\") " pod="openshift-network-diagnostics/network-check-target-nf7q8" Jul 10 02:48:03.993856 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:03.993787 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 02:48:03.993856 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:03.993806 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 02:48:03.993856 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:03.993819 2575 projected.go:196] Error preparing data for projected volume kube-api-access-jzkpc for pod openshift-network-diagnostics/network-check-target-nf7q8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 02:48:03.994043 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:03.993916 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25d9393a-d380-4ef0-a976-9fd998deb429-kube-api-access-jzkpc podName:25d9393a-d380-4ef0-a976-9fd998deb429 nodeName:}" failed. No retries permitted until 2026-07-10 02:48:04.993894803 +0000 UTC m=+3.275011070 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jzkpc" (UniqueName: "kubernetes.io/projected/25d9393a-d380-4ef0-a976-9fd998deb429-kube-api-access-jzkpc") pod "network-check-target-nf7q8" (UID: "25d9393a-d380-4ef0-a976-9fd998deb429") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 02:48:04.263366 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:04.263182 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 02:43:03 +0000 UTC" deadline="2028-01-31 12:36:02.592072078 +0000 UTC" Jul 10 02:48:04.263366 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:04.263253 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13689h47m58.328823636s" Jul 10 02:48:04.285035 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:04.284082 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nf7q8" Jul 10 02:48:04.285035 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:04.284218 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nf7q8" podUID="25d9393a-d380-4ef0-a976-9fd998deb429" Jul 10 02:48:04.285035 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:04.284676 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bgbpr" Jul 10 02:48:04.285035 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:04.284800 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bgbpr" podUID="7d037586-6a56-4354-ba36-a40c2cc2ab48" Jul 10 02:48:04.296768 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:04.296655 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kbfmm" event={"ID":"cc6bf0c3-5e10-4619-9748-a2e9e46754ed","Type":"ContainerStarted","Data":"bf072e2ea6df1628fd4470fa0674e1dfd708dd05da26748e891b766980af0eef"} Jul 10 02:48:04.303219 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:04.303184 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tfrkp" event={"ID":"ff323ccb-9efc-43fd-8ce7-9365fe9af996","Type":"ContainerStarted","Data":"975fbc9c64bbda0472f67fc40871d1af1131118ccc8f067bbd68c0b956e387ee"} Jul 10 02:48:04.315091 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:04.315060 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-139.ec2.internal" event={"ID":"e1e6758aab956feebe21df5c8cc1fa0a","Type":"ContainerStarted","Data":"9375de5548656311a2a3119d19de679582d9752406fdb50249d2d6d7fdee923f"} Jul 10 02:48:04.331024 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:04.330785 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-139.ec2.internal" event={"ID":"f238b60151ef7c6e38d6cb1b9cdf59c0","Type":"ContainerStarted","Data":"ef40b33d7eec85d3ebede19152bae4a419167abf00d3bef8967f7d6c7d34a240"} Jul 10 02:48:04.345984 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:04.345946 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7fp4" event={"ID":"a96aece5-40cd-4431-a0e9-1c54d4c6f066","Type":"ContainerStarted","Data":"78cb0e06838f7ad70c1c59e41a3fd5d3df707e07ccb7cbbd8692e2d0ec05c541"} Jul 10 02:48:04.350081 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:04.349979 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xdwmm" event={"ID":"f9bfec96-adaf-4cfd-9789-cce3b7276803","Type":"ContainerStarted","Data":"e56bea9294bd2345e4eb9b1265667d71d0ffcbd051b5541c03e2072cde2f7660"} Jul 10 02:48:04.354114 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:04.354084 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-shf9m" event={"ID":"731666e1-c05f-4169-98f4-df0efa01817a","Type":"ContainerStarted","Data":"7d18fd1d87e2ca84887017a3bb9b2b33ddf49c6734e96dd9115d69219e1044d5"} Jul 10 02:48:04.356477 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:04.356436 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-spcmm" event={"ID":"7a694fd5-981d-4946-8d5c-e8a62f67dff6","Type":"ContainerStarted","Data":"32cbe1a1618bf6b3a61768374b5a1d6e85934a9c0d155b02d526634b66a0379a"} Jul 10 02:48:04.359872 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:04.359605 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" event={"ID":"b93349d9-10f0-430c-b00f-74f141f508de","Type":"ContainerStarted","Data":"d5abf9244d6b2657b24f124c0b4e4dc63c12ef6d128c96e461a92093c743e3fc"} Jul 10 02:48:04.372699 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:04.372673 2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 02:48:04.376122 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:04.376090 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9hgr6" event={"ID":"29d54ad7-c565-4c22-a182-0e9fe8c8b981","Type":"ContainerStarted","Data":"be1c62d8d00263d6575a2ceb974c0573ee9fb9161c3973ade511827777f90814"} Jul 10 02:48:04.383517 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:04.383485 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gp2gc" event={"ID":"11f18b36-f920-47ba-8b04-10483f768ebf","Type":"ContainerStarted","Data":"3e74fd7ecd78fdc3def86127648200d853f94ad85241db84a973dd22ed95a3f7"} Jul 10 02:48:04.902536 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:04.902413 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d037586-6a56-4354-ba36-a40c2cc2ab48-metrics-certs\") pod \"network-metrics-daemon-bgbpr\" (UID: \"7d037586-6a56-4354-ba36-a40c2cc2ab48\") " pod="openshift-multus/network-metrics-daemon-bgbpr" Jul 10 02:48:04.902536 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:04.902462 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc1f4b26-669e-42a0-abd4-f858ac6b1543-original-pull-secret\") pod \"global-pull-secret-syncer-q28nr\" (UID: \"dc1f4b26-669e-42a0-abd4-f858ac6b1543\") " pod="kube-system/global-pull-secret-syncer-q28nr" Jul 10 02:48:04.902804 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:04.902624 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 02:48:04.902804 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:04.902693 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc1f4b26-669e-42a0-abd4-f858ac6b1543-original-pull-secret podName:dc1f4b26-669e-42a0-abd4-f858ac6b1543 nodeName:}" failed. No retries permitted until 2026-07-10 02:48:06.902670429 +0000 UTC m=+5.183786704 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dc1f4b26-669e-42a0-abd4-f858ac6b1543-original-pull-secret") pod "global-pull-secret-syncer-q28nr" (UID: "dc1f4b26-669e-42a0-abd4-f858ac6b1543") : object "kube-system"/"original-pull-secret" not registered Jul 10 02:48:04.903455 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:04.902934 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 02:48:04.903455 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:04.903000 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d037586-6a56-4354-ba36-a40c2cc2ab48-metrics-certs podName:7d037586-6a56-4354-ba36-a40c2cc2ab48 nodeName:}" failed. No retries permitted until 2026-07-10 02:48:06.902982849 +0000 UTC m=+5.184099130 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d037586-6a56-4354-ba36-a40c2cc2ab48-metrics-certs") pod "network-metrics-daemon-bgbpr" (UID: "7d037586-6a56-4354-ba36-a40c2cc2ab48") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 02:48:05.004113 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:05.003369 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzkpc\" (UniqueName: \"kubernetes.io/projected/25d9393a-d380-4ef0-a976-9fd998deb429-kube-api-access-jzkpc\") pod \"network-check-target-nf7q8\" (UID: \"25d9393a-d380-4ef0-a976-9fd998deb429\") " pod="openshift-network-diagnostics/network-check-target-nf7q8" Jul 10 02:48:05.004113 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:05.003614 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 02:48:05.004113 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:05.003637 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 02:48:05.004113 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:05.003650 2575 projected.go:196] Error preparing data for projected volume kube-api-access-jzkpc for pod openshift-network-diagnostics/network-check-target-nf7q8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 02:48:05.004113 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:05.003713 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25d9393a-d380-4ef0-a976-9fd998deb429-kube-api-access-jzkpc podName:25d9393a-d380-4ef0-a976-9fd998deb429 nodeName:}" failed. No retries permitted until 2026-07-10 02:48:07.003693747 +0000 UTC m=+5.284810034 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jzkpc" (UniqueName: "kubernetes.io/projected/25d9393a-d380-4ef0-a976-9fd998deb429-kube-api-access-jzkpc") pod "network-check-target-nf7q8" (UID: "25d9393a-d380-4ef0-a976-9fd998deb429") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 02:48:05.021051 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:05.021014 2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 02:48:05.264449 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:05.264350 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 02:43:03 +0000 UTC" deadline="2028-03-31 00:08:41.989986462 +0000 UTC" Jul 10 02:48:05.264449 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:05.264395 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15117h20m36.725596615s" Jul 10 02:48:05.282836 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:05.282797 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q28nr" Jul 10 02:48:05.283023 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:05.282966 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q28nr" podUID="dc1f4b26-669e-42a0-abd4-f858ac6b1543" Jul 10 02:48:06.284489 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:06.283988 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nf7q8" Jul 10 02:48:06.284489 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:06.284123 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nf7q8" podUID="25d9393a-d380-4ef0-a976-9fd998deb429" Jul 10 02:48:06.284489 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:06.284178 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bgbpr" Jul 10 02:48:06.284489 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:06.284308 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bgbpr" podUID="7d037586-6a56-4354-ba36-a40c2cc2ab48" Jul 10 02:48:06.921634 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:06.921591 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d037586-6a56-4354-ba36-a40c2cc2ab48-metrics-certs\") pod \"network-metrics-daemon-bgbpr\" (UID: \"7d037586-6a56-4354-ba36-a40c2cc2ab48\") " pod="openshift-multus/network-metrics-daemon-bgbpr" Jul 10 02:48:06.921828 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:06.921665 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc1f4b26-669e-42a0-abd4-f858ac6b1543-original-pull-secret\") pod \"global-pull-secret-syncer-q28nr\" (UID: \"dc1f4b26-669e-42a0-abd4-f858ac6b1543\") " pod="kube-system/global-pull-secret-syncer-q28nr" Jul 10 02:48:06.921937 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:06.921830 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 02:48:06.921937 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:06.921912 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc1f4b26-669e-42a0-abd4-f858ac6b1543-original-pull-secret podName:dc1f4b26-669e-42a0-abd4-f858ac6b1543 nodeName:}" failed. No retries permitted until 2026-07-10 02:48:10.921891775 +0000 UTC m=+9.203008041 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dc1f4b26-669e-42a0-abd4-f858ac6b1543-original-pull-secret") pod "global-pull-secret-syncer-q28nr" (UID: "dc1f4b26-669e-42a0-abd4-f858ac6b1543") : object "kube-system"/"original-pull-secret" not registered Jul 10 02:48:06.922358 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:06.922335 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 02:48:06.922449 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:06.922398 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d037586-6a56-4354-ba36-a40c2cc2ab48-metrics-certs podName:7d037586-6a56-4354-ba36-a40c2cc2ab48 nodeName:}" failed. No retries permitted until 2026-07-10 02:48:10.922380988 +0000 UTC m=+9.203497270 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d037586-6a56-4354-ba36-a40c2cc2ab48-metrics-certs") pod "network-metrics-daemon-bgbpr" (UID: "7d037586-6a56-4354-ba36-a40c2cc2ab48") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 02:48:07.023103 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:07.023066 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzkpc\" (UniqueName: \"kubernetes.io/projected/25d9393a-d380-4ef0-a976-9fd998deb429-kube-api-access-jzkpc\") pod \"network-check-target-nf7q8\" (UID: \"25d9393a-d380-4ef0-a976-9fd998deb429\") " pod="openshift-network-diagnostics/network-check-target-nf7q8" Jul 10 02:48:07.023335 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:07.023313 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 02:48:07.023411 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:07.023347 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 02:48:07.023411 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:07.023362 2575 projected.go:196] Error preparing data for projected volume kube-api-access-jzkpc for pod openshift-network-diagnostics/network-check-target-nf7q8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 02:48:07.023526 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:07.023431 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25d9393a-d380-4ef0-a976-9fd998deb429-kube-api-access-jzkpc podName:25d9393a-d380-4ef0-a976-9fd998deb429 nodeName:}" failed. No retries permitted until 2026-07-10 02:48:11.023410418 +0000 UTC m=+9.304526700 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jzkpc" (UniqueName: "kubernetes.io/projected/25d9393a-d380-4ef0-a976-9fd998deb429-kube-api-access-jzkpc") pod "network-check-target-nf7q8" (UID: "25d9393a-d380-4ef0-a976-9fd998deb429") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 02:48:07.282707 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:07.282622 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q28nr" Jul 10 02:48:07.282903 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:07.282758 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q28nr" podUID="dc1f4b26-669e-42a0-abd4-f858ac6b1543" Jul 10 02:48:08.283112 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:08.283076 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nf7q8" Jul 10 02:48:08.283112 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:08.283103 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bgbpr" Jul 10 02:48:08.283691 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:08.283233 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nf7q8" podUID="25d9393a-d380-4ef0-a976-9fd998deb429" Jul 10 02:48:08.283691 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:08.283386 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bgbpr" podUID="7d037586-6a56-4354-ba36-a40c2cc2ab48" Jul 10 02:48:09.283085 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:09.283036 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q28nr" Jul 10 02:48:09.283278 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:09.283190 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q28nr" podUID="dc1f4b26-669e-42a0-abd4-f858ac6b1543" Jul 10 02:48:10.282857 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:10.282800 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bgbpr" Jul 10 02:48:10.283041 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:10.282977 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bgbpr" podUID="7d037586-6a56-4354-ba36-a40c2cc2ab48" Jul 10 02:48:10.283350 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:10.282800 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nf7q8" Jul 10 02:48:10.283350 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:10.283305 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nf7q8" podUID="25d9393a-d380-4ef0-a976-9fd998deb429" Jul 10 02:48:10.961096 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:10.961004 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d037586-6a56-4354-ba36-a40c2cc2ab48-metrics-certs\") pod \"network-metrics-daemon-bgbpr\" (UID: \"7d037586-6a56-4354-ba36-a40c2cc2ab48\") " pod="openshift-multus/network-metrics-daemon-bgbpr" Jul 10 02:48:10.961096 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:10.961063 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc1f4b26-669e-42a0-abd4-f858ac6b1543-original-pull-secret\") pod \"global-pull-secret-syncer-q28nr\" (UID: \"dc1f4b26-669e-42a0-abd4-f858ac6b1543\") " pod="kube-system/global-pull-secret-syncer-q28nr" Jul 10 02:48:10.961343 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:10.961205 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 02:48:10.961343 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:10.961203 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 02:48:10.961343 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:10.961275 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc1f4b26-669e-42a0-abd4-f858ac6b1543-original-pull-secret podName:dc1f4b26-669e-42a0-abd4-f858ac6b1543 nodeName:}" failed. No retries permitted until 2026-07-10 02:48:18.96125762 +0000 UTC m=+17.242373888 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dc1f4b26-669e-42a0-abd4-f858ac6b1543-original-pull-secret") pod "global-pull-secret-syncer-q28nr" (UID: "dc1f4b26-669e-42a0-abd4-f858ac6b1543") : object "kube-system"/"original-pull-secret" not registered Jul 10 02:48:10.961343 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:10.961325 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d037586-6a56-4354-ba36-a40c2cc2ab48-metrics-certs podName:7d037586-6a56-4354-ba36-a40c2cc2ab48 nodeName:}" failed. No retries permitted until 2026-07-10 02:48:18.96130465 +0000 UTC m=+17.242420937 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d037586-6a56-4354-ba36-a40c2cc2ab48-metrics-certs") pod "network-metrics-daemon-bgbpr" (UID: "7d037586-6a56-4354-ba36-a40c2cc2ab48") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 02:48:11.062376 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:11.062326 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzkpc\" (UniqueName: \"kubernetes.io/projected/25d9393a-d380-4ef0-a976-9fd998deb429-kube-api-access-jzkpc\") pod \"network-check-target-nf7q8\" (UID: \"25d9393a-d380-4ef0-a976-9fd998deb429\") " pod="openshift-network-diagnostics/network-check-target-nf7q8" Jul 10 02:48:11.062564 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:11.062525 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 02:48:11.062564 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:11.062543 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 02:48:11.062564 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:11.062556 2575 projected.go:196] Error preparing data for projected volume kube-api-access-jzkpc for pod openshift-network-diagnostics/network-check-target-nf7q8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 02:48:11.062731 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:11.062617 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25d9393a-d380-4ef0-a976-9fd998deb429-kube-api-access-jzkpc podName:25d9393a-d380-4ef0-a976-9fd998deb429 nodeName:}" failed. No retries permitted until 2026-07-10 02:48:19.062599179 +0000 UTC m=+17.343715446 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jzkpc" (UniqueName: "kubernetes.io/projected/25d9393a-d380-4ef0-a976-9fd998deb429-kube-api-access-jzkpc") pod "network-check-target-nf7q8" (UID: "25d9393a-d380-4ef0-a976-9fd998deb429") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 02:48:11.283061 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:11.282977 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q28nr" Jul 10 02:48:11.283220 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:11.283107 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q28nr" podUID="dc1f4b26-669e-42a0-abd4-f858ac6b1543" Jul 10 02:48:12.283944 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:12.283911 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nf7q8" Jul 10 02:48:12.284397 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:12.284018 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nf7q8" podUID="25d9393a-d380-4ef0-a976-9fd998deb429" Jul 10 02:48:12.284397 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:12.284070 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bgbpr" Jul 10 02:48:12.284397 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:12.284128 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bgbpr" podUID="7d037586-6a56-4354-ba36-a40c2cc2ab48" Jul 10 02:48:13.282762 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:13.282725 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q28nr" Jul 10 02:48:13.282971 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:13.282876 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q28nr" podUID="dc1f4b26-669e-42a0-abd4-f858ac6b1543" Jul 10 02:48:14.283256 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:14.283224 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bgbpr" Jul 10 02:48:14.283711 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:14.283225 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nf7q8" Jul 10 02:48:14.283711 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:14.283349 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bgbpr" podUID="7d037586-6a56-4354-ba36-a40c2cc2ab48" Jul 10 02:48:14.283711 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:14.283408 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nf7q8" podUID="25d9393a-d380-4ef0-a976-9fd998deb429" Jul 10 02:48:15.282544 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:15.282511 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q28nr" Jul 10 02:48:15.282751 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:15.282624 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q28nr" podUID="dc1f4b26-669e-42a0-abd4-f858ac6b1543" Jul 10 02:48:16.282713 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:16.282672 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bgbpr" Jul 10 02:48:16.283189 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:16.282673 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nf7q8" Jul 10 02:48:16.283189 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:16.282834 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bgbpr" podUID="7d037586-6a56-4354-ba36-a40c2cc2ab48" Jul 10 02:48:16.283189 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:16.282899 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nf7q8" podUID="25d9393a-d380-4ef0-a976-9fd998deb429" Jul 10 02:48:17.283075 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:17.283037 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q28nr" Jul 10 02:48:17.283520 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:17.283165 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q28nr" podUID="dc1f4b26-669e-42a0-abd4-f858ac6b1543" Jul 10 02:48:18.282938 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:18.282902 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bgbpr" Jul 10 02:48:18.283167 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:18.282902 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nf7q8" Jul 10 02:48:18.283167 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:18.283046 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bgbpr" podUID="7d037586-6a56-4354-ba36-a40c2cc2ab48" Jul 10 02:48:18.283167 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:18.283115 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nf7q8" podUID="25d9393a-d380-4ef0-a976-9fd998deb429" Jul 10 02:48:19.023086 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:19.023048 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d037586-6a56-4354-ba36-a40c2cc2ab48-metrics-certs\") pod \"network-metrics-daemon-bgbpr\" (UID: \"7d037586-6a56-4354-ba36-a40c2cc2ab48\") " pod="openshift-multus/network-metrics-daemon-bgbpr" Jul 10 02:48:19.023086 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:19.023093 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc1f4b26-669e-42a0-abd4-f858ac6b1543-original-pull-secret\") pod \"global-pull-secret-syncer-q28nr\" (UID: \"dc1f4b26-669e-42a0-abd4-f858ac6b1543\") " pod="kube-system/global-pull-secret-syncer-q28nr" Jul 10 02:48:19.023363 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:19.023193 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 02:48:19.023363 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:19.023247 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc1f4b26-669e-42a0-abd4-f858ac6b1543-original-pull-secret podName:dc1f4b26-669e-42a0-abd4-f858ac6b1543 nodeName:}" failed. No retries permitted until 2026-07-10 02:48:35.023233821 +0000 UTC m=+33.304350084 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dc1f4b26-669e-42a0-abd4-f858ac6b1543-original-pull-secret") pod "global-pull-secret-syncer-q28nr" (UID: "dc1f4b26-669e-42a0-abd4-f858ac6b1543") : object "kube-system"/"original-pull-secret" not registered Jul 10 02:48:19.023363 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:19.023193 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 02:48:19.023363 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:19.023313 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d037586-6a56-4354-ba36-a40c2cc2ab48-metrics-certs podName:7d037586-6a56-4354-ba36-a40c2cc2ab48 nodeName:}" failed. No retries permitted until 2026-07-10 02:48:35.023301122 +0000 UTC m=+33.304417396 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d037586-6a56-4354-ba36-a40c2cc2ab48-metrics-certs") pod "network-metrics-daemon-bgbpr" (UID: "7d037586-6a56-4354-ba36-a40c2cc2ab48") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 02:48:19.124187 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:19.124147 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzkpc\" (UniqueName: \"kubernetes.io/projected/25d9393a-d380-4ef0-a976-9fd998deb429-kube-api-access-jzkpc\") pod \"network-check-target-nf7q8\" (UID: \"25d9393a-d380-4ef0-a976-9fd998deb429\") " pod="openshift-network-diagnostics/network-check-target-nf7q8" Jul 10 02:48:19.124376 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:19.124313 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 02:48:19.124376 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:19.124329 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 02:48:19.124376 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:19.124337 2575 projected.go:196] Error preparing data for projected volume kube-api-access-jzkpc for pod openshift-network-diagnostics/network-check-target-nf7q8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 02:48:19.124543 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:19.124389 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25d9393a-d380-4ef0-a976-9fd998deb429-kube-api-access-jzkpc podName:25d9393a-d380-4ef0-a976-9fd998deb429 nodeName:}" failed. No retries permitted until 2026-07-10 02:48:35.124376002 +0000 UTC m=+33.405492264 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jzkpc" (UniqueName: "kubernetes.io/projected/25d9393a-d380-4ef0-a976-9fd998deb429-kube-api-access-jzkpc") pod "network-check-target-nf7q8" (UID: "25d9393a-d380-4ef0-a976-9fd998deb429") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 02:48:19.283025 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:19.282945 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q28nr" Jul 10 02:48:19.283230 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:19.283067 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q28nr" podUID="dc1f4b26-669e-42a0-abd4-f858ac6b1543" Jul 10 02:48:20.282527 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:20.282494 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bgbpr" Jul 10 02:48:20.282734 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:20.282493 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nf7q8" Jul 10 02:48:20.282734 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:20.282629 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bgbpr" podUID="7d037586-6a56-4354-ba36-a40c2cc2ab48" Jul 10 02:48:20.282734 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:20.282669 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nf7q8" podUID="25d9393a-d380-4ef0-a976-9fd998deb429" Jul 10 02:48:21.282929 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:21.282889 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q28nr" Jul 10 02:48:21.283411 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:21.283011 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q28nr" podUID="dc1f4b26-669e-42a0-abd4-f858ac6b1543" Jul 10 02:48:22.283209 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:22.283180 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nf7q8" Jul 10 02:48:22.283654 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:22.283310 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nf7q8" podUID="25d9393a-d380-4ef0-a976-9fd998deb429" Jul 10 02:48:22.283654 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:22.283361 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bgbpr" Jul 10 02:48:22.283654 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:22.283477 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bgbpr" podUID="7d037586-6a56-4354-ba36-a40c2cc2ab48" Jul 10 02:48:23.282252 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:23.282217 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q28nr" Jul 10 02:48:23.282443 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:23.282346 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q28nr" podUID="dc1f4b26-669e-42a0-abd4-f858ac6b1543" Jul 10 02:48:24.282846 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:24.282774 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bgbpr" Jul 10 02:48:24.283310 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:24.282781 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nf7q8" Jul 10 02:48:24.283310 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:24.282942 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bgbpr" podUID="7d037586-6a56-4354-ba36-a40c2cc2ab48" Jul 10 02:48:24.283310 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:24.283006 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nf7q8" podUID="25d9393a-d380-4ef0-a976-9fd998deb429" Jul 10 02:48:24.435662 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:24.435627 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-139.ec2.internal" event={"ID":"f238b60151ef7c6e38d6cb1b9cdf59c0","Type":"ContainerStarted","Data":"4b19acf3c08e9b9979764309c9ae5c5a2d1beb47e4097b019b4f91d8fa9d2806"} Jul 10 02:48:24.437110 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:24.437088 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-spcmm" event={"ID":"7a694fd5-981d-4946-8d5c-e8a62f67dff6","Type":"ContainerStarted","Data":"f33fa6a09841fa442eeefbbd3923b16eec674a4dae5b804edffbedc68f7e7267"} Jul 10 02:48:24.443493 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:24.443470 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9hgr6" event={"ID":"29d54ad7-c565-4c22-a182-0e9fe8c8b981","Type":"ContainerStarted","Data":"b3b9a47bdad1747e44a1a553ce8fd47f12ae35d123c046158e63d96b3c36e020"} Jul 10 02:48:24.454129 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:24.454090 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-139.ec2.internal" podStartSLOduration=21.454078016 podStartE2EDuration="21.454078016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 02:48:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:48:24.453559996 +0000 UTC m=+22.734676282" watchObservedRunningTime="2026-07-10 02:48:24.454078016 +0000 UTC m=+22.735194301" Jul 10 02:48:24.470608 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:24.470569 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-9hgr6" podStartSLOduration=0.913994466 podStartE2EDuration="21.470556667s" totalImagesPullingTime="20.556562201s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 02:48:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:48:24.470225881 +0000 UTC m=+22.751342187" watchObservedRunningTime="2026-07-10 02:48:24.470556667 +0000 UTC m=+22.751672951" Jul 10 02:48:24.487414 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:24.487367 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-spcmm" podStartSLOduration=0.893837543 podStartE2EDuration="21.487350707s" totalImagesPullingTime="20.593513164s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 02:48:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:48:24.487158173 +0000 UTC m=+22.768274458" watchObservedRunningTime="2026-07-10 02:48:24.487350707 +0000 UTC m=+22.768466993" Jul 10 02:48:25.282984 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:25.282669 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q28nr" Jul 10 02:48:25.283651 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:25.283086 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q28nr" podUID="dc1f4b26-669e-42a0-abd4-f858ac6b1543" Jul 10 02:48:25.447388 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:25.447307 2575 generic.go:356] "Generic (PLEG): container finished" podID="e1e6758aab956feebe21df5c8cc1fa0a" containerID="3a64878c46df2286d71368faf826c43f6400f37206efeb48f8f8282a2adb77ea" exitCode=0 Jul 10 02:48:25.447538 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:25.447393 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-139.ec2.internal" event={"ID":"e1e6758aab956feebe21df5c8cc1fa0a","Type":"ContainerDied","Data":"3a64878c46df2286d71368faf826c43f6400f37206efeb48f8f8282a2adb77ea"} Jul 10 02:48:25.449174 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:25.449143 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7fp4" event={"ID":"a96aece5-40cd-4431-a0e9-1c54d4c6f066","Type":"ContainerStarted","Data":"a2ef40257c6f4fef8748e53ee2ba955f62f74d963972df876e55ef23fc10b071"} Jul 10 02:48:25.450792 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:25.450768 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-shf9m" event={"ID":"731666e1-c05f-4169-98f4-df0efa01817a","Type":"ContainerStarted","Data":"8264ac6874e494886abec92aee2369f818580bcf8d775801d33130ecfc8e9758"} Jul 10 02:48:25.453744 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:25.453720 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" event={"ID":"b93349d9-10f0-430c-b00f-74f141f508de","Type":"ContainerStarted","Data":"5a83e67e61a88d3dc443614289ef562dc5d72a1747a253ee8bcee1f59aadcf45"} Jul 10 02:48:25.453872 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:25.453794 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" event={"ID":"b93349d9-10f0-430c-b00f-74f141f508de","Type":"ContainerStarted","Data":"c4c1d86aa0e16c62e9994dac76f489ca522d99b90586c6be8b235714d87a1e3c"} Jul 10 02:48:25.453872 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:25.453854 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" event={"ID":"b93349d9-10f0-430c-b00f-74f141f508de","Type":"ContainerStarted","Data":"3685eb87e9fbf1d13d2bab31e8c937403d085addfc1df144d6893b973936bba1"} Jul 10 02:48:25.453988 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:25.453891 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" event={"ID":"b93349d9-10f0-430c-b00f-74f141f508de","Type":"ContainerStarted","Data":"8e44a3648e689f6f2b2ff8619b8907481d70eac9b7bfca5db314002afc1b886d"} Jul 10 02:48:25.453988 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:25.453935 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" event={"ID":"b93349d9-10f0-430c-b00f-74f141f508de","Type":"ContainerStarted","Data":"7536af557c16c6cb4a05d666f4cf3ff5b306d966c18d20b92433f6d4963a1f95"} Jul 10 02:48:25.455302 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:25.455280 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gp2gc" event={"ID":"11f18b36-f920-47ba-8b04-10483f768ebf","Type":"ContainerStarted","Data":"a4abb29628e3b696a080035368683ff59a8686f76e95e0b25dc5b17988d81969"} Jul 10 02:48:25.457073 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:25.457045 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kbfmm" event={"ID":"cc6bf0c3-5e10-4619-9748-a2e9e46754ed","Type":"ContainerStarted","Data":"99ff83bb7e03121ba0c538045cd894cac5195974b18b086b14fb999b15ae670a"} Jul 10 02:48:25.459224 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:25.459200 2575 generic.go:356] "Generic (PLEG): container finished" podID="ff323ccb-9efc-43fd-8ce7-9365fe9af996" containerID="0e6784ffb1d2be67f3e95e2dd9fdba29df2f3891bfa45f9a71df195761dd73d1" exitCode=0 Jul 10 02:48:25.459334 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:25.459246 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tfrkp" event={"ID":"ff323ccb-9efc-43fd-8ce7-9365fe9af996","Type":"ContainerDied","Data":"0e6784ffb1d2be67f3e95e2dd9fdba29df2f3891bfa45f9a71df195761dd73d1"} Jul 10 02:48:25.495287 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:25.495232 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-kbfmm" podStartSLOduration=1.9544241329999998 podStartE2EDuration="22.495210972s" totalImagesPullingTime="20.540786839s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:48:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:48:25.494383909 +0000 UTC m=+23.775500194" watchObservedRunningTime="2026-07-10 02:48:25.495210972 +0000 UTC m=+23.776327299" Jul 10 02:48:25.521940 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:25.521890 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-shf9m" podStartSLOduration=10.531116936 podStartE2EDuration="22.521869096s" totalImagesPullingTime="11.99075216s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 02:48:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:48:25.521511413 +0000 UTC m=+23.802627702" watchObservedRunningTime="2026-07-10 02:48:25.521869096 +0000 UTC m=+23.802985374" Jul 10 02:48:25.522619 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:25.522288 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-gp2gc" podStartSLOduration=2.316599233 podStartE2EDuration="22.522276748s" totalImagesPullingTime="20.205677515s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 02:48:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:48:25.508232599 +0000 UTC m=+23.789351019" watchObservedRunningTime="2026-07-10 02:48:25.522276748 +0000 UTC m=+23.803393035" Jul 10 02:48:25.738866 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:25.738775 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-kbfmm" Jul 10 02:48:26.180481 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:26.180457 2575 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 10 02:48:26.260871 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:26.260748 2575 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-10T02:48:26.180476254Z","UUID":"dc035f5b-ab59-4f2c-8241-d91c8137f5db","Handler":null,"Name":"","Endpoint":""} Jul 10 02:48:26.262411 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:26.262383 2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 10 02:48:26.262546 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:26.262431 2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 10 02:48:26.282278 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:26.282251 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bgbpr" Jul 10 02:48:26.282425 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:26.282351 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bgbpr" podUID="7d037586-6a56-4354-ba36-a40c2cc2ab48" Jul 10 02:48:26.282496 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:26.282434 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nf7q8" Jul 10 02:48:26.282550 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:26.282505 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nf7q8" podUID="25d9393a-d380-4ef0-a976-9fd998deb429" Jul 10 02:48:26.464254 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:26.464174 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" event={"ID":"b93349d9-10f0-430c-b00f-74f141f508de","Type":"ContainerStarted","Data":"9a1eb902160528ed23ee4e620879f3ddbbdb8a60b6929982836c214c5e77a989"} Jul 10 02:48:26.465847 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:26.465802 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-139.ec2.internal" event={"ID":"e1e6758aab956feebe21df5c8cc1fa0a","Type":"ContainerStarted","Data":"42ae627c16040a5b3b13a02323af1a699cfc25ff0afb32300fd3bc6527630c76"} Jul 10 02:48:26.467442 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:26.467421 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7fp4" event={"ID":"a96aece5-40cd-4431-a0e9-1c54d4c6f066","Type":"ContainerStarted","Data":"390a8e66a3c699de4e225d7206ea9f030d201f235d85e981910bf4e285563595"} Jul 10 02:48:26.468574 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:26.468550 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xdwmm" event={"ID":"f9bfec96-adaf-4cfd-9789-cce3b7276803","Type":"ContainerStarted","Data":"00f83ead428931f1ddac891fe4b712e83a7aad1f92f579673a858606f752fb1e"} Jul 10 02:48:26.480404 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:26.480359 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-139.ec2.internal" podStartSLOduration=23.480344933 podStartE2EDuration="23.480344933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 02:48:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:48:26.479954208 +0000 UTC m=+24.761070492" watchObservedRunningTime="2026-07-10 02:48:26.480344933 +0000 UTC m=+24.761461218" Jul 10 02:48:26.494145 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:26.494098 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xdwmm" podStartSLOduration=2.991217356 podStartE2EDuration="23.494085298s" totalImagesPullingTime="20.502867942s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 02:48:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:48:26.493625418 +0000 UTC m=+24.774741703" watchObservedRunningTime="2026-07-10 02:48:26.494085298 +0000 UTC m=+24.775201582" Jul 10 02:48:27.282896 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:27.282868 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q28nr" Jul 10 02:48:27.283165 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:27.282979 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q28nr" podUID="dc1f4b26-669e-42a0-abd4-f858ac6b1543" Jul 10 02:48:28.283121 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:28.282914 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bgbpr" Jul 10 02:48:28.283582 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:28.282914 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nf7q8" Jul 10 02:48:28.283582 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:28.283221 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bgbpr" podUID="7d037586-6a56-4354-ba36-a40c2cc2ab48" Jul 10 02:48:28.283582 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:28.283292 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nf7q8" podUID="25d9393a-d380-4ef0-a976-9fd998deb429" Jul 10 02:48:28.476701 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:28.476647 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" event={"ID":"b93349d9-10f0-430c-b00f-74f141f508de","Type":"ContainerStarted","Data":"f0fad757662460d94fb8fbd02215c042800d8005659c9809f921d71a5a34d738"} Jul 10 02:48:28.478952 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:28.478922 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7fp4" event={"ID":"a96aece5-40cd-4431-a0e9-1c54d4c6f066","Type":"ContainerStarted","Data":"7f81fe44e0f36313435ab70b868294babc76737efc85f9ce7f4d9f7b4f1eb74c"} Jul 10 02:48:28.496974 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:28.496928 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n7fp4" podStartSLOduration=6.898371585 podStartE2EDuration="25.496915137s" totalImagesPullingTime="18.598543552s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 02:48:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:48:28.496363672 +0000 UTC m=+26.777480164" watchObservedRunningTime="2026-07-10 02:48:28.496915137 +0000 UTC m=+26.778031422" Jul 10 02:48:28.521431 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:28.521409 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-kbfmm" Jul 10 02:48:28.521942 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:28.521925 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-kbfmm" Jul 10 02:48:29.282375 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:29.282342 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q28nr" Jul 10 02:48:29.282572 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:29.282474 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q28nr" podUID="dc1f4b26-669e-42a0-abd4-f858ac6b1543" Jul 10 02:48:29.481528 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:29.481505 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kbfmm" Jul 10 02:48:30.283071 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:30.283040 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nf7q8" Jul 10 02:48:30.283271 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:30.283045 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bgbpr" Jul 10 02:48:30.283271 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:30.283168 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nf7q8" podUID="25d9393a-d380-4ef0-a976-9fd998deb429" Jul 10 02:48:30.283380 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:30.283271 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bgbpr" podUID="7d037586-6a56-4354-ba36-a40c2cc2ab48" Jul 10 02:48:31.283306 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:31.283091 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q28nr" Jul 10 02:48:31.284067 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:31.283359 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q28nr" podUID="dc1f4b26-669e-42a0-abd4-f858ac6b1543" Jul 10 02:48:31.487840 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:31.487789 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tfrkp" event={"ID":"ff323ccb-9efc-43fd-8ce7-9365fe9af996","Type":"ContainerStarted","Data":"912e04e7d587fb1da12cad814c99ed214b8b42aca38ada566586c77e43523863"} Jul 10 02:48:31.491483 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:31.491449 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" event={"ID":"b93349d9-10f0-430c-b00f-74f141f508de","Type":"ContainerStarted","Data":"9d12d2a6f2a78a8f957aefd7d072219d6764cb89b7c318aad7583e79cc7c8d87"} Jul 10 02:48:31.491709 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:31.491684 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:31.491782 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:31.491774 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:31.508824 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:31.508775 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:31.533674 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:31.533602 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" podStartSLOduration=7.635731315 podStartE2EDuration="28.533587395s" totalImagesPullingTime="20.89785608s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 02:48:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:48:31.533218253 +0000 UTC m=+29.814334543" watchObservedRunningTime="2026-07-10 02:48:31.533587395 +0000 UTC m=+29.814703680" Jul 10 02:48:32.283481 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:32.283447 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nf7q8" Jul 10 02:48:32.283947 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:32.283553 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nf7q8" podUID="25d9393a-d380-4ef0-a976-9fd998deb429" Jul 10 02:48:32.283947 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:32.283588 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bgbpr" Jul 10 02:48:32.283947 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:32.283682 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bgbpr" podUID="7d037586-6a56-4354-ba36-a40c2cc2ab48" Jul 10 02:48:32.495482 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:32.495451 2575 generic.go:356] "Generic (PLEG): container finished" podID="ff323ccb-9efc-43fd-8ce7-9365fe9af996" containerID="912e04e7d587fb1da12cad814c99ed214b8b42aca38ada566586c77e43523863" exitCode=0 Jul 10 02:48:32.495652 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:32.495539 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tfrkp" event={"ID":"ff323ccb-9efc-43fd-8ce7-9365fe9af996","Type":"ContainerDied","Data":"912e04e7d587fb1da12cad814c99ed214b8b42aca38ada566586c77e43523863"} Jul 10 02:48:32.495960 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:32.495935 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:32.511781 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:32.511760 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:48:33.282979 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:33.282945 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q28nr" Jul 10 02:48:33.283137 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:33.283088 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q28nr" podUID="dc1f4b26-669e-42a0-abd4-f858ac6b1543" Jul 10 02:48:33.335242 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:33.335208 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-q28nr"] Jul 10 02:48:33.338451 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:33.338422 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nf7q8"] Jul 10 02:48:33.338577 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:33.338552 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nf7q8" Jul 10 02:48:33.338672 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:33.338650 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nf7q8" podUID="25d9393a-d380-4ef0-a976-9fd998deb429" Jul 10 02:48:33.339220 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:33.339196 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bgbpr"] Jul 10 02:48:33.339356 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:33.339315 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bgbpr" Jul 10 02:48:33.339459 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:33.339435 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bgbpr" podUID="7d037586-6a56-4354-ba36-a40c2cc2ab48" Jul 10 02:48:33.498066 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:33.498034 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q28nr" Jul 10 02:48:33.498215 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:33.498168 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q28nr" podUID="dc1f4b26-669e-42a0-abd4-f858ac6b1543" Jul 10 02:48:34.501940 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:34.501861 2575 generic.go:356] "Generic (PLEG): container finished" podID="ff323ccb-9efc-43fd-8ce7-9365fe9af996" containerID="3ba91dd9127ba5b6ceb37b1e04436db3cb7ac5859e3feca525a26a5b56987e5b" exitCode=0 Jul 10 02:48:34.502272 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:34.501935 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tfrkp" event={"ID":"ff323ccb-9efc-43fd-8ce7-9365fe9af996","Type":"ContainerDied","Data":"3ba91dd9127ba5b6ceb37b1e04436db3cb7ac5859e3feca525a26a5b56987e5b"} Jul 10 02:48:35.038555 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:35.038517 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d037586-6a56-4354-ba36-a40c2cc2ab48-metrics-certs\") pod \"network-metrics-daemon-bgbpr\" (UID: \"7d037586-6a56-4354-ba36-a40c2cc2ab48\") " pod="openshift-multus/network-metrics-daemon-bgbpr" Jul 10 02:48:35.038728 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:35.038575 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc1f4b26-669e-42a0-abd4-f858ac6b1543-original-pull-secret\") pod \"global-pull-secret-syncer-q28nr\" (UID: \"dc1f4b26-669e-42a0-abd4-f858ac6b1543\") " pod="kube-system/global-pull-secret-syncer-q28nr" Jul 10 02:48:35.038728 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:35.038691 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 02:48:35.038862 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:35.038760 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d037586-6a56-4354-ba36-a40c2cc2ab48-metrics-certs podName:7d037586-6a56-4354-ba36-a40c2cc2ab48 nodeName:}" failed. No retries permitted until 2026-07-10 02:49:07.038745395 +0000 UTC m=+65.319861663 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d037586-6a56-4354-ba36-a40c2cc2ab48-metrics-certs") pod "network-metrics-daemon-bgbpr" (UID: "7d037586-6a56-4354-ba36-a40c2cc2ab48") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 02:48:35.038862 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:35.038696 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 02:48:35.038862 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:35.038847 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc1f4b26-669e-42a0-abd4-f858ac6b1543-original-pull-secret podName:dc1f4b26-669e-42a0-abd4-f858ac6b1543 nodeName:}" failed. No retries permitted until 2026-07-10 02:49:07.038835073 +0000 UTC m=+65.319951348 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dc1f4b26-669e-42a0-abd4-f858ac6b1543-original-pull-secret") pod "global-pull-secret-syncer-q28nr" (UID: "dc1f4b26-669e-42a0-abd4-f858ac6b1543") : object "kube-system"/"original-pull-secret" not registered Jul 10 02:48:35.139554 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:35.139517 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzkpc\" (UniqueName: \"kubernetes.io/projected/25d9393a-d380-4ef0-a976-9fd998deb429-kube-api-access-jzkpc\") pod \"network-check-target-nf7q8\" (UID: \"25d9393a-d380-4ef0-a976-9fd998deb429\") " pod="openshift-network-diagnostics/network-check-target-nf7q8" Jul 10 02:48:35.139726 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:35.139683 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 02:48:35.139726 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:35.139704 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 02:48:35.139726 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:35.139717 2575 projected.go:196] Error preparing data for projected volume kube-api-access-jzkpc for pod openshift-network-diagnostics/network-check-target-nf7q8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 02:48:35.139854 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:35.139770 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25d9393a-d380-4ef0-a976-9fd998deb429-kube-api-access-jzkpc podName:25d9393a-d380-4ef0-a976-9fd998deb429 nodeName:}" failed. No retries permitted until 2026-07-10 02:49:07.139753788 +0000 UTC m=+65.420870052 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-jzkpc" (UniqueName: "kubernetes.io/projected/25d9393a-d380-4ef0-a976-9fd998deb429-kube-api-access-jzkpc") pod "network-check-target-nf7q8" (UID: "25d9393a-d380-4ef0-a976-9fd998deb429") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 02:48:35.282864 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:35.282664 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nf7q8" Jul 10 02:48:35.283020 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:35.282685 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q28nr" Jul 10 02:48:35.283020 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:35.282958 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nf7q8" podUID="25d9393a-d380-4ef0-a976-9fd998deb429" Jul 10 02:48:35.283020 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:35.282726 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bgbpr" Jul 10 02:48:35.283020 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:35.283013 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q28nr" podUID="dc1f4b26-669e-42a0-abd4-f858ac6b1543" Jul 10 02:48:35.283231 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:35.283076 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bgbpr" podUID="7d037586-6a56-4354-ba36-a40c2cc2ab48" Jul 10 02:48:36.507874 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:36.507834 2575 generic.go:356] "Generic (PLEG): container finished" podID="ff323ccb-9efc-43fd-8ce7-9365fe9af996" containerID="0e37aa5a4a601e216a0d4c93a3745a2591a008b092c0e75c37d1c81fa2488a26" exitCode=0 Jul 10 02:48:36.508228 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:36.507885 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tfrkp" event={"ID":"ff323ccb-9efc-43fd-8ce7-9365fe9af996","Type":"ContainerDied","Data":"0e37aa5a4a601e216a0d4c93a3745a2591a008b092c0e75c37d1c81fa2488a26"} Jul 10 02:48:37.282943 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:37.282910 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nf7q8" Jul 10 02:48:37.283115 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:37.282954 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q28nr" Jul 10 02:48:37.283115 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:37.283003 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bgbpr" Jul 10 02:48:37.283115 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:37.283016 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nf7q8" podUID="25d9393a-d380-4ef0-a976-9fd998deb429" Jul 10 02:48:37.283115 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:37.283091 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bgbpr" podUID="7d037586-6a56-4354-ba36-a40c2cc2ab48" Jul 10 02:48:37.283240 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:37.283135 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q28nr" podUID="dc1f4b26-669e-42a0-abd4-f858ac6b1543" Jul 10 02:48:38.055821 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.055791 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-139.ec2.internal" event="NodeReady" Jul 10 02:48:38.056389 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.055967 2575 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 10 02:48:38.094010 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.093881 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-bfcf8947b-7fz4l"] Jul 10 02:48:38.097198 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.097174 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-x22bq"] Jul 10 02:48:38.097355 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.097321 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-bfcf8947b-7fz4l" Jul 10 02:48:38.099667 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.099638 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-27kdw\"" Jul 10 02:48:38.100915 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.100642 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 10 02:48:38.100915 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.100744 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 10 02:48:38.100915 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.100768 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 10 02:48:38.100915 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.100828 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-28scv"] Jul 10 02:48:38.100915 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.100870 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 10 02:48:38.101196 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.100952 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 10 02:48:38.101196 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.101129 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-x22bq" Jul 10 02:48:38.101425 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.101407 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 10 02:48:38.103868 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.103831 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-xntdx\"" Jul 10 02:48:38.104238 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.104218 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 10 02:48:38.104654 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.104590 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 10 02:48:38.105061 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.105038 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 10 02:48:38.107730 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.107497 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-fbd7648c9-vtvdf"] Jul 10 02:48:38.107730 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.107528 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 10 02:48:38.107999 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.107958 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-28scv" Jul 10 02:48:38.110358 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.110320 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-7hqqk\"" Jul 10 02:48:38.110544 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.110520 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 10 02:48:38.110642 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.110596 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 10 02:48:38.110642 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.110611 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 10 02:48:38.110744 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.110520 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 10 02:48:38.113822 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.112553 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-q44gx"] Jul 10 02:48:38.113822 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.113550 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fbd7648c9-vtvdf" Jul 10 02:48:38.116206 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.116187 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 10 02:48:38.116384 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.116352 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 10 02:48:38.116756 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.116510 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-c87sb\"" Jul 10 02:48:38.117335 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.117317 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 10 02:48:38.118657 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.118628 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-28scv"] Jul 10 02:48:38.118746 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.118686 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-q44gx"] Jul 10 02:48:38.118746 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.118715 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-x22bq"] Jul 10 02:48:38.118746 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.118732 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-bfcf8947b-7fz4l"] Jul 10 02:48:38.118746 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.118732 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 10 02:48:38.119010 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.118923 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-q44gx" Jul 10 02:48:38.119519 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.119498 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-cc72k"] Jul 10 02:48:38.121384 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.121357 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 10 02:48:38.121612 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.121594 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 10 02:48:38.121924 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.121905 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-mkh7g\"" Jul 10 02:48:38.122098 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.122083 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 10 02:48:38.124611 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.124591 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 10 02:48:38.126368 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.126343 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-n8kpv"] Jul 10 02:48:38.126518 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.126503 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cc72k" Jul 10 02:48:38.130734 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.130709 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-l9gjz\"" Jul 10 02:48:38.131014 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.130985 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 10 02:48:38.131480 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.131311 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 10 02:48:38.132960 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.132942 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7866497494-xxlp6"] Jul 10 02:48:38.137820 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.135773 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-n8kpv" Jul 10 02:48:38.139380 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.138541 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 10 02:48:38.139380 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.138743 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-9n922\"" Jul 10 02:48:38.139727 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.139605 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 10 02:48:38.140090 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.140023 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:38.144274 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.144255 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-2rh2j"] Jul 10 02:48:38.144939 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.144913 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-xxlp6" Jul 10 02:48:38.146770 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.146751 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 10 02:48:38.147341 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.147320 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 10 02:48:38.147477 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.147457 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-f9zdg\"" Jul 10 02:48:38.147615 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.147595 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 10 02:48:38.147696 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.147670 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 10 02:48:38.147924 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.147908 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-t48ft"] Jul 10 02:48:38.148214 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.147989 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-2rh2j" Jul 10 02:48:38.150545 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.150528 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 10 02:48:38.151192 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.151176 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 10 02:48:38.151483 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.151465 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 10 02:48:38.152310 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.152288 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-gtqxb"] Jul 10 02:48:38.153425 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.153371 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-t48ft" Jul 10 02:48:38.154157 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.153955 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 10 02:48:38.154157 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.152289 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-k9srp\"" Jul 10 02:48:38.154157 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.154149 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 10 02:48:38.155472 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.155451 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 02:48:38.155569 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.155546 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 02:48:38.155692 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.155673 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-qss29\"" Jul 10 02:48:38.156692 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.156630 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-fbd7648c9-vtvdf"] Jul 10 02:48:38.156868 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.156854 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-n8kpv"] Jul 10 02:48:38.156947 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.156914 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-t48ft"] Jul 10 02:48:38.156947 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.156938 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cc72k"] Jul 10 02:48:38.157044 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.156962 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-2rh2j"] Jul 10 02:48:38.157044 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.156983 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9qrnw"] Jul 10 02:48:38.157140 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.156716 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-gtqxb" Jul 10 02:48:38.159154 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.159133 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 10 02:48:38.159772 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.159307 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 10 02:48:38.159772 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.159502 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 10 02:48:38.159772 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.159712 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 10 02:48:38.160001 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.159853 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-lnkk8\"" Jul 10 02:48:38.160909 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.160852 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-gtqxb"] Jul 10 02:48:38.160909 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.160895 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9qrnw"] Jul 10 02:48:38.161046 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.160918 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-xxlp6"] Jul 10 02:48:38.161046 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.161002 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9qrnw" Jul 10 02:48:38.163045 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.163026 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 10 02:48:38.163136 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.163058 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-45zzw\"" Jul 10 02:48:38.163451 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.163436 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 10 02:48:38.163619 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.163602 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 10 02:48:38.231247 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.231210 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:38.263610 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.263568 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/167ea7f3-1d8f-4892-bc0e-e9247912fef6-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-q44gx\" (UID: \"167ea7f3-1d8f-4892-bc0e-e9247912fef6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-q44gx" Jul 10 02:48:38.263610 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.263614 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trpjk\" (UniqueName: \"kubernetes.io/projected/0cd9606f-fafb-4005-91fb-1579f4b0cc75-kube-api-access-trpjk\") pod \"service-ca-operator-7695fc6c8c-28scv\" (UID: \"0cd9606f-fafb-4005-91fb-1579f4b0cc75\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-28scv" Jul 10 02:48:38.263875 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.263643 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4t5q\" (UniqueName: \"kubernetes.io/projected/d653dbf1-1efa-41b0-889c-f78d4f1d1ed8-kube-api-access-q4t5q\") pod \"network-check-source-869dcfcdcd-t48ft\" (UID: \"d653dbf1-1efa-41b0-889c-f78d4f1d1ed8\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-t48ft" Jul 10 02:48:38.263875 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.263676 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64e37f35-938c-457a-a377-354c4bc12d73-tmp\") pod \"insights-operator-68d9dd94cf-x22bq\" (UID: \"64e37f35-938c-457a-a377-354c4bc12d73\") " pod="openshift-insights/insights-operator-68d9dd94cf-x22bq" Jul 10 02:48:38.263875 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.263703 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nprvf\" (UniqueName: \"kubernetes.io/projected/372be3ce-a380-4c0e-bbee-49c2e3b80095-kube-api-access-nprvf\") pod \"console-operator-7866497494-xxlp6\" (UID: \"372be3ce-a380-4c0e-bbee-49c2e3b80095\") " pod="openshift-console-operator/console-operator-7866497494-xxlp6" Jul 10 02:48:38.263875 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.263739 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4094383d-2bb9-441b-8c30-94ff7551ce18-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-gtqxb\" (UID: \"4094383d-2bb9-441b-8c30-94ff7551ce18\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-gtqxb" Jul 10 02:48:38.263875 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.263804 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6136f9e5-f2c2-4e54-94e5-f86957cc814e-cert\") pod \"ingress-canary-9qrnw\" (UID: \"6136f9e5-f2c2-4e54-94e5-f86957cc814e\") " pod="openshift-ingress-canary/ingress-canary-9qrnw" Jul 10 02:48:38.263875 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.263859 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtqbp\" (UniqueName: \"kubernetes.io/projected/f540bce4-0371-4500-8e77-3bb4291a2e3d-kube-api-access-vtqbp\") pod \"image-registry-fbd7648c9-vtvdf\" (UID: \"f540bce4-0371-4500-8e77-3bb4291a2e3d\") " pod="openshift-image-registry/image-registry-fbd7648c9-vtvdf" Jul 10 02:48:38.264162 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.263900 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5176df10-dfd5-4514-9a59-c854c4a0ec21-metrics-tls\") pod \"dns-default-cc72k\" (UID: \"5176df10-dfd5-4514-9a59-c854c4a0ec21\") " pod="openshift-dns/dns-default-cc72k" Jul 10 02:48:38.264162 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.263935 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdhq8\" (UniqueName: \"kubernetes.io/projected/00377a57-ff2d-4c74-a9de-1fb20db00f5e-kube-api-access-zdhq8\") pod \"volume-data-source-validator-78495597f6-n8kpv\" (UID: \"00377a57-ff2d-4c74-a9de-1fb20db00f5e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-n8kpv" Jul 10 02:48:38.264162 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.263953 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f540bce4-0371-4500-8e77-3bb4291a2e3d-ca-trust-extracted\") pod \"image-registry-fbd7648c9-vtvdf\" (UID: \"f540bce4-0371-4500-8e77-3bb4291a2e3d\") " pod="openshift-image-registry/image-registry-fbd7648c9-vtvdf" Jul 10 02:48:38.264162 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.263982 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f540bce4-0371-4500-8e77-3bb4291a2e3d-installation-pull-secrets\") pod \"image-registry-fbd7648c9-vtvdf\" (UID: \"f540bce4-0371-4500-8e77-3bb4291a2e3d\") " pod="openshift-image-registry/image-registry-fbd7648c9-vtvdf" Jul 10 02:48:38.264162 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.264006 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f540bce4-0371-4500-8e77-3bb4291a2e3d-bound-sa-token\") pod \"image-registry-fbd7648c9-vtvdf\" (UID: \"f540bce4-0371-4500-8e77-3bb4291a2e3d\") " pod="openshift-image-registry/image-registry-fbd7648c9-vtvdf" Jul 10 02:48:38.264162 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.264029 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f540bce4-0371-4500-8e77-3bb4291a2e3d-trusted-ca\") pod \"image-registry-fbd7648c9-vtvdf\" (UID: \"f540bce4-0371-4500-8e77-3bb4291a2e3d\") " pod="openshift-image-registry/image-registry-fbd7648c9-vtvdf" Jul 10 02:48:38.264162 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.264055 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5b6v\" (UniqueName: \"kubernetes.io/projected/4094383d-2bb9-441b-8c30-94ff7551ce18-kube-api-access-d5b6v\") pod \"kube-storage-version-migrator-operator-799cb7f958-gtqxb\" (UID: \"4094383d-2bb9-441b-8c30-94ff7551ce18\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-gtqxb" Jul 10 02:48:38.264162 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.264075 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bcda0e99-fdd8-498f-8e73-ecf4c279be71-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-2rh2j\" (UID: \"bcda0e99-fdd8-498f-8e73-ecf4c279be71\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-2rh2j" Jul 10 02:48:38.264162 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.264097 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/64e37f35-938c-457a-a377-354c4bc12d73-snapshots\") pod \"insights-operator-68d9dd94cf-x22bq\" (UID: \"64e37f35-938c-457a-a377-354c4bc12d73\") " pod="openshift-insights/insights-operator-68d9dd94cf-x22bq" Jul 10 02:48:38.264162 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.264120 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f540bce4-0371-4500-8e77-3bb4291a2e3d-image-registry-private-configuration\") pod \"image-registry-fbd7648c9-vtvdf\" (UID: \"f540bce4-0371-4500-8e77-3bb4291a2e3d\") " pod="openshift-image-registry/image-registry-fbd7648c9-vtvdf" Jul 10 02:48:38.264162 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.264145 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64e37f35-938c-457a-a377-354c4bc12d73-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-x22bq\" (UID: \"64e37f35-938c-457a-a377-354c4bc12d73\") " pod="openshift-insights/insights-operator-68d9dd94cf-x22bq" Jul 10 02:48:38.264680 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.264171 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64e37f35-938c-457a-a377-354c4bc12d73-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-x22bq\" (UID: \"64e37f35-938c-457a-a377-354c4bc12d73\") " pod="openshift-insights/insights-operator-68d9dd94cf-x22bq" Jul 10 02:48:38.264680 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.264218 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f540bce4-0371-4500-8e77-3bb4291a2e3d-registry-certificates\") pod \"image-registry-fbd7648c9-vtvdf\" (UID: \"f540bce4-0371-4500-8e77-3bb4291a2e3d\") " pod="openshift-image-registry/image-registry-fbd7648c9-vtvdf" Jul 10 02:48:38.264680 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.264265 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvlrc\" (UniqueName: \"kubernetes.io/projected/6136f9e5-f2c2-4e54-94e5-f86957cc814e-kube-api-access-jvlrc\") pod \"ingress-canary-9qrnw\" (UID: \"6136f9e5-f2c2-4e54-94e5-f86957cc814e\") " pod="openshift-ingress-canary/ingress-canary-9qrnw" Jul 10 02:48:38.264680 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.264289 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0cd9606f-fafb-4005-91fb-1579f4b0cc75-serving-cert\") pod \"service-ca-operator-7695fc6c8c-28scv\" (UID: \"0cd9606f-fafb-4005-91fb-1579f4b0cc75\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-28scv" Jul 10 02:48:38.264680 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.264334 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fjg5\" (UniqueName: \"kubernetes.io/projected/3adc0ae6-f77b-4909-8bef-8534f19e6601-kube-api-access-6fjg5\") pod \"router-default-bfcf8947b-7fz4l\" (UID: \"3adc0ae6-f77b-4909-8bef-8534f19e6601\") " pod="openshift-ingress/router-default-bfcf8947b-7fz4l" Jul 10 02:48:38.264680 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.264371 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5176df10-dfd5-4514-9a59-c854c4a0ec21-tmp-dir\") pod \"dns-default-cc72k\" (UID: \"5176df10-dfd5-4514-9a59-c854c4a0ec21\") " pod="openshift-dns/dns-default-cc72k" Jul 10 02:48:38.264680 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.264410 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4094383d-2bb9-441b-8c30-94ff7551ce18-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-gtqxb\" (UID: \"4094383d-2bb9-441b-8c30-94ff7551ce18\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-gtqxb" Jul 10 02:48:38.264680 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.264436 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f540bce4-0371-4500-8e77-3bb4291a2e3d-registry-tls\") pod \"image-registry-fbd7648c9-vtvdf\" (UID: \"f540bce4-0371-4500-8e77-3bb4291a2e3d\") " pod="openshift-image-registry/image-registry-fbd7648c9-vtvdf" Jul 10 02:48:38.264680 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.264465 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5176df10-dfd5-4514-9a59-c854c4a0ec21-config-volume\") pod \"dns-default-cc72k\" (UID: \"5176df10-dfd5-4514-9a59-c854c4a0ec21\") " pod="openshift-dns/dns-default-cc72k" Jul 10 02:48:38.264680 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.264489 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/372be3ce-a380-4c0e-bbee-49c2e3b80095-config\") pod \"console-operator-7866497494-xxlp6\" (UID: \"372be3ce-a380-4c0e-bbee-49c2e3b80095\") " pod="openshift-console-operator/console-operator-7866497494-xxlp6" Jul 10 02:48:38.264680 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.264514 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bcda0e99-fdd8-498f-8e73-ecf4c279be71-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-2rh2j\" (UID: \"bcda0e99-fdd8-498f-8e73-ecf4c279be71\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-2rh2j" Jul 10 02:48:38.264680 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.264546 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3adc0ae6-f77b-4909-8bef-8534f19e6601-service-ca-bundle\") pod \"router-default-bfcf8947b-7fz4l\" (UID: \"3adc0ae6-f77b-4909-8bef-8534f19e6601\") " pod="openshift-ingress/router-default-bfcf8947b-7fz4l" Jul 10 02:48:38.264680 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.264570 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/64e37f35-938c-457a-a377-354c4bc12d73-serving-cert\") pod \"insights-operator-68d9dd94cf-x22bq\" (UID: \"64e37f35-938c-457a-a377-354c4bc12d73\") " pod="openshift-insights/insights-operator-68d9dd94cf-x22bq" Jul 10 02:48:38.264680 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.264594 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzn54\" (UniqueName: \"kubernetes.io/projected/5176df10-dfd5-4514-9a59-c854c4a0ec21-kube-api-access-qzn54\") pod \"dns-default-cc72k\" (UID: \"5176df10-dfd5-4514-9a59-c854c4a0ec21\") " pod="openshift-dns/dns-default-cc72k" Jul 10 02:48:38.264680 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.264619 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0cd9606f-fafb-4005-91fb-1579f4b0cc75-config\") pod \"service-ca-operator-7695fc6c8c-28scv\" (UID: \"0cd9606f-fafb-4005-91fb-1579f4b0cc75\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-28scv" Jul 10 02:48:38.265322 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.264645 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3adc0ae6-f77b-4909-8bef-8534f19e6601-metrics-certs\") pod \"router-default-bfcf8947b-7fz4l\" (UID: \"3adc0ae6-f77b-4909-8bef-8534f19e6601\") " pod="openshift-ingress/router-default-bfcf8947b-7fz4l" Jul 10 02:48:38.265322 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.264668 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3adc0ae6-f77b-4909-8bef-8534f19e6601-default-certificate\") pod \"router-default-bfcf8947b-7fz4l\" (UID: \"3adc0ae6-f77b-4909-8bef-8534f19e6601\") " pod="openshift-ingress/router-default-bfcf8947b-7fz4l" Jul 10 02:48:38.265322 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.264698 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/372be3ce-a380-4c0e-bbee-49c2e3b80095-trusted-ca\") pod \"console-operator-7866497494-xxlp6\" (UID: \"372be3ce-a380-4c0e-bbee-49c2e3b80095\") " pod="openshift-console-operator/console-operator-7866497494-xxlp6" Jul 10 02:48:38.265322 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.264768 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3adc0ae6-f77b-4909-8bef-8534f19e6601-stats-auth\") pod \"router-default-bfcf8947b-7fz4l\" (UID: \"3adc0ae6-f77b-4909-8bef-8534f19e6601\") " pod="openshift-ingress/router-default-bfcf8947b-7fz4l" Jul 10 02:48:38.265322 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.264820 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8nr4\" (UniqueName: \"kubernetes.io/projected/64e37f35-938c-457a-a377-354c4bc12d73-kube-api-access-m8nr4\") pod \"insights-operator-68d9dd94cf-x22bq\" (UID: \"64e37f35-938c-457a-a377-354c4bc12d73\") " pod="openshift-insights/insights-operator-68d9dd94cf-x22bq" Jul 10 02:48:38.265322 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.264863 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/372be3ce-a380-4c0e-bbee-49c2e3b80095-serving-cert\") pod \"console-operator-7866497494-xxlp6\" (UID: \"372be3ce-a380-4c0e-bbee-49c2e3b80095\") " pod="openshift-console-operator/console-operator-7866497494-xxlp6" Jul 10 02:48:38.265322 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.264889 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcdv6\" (UniqueName: \"kubernetes.io/projected/bcda0e99-fdd8-498f-8e73-ecf4c279be71-kube-api-access-wcdv6\") pod \"cluster-monitoring-operator-5f7f494dd4-2rh2j\" (UID: \"bcda0e99-fdd8-498f-8e73-ecf4c279be71\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-2rh2j" Jul 10 02:48:38.265322 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.264918 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-757px\" (UniqueName: \"kubernetes.io/projected/167ea7f3-1d8f-4892-bc0e-e9247912fef6-kube-api-access-757px\") pod \"cluster-samples-operator-65db858894-q44gx\" (UID: \"167ea7f3-1d8f-4892-bc0e-e9247912fef6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-q44gx" Jul 10 02:48:38.293264 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.293225 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:38.365997 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.365960 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-757px\" (UniqueName: \"kubernetes.io/projected/167ea7f3-1d8f-4892-bc0e-e9247912fef6-kube-api-access-757px\") pod \"cluster-samples-operator-65db858894-q44gx\" (UID: \"167ea7f3-1d8f-4892-bc0e-e9247912fef6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-q44gx" Jul 10 02:48:38.366165 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.366000 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/167ea7f3-1d8f-4892-bc0e-e9247912fef6-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-q44gx\" (UID: \"167ea7f3-1d8f-4892-bc0e-e9247912fef6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-q44gx" Jul 10 02:48:38.366165 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.366025 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trpjk\" (UniqueName: \"kubernetes.io/projected/0cd9606f-fafb-4005-91fb-1579f4b0cc75-kube-api-access-trpjk\") pod \"service-ca-operator-7695fc6c8c-28scv\" (UID: \"0cd9606f-fafb-4005-91fb-1579f4b0cc75\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-28scv" Jul 10 02:48:38.366165 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.366044 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4t5q\" (UniqueName: \"kubernetes.io/projected/d653dbf1-1efa-41b0-889c-f78d4f1d1ed8-kube-api-access-q4t5q\") pod \"network-check-source-869dcfcdcd-t48ft\" (UID: \"d653dbf1-1efa-41b0-889c-f78d4f1d1ed8\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-t48ft" Jul 10 02:48:38.366165 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:38.366123 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 02:48:38.366368 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.366179 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64e37f35-938c-457a-a377-354c4bc12d73-tmp\") pod \"insights-operator-68d9dd94cf-x22bq\" (UID: \"64e37f35-938c-457a-a377-354c4bc12d73\") " pod="openshift-insights/insights-operator-68d9dd94cf-x22bq" Jul 10 02:48:38.366368 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:38.366194 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/167ea7f3-1d8f-4892-bc0e-e9247912fef6-samples-operator-tls podName:167ea7f3-1d8f-4892-bc0e-e9247912fef6 nodeName:}" failed. No retries permitted until 2026-07-10 02:48:38.866175383 +0000 UTC m=+37.147291664 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/167ea7f3-1d8f-4892-bc0e-e9247912fef6-samples-operator-tls") pod "cluster-samples-operator-65db858894-q44gx" (UID: "167ea7f3-1d8f-4892-bc0e-e9247912fef6") : secret "samples-operator-tls" not found Jul 10 02:48:38.366368 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.366233 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nprvf\" (UniqueName: \"kubernetes.io/projected/372be3ce-a380-4c0e-bbee-49c2e3b80095-kube-api-access-nprvf\") pod \"console-operator-7866497494-xxlp6\" (UID: \"372be3ce-a380-4c0e-bbee-49c2e3b80095\") " pod="openshift-console-operator/console-operator-7866497494-xxlp6" Jul 10 02:48:38.366368 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.366279 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4094383d-2bb9-441b-8c30-94ff7551ce18-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-gtqxb\" (UID: \"4094383d-2bb9-441b-8c30-94ff7551ce18\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-gtqxb" Jul 10 02:48:38.366368 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.366312 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6136f9e5-f2c2-4e54-94e5-f86957cc814e-cert\") pod \"ingress-canary-9qrnw\" (UID: \"6136f9e5-f2c2-4e54-94e5-f86957cc814e\") " pod="openshift-ingress-canary/ingress-canary-9qrnw" Jul 10 02:48:38.366619 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.366372 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtqbp\" (UniqueName: \"kubernetes.io/projected/f540bce4-0371-4500-8e77-3bb4291a2e3d-kube-api-access-vtqbp\") pod \"image-registry-fbd7648c9-vtvdf\" (UID: \"f540bce4-0371-4500-8e77-3bb4291a2e3d\") " pod="openshift-image-registry/image-registry-fbd7648c9-vtvdf" Jul 10 02:48:38.366619 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.366403 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5176df10-dfd5-4514-9a59-c854c4a0ec21-metrics-tls\") pod \"dns-default-cc72k\" (UID: \"5176df10-dfd5-4514-9a59-c854c4a0ec21\") " pod="openshift-dns/dns-default-cc72k" Jul 10 02:48:38.366619 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.366452 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdhq8\" (UniqueName: \"kubernetes.io/projected/00377a57-ff2d-4c74-a9de-1fb20db00f5e-kube-api-access-zdhq8\") pod \"volume-data-source-validator-78495597f6-n8kpv\" (UID: \"00377a57-ff2d-4c74-a9de-1fb20db00f5e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-n8kpv" Jul 10 02:48:38.366619 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.366487 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f540bce4-0371-4500-8e77-3bb4291a2e3d-ca-trust-extracted\") pod \"image-registry-fbd7648c9-vtvdf\" (UID: \"f540bce4-0371-4500-8e77-3bb4291a2e3d\") " pod="openshift-image-registry/image-registry-fbd7648c9-vtvdf" Jul 10 02:48:38.366619 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.366518 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64e37f35-938c-457a-a377-354c4bc12d73-tmp\") pod \"insights-operator-68d9dd94cf-x22bq\" (UID: \"64e37f35-938c-457a-a377-354c4bc12d73\") " pod="openshift-insights/insights-operator-68d9dd94cf-x22bq" Jul 10 02:48:38.366619 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.366525 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f540bce4-0371-4500-8e77-3bb4291a2e3d-installation-pull-secrets\") pod \"image-registry-fbd7648c9-vtvdf\" (UID: \"f540bce4-0371-4500-8e77-3bb4291a2e3d\") " pod="openshift-image-registry/image-registry-fbd7648c9-vtvdf" Jul 10 02:48:38.366619 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:38.366551 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 02:48:38.366619 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.366570 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f540bce4-0371-4500-8e77-3bb4291a2e3d-bound-sa-token\") pod \"image-registry-fbd7648c9-vtvdf\" (UID: \"f540bce4-0371-4500-8e77-3bb4291a2e3d\") " pod="openshift-image-registry/image-registry-fbd7648c9-vtvdf" Jul 10 02:48:38.366619 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.366599 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f540bce4-0371-4500-8e77-3bb4291a2e3d-trusted-ca\") pod \"image-registry-fbd7648c9-vtvdf\" (UID: \"f540bce4-0371-4500-8e77-3bb4291a2e3d\") " pod="openshift-image-registry/image-registry-fbd7648c9-vtvdf" Jul 10 02:48:38.366619 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:38.366614 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5176df10-dfd5-4514-9a59-c854c4a0ec21-metrics-tls podName:5176df10-dfd5-4514-9a59-c854c4a0ec21 nodeName:}" failed. No retries permitted until 2026-07-10 02:48:38.866597873 +0000 UTC m=+37.147714156 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5176df10-dfd5-4514-9a59-c854c4a0ec21-metrics-tls") pod "dns-default-cc72k" (UID: "5176df10-dfd5-4514-9a59-c854c4a0ec21") : secret "dns-default-metrics-tls" not found Jul 10 02:48:38.367128 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.366652 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5b6v\" (UniqueName: \"kubernetes.io/projected/4094383d-2bb9-441b-8c30-94ff7551ce18-kube-api-access-d5b6v\") pod \"kube-storage-version-migrator-operator-799cb7f958-gtqxb\" (UID: \"4094383d-2bb9-441b-8c30-94ff7551ce18\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-gtqxb" Jul 10 02:48:38.367128 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.366682 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bcda0e99-fdd8-498f-8e73-ecf4c279be71-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-2rh2j\" (UID: \"bcda0e99-fdd8-498f-8e73-ecf4c279be71\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-2rh2j" Jul 10 02:48:38.367128 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.366710 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/64e37f35-938c-457a-a377-354c4bc12d73-snapshots\") pod \"insights-operator-68d9dd94cf-x22bq\" (UID: \"64e37f35-938c-457a-a377-354c4bc12d73\") " pod="openshift-insights/insights-operator-68d9dd94cf-x22bq" Jul 10 02:48:38.367128 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.366738 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f540bce4-0371-4500-8e77-3bb4291a2e3d-image-registry-private-configuration\") pod \"image-registry-fbd7648c9-vtvdf\" (UID: \"f540bce4-0371-4500-8e77-3bb4291a2e3d\") " pod="openshift-image-registry/image-registry-fbd7648c9-vtvdf" Jul 10 02:48:38.367128 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.366767 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64e37f35-938c-457a-a377-354c4bc12d73-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-x22bq\" (UID: \"64e37f35-938c-457a-a377-354c4bc12d73\") " pod="openshift-insights/insights-operator-68d9dd94cf-x22bq" Jul 10 02:48:38.367128 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.366792 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64e37f35-938c-457a-a377-354c4bc12d73-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-x22bq\" (UID: \"64e37f35-938c-457a-a377-354c4bc12d73\") " pod="openshift-insights/insights-operator-68d9dd94cf-x22bq" Jul 10 02:48:38.367128 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.366838 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f540bce4-0371-4500-8e77-3bb4291a2e3d-registry-certificates\") pod \"image-registry-fbd7648c9-vtvdf\" (UID: \"f540bce4-0371-4500-8e77-3bb4291a2e3d\") " pod="openshift-image-registry/image-registry-fbd7648c9-vtvdf" Jul 10 02:48:38.367128 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.366871 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvlrc\" (UniqueName: \"kubernetes.io/projected/6136f9e5-f2c2-4e54-94e5-f86957cc814e-kube-api-access-jvlrc\") pod \"ingress-canary-9qrnw\" (UID: \"6136f9e5-f2c2-4e54-94e5-f86957cc814e\") " pod="openshift-ingress-canary/ingress-canary-9qrnw" Jul 10 02:48:38.367128 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.366894 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0cd9606f-fafb-4005-91fb-1579f4b0cc75-serving-cert\") pod \"service-ca-operator-7695fc6c8c-28scv\" (UID: \"0cd9606f-fafb-4005-91fb-1579f4b0cc75\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-28scv" Jul 10 02:48:38.367128 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.366907 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f540bce4-0371-4500-8e77-3bb4291a2e3d-ca-trust-extracted\") pod \"image-registry-fbd7648c9-vtvdf\" (UID: \"f540bce4-0371-4500-8e77-3bb4291a2e3d\") " pod="openshift-image-registry/image-registry-fbd7648c9-vtvdf" Jul 10 02:48:38.367612 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.367592 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64e37f35-938c-457a-a377-354c4bc12d73-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-x22bq\" (UID: \"64e37f35-938c-457a-a377-354c4bc12d73\") " pod="openshift-insights/insights-operator-68d9dd94cf-x22bq" Jul 10 02:48:38.367739 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.367716 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f540bce4-0371-4500-8e77-3bb4291a2e3d-registry-certificates\") pod \"image-registry-fbd7648c9-vtvdf\" (UID: \"f540bce4-0371-4500-8e77-3bb4291a2e3d\") " pod="openshift-image-registry/image-registry-fbd7648c9-vtvdf" Jul 10 02:48:38.367824 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.367741 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f540bce4-0371-4500-8e77-3bb4291a2e3d-trusted-ca\") pod \"image-registry-fbd7648c9-vtvdf\" (UID: \"f540bce4-0371-4500-8e77-3bb4291a2e3d\") " pod="openshift-image-registry/image-registry-fbd7648c9-vtvdf" Jul 10 02:48:38.367887 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:38.367838 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 10 02:48:38.367940 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:38.367891 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcda0e99-fdd8-498f-8e73-ecf4c279be71-cluster-monitoring-operator-tls podName:bcda0e99-fdd8-498f-8e73-ecf4c279be71 nodeName:}" failed. No retries permitted until 2026-07-10 02:48:38.867876738 +0000 UTC m=+37.148993008 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bcda0e99-fdd8-498f-8e73-ecf4c279be71-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-2rh2j" (UID: "bcda0e99-fdd8-498f-8e73-ecf4c279be71") : secret "cluster-monitoring-operator-tls" not found Jul 10 02:48:38.368176 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.368156 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fjg5\" (UniqueName: \"kubernetes.io/projected/3adc0ae6-f77b-4909-8bef-8534f19e6601-kube-api-access-6fjg5\") pod \"router-default-bfcf8947b-7fz4l\" (UID: \"3adc0ae6-f77b-4909-8bef-8534f19e6601\") " pod="openshift-ingress/router-default-bfcf8947b-7fz4l" Jul 10 02:48:38.368268 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.368188 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5176df10-dfd5-4514-9a59-c854c4a0ec21-tmp-dir\") pod \"dns-default-cc72k\" (UID: \"5176df10-dfd5-4514-9a59-c854c4a0ec21\") " pod="openshift-dns/dns-default-cc72k" Jul 10 02:48:38.368268 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.368227 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4094383d-2bb9-441b-8c30-94ff7551ce18-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-gtqxb\" (UID: \"4094383d-2bb9-441b-8c30-94ff7551ce18\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-gtqxb" Jul 10 02:48:38.368268 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.368254 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f540bce4-0371-4500-8e77-3bb4291a2e3d-registry-tls\") pod \"image-registry-fbd7648c9-vtvdf\" (UID: \"f540bce4-0371-4500-8e77-3bb4291a2e3d\") " pod="openshift-image-registry/image-registry-fbd7648c9-vtvdf" Jul 10 02:48:38.368417 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.368282 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5176df10-dfd5-4514-9a59-c854c4a0ec21-config-volume\") pod \"dns-default-cc72k\" (UID: \"5176df10-dfd5-4514-9a59-c854c4a0ec21\") " pod="openshift-dns/dns-default-cc72k" Jul 10 02:48:38.368417 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.368302 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/64e37f35-938c-457a-a377-354c4bc12d73-snapshots\") pod \"insights-operator-68d9dd94cf-x22bq\" (UID: \"64e37f35-938c-457a-a377-354c4bc12d73\") " pod="openshift-insights/insights-operator-68d9dd94cf-x22bq" Jul 10 02:48:38.368417 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.368308 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/372be3ce-a380-4c0e-bbee-49c2e3b80095-config\") pod \"console-operator-7866497494-xxlp6\" (UID: \"372be3ce-a380-4c0e-bbee-49c2e3b80095\") " pod="openshift-console-operator/console-operator-7866497494-xxlp6" Jul 10 02:48:38.368417 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.368332 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bcda0e99-fdd8-498f-8e73-ecf4c279be71-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-2rh2j\" (UID: \"bcda0e99-fdd8-498f-8e73-ecf4c279be71\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-2rh2j" Jul 10 02:48:38.368417 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.368361 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3adc0ae6-f77b-4909-8bef-8534f19e6601-service-ca-bundle\") pod \"router-default-bfcf8947b-7fz4l\" (UID: \"3adc0ae6-f77b-4909-8bef-8534f19e6601\") " pod="openshift-ingress/router-default-bfcf8947b-7fz4l" Jul 10 02:48:38.368417 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.368389 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/64e37f35-938c-457a-a377-354c4bc12d73-serving-cert\") pod \"insights-operator-68d9dd94cf-x22bq\" (UID: \"64e37f35-938c-457a-a377-354c4bc12d73\") " pod="openshift-insights/insights-operator-68d9dd94cf-x22bq" Jul 10 02:48:38.368417 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.368416 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzn54\" (UniqueName: \"kubernetes.io/projected/5176df10-dfd5-4514-9a59-c854c4a0ec21-kube-api-access-qzn54\") pod \"dns-default-cc72k\" (UID: \"5176df10-dfd5-4514-9a59-c854c4a0ec21\") " pod="openshift-dns/dns-default-cc72k" Jul 10 02:48:38.368733 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.368421 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64e37f35-938c-457a-a377-354c4bc12d73-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-x22bq\" (UID: \"64e37f35-938c-457a-a377-354c4bc12d73\") " pod="openshift-insights/insights-operator-68d9dd94cf-x22bq" Jul 10 02:48:38.368733 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.368440 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0cd9606f-fafb-4005-91fb-1579f4b0cc75-config\") pod \"service-ca-operator-7695fc6c8c-28scv\" (UID: \"0cd9606f-fafb-4005-91fb-1579f4b0cc75\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-28scv" Jul 10 02:48:38.368733 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.368471 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3adc0ae6-f77b-4909-8bef-8534f19e6601-metrics-certs\") pod \"router-default-bfcf8947b-7fz4l\" (UID: \"3adc0ae6-f77b-4909-8bef-8534f19e6601\") " pod="openshift-ingress/router-default-bfcf8947b-7fz4l" Jul 10 02:48:38.368733 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.368497 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3adc0ae6-f77b-4909-8bef-8534f19e6601-default-certificate\") pod \"router-default-bfcf8947b-7fz4l\" (UID: \"3adc0ae6-f77b-4909-8bef-8534f19e6601\") " pod="openshift-ingress/router-default-bfcf8947b-7fz4l" Jul 10 02:48:38.368733 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.368528 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/372be3ce-a380-4c0e-bbee-49c2e3b80095-trusted-ca\") pod \"console-operator-7866497494-xxlp6\" (UID: \"372be3ce-a380-4c0e-bbee-49c2e3b80095\") " pod="openshift-console-operator/console-operator-7866497494-xxlp6" Jul 10 02:48:38.368733 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.368542 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5176df10-dfd5-4514-9a59-c854c4a0ec21-tmp-dir\") pod \"dns-default-cc72k\" (UID: \"5176df10-dfd5-4514-9a59-c854c4a0ec21\") " pod="openshift-dns/dns-default-cc72k" Jul 10 02:48:38.368733 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.368590 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3adc0ae6-f77b-4909-8bef-8534f19e6601-stats-auth\") pod \"router-default-bfcf8947b-7fz4l\" (UID: \"3adc0ae6-f77b-4909-8bef-8534f19e6601\") " pod="openshift-ingress/router-default-bfcf8947b-7fz4l" Jul 10 02:48:38.368733 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:38.368604 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 02:48:38.368733 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:38.368620 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fbd7648c9-vtvdf: secret "image-registry-tls" not found Jul 10 02:48:38.368733 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.368623 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8nr4\" (UniqueName: \"kubernetes.io/projected/64e37f35-938c-457a-a377-354c4bc12d73-kube-api-access-m8nr4\") pod \"insights-operator-68d9dd94cf-x22bq\" (UID: \"64e37f35-938c-457a-a377-354c4bc12d73\") " pod="openshift-insights/insights-operator-68d9dd94cf-x22bq" Jul 10 02:48:38.368733 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.368651 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/372be3ce-a380-4c0e-bbee-49c2e3b80095-serving-cert\") pod \"console-operator-7866497494-xxlp6\" (UID: \"372be3ce-a380-4c0e-bbee-49c2e3b80095\") " pod="openshift-console-operator/console-operator-7866497494-xxlp6" Jul 10 02:48:38.368733 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:38.368674 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f540bce4-0371-4500-8e77-3bb4291a2e3d-registry-tls podName:f540bce4-0371-4500-8e77-3bb4291a2e3d nodeName:}" failed. No retries permitted until 2026-07-10 02:48:38.868657208 +0000 UTC m=+37.149773473 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f540bce4-0371-4500-8e77-3bb4291a2e3d-registry-tls") pod "image-registry-fbd7648c9-vtvdf" (UID: "f540bce4-0371-4500-8e77-3bb4291a2e3d") : secret "image-registry-tls" not found Jul 10 02:48:38.368733 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.368707 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcdv6\" (UniqueName: \"kubernetes.io/projected/bcda0e99-fdd8-498f-8e73-ecf4c279be71-kube-api-access-wcdv6\") pod \"cluster-monitoring-operator-5f7f494dd4-2rh2j\" (UID: \"bcda0e99-fdd8-498f-8e73-ecf4c279be71\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-2rh2j" Jul 10 02:48:38.369375 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.369239 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/372be3ce-a380-4c0e-bbee-49c2e3b80095-config\") pod \"console-operator-7866497494-xxlp6\" (UID: \"372be3ce-a380-4c0e-bbee-49c2e3b80095\") " pod="openshift-console-operator/console-operator-7866497494-xxlp6" Jul 10 02:48:38.369534 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.369508 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4094383d-2bb9-441b-8c30-94ff7551ce18-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-gtqxb\" (UID: \"4094383d-2bb9-441b-8c30-94ff7551ce18\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-gtqxb" Jul 10 02:48:38.369751 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:38.369732 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3adc0ae6-f77b-4909-8bef-8534f19e6601-service-ca-bundle podName:3adc0ae6-f77b-4909-8bef-8534f19e6601 nodeName:}" failed. No retries permitted until 2026-07-10 02:48:38.869715438 +0000 UTC m=+37.150831723 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3adc0ae6-f77b-4909-8bef-8534f19e6601-service-ca-bundle") pod "router-default-bfcf8947b-7fz4l" (UID: "3adc0ae6-f77b-4909-8bef-8534f19e6601") : configmap references non-existent config key: service-ca.crt Jul 10 02:48:38.370342 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.369974 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5176df10-dfd5-4514-9a59-c854c4a0ec21-config-volume\") pod \"dns-default-cc72k\" (UID: \"5176df10-dfd5-4514-9a59-c854c4a0ec21\") " pod="openshift-dns/dns-default-cc72k" Jul 10 02:48:38.370342 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.370117 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bcda0e99-fdd8-498f-8e73-ecf4c279be71-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-2rh2j\" (UID: \"bcda0e99-fdd8-498f-8e73-ecf4c279be71\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-2rh2j" Jul 10 02:48:38.371222 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:38.370574 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 10 02:48:38.371222 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:38.370636 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3adc0ae6-f77b-4909-8bef-8534f19e6601-metrics-certs podName:3adc0ae6-f77b-4909-8bef-8534f19e6601 nodeName:}" failed. No retries permitted until 2026-07-10 02:48:38.870620917 +0000 UTC m=+37.151737180 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3adc0ae6-f77b-4909-8bef-8534f19e6601-metrics-certs") pod "router-default-bfcf8947b-7fz4l" (UID: "3adc0ae6-f77b-4909-8bef-8534f19e6601") : secret "router-metrics-certs-default" not found Jul 10 02:48:38.371222 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.370579 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0cd9606f-fafb-4005-91fb-1579f4b0cc75-config\") pod \"service-ca-operator-7695fc6c8c-28scv\" (UID: \"0cd9606f-fafb-4005-91fb-1579f4b0cc75\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-28scv" Jul 10 02:48:38.371639 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.371595 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/372be3ce-a380-4c0e-bbee-49c2e3b80095-trusted-ca\") pod \"console-operator-7866497494-xxlp6\" (UID: \"372be3ce-a380-4c0e-bbee-49c2e3b80095\") " pod="openshift-console-operator/console-operator-7866497494-xxlp6" Jul 10 02:48:38.372579 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.372233 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/372be3ce-a380-4c0e-bbee-49c2e3b80095-serving-cert\") pod \"console-operator-7866497494-xxlp6\" (UID: \"372be3ce-a380-4c0e-bbee-49c2e3b80095\") " pod="openshift-console-operator/console-operator-7866497494-xxlp6" Jul 10 02:48:38.372579 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.372249 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f540bce4-0371-4500-8e77-3bb4291a2e3d-image-registry-private-configuration\") pod \"image-registry-fbd7648c9-vtvdf\" (UID: \"f540bce4-0371-4500-8e77-3bb4291a2e3d\") " pod="openshift-image-registry/image-registry-fbd7648c9-vtvdf" Jul 10 02:48:38.372579 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.372332 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6136f9e5-f2c2-4e54-94e5-f86957cc814e-cert\") pod \"ingress-canary-9qrnw\" (UID: \"6136f9e5-f2c2-4e54-94e5-f86957cc814e\") " pod="openshift-ingress-canary/ingress-canary-9qrnw" Jul 10 02:48:38.372579 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.372548 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4094383d-2bb9-441b-8c30-94ff7551ce18-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-gtqxb\" (UID: \"4094383d-2bb9-441b-8c30-94ff7551ce18\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-gtqxb" Jul 10 02:48:38.372579 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.372551 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/64e37f35-938c-457a-a377-354c4bc12d73-serving-cert\") pod \"insights-operator-68d9dd94cf-x22bq\" (UID: \"64e37f35-938c-457a-a377-354c4bc12d73\") " pod="openshift-insights/insights-operator-68d9dd94cf-x22bq" Jul 10 02:48:38.373177 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.373153 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0cd9606f-fafb-4005-91fb-1579f4b0cc75-serving-cert\") pod \"service-ca-operator-7695fc6c8c-28scv\" (UID: \"0cd9606f-fafb-4005-91fb-1579f4b0cc75\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-28scv" Jul 10 02:48:38.373386 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.373357 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f540bce4-0371-4500-8e77-3bb4291a2e3d-installation-pull-secrets\") pod \"image-registry-fbd7648c9-vtvdf\" (UID: \"f540bce4-0371-4500-8e77-3bb4291a2e3d\") " pod="openshift-image-registry/image-registry-fbd7648c9-vtvdf" Jul 10 02:48:38.374312 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.374242 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3adc0ae6-f77b-4909-8bef-8534f19e6601-default-certificate\") pod \"router-default-bfcf8947b-7fz4l\" (UID: \"3adc0ae6-f77b-4909-8bef-8534f19e6601\") " pod="openshift-ingress/router-default-bfcf8947b-7fz4l" Jul 10 02:48:38.374312 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.374245 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3adc0ae6-f77b-4909-8bef-8534f19e6601-stats-auth\") pod \"router-default-bfcf8947b-7fz4l\" (UID: \"3adc0ae6-f77b-4909-8bef-8534f19e6601\") " pod="openshift-ingress/router-default-bfcf8947b-7fz4l" Jul 10 02:48:38.375856 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.375745 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f540bce4-0371-4500-8e77-3bb4291a2e3d-bound-sa-token\") pod \"image-registry-fbd7648c9-vtvdf\" (UID: \"f540bce4-0371-4500-8e77-3bb4291a2e3d\") " pod="openshift-image-registry/image-registry-fbd7648c9-vtvdf" Jul 10 02:48:38.375856 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.375769 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-757px\" (UniqueName: \"kubernetes.io/projected/167ea7f3-1d8f-4892-bc0e-e9247912fef6-kube-api-access-757px\") pod \"cluster-samples-operator-65db858894-q44gx\" (UID: \"167ea7f3-1d8f-4892-bc0e-e9247912fef6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-q44gx" Jul 10 02:48:38.376578 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.376305 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fjg5\" (UniqueName: \"kubernetes.io/projected/3adc0ae6-f77b-4909-8bef-8534f19e6601-kube-api-access-6fjg5\") pod \"router-default-bfcf8947b-7fz4l\" (UID: \"3adc0ae6-f77b-4909-8bef-8534f19e6601\") " pod="openshift-ingress/router-default-bfcf8947b-7fz4l" Jul 10 02:48:38.376578 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.376528 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdhq8\" (UniqueName: \"kubernetes.io/projected/00377a57-ff2d-4c74-a9de-1fb20db00f5e-kube-api-access-zdhq8\") pod \"volume-data-source-validator-78495597f6-n8kpv\" (UID: \"00377a57-ff2d-4c74-a9de-1fb20db00f5e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-n8kpv" Jul 10 02:48:38.377486 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.377436 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nprvf\" (UniqueName: \"kubernetes.io/projected/372be3ce-a380-4c0e-bbee-49c2e3b80095-kube-api-access-nprvf\") pod \"console-operator-7866497494-xxlp6\" (UID: \"372be3ce-a380-4c0e-bbee-49c2e3b80095\") " pod="openshift-console-operator/console-operator-7866497494-xxlp6" Jul 10 02:48:38.378080 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.378032 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trpjk\" (UniqueName: \"kubernetes.io/projected/0cd9606f-fafb-4005-91fb-1579f4b0cc75-kube-api-access-trpjk\") pod \"service-ca-operator-7695fc6c8c-28scv\" (UID: \"0cd9606f-fafb-4005-91fb-1579f4b0cc75\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-28scv" Jul 10 02:48:38.378175 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.378151 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcdv6\" (UniqueName: \"kubernetes.io/projected/bcda0e99-fdd8-498f-8e73-ecf4c279be71-kube-api-access-wcdv6\") pod \"cluster-monitoring-operator-5f7f494dd4-2rh2j\" (UID: \"bcda0e99-fdd8-498f-8e73-ecf4c279be71\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-2rh2j" Jul 10 02:48:38.378278 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.378257 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvlrc\" (UniqueName: \"kubernetes.io/projected/6136f9e5-f2c2-4e54-94e5-f86957cc814e-kube-api-access-jvlrc\") pod \"ingress-canary-9qrnw\" (UID: \"6136f9e5-f2c2-4e54-94e5-f86957cc814e\") " pod="openshift-ingress-canary/ingress-canary-9qrnw" Jul 10 02:48:38.378725 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.378692 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4t5q\" (UniqueName: \"kubernetes.io/projected/d653dbf1-1efa-41b0-889c-f78d4f1d1ed8-kube-api-access-q4t5q\") pod \"network-check-source-869dcfcdcd-t48ft\" (UID: \"d653dbf1-1efa-41b0-889c-f78d4f1d1ed8\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-t48ft" Jul 10 02:48:38.378927 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.378902 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtqbp\" (UniqueName: \"kubernetes.io/projected/f540bce4-0371-4500-8e77-3bb4291a2e3d-kube-api-access-vtqbp\") pod \"image-registry-fbd7648c9-vtvdf\" (UID: \"f540bce4-0371-4500-8e77-3bb4291a2e3d\") " pod="openshift-image-registry/image-registry-fbd7648c9-vtvdf" Jul 10 02:48:38.379120 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.379101 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzn54\" (UniqueName: \"kubernetes.io/projected/5176df10-dfd5-4514-9a59-c854c4a0ec21-kube-api-access-qzn54\") pod \"dns-default-cc72k\" (UID: \"5176df10-dfd5-4514-9a59-c854c4a0ec21\") " pod="openshift-dns/dns-default-cc72k" Jul 10 02:48:38.379442 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.379424 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8nr4\" (UniqueName: \"kubernetes.io/projected/64e37f35-938c-457a-a377-354c4bc12d73-kube-api-access-m8nr4\") pod \"insights-operator-68d9dd94cf-x22bq\" (UID: \"64e37f35-938c-457a-a377-354c4bc12d73\") " pod="openshift-insights/insights-operator-68d9dd94cf-x22bq" Jul 10 02:48:38.379656 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.379637 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5b6v\" (UniqueName: \"kubernetes.io/projected/4094383d-2bb9-441b-8c30-94ff7551ce18-kube-api-access-d5b6v\") pod \"kube-storage-version-migrator-operator-799cb7f958-gtqxb\" (UID: \"4094383d-2bb9-441b-8c30-94ff7551ce18\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-gtqxb" Jul 10 02:48:38.426279 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.426212 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-x22bq" Jul 10 02:48:38.439408 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.439378 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-28scv" Jul 10 02:48:38.472296 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.472269 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-n8kpv" Jul 10 02:48:38.481538 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.481508 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-xxlp6" Jul 10 02:48:38.497993 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.497722 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-t48ft" Jul 10 02:48:38.505408 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.504957 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-gtqxb" Jul 10 02:48:38.509490 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.509460 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9qrnw" Jul 10 02:48:38.632223 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.631932 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-x22bq"] Jul 10 02:48:38.672835 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.672257 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-28scv"] Jul 10 02:48:38.713307 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.713280 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-xxlp6"] Jul 10 02:48:38.731772 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.731744 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-n8kpv"] Jul 10 02:48:38.735703 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:38.735667 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00377a57_ff2d_4c74_a9de_1fb20db00f5e.slice/crio-75c54f36a6c6661d754a487e0adfe6e0188827f15829e3d81b082f8db6e051cd WatchSource:0}: Error finding container 75c54f36a6c6661d754a487e0adfe6e0188827f15829e3d81b082f8db6e051cd: Status 404 returned error can't find the container with id 75c54f36a6c6661d754a487e0adfe6e0188827f15829e3d81b082f8db6e051cd Jul 10 02:48:38.757164 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.757131 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-gtqxb"] Jul 10 02:48:38.760471 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:38.760445 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4094383d_2bb9_441b_8c30_94ff7551ce18.slice/crio-5d039510eac39913ce134a0f19bff2777e75ca2d9fed6583a4c21dbea4e27d35 WatchSource:0}: Error finding container 5d039510eac39913ce134a0f19bff2777e75ca2d9fed6583a4c21dbea4e27d35: Status 404 returned error can't find the container with id 5d039510eac39913ce134a0f19bff2777e75ca2d9fed6583a4c21dbea4e27d35 Jul 10 02:48:38.766697 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.766663 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-t48ft"] Jul 10 02:48:38.770003 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:38.769980 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd653dbf1_1efa_41b0_889c_f78d4f1d1ed8.slice/crio-65014eeb75adb3ebe737d336d5dc5653b22c0da2063b13ccc4f09c4142b8cee2 WatchSource:0}: Error finding container 65014eeb75adb3ebe737d336d5dc5653b22c0da2063b13ccc4f09c4142b8cee2: Status 404 returned error can't find the container with id 65014eeb75adb3ebe737d336d5dc5653b22c0da2063b13ccc4f09c4142b8cee2 Jul 10 02:48:38.775154 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.775128 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9qrnw"] Jul 10 02:48:38.778569 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:38.778542 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6136f9e5_f2c2_4e54_94e5_f86957cc814e.slice/crio-4007d108f268303c4e0f72688b79e6002a6db44e12f274002670f12c96635702 WatchSource:0}: Error finding container 4007d108f268303c4e0f72688b79e6002a6db44e12f274002670f12c96635702: Status 404 returned error can't find the container with id 4007d108f268303c4e0f72688b79e6002a6db44e12f274002670f12c96635702 Jul 10 02:48:38.863898 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.863863 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:38.880842 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.880796 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bcda0e99-fdd8-498f-8e73-ecf4c279be71-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-2rh2j\" (UID: \"bcda0e99-fdd8-498f-8e73-ecf4c279be71\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-2rh2j" Jul 10 02:48:38.880984 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.880856 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f540bce4-0371-4500-8e77-3bb4291a2e3d-registry-tls\") pod \"image-registry-fbd7648c9-vtvdf\" (UID: \"f540bce4-0371-4500-8e77-3bb4291a2e3d\") " pod="openshift-image-registry/image-registry-fbd7648c9-vtvdf" Jul 10 02:48:38.880984 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.880880 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3adc0ae6-f77b-4909-8bef-8534f19e6601-service-ca-bundle\") pod \"router-default-bfcf8947b-7fz4l\" (UID: \"3adc0ae6-f77b-4909-8bef-8534f19e6601\") " pod="openshift-ingress/router-default-bfcf8947b-7fz4l" Jul 10 02:48:38.880984 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.880901 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3adc0ae6-f77b-4909-8bef-8534f19e6601-metrics-certs\") pod \"router-default-bfcf8947b-7fz4l\" (UID: \"3adc0ae6-f77b-4909-8bef-8534f19e6601\") " pod="openshift-ingress/router-default-bfcf8947b-7fz4l" Jul 10 02:48:38.880984 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:38.880950 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 10 02:48:38.880984 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.880973 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/167ea7f3-1d8f-4892-bc0e-e9247912fef6-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-q44gx\" (UID: \"167ea7f3-1d8f-4892-bc0e-e9247912fef6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-q44gx" Jul 10 02:48:38.881224 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:38.881006 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcda0e99-fdd8-498f-8e73-ecf4c279be71-cluster-monitoring-operator-tls podName:bcda0e99-fdd8-498f-8e73-ecf4c279be71 nodeName:}" failed. No retries permitted until 2026-07-10 02:48:39.880991243 +0000 UTC m=+38.162107506 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bcda0e99-fdd8-498f-8e73-ecf4c279be71-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-2rh2j" (UID: "bcda0e99-fdd8-498f-8e73-ecf4c279be71") : secret "cluster-monitoring-operator-tls" not found Jul 10 02:48:38.881224 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:38.881030 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 10 02:48:38.881224 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:38.881044 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 02:48:38.881224 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:38.881062 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3adc0ae6-f77b-4909-8bef-8534f19e6601-service-ca-bundle podName:3adc0ae6-f77b-4909-8bef-8534f19e6601 nodeName:}" failed. No retries permitted until 2026-07-10 02:48:39.8810364 +0000 UTC m=+38.162152679 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3adc0ae6-f77b-4909-8bef-8534f19e6601-service-ca-bundle") pod "router-default-bfcf8947b-7fz4l" (UID: "3adc0ae6-f77b-4909-8bef-8534f19e6601") : configmap references non-existent config key: service-ca.crt Jul 10 02:48:38.881224 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.881098 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5176df10-dfd5-4514-9a59-c854c4a0ec21-metrics-tls\") pod \"dns-default-cc72k\" (UID: \"5176df10-dfd5-4514-9a59-c854c4a0ec21\") " pod="openshift-dns/dns-default-cc72k" Jul 10 02:48:38.881224 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:38.881119 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 02:48:38.881224 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:38.881131 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fbd7648c9-vtvdf: secret "image-registry-tls" not found Jul 10 02:48:38.881224 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:38.881149 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 02:48:38.881224 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:38.881158 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3adc0ae6-f77b-4909-8bef-8534f19e6601-metrics-certs podName:3adc0ae6-f77b-4909-8bef-8534f19e6601 nodeName:}" failed. No retries permitted until 2026-07-10 02:48:39.881130332 +0000 UTC m=+38.162246607 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3adc0ae6-f77b-4909-8bef-8534f19e6601-metrics-certs") pod "router-default-bfcf8947b-7fz4l" (UID: "3adc0ae6-f77b-4909-8bef-8534f19e6601") : secret "router-metrics-certs-default" not found Jul 10 02:48:38.881224 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:38.881186 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5176df10-dfd5-4514-9a59-c854c4a0ec21-metrics-tls podName:5176df10-dfd5-4514-9a59-c854c4a0ec21 nodeName:}" failed. No retries permitted until 2026-07-10 02:48:39.881175927 +0000 UTC m=+38.162292192 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5176df10-dfd5-4514-9a59-c854c4a0ec21-metrics-tls") pod "dns-default-cc72k" (UID: "5176df10-dfd5-4514-9a59-c854c4a0ec21") : secret "dns-default-metrics-tls" not found Jul 10 02:48:38.881224 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:38.881203 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/167ea7f3-1d8f-4892-bc0e-e9247912fef6-samples-operator-tls podName:167ea7f3-1d8f-4892-bc0e-e9247912fef6 nodeName:}" failed. No retries permitted until 2026-07-10 02:48:39.881194246 +0000 UTC m=+38.162310530 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/167ea7f3-1d8f-4892-bc0e-e9247912fef6-samples-operator-tls") pod "cluster-samples-operator-65db858894-q44gx" (UID: "167ea7f3-1d8f-4892-bc0e-e9247912fef6") : secret "samples-operator-tls" not found Jul 10 02:48:38.881689 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:38.881274 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f540bce4-0371-4500-8e77-3bb4291a2e3d-registry-tls podName:f540bce4-0371-4500-8e77-3bb4291a2e3d nodeName:}" failed. No retries permitted until 2026-07-10 02:48:39.881262449 +0000 UTC m=+38.162378712 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f540bce4-0371-4500-8e77-3bb4291a2e3d-registry-tls") pod "image-registry-fbd7648c9-vtvdf" (UID: "f540bce4-0371-4500-8e77-3bb4291a2e3d") : secret "image-registry-tls" not found Jul 10 02:48:38.964562 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:38.964470 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:39.266070 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:39.265994 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:39.283988 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:39.282453 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bgbpr" Jul 10 02:48:39.283988 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:39.283029 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q28nr" Jul 10 02:48:39.283988 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:39.283604 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nf7q8" Jul 10 02:48:39.286773 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:39.285294 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r5msv\"" Jul 10 02:48:39.286773 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:39.285536 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 02:48:39.286773 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:39.285718 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 02:48:39.286773 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:39.286088 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-p9j2r\"" Jul 10 02:48:39.364918 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:39.364869 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:39.524952 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:39.524858 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-xxlp6" event={"ID":"372be3ce-a380-4c0e-bbee-49c2e3b80095","Type":"ContainerStarted","Data":"2a464ceda2a1ca62a13325d65505994bfcef40e4c12f3d2cff9bee71bddebfb3"} Jul 10 02:48:39.533462 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:39.533423 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9qrnw" event={"ID":"6136f9e5-f2c2-4e54-94e5-f86957cc814e","Type":"ContainerStarted","Data":"4007d108f268303c4e0f72688b79e6002a6db44e12f274002670f12c96635702"} Jul 10 02:48:39.539555 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:39.539480 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-gtqxb" event={"ID":"4094383d-2bb9-441b-8c30-94ff7551ce18","Type":"ContainerStarted","Data":"5d039510eac39913ce134a0f19bff2777e75ca2d9fed6583a4c21dbea4e27d35"} Jul 10 02:48:39.553270 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:39.553172 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-t48ft" event={"ID":"d653dbf1-1efa-41b0-889c-f78d4f1d1ed8","Type":"ContainerStarted","Data":"65014eeb75adb3ebe737d336d5dc5653b22c0da2063b13ccc4f09c4142b8cee2"} Jul 10 02:48:39.560708 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:39.560672 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-28scv" event={"ID":"0cd9606f-fafb-4005-91fb-1579f4b0cc75","Type":"ContainerStarted","Data":"1e31fab2030ddbbc00107cab4dcfe272330b93ea3e75bf2741f6625644d498e1"} Jul 10 02:48:39.564871 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:39.564773 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-x22bq" event={"ID":"64e37f35-938c-457a-a377-354c4bc12d73","Type":"ContainerStarted","Data":"f0c39bc421e91a1cffa4a7ceadbf123cc8bd83baa2024d86c484e5c269766714"} Jul 10 02:48:39.594659 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:39.594571 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-n8kpv" event={"ID":"00377a57-ff2d-4c74-a9de-1fb20db00f5e","Type":"ContainerStarted","Data":"75c54f36a6c6661d754a487e0adfe6e0188827f15829e3d81b082f8db6e051cd"} Jul 10 02:48:39.897768 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:39.897673 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f540bce4-0371-4500-8e77-3bb4291a2e3d-registry-tls\") pod \"image-registry-fbd7648c9-vtvdf\" (UID: \"f540bce4-0371-4500-8e77-3bb4291a2e3d\") " pod="openshift-image-registry/image-registry-fbd7648c9-vtvdf" Jul 10 02:48:39.897768 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:39.897729 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3adc0ae6-f77b-4909-8bef-8534f19e6601-service-ca-bundle\") pod \"router-default-bfcf8947b-7fz4l\" (UID: \"3adc0ae6-f77b-4909-8bef-8534f19e6601\") " pod="openshift-ingress/router-default-bfcf8947b-7fz4l" Jul 10 02:48:39.897768 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:39.897760 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3adc0ae6-f77b-4909-8bef-8534f19e6601-metrics-certs\") pod \"router-default-bfcf8947b-7fz4l\" (UID: \"3adc0ae6-f77b-4909-8bef-8534f19e6601\") " pod="openshift-ingress/router-default-bfcf8947b-7fz4l" Jul 10 02:48:39.898064 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:39.897823 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/167ea7f3-1d8f-4892-bc0e-e9247912fef6-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-q44gx\" (UID: \"167ea7f3-1d8f-4892-bc0e-e9247912fef6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-q44gx" Jul 10 02:48:39.898064 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:39.897865 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5176df10-dfd5-4514-9a59-c854c4a0ec21-metrics-tls\") pod \"dns-default-cc72k\" (UID: \"5176df10-dfd5-4514-9a59-c854c4a0ec21\") " pod="openshift-dns/dns-default-cc72k" Jul 10 02:48:39.898064 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:39.897922 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bcda0e99-fdd8-498f-8e73-ecf4c279be71-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-2rh2j\" (UID: \"bcda0e99-fdd8-498f-8e73-ecf4c279be71\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-2rh2j" Jul 10 02:48:39.898197 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:39.898068 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 10 02:48:39.898197 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:39.898134 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcda0e99-fdd8-498f-8e73-ecf4c279be71-cluster-monitoring-operator-tls podName:bcda0e99-fdd8-498f-8e73-ecf4c279be71 nodeName:}" failed. No retries permitted until 2026-07-10 02:48:41.898115754 +0000 UTC m=+40.179232037 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bcda0e99-fdd8-498f-8e73-ecf4c279be71-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-2rh2j" (UID: "bcda0e99-fdd8-498f-8e73-ecf4c279be71") : secret "cluster-monitoring-operator-tls" not found Jul 10 02:48:39.898575 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:39.898553 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 02:48:39.898575 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:39.898575 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fbd7648c9-vtvdf: secret "image-registry-tls" not found Jul 10 02:48:39.898733 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:39.898620 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f540bce4-0371-4500-8e77-3bb4291a2e3d-registry-tls podName:f540bce4-0371-4500-8e77-3bb4291a2e3d nodeName:}" failed. No retries permitted until 2026-07-10 02:48:41.898605412 +0000 UTC m=+40.179721676 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f540bce4-0371-4500-8e77-3bb4291a2e3d-registry-tls") pod "image-registry-fbd7648c9-vtvdf" (UID: "f540bce4-0371-4500-8e77-3bb4291a2e3d") : secret "image-registry-tls" not found Jul 10 02:48:39.898733 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:39.898690 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3adc0ae6-f77b-4909-8bef-8534f19e6601-service-ca-bundle podName:3adc0ae6-f77b-4909-8bef-8534f19e6601 nodeName:}" failed. No retries permitted until 2026-07-10 02:48:41.898680347 +0000 UTC m=+40.179796624 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3adc0ae6-f77b-4909-8bef-8534f19e6601-service-ca-bundle") pod "router-default-bfcf8947b-7fz4l" (UID: "3adc0ae6-f77b-4909-8bef-8534f19e6601") : configmap references non-existent config key: service-ca.crt Jul 10 02:48:39.898924 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:39.898754 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 10 02:48:39.898924 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:39.898801 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3adc0ae6-f77b-4909-8bef-8534f19e6601-metrics-certs podName:3adc0ae6-f77b-4909-8bef-8534f19e6601 nodeName:}" failed. No retries permitted until 2026-07-10 02:48:41.898791074 +0000 UTC m=+40.179907350 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3adc0ae6-f77b-4909-8bef-8534f19e6601-metrics-certs") pod "router-default-bfcf8947b-7fz4l" (UID: "3adc0ae6-f77b-4909-8bef-8534f19e6601") : secret "router-metrics-certs-default" not found Jul 10 02:48:39.898924 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:39.898874 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 02:48:39.898924 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:39.898910 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/167ea7f3-1d8f-4892-bc0e-e9247912fef6-samples-operator-tls podName:167ea7f3-1d8f-4892-bc0e-e9247912fef6 nodeName:}" failed. No retries permitted until 2026-07-10 02:48:41.898898688 +0000 UTC m=+40.180014957 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/167ea7f3-1d8f-4892-bc0e-e9247912fef6-samples-operator-tls") pod "cluster-samples-operator-65db858894-q44gx" (UID: "167ea7f3-1d8f-4892-bc0e-e9247912fef6") : secret "samples-operator-tls" not found Jul 10 02:48:39.899137 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:39.898968 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 02:48:39.899137 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:39.898998 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5176df10-dfd5-4514-9a59-c854c4a0ec21-metrics-tls podName:5176df10-dfd5-4514-9a59-c854c4a0ec21 nodeName:}" failed. No retries permitted until 2026-07-10 02:48:41.89898831 +0000 UTC m=+40.180104573 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5176df10-dfd5-4514-9a59-c854c4a0ec21-metrics-tls") pod "dns-default-cc72k" (UID: "5176df10-dfd5-4514-9a59-c854c4a0ec21") : secret "dns-default-metrics-tls" not found Jul 10 02:48:41.609238 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:41.608063 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-n8kpv" event={"ID":"00377a57-ff2d-4c74-a9de-1fb20db00f5e","Type":"ContainerStarted","Data":"7687175a8779b0a8b540da676f6961aa7d2fb20c21385d82ec89ffe7b0170ac6"} Jul 10 02:48:41.698881 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:41.698840 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:41.752010 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:41.751973 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:41.916514 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:41.916431 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3adc0ae6-f77b-4909-8bef-8534f19e6601-service-ca-bundle\") pod \"router-default-bfcf8947b-7fz4l\" (UID: \"3adc0ae6-f77b-4909-8bef-8534f19e6601\") " pod="openshift-ingress/router-default-bfcf8947b-7fz4l" Jul 10 02:48:41.916514 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:41.916474 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3adc0ae6-f77b-4909-8bef-8534f19e6601-metrics-certs\") pod \"router-default-bfcf8947b-7fz4l\" (UID: \"3adc0ae6-f77b-4909-8bef-8534f19e6601\") " pod="openshift-ingress/router-default-bfcf8947b-7fz4l" Jul 10 02:48:41.916723 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:41.916528 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/167ea7f3-1d8f-4892-bc0e-e9247912fef6-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-q44gx\" (UID: \"167ea7f3-1d8f-4892-bc0e-e9247912fef6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-q44gx" Jul 10 02:48:41.916723 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:41.916571 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5176df10-dfd5-4514-9a59-c854c4a0ec21-metrics-tls\") pod \"dns-default-cc72k\" (UID: \"5176df10-dfd5-4514-9a59-c854c4a0ec21\") " pod="openshift-dns/dns-default-cc72k" Jul 10 02:48:41.916723 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:41.916599 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3adc0ae6-f77b-4909-8bef-8534f19e6601-service-ca-bundle podName:3adc0ae6-f77b-4909-8bef-8534f19e6601 nodeName:}" failed. No retries permitted until 2026-07-10 02:48:45.916578065 +0000 UTC m=+44.197694345 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3adc0ae6-f77b-4909-8bef-8534f19e6601-service-ca-bundle") pod "router-default-bfcf8947b-7fz4l" (UID: "3adc0ae6-f77b-4909-8bef-8534f19e6601") : configmap references non-existent config key: service-ca.crt Jul 10 02:48:41.916723 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:41.916648 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 02:48:41.916723 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:41.916664 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 10 02:48:41.916723 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:41.916683 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5176df10-dfd5-4514-9a59-c854c4a0ec21-metrics-tls podName:5176df10-dfd5-4514-9a59-c854c4a0ec21 nodeName:}" failed. No retries permitted until 2026-07-10 02:48:45.916672689 +0000 UTC m=+44.197788970 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5176df10-dfd5-4514-9a59-c854c4a0ec21-metrics-tls") pod "dns-default-cc72k" (UID: "5176df10-dfd5-4514-9a59-c854c4a0ec21") : secret "dns-default-metrics-tls" not found Jul 10 02:48:41.916723 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:41.916685 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 02:48:41.916723 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:41.916669 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bcda0e99-fdd8-498f-8e73-ecf4c279be71-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-2rh2j\" (UID: \"bcda0e99-fdd8-498f-8e73-ecf4c279be71\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-2rh2j" Jul 10 02:48:41.917125 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:41.916725 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 10 02:48:41.917125 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:41.916756 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/167ea7f3-1d8f-4892-bc0e-e9247912fef6-samples-operator-tls podName:167ea7f3-1d8f-4892-bc0e-e9247912fef6 nodeName:}" failed. No retries permitted until 2026-07-10 02:48:45.916737474 +0000 UTC m=+44.197853752 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/167ea7f3-1d8f-4892-bc0e-e9247912fef6-samples-operator-tls") pod "cluster-samples-operator-65db858894-q44gx" (UID: "167ea7f3-1d8f-4892-bc0e-e9247912fef6") : secret "samples-operator-tls" not found Jul 10 02:48:41.917125 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:41.916780 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3adc0ae6-f77b-4909-8bef-8534f19e6601-metrics-certs podName:3adc0ae6-f77b-4909-8bef-8534f19e6601 nodeName:}" failed. No retries permitted until 2026-07-10 02:48:45.916769282 +0000 UTC m=+44.197885546 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3adc0ae6-f77b-4909-8bef-8534f19e6601-metrics-certs") pod "router-default-bfcf8947b-7fz4l" (UID: "3adc0ae6-f77b-4909-8bef-8534f19e6601") : secret "router-metrics-certs-default" not found Jul 10 02:48:41.917125 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:41.916795 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcda0e99-fdd8-498f-8e73-ecf4c279be71-cluster-monitoring-operator-tls podName:bcda0e99-fdd8-498f-8e73-ecf4c279be71 nodeName:}" failed. No retries permitted until 2026-07-10 02:48:45.916786504 +0000 UTC m=+44.197902769 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bcda0e99-fdd8-498f-8e73-ecf4c279be71-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-2rh2j" (UID: "bcda0e99-fdd8-498f-8e73-ecf4c279be71") : secret "cluster-monitoring-operator-tls" not found Jul 10 02:48:41.917125 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:41.916875 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f540bce4-0371-4500-8e77-3bb4291a2e3d-registry-tls\") pod \"image-registry-fbd7648c9-vtvdf\" (UID: \"f540bce4-0371-4500-8e77-3bb4291a2e3d\") " pod="openshift-image-registry/image-registry-fbd7648c9-vtvdf" Jul 10 02:48:41.917125 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:41.916983 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 02:48:41.917125 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:41.916995 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fbd7648c9-vtvdf: secret "image-registry-tls" not found Jul 10 02:48:41.917125 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:41.917034 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f540bce4-0371-4500-8e77-3bb4291a2e3d-registry-tls podName:f540bce4-0371-4500-8e77-3bb4291a2e3d nodeName:}" failed. No retries permitted until 2026-07-10 02:48:45.9170233 +0000 UTC m=+44.198139598 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f540bce4-0371-4500-8e77-3bb4291a2e3d-registry-tls") pod "image-registry-fbd7648c9-vtvdf" (UID: "f540bce4-0371-4500-8e77-3bb4291a2e3d") : secret "image-registry-tls" not found Jul 10 02:48:42.258152 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:42.258110 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:42.358777 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:42.358598 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:42.459990 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:42.459949 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:42.520549 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:42.520455 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:45.950501 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:45.950459 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5176df10-dfd5-4514-9a59-c854c4a0ec21-metrics-tls\") pod \"dns-default-cc72k\" (UID: \"5176df10-dfd5-4514-9a59-c854c4a0ec21\") " pod="openshift-dns/dns-default-cc72k" Jul 10 02:48:45.951006 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:45.950552 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bcda0e99-fdd8-498f-8e73-ecf4c279be71-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-2rh2j\" (UID: \"bcda0e99-fdd8-498f-8e73-ecf4c279be71\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-2rh2j" Jul 10 02:48:45.951006 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:45.950593 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f540bce4-0371-4500-8e77-3bb4291a2e3d-registry-tls\") pod \"image-registry-fbd7648c9-vtvdf\" (UID: \"f540bce4-0371-4500-8e77-3bb4291a2e3d\") " pod="openshift-image-registry/image-registry-fbd7648c9-vtvdf" Jul 10 02:48:45.951006 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:45.950622 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3adc0ae6-f77b-4909-8bef-8534f19e6601-service-ca-bundle\") pod \"router-default-bfcf8947b-7fz4l\" (UID: \"3adc0ae6-f77b-4909-8bef-8534f19e6601\") " pod="openshift-ingress/router-default-bfcf8947b-7fz4l" Jul 10 02:48:45.951006 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:45.950642 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 02:48:45.951006 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:45.950717 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5176df10-dfd5-4514-9a59-c854c4a0ec21-metrics-tls podName:5176df10-dfd5-4514-9a59-c854c4a0ec21 nodeName:}" failed. No retries permitted until 2026-07-10 02:48:53.950696916 +0000 UTC m=+52.231813197 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5176df10-dfd5-4514-9a59-c854c4a0ec21-metrics-tls") pod "dns-default-cc72k" (UID: "5176df10-dfd5-4514-9a59-c854c4a0ec21") : secret "dns-default-metrics-tls" not found Jul 10 02:48:45.951006 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:45.950736 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 10 02:48:45.951006 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:45.950750 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 02:48:45.951006 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:45.950764 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3adc0ae6-f77b-4909-8bef-8534f19e6601-service-ca-bundle podName:3adc0ae6-f77b-4909-8bef-8534f19e6601 nodeName:}" failed. No retries permitted until 2026-07-10 02:48:53.950752187 +0000 UTC m=+52.231868455 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3adc0ae6-f77b-4909-8bef-8534f19e6601-service-ca-bundle") pod "router-default-bfcf8947b-7fz4l" (UID: "3adc0ae6-f77b-4909-8bef-8534f19e6601") : configmap references non-existent config key: service-ca.crt Jul 10 02:48:45.951006 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:45.950653 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3adc0ae6-f77b-4909-8bef-8534f19e6601-metrics-certs\") pod \"router-default-bfcf8947b-7fz4l\" (UID: \"3adc0ae6-f77b-4909-8bef-8534f19e6601\") " pod="openshift-ingress/router-default-bfcf8947b-7fz4l" Jul 10 02:48:45.951006 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:45.950770 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 10 02:48:45.951006 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:45.950786 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3adc0ae6-f77b-4909-8bef-8534f19e6601-metrics-certs podName:3adc0ae6-f77b-4909-8bef-8534f19e6601 nodeName:}" failed. No retries permitted until 2026-07-10 02:48:53.950776642 +0000 UTC m=+52.231892905 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3adc0ae6-f77b-4909-8bef-8534f19e6601-metrics-certs") pod "router-default-bfcf8947b-7fz4l" (UID: "3adc0ae6-f77b-4909-8bef-8534f19e6601") : secret "router-metrics-certs-default" not found Jul 10 02:48:45.951006 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:45.950772 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fbd7648c9-vtvdf: secret "image-registry-tls" not found Jul 10 02:48:45.951006 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:45.950838 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcda0e99-fdd8-498f-8e73-ecf4c279be71-cluster-monitoring-operator-tls podName:bcda0e99-fdd8-498f-8e73-ecf4c279be71 nodeName:}" failed. No retries permitted until 2026-07-10 02:48:53.950797086 +0000 UTC m=+52.231913354 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bcda0e99-fdd8-498f-8e73-ecf4c279be71-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-2rh2j" (UID: "bcda0e99-fdd8-498f-8e73-ecf4c279be71") : secret "cluster-monitoring-operator-tls" not found Jul 10 02:48:45.951006 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:45.950872 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f540bce4-0371-4500-8e77-3bb4291a2e3d-registry-tls podName:f540bce4-0371-4500-8e77-3bb4291a2e3d nodeName:}" failed. No retries permitted until 2026-07-10 02:48:53.950855506 +0000 UTC m=+52.231971785 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f540bce4-0371-4500-8e77-3bb4291a2e3d-registry-tls") pod "image-registry-fbd7648c9-vtvdf" (UID: "f540bce4-0371-4500-8e77-3bb4291a2e3d") : secret "image-registry-tls" not found Jul 10 02:48:45.951006 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:45.950932 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/167ea7f3-1d8f-4892-bc0e-e9247912fef6-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-q44gx\" (UID: \"167ea7f3-1d8f-4892-bc0e-e9247912fef6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-q44gx" Jul 10 02:48:45.951697 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:45.951050 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 02:48:45.951697 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:45.951093 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/167ea7f3-1d8f-4892-bc0e-e9247912fef6-samples-operator-tls podName:167ea7f3-1d8f-4892-bc0e-e9247912fef6 nodeName:}" failed. No retries permitted until 2026-07-10 02:48:53.951081091 +0000 UTC m=+52.232197365 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/167ea7f3-1d8f-4892-bc0e-e9247912fef6-samples-operator-tls") pod "cluster-samples-operator-65db858894-q44gx" (UID: "167ea7f3-1d8f-4892-bc0e-e9247912fef6") : secret "samples-operator-tls" not found Jul 10 02:48:47.447349 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:47.447309 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:47.526794 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:47.526749 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:47.602214 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:47.602174 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:47.700849 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:47.700754 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:47.801746 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:47.801709 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:47.905849 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:47.905795 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:48.002618 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:48.002542 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:48.631465 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:48.631430 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-xxlp6_372be3ce-a380-4c0e-bbee-49c2e3b80095/console-operator/0.log" Jul 10 02:48:48.631798 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:48.631482 2575 generic.go:356] "Generic (PLEG): container finished" podID="372be3ce-a380-4c0e-bbee-49c2e3b80095" containerID="6533e3df8b998f1174dd49c125957f1c21be38e4e0c46d88f69c70834c21c6b2" exitCode=255 Jul 10 02:48:48.631995 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:48.631894 2575 scope.go:117] "RemoveContainer" containerID="6533e3df8b998f1174dd49c125957f1c21be38e4e0c46d88f69c70834c21c6b2" Jul 10 02:48:48.631995 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:48.631916 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-xxlp6" event={"ID":"372be3ce-a380-4c0e-bbee-49c2e3b80095","Type":"ContainerDied","Data":"6533e3df8b998f1174dd49c125957f1c21be38e4e0c46d88f69c70834c21c6b2"} Jul 10 02:48:48.635134 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:48.634897 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9qrnw" event={"ID":"6136f9e5-f2c2-4e54-94e5-f86957cc814e","Type":"ContainerStarted","Data":"64ad4d2a736eb1446e08ca78c3d0b83b7b049f655127a2f6337ef2f4de757a3c"} Jul 10 02:48:48.636899 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:48.636865 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-gtqxb" event={"ID":"4094383d-2bb9-441b-8c30-94ff7551ce18","Type":"ContainerStarted","Data":"89fb07e085b038527a8b31c5a072286c88c3172ada0f1f95cc471a74fed8fe81"} Jul 10 02:48:48.638973 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:48.638794 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-28scv" event={"ID":"0cd9606f-fafb-4005-91fb-1579f4b0cc75","Type":"ContainerStarted","Data":"f5d6c83e4b4bb95a911cf9f38c6ed99ec3bb28fd5f81e5e6a5ef9023c273eda8"} Jul 10 02:48:48.642082 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:48.641090 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-x22bq" event={"ID":"64e37f35-938c-457a-a377-354c4bc12d73","Type":"ContainerStarted","Data":"f1aa012ec41c9cd7913e8bd040e9db4acb2e365c66afd0920c65d64874316458"} Jul 10 02:48:48.650923 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:48.649740 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-n8kpv" podStartSLOduration=18.583634457 podStartE2EDuration="20.649723791s" totalImagesPullingTime="2.066089334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:48:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:48:41.625216297 +0000 UTC m=+39.906332602" watchObservedRunningTime="2026-07-10 02:48:48.649723791 +0000 UTC m=+46.930840077" Jul 10 02:48:48.668065 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:48.668015 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-68d9dd94cf-x22bq" podStartSLOduration=11.981337599 podStartE2EDuration="21.667994416s" totalImagesPullingTime="9.686656817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:48:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:48:48.667397152 +0000 UTC m=+46.948513441" watchObservedRunningTime="2026-07-10 02:48:48.667994416 +0000 UTC m=+46.949110702" Jul 10 02:48:48.687946 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:48.687680 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-28scv" podStartSLOduration=11.002599436 podStartE2EDuration="20.687653992s" totalImagesPullingTime="9.685054556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:48:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:48:48.686185769 +0000 UTC m=+46.967302080" watchObservedRunningTime="2026-07-10 02:48:48.687653992 +0000 UTC m=+46.968770279" Jul 10 02:48:48.707552 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:48.707487 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-gtqxb" podStartSLOduration=11.099500902 podStartE2EDuration="20.707466841s" totalImagesPullingTime="9.607965939s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:48:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:48:48.70638994 +0000 UTC m=+46.987506227" watchObservedRunningTime="2026-07-10 02:48:48.707466841 +0000 UTC m=+46.988583127" Jul 10 02:48:49.648217 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:49.648179 2575 generic.go:356] "Generic (PLEG): container finished" podID="ff323ccb-9efc-43fd-8ce7-9365fe9af996" containerID="fe4385b2d938aff709f40bb6526ed0a31482bd96c8033adc75df825f8ba87185" exitCode=0 Jul 10 02:48:49.648638 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:49.648409 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tfrkp" event={"ID":"ff323ccb-9efc-43fd-8ce7-9365fe9af996","Type":"ContainerDied","Data":"fe4385b2d938aff709f40bb6526ed0a31482bd96c8033adc75df825f8ba87185"} Jul 10 02:48:49.659586 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:49.659508 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-xxlp6_372be3ce-a380-4c0e-bbee-49c2e3b80095/console-operator/1.log" Jul 10 02:48:49.660096 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:49.660074 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-xxlp6_372be3ce-a380-4c0e-bbee-49c2e3b80095/console-operator/0.log" Jul 10 02:48:49.660227 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:49.660111 2575 generic.go:356] "Generic (PLEG): container finished" podID="372be3ce-a380-4c0e-bbee-49c2e3b80095" containerID="c2018deea9d6279c181cf3ccbdd4414834ea69449237a7dc619c75a8159aa2a1" exitCode=255 Jul 10 02:48:49.660227 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:49.660198 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-xxlp6" event={"ID":"372be3ce-a380-4c0e-bbee-49c2e3b80095","Type":"ContainerDied","Data":"c2018deea9d6279c181cf3ccbdd4414834ea69449237a7dc619c75a8159aa2a1"} Jul 10 02:48:49.660365 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:49.660246 2575 scope.go:117] "RemoveContainer" containerID="6533e3df8b998f1174dd49c125957f1c21be38e4e0c46d88f69c70834c21c6b2" Jul 10 02:48:49.660664 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:49.660637 2575 scope.go:117] "RemoveContainer" containerID="c2018deea9d6279c181cf3ccbdd4414834ea69449237a7dc619c75a8159aa2a1" Jul 10 02:48:49.660852 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:49.660831 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-xxlp6_openshift-console-operator(372be3ce-a380-4c0e-bbee-49c2e3b80095)\"" pod="openshift-console-operator/console-operator-7866497494-xxlp6" podUID="372be3ce-a380-4c0e-bbee-49c2e3b80095" Jul 10 02:48:49.663065 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:49.663038 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-t48ft" event={"ID":"d653dbf1-1efa-41b0-889c-f78d4f1d1ed8","Type":"ContainerStarted","Data":"57042b77262a30c9622aee5021ff78a7419778e4ea7ef17433dc386826621743"} Jul 10 02:48:49.674509 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:49.674418 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9qrnw" podStartSLOduration=2.084496261 podStartE2EDuration="11.674399308s" totalImagesPullingTime="9.589903047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:48:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:48:48.724780928 +0000 UTC m=+47.005897257" watchObservedRunningTime="2026-07-10 02:48:49.674399308 +0000 UTC m=+47.955515592" Jul 10 02:48:49.705766 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:49.705713 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-w7npn"] Jul 10 02:48:49.712217 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:49.712021 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-t48ft" podStartSLOduration=12.10335901 podStartE2EDuration="21.71200524s" totalImagesPullingTime="9.60864623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:48:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:48:49.7109912 +0000 UTC m=+47.992107514" watchObservedRunningTime="2026-07-10 02:48:49.71200524 +0000 UTC m=+47.993121525" Jul 10 02:48:49.738619 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:49.738591 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-w7npn"] Jul 10 02:48:49.739145 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:49.739122 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-w7npn" Jul 10 02:48:49.743602 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:49.743572 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-f756g\"" Jul 10 02:48:49.743876 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:49.743853 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 10 02:48:49.743986 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:49.743931 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 10 02:48:49.762304 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:49.762272 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:49.789061 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:49.789029 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b3661326-52c2-4c9b-b2db-420720fe4a86-data-volume\") pod \"insights-runtime-extractor-w7npn\" (UID: \"b3661326-52c2-4c9b-b2db-420720fe4a86\") " pod="openshift-insights/insights-runtime-extractor-w7npn" Jul 10 02:48:49.789211 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:49.789072 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lsp9\" (UniqueName: \"kubernetes.io/projected/b3661326-52c2-4c9b-b2db-420720fe4a86-kube-api-access-7lsp9\") pod \"insights-runtime-extractor-w7npn\" (UID: \"b3661326-52c2-4c9b-b2db-420720fe4a86\") " pod="openshift-insights/insights-runtime-extractor-w7npn" Jul 10 02:48:49.789211 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:49.789134 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b3661326-52c2-4c9b-b2db-420720fe4a86-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w7npn\" (UID: \"b3661326-52c2-4c9b-b2db-420720fe4a86\") " pod="openshift-insights/insights-runtime-extractor-w7npn" Jul 10 02:48:49.789211 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:49.789170 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b3661326-52c2-4c9b-b2db-420720fe4a86-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w7npn\" (UID: \"b3661326-52c2-4c9b-b2db-420720fe4a86\") " pod="openshift-insights/insights-runtime-extractor-w7npn" Jul 10 02:48:49.789359 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:49.789294 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b3661326-52c2-4c9b-b2db-420720fe4a86-crio-socket\") pod \"insights-runtime-extractor-w7npn\" (UID: \"b3661326-52c2-4c9b-b2db-420720fe4a86\") " pod="openshift-insights/insights-runtime-extractor-w7npn" Jul 10 02:48:49.828066 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:49.828036 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:49.890378 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:49.890342 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b3661326-52c2-4c9b-b2db-420720fe4a86-data-volume\") pod \"insights-runtime-extractor-w7npn\" (UID: \"b3661326-52c2-4c9b-b2db-420720fe4a86\") " pod="openshift-insights/insights-runtime-extractor-w7npn" Jul 10 02:48:49.890530 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:49.890389 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lsp9\" (UniqueName: \"kubernetes.io/projected/b3661326-52c2-4c9b-b2db-420720fe4a86-kube-api-access-7lsp9\") pod \"insights-runtime-extractor-w7npn\" (UID: \"b3661326-52c2-4c9b-b2db-420720fe4a86\") " pod="openshift-insights/insights-runtime-extractor-w7npn" Jul 10 02:48:49.890530 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:49.890446 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b3661326-52c2-4c9b-b2db-420720fe4a86-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w7npn\" (UID: \"b3661326-52c2-4c9b-b2db-420720fe4a86\") " pod="openshift-insights/insights-runtime-extractor-w7npn" Jul 10 02:48:49.890530 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:49.890482 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b3661326-52c2-4c9b-b2db-420720fe4a86-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w7npn\" (UID: \"b3661326-52c2-4c9b-b2db-420720fe4a86\") " pod="openshift-insights/insights-runtime-extractor-w7npn" Jul 10 02:48:49.890673 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:49.890584 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 02:48:49.890673 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:49.890647 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b3661326-52c2-4c9b-b2db-420720fe4a86-kube-rbac-proxy-cm podName:b3661326-52c2-4c9b-b2db-420720fe4a86 nodeName:}" failed. No retries permitted until 2026-07-10 02:48:50.390626513 +0000 UTC m=+48.671742790 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b3661326-52c2-4c9b-b2db-420720fe4a86-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w7npn" (UID: "b3661326-52c2-4c9b-b2db-420720fe4a86") : configmap "kube-rbac-proxy" not found Jul 10 02:48:49.890775 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:49.890740 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b3661326-52c2-4c9b-b2db-420720fe4a86-data-volume\") pod \"insights-runtime-extractor-w7npn\" (UID: \"b3661326-52c2-4c9b-b2db-420720fe4a86\") " pod="openshift-insights/insights-runtime-extractor-w7npn" Jul 10 02:48:49.890875 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:49.890845 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 02:48:49.890952 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:49.890896 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3661326-52c2-4c9b-b2db-420720fe4a86-insights-runtime-extractor-tls podName:b3661326-52c2-4c9b-b2db-420720fe4a86 nodeName:}" failed. No retries permitted until 2026-07-10 02:48:50.390878992 +0000 UTC m=+48.671995268 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b3661326-52c2-4c9b-b2db-420720fe4a86-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w7npn" (UID: "b3661326-52c2-4c9b-b2db-420720fe4a86") : secret "insights-runtime-extractor-tls" not found Jul 10 02:48:49.890952 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:49.890918 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b3661326-52c2-4c9b-b2db-420720fe4a86-crio-socket\") pod \"insights-runtime-extractor-w7npn\" (UID: \"b3661326-52c2-4c9b-b2db-420720fe4a86\") " pod="openshift-insights/insights-runtime-extractor-w7npn" Jul 10 02:48:49.891150 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:49.891133 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b3661326-52c2-4c9b-b2db-420720fe4a86-crio-socket\") pod \"insights-runtime-extractor-w7npn\" (UID: \"b3661326-52c2-4c9b-b2db-420720fe4a86\") " pod="openshift-insights/insights-runtime-extractor-w7npn" Jul 10 02:48:49.904129 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:49.904100 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lsp9\" (UniqueName: \"kubernetes.io/projected/b3661326-52c2-4c9b-b2db-420720fe4a86-kube-api-access-7lsp9\") pod \"insights-runtime-extractor-w7npn\" (UID: \"b3661326-52c2-4c9b-b2db-420720fe4a86\") " pod="openshift-insights/insights-runtime-extractor-w7npn" Jul 10 02:48:50.236540 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:50.236492 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:50.289257 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:50.289219 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-7bqcj"] Jul 10 02:48:50.312302 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:50.312269 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-7bqcj"] Jul 10 02:48:50.312466 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:50.312359 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-7bqcj" Jul 10 02:48:50.315007 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:50.314977 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 10 02:48:50.315138 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:50.315032 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-pbgfv\"" Jul 10 02:48:50.315138 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:50.315049 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 10 02:48:50.332331 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:50.332305 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:50.391850 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:50.391792 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:50.395791 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:50.395770 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m4dl\" (UniqueName: \"kubernetes.io/projected/c568f52e-6afa-453d-8b32-e5cb8b4f762e-kube-api-access-9m4dl\") pod \"migrator-7f59b54f6c-7bqcj\" (UID: \"c568f52e-6afa-453d-8b32-e5cb8b4f762e\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-7bqcj" Jul 10 02:48:50.395907 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:50.395890 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b3661326-52c2-4c9b-b2db-420720fe4a86-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w7npn\" (UID: \"b3661326-52c2-4c9b-b2db-420720fe4a86\") " pod="openshift-insights/insights-runtime-extractor-w7npn" Jul 10 02:48:50.395949 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:50.395929 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b3661326-52c2-4c9b-b2db-420720fe4a86-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w7npn\" (UID: \"b3661326-52c2-4c9b-b2db-420720fe4a86\") " pod="openshift-insights/insights-runtime-extractor-w7npn" Jul 10 02:48:50.396056 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:50.396043 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 02:48:50.396090 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:50.396066 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 02:48:50.396119 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:50.396101 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3661326-52c2-4c9b-b2db-420720fe4a86-insights-runtime-extractor-tls podName:b3661326-52c2-4c9b-b2db-420720fe4a86 nodeName:}" failed. No retries permitted until 2026-07-10 02:48:51.396087797 +0000 UTC m=+49.677204060 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b3661326-52c2-4c9b-b2db-420720fe4a86-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w7npn" (UID: "b3661326-52c2-4c9b-b2db-420720fe4a86") : secret "insights-runtime-extractor-tls" not found Jul 10 02:48:50.396162 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:50.396120 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b3661326-52c2-4c9b-b2db-420720fe4a86-kube-rbac-proxy-cm podName:b3661326-52c2-4c9b-b2db-420720fe4a86 nodeName:}" failed. No retries permitted until 2026-07-10 02:48:51.396107609 +0000 UTC m=+49.677223872 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b3661326-52c2-4c9b-b2db-420720fe4a86-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w7npn" (UID: "b3661326-52c2-4c9b-b2db-420720fe4a86") : configmap "kube-rbac-proxy" not found Jul 10 02:48:50.496830 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:50.496738 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m4dl\" (UniqueName: \"kubernetes.io/projected/c568f52e-6afa-453d-8b32-e5cb8b4f762e-kube-api-access-9m4dl\") pod \"migrator-7f59b54f6c-7bqcj\" (UID: \"c568f52e-6afa-453d-8b32-e5cb8b4f762e\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-7bqcj" Jul 10 02:48:50.504274 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:50.504245 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m4dl\" (UniqueName: \"kubernetes.io/projected/c568f52e-6afa-453d-8b32-e5cb8b4f762e-kube-api-access-9m4dl\") pod \"migrator-7f59b54f6c-7bqcj\" (UID: \"c568f52e-6afa-453d-8b32-e5cb8b4f762e\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-7bqcj" Jul 10 02:48:50.621576 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:50.621543 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-7bqcj" Jul 10 02:48:50.669972 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:50.669940 2575 generic.go:356] "Generic (PLEG): container finished" podID="ff323ccb-9efc-43fd-8ce7-9365fe9af996" containerID="c76dd244d60fb316d51183152c5b0b2ca25342760fda92f0a905afbf6864f0dd" exitCode=0 Jul 10 02:48:50.670365 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:50.670051 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tfrkp" event={"ID":"ff323ccb-9efc-43fd-8ce7-9365fe9af996","Type":"ContainerDied","Data":"c76dd244d60fb316d51183152c5b0b2ca25342760fda92f0a905afbf6864f0dd"} Jul 10 02:48:50.671727 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:50.671709 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-xxlp6_372be3ce-a380-4c0e-bbee-49c2e3b80095/console-operator/1.log" Jul 10 02:48:50.672056 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:50.672040 2575 scope.go:117] "RemoveContainer" containerID="c2018deea9d6279c181cf3ccbdd4414834ea69449237a7dc619c75a8159aa2a1" Jul 10 02:48:50.672241 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:50.672218 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-xxlp6_openshift-console-operator(372be3ce-a380-4c0e-bbee-49c2e3b80095)\"" pod="openshift-console-operator/console-operator-7866497494-xxlp6" podUID="372be3ce-a380-4c0e-bbee-49c2e3b80095" Jul 10 02:48:50.789597 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:50.789570 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-7bqcj"] Jul 10 02:48:50.792272 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:50.792247 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc568f52e_6afa_453d_8b32_e5cb8b4f762e.slice/crio-18924b9f5dabbfaa9f837c9bae50041adac2239285795e5aa49206c1563de490 WatchSource:0}: Error finding container 18924b9f5dabbfaa9f837c9bae50041adac2239285795e5aa49206c1563de490: Status 404 returned error can't find the container with id 18924b9f5dabbfaa9f837c9bae50041adac2239285795e5aa49206c1563de490 Jul 10 02:48:51.025674 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:51.025589 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:51.126521 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:51.126486 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:51.127180 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:51.127164 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-shf9m_731666e1-c05f-4169-98f4-df0efa01817a/dns-node-resolver/0.log" Jul 10 02:48:51.404817 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:51.404730 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b3661326-52c2-4c9b-b2db-420720fe4a86-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w7npn\" (UID: \"b3661326-52c2-4c9b-b2db-420720fe4a86\") " pod="openshift-insights/insights-runtime-extractor-w7npn" Jul 10 02:48:51.404817 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:51.404771 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b3661326-52c2-4c9b-b2db-420720fe4a86-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w7npn\" (UID: \"b3661326-52c2-4c9b-b2db-420720fe4a86\") " pod="openshift-insights/insights-runtime-extractor-w7npn" Jul 10 02:48:51.405008 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:51.404904 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 02:48:51.405008 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:51.404937 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 02:48:51.405008 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:51.404977 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3661326-52c2-4c9b-b2db-420720fe4a86-insights-runtime-extractor-tls podName:b3661326-52c2-4c9b-b2db-420720fe4a86 nodeName:}" failed. No retries permitted until 2026-07-10 02:48:53.404957574 +0000 UTC m=+51.686073840 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b3661326-52c2-4c9b-b2db-420720fe4a86-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w7npn" (UID: "b3661326-52c2-4c9b-b2db-420720fe4a86") : secret "insights-runtime-extractor-tls" not found Jul 10 02:48:51.405008 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:51.405001 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b3661326-52c2-4c9b-b2db-420720fe4a86-kube-rbac-proxy-cm podName:b3661326-52c2-4c9b-b2db-420720fe4a86 nodeName:}" failed. No retries permitted until 2026-07-10 02:48:53.404988841 +0000 UTC m=+51.686105106 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b3661326-52c2-4c9b-b2db-420720fe4a86-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w7npn" (UID: "b3661326-52c2-4c9b-b2db-420720fe4a86") : configmap "kube-rbac-proxy" not found Jul 10 02:48:51.679042 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:51.678955 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tfrkp" event={"ID":"ff323ccb-9efc-43fd-8ce7-9365fe9af996","Type":"ContainerStarted","Data":"1ede09e5d8de8ca120bf1627defe5c7512e9410284f9da88a97275c9b9fac85d"} Jul 10 02:48:51.680003 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:51.679984 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-7bqcj" event={"ID":"c568f52e-6afa-453d-8b32-e5cb8b4f762e","Type":"ContainerStarted","Data":"18924b9f5dabbfaa9f837c9bae50041adac2239285795e5aa49206c1563de490"} Jul 10 02:48:51.703764 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:51.703708 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-tfrkp" podStartSLOduration=7.752769933 podStartE2EDuration="48.703694409s" totalImagesPullingTime="40.950924476s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 02:48:03 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:48:51.702914815 +0000 UTC m=+49.984031102" watchObservedRunningTime="2026-07-10 02:48:51.703694409 +0000 UTC m=+49.984810671" Jul 10 02:48:51.824418 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:51.824379 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:51.886788 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:51.886757 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:51.928119 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:51.928096 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gp2gc_11f18b36-f920-47ba-8b04-10483f768ebf/node-ca/0.log" Jul 10 02:48:51.949889 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:51.949800 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:52.526079 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:52.526046 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9qrnw_6136f9e5-f2c2-4e54-94e5-f86957cc814e/serve-healthcheck-canary/0.log" Jul 10 02:48:52.631051 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:52.631028 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-lgtq8"] Jul 10 02:48:52.646073 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:52.646051 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-lgtq8" Jul 10 02:48:52.647784 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:52.647755 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-lgtq8"] Jul 10 02:48:52.648458 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:52.648406 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 10 02:48:52.648557 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:52.648506 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 10 02:48:52.648557 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:52.648541 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 10 02:48:52.649261 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:52.649244 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-kknkn\"" Jul 10 02:48:52.649368 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:52.649349 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 10 02:48:52.671801 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:52.671775 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:52.684929 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:52.684905 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-7bqcj" event={"ID":"c568f52e-6afa-453d-8b32-e5cb8b4f762e","Type":"ContainerStarted","Data":"883e6c7d516db9b2bd237963079bf11ca80084cca42416690786e211b331d96e"} Jul 10 02:48:52.718310 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:52.718281 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-294rz\" (UniqueName: \"kubernetes.io/projected/ef3e9c90-1fff-48d2-8133-7a82c30588e5-kube-api-access-294rz\") pod \"service-ca-79f68b9f4f-lgtq8\" (UID: \"ef3e9c90-1fff-48d2-8133-7a82c30588e5\") " pod="openshift-service-ca/service-ca-79f68b9f4f-lgtq8" Jul 10 02:48:52.718588 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:52.718413 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ef3e9c90-1fff-48d2-8133-7a82c30588e5-signing-key\") pod \"service-ca-79f68b9f4f-lgtq8\" (UID: \"ef3e9c90-1fff-48d2-8133-7a82c30588e5\") " pod="openshift-service-ca/service-ca-79f68b9f4f-lgtq8" Jul 10 02:48:52.718588 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:52.718505 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ef3e9c90-1fff-48d2-8133-7a82c30588e5-signing-cabundle\") pod \"service-ca-79f68b9f4f-lgtq8\" (UID: \"ef3e9c90-1fff-48d2-8133-7a82c30588e5\") " pod="openshift-service-ca/service-ca-79f68b9f4f-lgtq8" Jul 10 02:48:52.729077 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:52.729039 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:52.819680 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:52.819592 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-294rz\" (UniqueName: \"kubernetes.io/projected/ef3e9c90-1fff-48d2-8133-7a82c30588e5-kube-api-access-294rz\") pod \"service-ca-79f68b9f4f-lgtq8\" (UID: \"ef3e9c90-1fff-48d2-8133-7a82c30588e5\") " pod="openshift-service-ca/service-ca-79f68b9f4f-lgtq8" Jul 10 02:48:52.819680 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:52.819640 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ef3e9c90-1fff-48d2-8133-7a82c30588e5-signing-key\") pod \"service-ca-79f68b9f4f-lgtq8\" (UID: \"ef3e9c90-1fff-48d2-8133-7a82c30588e5\") " pod="openshift-service-ca/service-ca-79f68b9f4f-lgtq8" Jul 10 02:48:52.819907 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:52.819716 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ef3e9c90-1fff-48d2-8133-7a82c30588e5-signing-cabundle\") pod \"service-ca-79f68b9f4f-lgtq8\" (UID: \"ef3e9c90-1fff-48d2-8133-7a82c30588e5\") " pod="openshift-service-ca/service-ca-79f68b9f4f-lgtq8" Jul 10 02:48:52.820352 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:52.820327 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ef3e9c90-1fff-48d2-8133-7a82c30588e5-signing-cabundle\") pod \"service-ca-79f68b9f4f-lgtq8\" (UID: \"ef3e9c90-1fff-48d2-8133-7a82c30588e5\") " pod="openshift-service-ca/service-ca-79f68b9f4f-lgtq8" Jul 10 02:48:52.822081 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:52.822059 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ef3e9c90-1fff-48d2-8133-7a82c30588e5-signing-key\") pod \"service-ca-79f68b9f4f-lgtq8\" (UID: \"ef3e9c90-1fff-48d2-8133-7a82c30588e5\") " pod="openshift-service-ca/service-ca-79f68b9f4f-lgtq8" Jul 10 02:48:52.828518 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:52.828496 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-294rz\" (UniqueName: \"kubernetes.io/projected/ef3e9c90-1fff-48d2-8133-7a82c30588e5-kube-api-access-294rz\") pod \"service-ca-79f68b9f4f-lgtq8\" (UID: \"ef3e9c90-1fff-48d2-8133-7a82c30588e5\") " pod="openshift-service-ca/service-ca-79f68b9f4f-lgtq8" Jul 10 02:48:52.956315 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:52.956274 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-lgtq8" Jul 10 02:48:53.089165 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:53.089085 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-lgtq8"] Jul 10 02:48:53.093374 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:48:53.093349 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef3e9c90_1fff_48d2_8133_7a82c30588e5.slice/crio-57e21a53de8bfabaf18cb3809eae80295a47a0380fa7a151e37068e187b3a227 WatchSource:0}: Error finding container 57e21a53de8bfabaf18cb3809eae80295a47a0380fa7a151e37068e187b3a227: Status 404 returned error can't find the container with id 57e21a53de8bfabaf18cb3809eae80295a47a0380fa7a151e37068e187b3a227 Jul 10 02:48:53.108050 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:53.108020 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:53.424730 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:53.424631 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b3661326-52c2-4c9b-b2db-420720fe4a86-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w7npn\" (UID: \"b3661326-52c2-4c9b-b2db-420720fe4a86\") " pod="openshift-insights/insights-runtime-extractor-w7npn" Jul 10 02:48:53.424730 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:53.424699 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b3661326-52c2-4c9b-b2db-420720fe4a86-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w7npn\" (UID: \"b3661326-52c2-4c9b-b2db-420720fe4a86\") " pod="openshift-insights/insights-runtime-extractor-w7npn" Jul 10 02:48:53.424933 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:53.424793 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 02:48:53.424933 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:53.424889 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3661326-52c2-4c9b-b2db-420720fe4a86-insights-runtime-extractor-tls podName:b3661326-52c2-4c9b-b2db-420720fe4a86 nodeName:}" failed. No retries permitted until 2026-07-10 02:48:57.424873107 +0000 UTC m=+55.705989374 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b3661326-52c2-4c9b-b2db-420720fe4a86-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w7npn" (UID: "b3661326-52c2-4c9b-b2db-420720fe4a86") : secret "insights-runtime-extractor-tls" not found Jul 10 02:48:53.425315 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:53.425297 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b3661326-52c2-4c9b-b2db-420720fe4a86-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w7npn\" (UID: \"b3661326-52c2-4c9b-b2db-420720fe4a86\") " pod="openshift-insights/insights-runtime-extractor-w7npn" Jul 10 02:48:53.516851 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:53.516799 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:53.689464 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:53.689372 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-lgtq8" event={"ID":"ef3e9c90-1fff-48d2-8133-7a82c30588e5","Type":"ContainerStarted","Data":"4b2a72a29430d8cb3459cb779ca6b7ddf8bafa7e5fcf091851d3b4f33cf3b427"} Jul 10 02:48:53.689464 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:53.689428 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-lgtq8" event={"ID":"ef3e9c90-1fff-48d2-8133-7a82c30588e5","Type":"ContainerStarted","Data":"57e21a53de8bfabaf18cb3809eae80295a47a0380fa7a151e37068e187b3a227"} Jul 10 02:48:53.691246 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:53.691222 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-7bqcj" event={"ID":"c568f52e-6afa-453d-8b32-e5cb8b4f762e","Type":"ContainerStarted","Data":"08d44b03bd38518891fba10435b1f408be74e085066e9c19b812780e877c13c4"} Jul 10 02:48:53.706352 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:53.706313 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-lgtq8" podStartSLOduration=1.706298852 podStartE2EDuration="1.706298852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:48:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:48:53.705891953 +0000 UTC m=+51.987008265" watchObservedRunningTime="2026-07-10 02:48:53.706298852 +0000 UTC m=+51.987415136" Jul 10 02:48:53.722499 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:53.722462 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-7bqcj" podStartSLOduration=1.9679570210000001 podStartE2EDuration="3.722451069s" totalImagesPullingTime="1.754494048s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:48:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:48:53.721621714 +0000 UTC m=+52.002738034" watchObservedRunningTime="2026-07-10 02:48:53.722451069 +0000 UTC m=+52.003567355" Jul 10 02:48:54.031329 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:54.031292 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3adc0ae6-f77b-4909-8bef-8534f19e6601-service-ca-bundle\") pod \"router-default-bfcf8947b-7fz4l\" (UID: \"3adc0ae6-f77b-4909-8bef-8534f19e6601\") " pod="openshift-ingress/router-default-bfcf8947b-7fz4l" Jul 10 02:48:54.031481 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:54.031336 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3adc0ae6-f77b-4909-8bef-8534f19e6601-metrics-certs\") pod \"router-default-bfcf8947b-7fz4l\" (UID: \"3adc0ae6-f77b-4909-8bef-8534f19e6601\") " pod="openshift-ingress/router-default-bfcf8947b-7fz4l" Jul 10 02:48:54.031481 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:54.031443 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/167ea7f3-1d8f-4892-bc0e-e9247912fef6-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-q44gx\" (UID: \"167ea7f3-1d8f-4892-bc0e-e9247912fef6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-q44gx" Jul 10 02:48:54.031594 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:54.031489 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3adc0ae6-f77b-4909-8bef-8534f19e6601-service-ca-bundle podName:3adc0ae6-f77b-4909-8bef-8534f19e6601 nodeName:}" failed. No retries permitted until 2026-07-10 02:49:10.031462544 +0000 UTC m=+68.312578815 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3adc0ae6-f77b-4909-8bef-8534f19e6601-service-ca-bundle") pod "router-default-bfcf8947b-7fz4l" (UID: "3adc0ae6-f77b-4909-8bef-8534f19e6601") : configmap references non-existent config key: service-ca.crt Jul 10 02:48:54.031594 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:54.031515 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 02:48:54.031594 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:54.031539 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5176df10-dfd5-4514-9a59-c854c4a0ec21-metrics-tls\") pod \"dns-default-cc72k\" (UID: \"5176df10-dfd5-4514-9a59-c854c4a0ec21\") " pod="openshift-dns/dns-default-cc72k" Jul 10 02:48:54.031594 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:54.031564 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/167ea7f3-1d8f-4892-bc0e-e9247912fef6-samples-operator-tls podName:167ea7f3-1d8f-4892-bc0e-e9247912fef6 nodeName:}" failed. No retries permitted until 2026-07-10 02:49:10.031553433 +0000 UTC m=+68.312669695 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/167ea7f3-1d8f-4892-bc0e-e9247912fef6-samples-operator-tls") pod "cluster-samples-operator-65db858894-q44gx" (UID: "167ea7f3-1d8f-4892-bc0e-e9247912fef6") : secret "samples-operator-tls" not found Jul 10 02:48:54.031594 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:54.031515 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 10 02:48:54.031859 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:54.031607 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 02:48:54.031859 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:54.031610 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bcda0e99-fdd8-498f-8e73-ecf4c279be71-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-2rh2j\" (UID: \"bcda0e99-fdd8-498f-8e73-ecf4c279be71\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-2rh2j" Jul 10 02:48:54.031859 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:54.031633 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3adc0ae6-f77b-4909-8bef-8534f19e6601-metrics-certs podName:3adc0ae6-f77b-4909-8bef-8534f19e6601 nodeName:}" failed. No retries permitted until 2026-07-10 02:49:10.031619388 +0000 UTC m=+68.312735666 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3adc0ae6-f77b-4909-8bef-8534f19e6601-metrics-certs") pod "router-default-bfcf8947b-7fz4l" (UID: "3adc0ae6-f77b-4909-8bef-8534f19e6601") : secret "router-metrics-certs-default" not found Jul 10 02:48:54.031859 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:54.031657 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 10 02:48:54.031859 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:54.031664 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5176df10-dfd5-4514-9a59-c854c4a0ec21-metrics-tls podName:5176df10-dfd5-4514-9a59-c854c4a0ec21 nodeName:}" failed. No retries permitted until 2026-07-10 02:49:10.031647119 +0000 UTC m=+68.312763395 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5176df10-dfd5-4514-9a59-c854c4a0ec21-metrics-tls") pod "dns-default-cc72k" (UID: "5176df10-dfd5-4514-9a59-c854c4a0ec21") : secret "dns-default-metrics-tls" not found Jul 10 02:48:54.031859 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:54.031700 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcda0e99-fdd8-498f-8e73-ecf4c279be71-cluster-monitoring-operator-tls podName:bcda0e99-fdd8-498f-8e73-ecf4c279be71 nodeName:}" failed. No retries permitted until 2026-07-10 02:49:10.031690163 +0000 UTC m=+68.312806429 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bcda0e99-fdd8-498f-8e73-ecf4c279be71-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-2rh2j" (UID: "bcda0e99-fdd8-498f-8e73-ecf4c279be71") : secret "cluster-monitoring-operator-tls" not found Jul 10 02:48:54.031859 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:54.031733 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f540bce4-0371-4500-8e77-3bb4291a2e3d-registry-tls\") pod \"image-registry-fbd7648c9-vtvdf\" (UID: \"f540bce4-0371-4500-8e77-3bb4291a2e3d\") " pod="openshift-image-registry/image-registry-fbd7648c9-vtvdf" Jul 10 02:48:54.031859 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:54.031800 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 02:48:54.031859 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:54.031832 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fbd7648c9-vtvdf: secret "image-registry-tls" not found Jul 10 02:48:54.032202 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:54.031870 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f540bce4-0371-4500-8e77-3bb4291a2e3d-registry-tls podName:f540bce4-0371-4500-8e77-3bb4291a2e3d nodeName:}" failed. No retries permitted until 2026-07-10 02:49:10.031859428 +0000 UTC m=+68.312975723 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f540bce4-0371-4500-8e77-3bb4291a2e3d-registry-tls") pod "image-registry-fbd7648c9-vtvdf" (UID: "f540bce4-0371-4500-8e77-3bb4291a2e3d") : secret "image-registry-tls" not found Jul 10 02:48:54.802427 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:54.802389 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:54.941862 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:54.941803 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:55.097203 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:55.097111 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:55.197350 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:55.197321 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:55.297114 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:55.297077 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:55.891111 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:55.891080 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:56.124577 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:56.124534 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:56.492987 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:56.492953 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:56.693696 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:56.693659 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:56.890235 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:56.890202 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:57.284082 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:57.284051 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:57.461382 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:57.461345 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b3661326-52c2-4c9b-b2db-420720fe4a86-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w7npn\" (UID: \"b3661326-52c2-4c9b-b2db-420720fe4a86\") " pod="openshift-insights/insights-runtime-extractor-w7npn" Jul 10 02:48:57.461538 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:57.461504 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 02:48:57.461611 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:57.461569 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3661326-52c2-4c9b-b2db-420720fe4a86-insights-runtime-extractor-tls podName:b3661326-52c2-4c9b-b2db-420720fe4a86 nodeName:}" failed. No retries permitted until 2026-07-10 02:49:05.461553669 +0000 UTC m=+63.742669931 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b3661326-52c2-4c9b-b2db-420720fe4a86-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w7npn" (UID: "b3661326-52c2-4c9b-b2db-420720fe4a86") : secret "insights-runtime-extractor-tls" not found Jul 10 02:48:57.886799 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:57.886764 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:58.282168 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:58.282115 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:58.482761 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:58.482726 2575 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-xxlp6" Jul 10 02:48:58.483240 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:58.482786 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-xxlp6" Jul 10 02:48:58.483240 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:58.483162 2575 scope.go:117] "RemoveContainer" containerID="c2018deea9d6279c181cf3ccbdd4414834ea69449237a7dc619c75a8159aa2a1" Jul 10 02:48:58.483337 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:48:58.483319 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-xxlp6_openshift-console-operator(372be3ce-a380-4c0e-bbee-49c2e3b80095)\"" pod="openshift-console-operator/console-operator-7866497494-xxlp6" podUID="372be3ce-a380-4c0e-bbee-49c2e3b80095" Jul 10 02:48:58.484340 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:58.484312 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:58.619190 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:58.619099 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:58.706724 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:58.706687 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:59.011146 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:59.011110 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:59.112109 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:59.112073 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:59.484980 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:59.484904 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:59.536455 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:59.536420 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:59.689939 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:59.689903 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:48:59.889013 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:48:59.888977 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:49:00.289761 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:00.289724 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:49:00.390356 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:00.390316 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:49:00.443918 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:00.443884 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:49:00.503426 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:00.503393 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:49:01.006252 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:01.006219 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:49:01.110159 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:01.110129 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:49:01.487285 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:01.487200 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:49:01.888675 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:01.888639 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:49:02.090410 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:02.090375 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:49:02.687558 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:02.687518 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:49:03.088085 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:03.088052 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:49:03.487627 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:03.487544 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:49:03.693202 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:03.693165 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:49:04.017981 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:04.017942 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:49:04.118563 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:04.118526 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:49:04.288097 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:04.288011 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:49:04.512768 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:04.512744 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-25tvc" Jul 10 02:49:04.687622 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:04.687532 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:49:05.081699 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:05.081666 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:49:05.283927 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:05.283895 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:49:05.488090 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:05.488000 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" limit=1024 Jul 10 02:49:05.534271 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:05.534238 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b3661326-52c2-4c9b-b2db-420720fe4a86-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w7npn\" (UID: \"b3661326-52c2-4c9b-b2db-420720fe4a86\") " pod="openshift-insights/insights-runtime-extractor-w7npn" Jul 10 02:49:05.536575 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:05.536547 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b3661326-52c2-4c9b-b2db-420720fe4a86-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w7npn\" (UID: \"b3661326-52c2-4c9b-b2db-420720fe4a86\") " pod="openshift-insights/insights-runtime-extractor-w7npn" Jul 10 02:49:05.653146 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:05.653115 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-f756g\"" Jul 10 02:49:05.661945 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:05.661926 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-w7npn" Jul 10 02:49:05.787747 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:05.787719 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-w7npn"] Jul 10 02:49:05.790636 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:49:05.790611 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3661326_52c2_4c9b_b2db_420720fe4a86.slice/crio-920b595a56751b567ca7d9a67aa3fe6456d300280140b99245faa169b6bb90be WatchSource:0}: Error finding container 920b595a56751b567ca7d9a67aa3fe6456d300280140b99245faa169b6bb90be: Status 404 returned error can't find the container with id 920b595a56751b567ca7d9a67aa3fe6456d300280140b99245faa169b6bb90be Jul 10 02:49:06.739851 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:06.739785 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w7npn" event={"ID":"b3661326-52c2-4c9b-b2db-420720fe4a86","Type":"ContainerStarted","Data":"920b595a56751b567ca7d9a67aa3fe6456d300280140b99245faa169b6bb90be"} Jul 10 02:49:07.051218 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:07.051178 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc1f4b26-669e-42a0-abd4-f858ac6b1543-original-pull-secret\") pod \"global-pull-secret-syncer-q28nr\" (UID: \"dc1f4b26-669e-42a0-abd4-f858ac6b1543\") " pod="kube-system/global-pull-secret-syncer-q28nr" Jul 10 02:49:07.051408 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:07.051287 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d037586-6a56-4354-ba36-a40c2cc2ab48-metrics-certs\") pod \"network-metrics-daemon-bgbpr\" (UID: \"7d037586-6a56-4354-ba36-a40c2cc2ab48\") " pod="openshift-multus/network-metrics-daemon-bgbpr" Jul 10 02:49:07.054144 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:07.054089 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 02:49:07.054690 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:07.054671 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 02:49:07.064775 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:07.064754 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc1f4b26-669e-42a0-abd4-f858ac6b1543-original-pull-secret\") pod \"global-pull-secret-syncer-q28nr\" (UID: \"dc1f4b26-669e-42a0-abd4-f858ac6b1543\") " pod="kube-system/global-pull-secret-syncer-q28nr" Jul 10 02:49:07.065081 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:07.065062 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d037586-6a56-4354-ba36-a40c2cc2ab48-metrics-certs\") pod \"network-metrics-daemon-bgbpr\" (UID: \"7d037586-6a56-4354-ba36-a40c2cc2ab48\") " pod="openshift-multus/network-metrics-daemon-bgbpr" Jul 10 02:49:07.152519 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:07.152482 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzkpc\" (UniqueName: \"kubernetes.io/projected/25d9393a-d380-4ef0-a976-9fd998deb429-kube-api-access-jzkpc\") pod \"network-check-target-nf7q8\" (UID: \"25d9393a-d380-4ef0-a976-9fd998deb429\") " pod="openshift-network-diagnostics/network-check-target-nf7q8" Jul 10 02:49:07.155409 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:07.155378 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzkpc\" (UniqueName: \"kubernetes.io/projected/25d9393a-d380-4ef0-a976-9fd998deb429-kube-api-access-jzkpc\") pod \"network-check-target-nf7q8\" (UID: \"25d9393a-d380-4ef0-a976-9fd998deb429\") " pod="openshift-network-diagnostics/network-check-target-nf7q8" Jul 10 02:49:07.210660 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:07.210629 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r5msv\"" Jul 10 02:49:07.218668 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:07.218640 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bgbpr" Jul 10 02:49:07.220912 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:07.220890 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q28nr" Jul 10 02:49:07.232034 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:07.231965 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-p9j2r\"" Jul 10 02:49:07.240090 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:07.239872 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nf7q8" Jul 10 02:49:07.371279 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:07.371202 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-q28nr"] Jul 10 02:49:07.388550 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:07.388511 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bgbpr"] Jul 10 02:49:07.409248 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:07.409219 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nf7q8"] Jul 10 02:49:08.588033 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:49:08.587991 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc1f4b26_669e_42a0_abd4_f858ac6b1543.slice/crio-5dfadffa260461cf5e2227d757f01940c54199725a4e527a9739a12fc7e81733 WatchSource:0}: Error finding container 5dfadffa260461cf5e2227d757f01940c54199725a4e527a9739a12fc7e81733: Status 404 returned error can't find the container with id 5dfadffa260461cf5e2227d757f01940c54199725a4e527a9739a12fc7e81733 Jul 10 02:49:08.588708 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:49:08.588465 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25d9393a_d380_4ef0_a976_9fd998deb429.slice/crio-645b9e78abe4f0529ca6e11e17c7a6974f7269099e81f28a29ff1c91b0fd15bc WatchSource:0}: Error finding container 645b9e78abe4f0529ca6e11e17c7a6974f7269099e81f28a29ff1c91b0fd15bc: Status 404 returned error can't find the container with id 645b9e78abe4f0529ca6e11e17c7a6974f7269099e81f28a29ff1c91b0fd15bc Jul 10 02:49:08.629441 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:49:08.629419 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d037586_6a56_4354_ba36_a40c2cc2ab48.slice/crio-8a13f0da3d249b7d04790048ad6ac550ec14c5b8bd6d962244ad4bc0cb125a3e WatchSource:0}: Error finding container 8a13f0da3d249b7d04790048ad6ac550ec14c5b8bd6d962244ad4bc0cb125a3e: Status 404 returned error can't find the container with id 8a13f0da3d249b7d04790048ad6ac550ec14c5b8bd6d962244ad4bc0cb125a3e Jul 10 02:49:08.748108 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:08.748072 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-q28nr" event={"ID":"dc1f4b26-669e-42a0-abd4-f858ac6b1543","Type":"ContainerStarted","Data":"5dfadffa260461cf5e2227d757f01940c54199725a4e527a9739a12fc7e81733"} Jul 10 02:49:08.749219 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:08.749184 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nf7q8" event={"ID":"25d9393a-d380-4ef0-a976-9fd998deb429","Type":"ContainerStarted","Data":"645b9e78abe4f0529ca6e11e17c7a6974f7269099e81f28a29ff1c91b0fd15bc"} Jul 10 02:49:08.750748 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:08.750717 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bgbpr" event={"ID":"7d037586-6a56-4354-ba36-a40c2cc2ab48","Type":"ContainerStarted","Data":"8a13f0da3d249b7d04790048ad6ac550ec14c5b8bd6d962244ad4bc0cb125a3e"} Jul 10 02:49:09.757069 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:09.756970 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w7npn" event={"ID":"b3661326-52c2-4c9b-b2db-420720fe4a86","Type":"ContainerStarted","Data":"2f5a3c01da4166fdca9249524c202eca935be2b31ff5ad4680ba865458fb5a89"} Jul 10 02:49:09.759463 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:09.759432 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nf7q8" event={"ID":"25d9393a-d380-4ef0-a976-9fd998deb429","Type":"ContainerStarted","Data":"3cf975b14d91dd3a0fbaced2e35d7bc90330fdf0a035aff6b547518c91c8081c"} Jul 10 02:49:09.759699 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:09.759631 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-nf7q8" Jul 10 02:49:10.078912 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:10.078872 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5176df10-dfd5-4514-9a59-c854c4a0ec21-metrics-tls\") pod \"dns-default-cc72k\" (UID: \"5176df10-dfd5-4514-9a59-c854c4a0ec21\") " pod="openshift-dns/dns-default-cc72k" Jul 10 02:49:10.079109 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:10.078946 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bcda0e99-fdd8-498f-8e73-ecf4c279be71-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-2rh2j\" (UID: \"bcda0e99-fdd8-498f-8e73-ecf4c279be71\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-2rh2j" Jul 10 02:49:10.079109 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:10.079000 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f540bce4-0371-4500-8e77-3bb4291a2e3d-registry-tls\") pod \"image-registry-fbd7648c9-vtvdf\" (UID: \"f540bce4-0371-4500-8e77-3bb4291a2e3d\") " pod="openshift-image-registry/image-registry-fbd7648c9-vtvdf" Jul 10 02:49:10.079109 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:10.079037 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3adc0ae6-f77b-4909-8bef-8534f19e6601-service-ca-bundle\") pod \"router-default-bfcf8947b-7fz4l\" (UID: \"3adc0ae6-f77b-4909-8bef-8534f19e6601\") " pod="openshift-ingress/router-default-bfcf8947b-7fz4l" Jul 10 02:49:10.079109 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:10.079064 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3adc0ae6-f77b-4909-8bef-8534f19e6601-metrics-certs\") pod \"router-default-bfcf8947b-7fz4l\" (UID: \"3adc0ae6-f77b-4909-8bef-8534f19e6601\") " pod="openshift-ingress/router-default-bfcf8947b-7fz4l" Jul 10 02:49:10.079109 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:10.079107 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/167ea7f3-1d8f-4892-bc0e-e9247912fef6-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-q44gx\" (UID: \"167ea7f3-1d8f-4892-bc0e-e9247912fef6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-q44gx" Jul 10 02:49:10.080408 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:10.080375 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3adc0ae6-f77b-4909-8bef-8534f19e6601-service-ca-bundle\") pod \"router-default-bfcf8947b-7fz4l\" (UID: \"3adc0ae6-f77b-4909-8bef-8534f19e6601\") " pod="openshift-ingress/router-default-bfcf8947b-7fz4l" Jul 10 02:49:10.082499 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:10.082422 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5176df10-dfd5-4514-9a59-c854c4a0ec21-metrics-tls\") pod \"dns-default-cc72k\" (UID: \"5176df10-dfd5-4514-9a59-c854c4a0ec21\") " pod="openshift-dns/dns-default-cc72k" Jul 10 02:49:10.082499 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:10.082425 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/167ea7f3-1d8f-4892-bc0e-e9247912fef6-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-q44gx\" (UID: \"167ea7f3-1d8f-4892-bc0e-e9247912fef6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-q44gx" Jul 10 02:49:10.083192 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:10.083126 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bcda0e99-fdd8-498f-8e73-ecf4c279be71-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-2rh2j\" (UID: \"bcda0e99-fdd8-498f-8e73-ecf4c279be71\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-2rh2j" Jul 10 02:49:10.083287 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:10.083207 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f540bce4-0371-4500-8e77-3bb4291a2e3d-registry-tls\") pod \"image-registry-fbd7648c9-vtvdf\" (UID: \"f540bce4-0371-4500-8e77-3bb4291a2e3d\") " pod="openshift-image-registry/image-registry-fbd7648c9-vtvdf" Jul 10 02:49:10.084012 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:10.083969 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3adc0ae6-f77b-4909-8bef-8534f19e6601-metrics-certs\") pod \"router-default-bfcf8947b-7fz4l\" (UID: \"3adc0ae6-f77b-4909-8bef-8534f19e6601\") " pod="openshift-ingress/router-default-bfcf8947b-7fz4l" Jul 10 02:49:10.217915 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:10.217884 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-27kdw\"" Jul 10 02:49:10.225721 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:10.225681 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-bfcf8947b-7fz4l" Jul 10 02:49:10.251295 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:10.251264 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-c87sb\"" Jul 10 02:49:10.259242 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:10.259216 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fbd7648c9-vtvdf" Jul 10 02:49:10.260181 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:10.260161 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-mkh7g\"" Jul 10 02:49:10.267560 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:10.267537 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-l9gjz\"" Jul 10 02:49:10.268649 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:10.268566 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-q44gx" Jul 10 02:49:10.275822 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:10.275783 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cc72k" Jul 10 02:49:10.283221 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:10.283194 2575 scope.go:117] "RemoveContainer" containerID="c2018deea9d6279c181cf3ccbdd4414834ea69449237a7dc619c75a8159aa2a1" Jul 10 02:49:10.291760 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:10.291733 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-k9srp\"" Jul 10 02:49:10.299737 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:10.299698 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-2rh2j" Jul 10 02:49:10.659439 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:10.659314 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-nf7q8" podStartSLOduration=67.659290555 podStartE2EDuration="1m7.659290555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:48:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:49:09.773944063 +0000 UTC m=+68.055060349" watchObservedRunningTime="2026-07-10 02:49:10.659290555 +0000 UTC m=+68.940406841" Jul 10 02:49:10.660787 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:10.660686 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-2rh2j"] Jul 10 02:49:10.663498 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:10.663433 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-fbd7648c9-vtvdf"] Jul 10 02:49:10.668184 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:49:10.668149 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcda0e99_fdd8_498f_8e73_ecf4c279be71.slice/crio-eb30ec78bf27f09f8cdf5d2d72effe5bc79ee2f3681e64222f65f98861c71d22 WatchSource:0}: Error finding container eb30ec78bf27f09f8cdf5d2d72effe5bc79ee2f3681e64222f65f98861c71d22: Status 404 returned error can't find the container with id eb30ec78bf27f09f8cdf5d2d72effe5bc79ee2f3681e64222f65f98861c71d22 Jul 10 02:49:10.670957 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:49:10.670441 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf540bce4_0371_4500_8e77_3bb4291a2e3d.slice/crio-6847b98461069af6b98c3b0610c1a328ae7e7195fabd1955d1770d18715f4668 WatchSource:0}: Error finding container 6847b98461069af6b98c3b0610c1a328ae7e7195fabd1955d1770d18715f4668: Status 404 returned error can't find the container with id 6847b98461069af6b98c3b0610c1a328ae7e7195fabd1955d1770d18715f4668 Jul 10 02:49:10.693633 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:10.693608 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-q44gx"] Jul 10 02:49:10.768029 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:10.767995 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w7npn" event={"ID":"b3661326-52c2-4c9b-b2db-420720fe4a86","Type":"ContainerStarted","Data":"846204814f49435b32d602e9f48600b55da43747c5c8bd5d105429190effe5a5"} Jul 10 02:49:10.770667 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:10.770623 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-xxlp6_372be3ce-a380-4c0e-bbee-49c2e3b80095/console-operator/1.log" Jul 10 02:49:10.770778 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:10.770721 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-xxlp6" event={"ID":"372be3ce-a380-4c0e-bbee-49c2e3b80095","Type":"ContainerStarted","Data":"08dc4b94aa0c84079c358e4c88526b7d5089a4e9f994432c7c48192d01119f5b"} Jul 10 02:49:10.771359 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:10.771045 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-xxlp6" Jul 10 02:49:10.774078 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:10.773853 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fbd7648c9-vtvdf" event={"ID":"f540bce4-0371-4500-8e77-3bb4291a2e3d","Type":"ContainerStarted","Data":"6847b98461069af6b98c3b0610c1a328ae7e7195fabd1955d1770d18715f4668"} Jul 10 02:49:10.775991 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:10.775966 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-2rh2j" event={"ID":"bcda0e99-fdd8-498f-8e73-ecf4c279be71","Type":"ContainerStarted","Data":"eb30ec78bf27f09f8cdf5d2d72effe5bc79ee2f3681e64222f65f98861c71d22"} Jul 10 02:49:10.778872 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:10.778461 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bgbpr" event={"ID":"7d037586-6a56-4354-ba36-a40c2cc2ab48","Type":"ContainerStarted","Data":"47a0a9da3788f8fd5038c5bc8cf743ace48899e8c19116aa3ac434a91b225ae2"} Jul 10 02:49:10.788301 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:10.787929 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7866497494-xxlp6" podStartSLOduration=33.137995045 podStartE2EDuration="42.787909915s" totalImagesPullingTime="9.64991487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:48:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:49:10.78728246 +0000 UTC m=+69.068398747" watchObservedRunningTime="2026-07-10 02:49:10.787909915 +0000 UTC m=+69.069026201" Jul 10 02:49:10.803707 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:10.803545 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-bgbpr" podStartSLOduration=66.056019028 podStartE2EDuration="1m7.803522476s" totalImagesPullingTime="1.747503448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:48:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:49:10.80272021 +0000 UTC m=+69.083836494" watchObservedRunningTime="2026-07-10 02:49:10.803522476 +0000 UTC m=+69.084638777" Jul 10 02:49:10.856941 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:10.856886 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cc72k"] Jul 10 02:49:10.860500 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:10.860466 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-bfcf8947b-7fz4l"] Jul 10 02:49:10.861562 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:49:10.861518 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5176df10_dfd5_4514_9a59_c854c4a0ec21.slice/crio-11959d5a0f7a92067acafff318ae8827ded7223375eb0cf4743f6d3b1b6f364e WatchSource:0}: Error finding container 11959d5a0f7a92067acafff318ae8827ded7223375eb0cf4743f6d3b1b6f364e: Status 404 returned error can't find the container with id 11959d5a0f7a92067acafff318ae8827ded7223375eb0cf4743f6d3b1b6f364e Jul 10 02:49:10.864182 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:49:10.864035 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3adc0ae6_f77b_4909_8bef_8534f19e6601.slice/crio-c4f9cb9ef24bd1a0ee166128bac0f5880d7a7046c5073d126e25ac7c8e24049e WatchSource:0}: Error finding container c4f9cb9ef24bd1a0ee166128bac0f5880d7a7046c5073d126e25ac7c8e24049e: Status 404 returned error can't find the container with id c4f9cb9ef24bd1a0ee166128bac0f5880d7a7046c5073d126e25ac7c8e24049e Jul 10 02:49:11.713776 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:11.713550 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7866497494-xxlp6" Jul 10 02:49:11.787335 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:11.787294 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fbd7648c9-vtvdf" event={"ID":"f540bce4-0371-4500-8e77-3bb4291a2e3d","Type":"ContainerStarted","Data":"73ff6532d45dd0f30864bb446dbae5fbc2e4352751afd7432965742f5269b75b"} Jul 10 02:49:11.787789 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:11.787543 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-fbd7648c9-vtvdf" Jul 10 02:49:11.797386 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:11.797355 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bgbpr" event={"ID":"7d037586-6a56-4354-ba36-a40c2cc2ab48","Type":"ContainerStarted","Data":"9e03ac2deeae783787f55b64f8683dd830cfcb8bc775c3b26d48e183d5921a44"} Jul 10 02:49:11.802064 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:11.801947 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cc72k" event={"ID":"5176df10-dfd5-4514-9a59-c854c4a0ec21","Type":"ContainerStarted","Data":"11959d5a0f7a92067acafff318ae8827ded7223375eb0cf4743f6d3b1b6f364e"} Jul 10 02:49:11.807593 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:11.807521 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-bfcf8947b-7fz4l" event={"ID":"3adc0ae6-f77b-4909-8bef-8534f19e6601","Type":"ContainerStarted","Data":"75e79067796e79bdc8543da6684815b69c0d564cfa302fa27436eebcbe4fa57d"} Jul 10 02:49:11.807593 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:11.807563 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-bfcf8947b-7fz4l" event={"ID":"3adc0ae6-f77b-4909-8bef-8534f19e6601","Type":"ContainerStarted","Data":"c4f9cb9ef24bd1a0ee166128bac0f5880d7a7046c5073d126e25ac7c8e24049e"} Jul 10 02:49:11.809999 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:11.808485 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-fbd7648c9-vtvdf" podStartSLOduration=46.808467746 podStartE2EDuration="46.808467746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:49:11.804857659 +0000 UTC m=+70.085973964" watchObservedRunningTime="2026-07-10 02:49:11.808467746 +0000 UTC m=+70.089584032" Jul 10 02:49:11.812691 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:11.812638 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-q44gx" event={"ID":"167ea7f3-1d8f-4892-bc0e-e9247912fef6","Type":"ContainerStarted","Data":"617b6eef0ceed0144fb4f6b2e4b9a71b3ca5a6d2638d1732612c3648dcd0b259"} Jul 10 02:49:11.826174 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:11.826121 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-bfcf8947b-7fz4l" podStartSLOduration=44.826100808 podStartE2EDuration="44.826100808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:48:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:49:11.824380964 +0000 UTC m=+70.105497262" watchObservedRunningTime="2026-07-10 02:49:11.826100808 +0000 UTC m=+70.107217093" Jul 10 02:49:12.227270 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:12.227192 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-bfcf8947b-7fz4l" Jul 10 02:49:12.230191 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:12.229990 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-bfcf8947b-7fz4l" Jul 10 02:49:12.817695 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:12.817439 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-bfcf8947b-7fz4l" Jul 10 02:49:12.819067 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:12.819017 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-bfcf8947b-7fz4l" Jul 10 02:49:14.196228 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.196180 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f959b7f6b-8jxpd"] Jul 10 02:49:14.200133 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.200109 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7bdf5774c4-6c9wp"] Jul 10 02:49:14.200319 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.200259 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f959b7f6b-8jxpd" Jul 10 02:49:14.204624 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.203933 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 10 02:49:14.204624 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.204154 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 10 02:49:14.204624 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.204174 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 10 02:49:14.204624 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.204396 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b66749db6-zpm4b"] Jul 10 02:49:14.204624 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.204438 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 10 02:49:14.207042 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.205044 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7bdf5774c4-6c9wp" Jul 10 02:49:14.207042 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.205083 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 10 02:49:14.207042 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.205311 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 10 02:49:14.207042 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.205540 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 10 02:49:14.209783 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.209585 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 10 02:49:14.210260 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.210239 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-xcjth\"" Jul 10 02:49:14.210678 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.210659 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b66749db6-zpm4b" Jul 10 02:49:14.211371 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.211337 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-7v4dt"] Jul 10 02:49:14.213046 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.212919 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 10 02:49:14.214772 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.214701 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-7v4dt" Jul 10 02:49:14.216988 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.216957 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 10 02:49:14.216988 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.216959 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-xtjsv\"" Jul 10 02:49:14.217134 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.217055 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 10 02:49:14.223626 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.223586 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7bdf5774c4-6c9wp"] Jul 10 02:49:14.225069 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.224959 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f959b7f6b-8jxpd"] Jul 10 02:49:14.226313 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.226293 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b66749db6-zpm4b"] Jul 10 02:49:14.248041 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.248015 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-7v4dt"] Jul 10 02:49:14.281155 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.281130 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-nnk5x"] Jul 10 02:49:14.284941 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.284919 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-nnk5x" Jul 10 02:49:14.288142 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.288117 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 10 02:49:14.288423 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.288357 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 10 02:49:14.288533 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.288463 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-f56h6\"" Jul 10 02:49:14.294750 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.294718 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-nnk5x"] Jul 10 02:49:14.325376 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.325344 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c6492dae-a18a-45ed-95be-93bc92198ea9-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-7v4dt\" (UID: \"c6492dae-a18a-45ed-95be-93bc92198ea9\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-7v4dt" Jul 10 02:49:14.325538 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.325390 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3d71fd15-80c3-42ef-b772-f5af5436012d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7bdf5774c4-6c9wp\" (UID: \"3d71fd15-80c3-42ef-b772-f5af5436012d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7bdf5774c4-6c9wp" Jul 10 02:49:14.325538 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.325468 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/32d3e7bf-5f00-4488-9e3f-2687d0453377-additional-service-ca\") pod \"cluster-proxy-proxy-agent-f959b7f6b-8jxpd\" (UID: \"32d3e7bf-5f00-4488-9e3f-2687d0453377\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f959b7f6b-8jxpd" Jul 10 02:49:14.325538 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.325504 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prxjn\" (UniqueName: \"kubernetes.io/projected/c544c7ea-9123-4b52-9cd6-9865d8e552a0-kube-api-access-prxjn\") pod \"klusterlet-addon-workmgr-5b66749db6-zpm4b\" (UID: \"c544c7ea-9123-4b52-9cd6-9865d8e552a0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b66749db6-zpm4b" Jul 10 02:49:14.325538 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.325534 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c544c7ea-9123-4b52-9cd6-9865d8e552a0-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b66749db6-zpm4b\" (UID: \"c544c7ea-9123-4b52-9cd6-9865d8e552a0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b66749db6-zpm4b" Jul 10 02:49:14.325702 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.325565 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c6492dae-a18a-45ed-95be-93bc92198ea9-nginx-conf\") pod \"networking-console-plugin-6cf856758c-7v4dt\" (UID: \"c6492dae-a18a-45ed-95be-93bc92198ea9\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-7v4dt" Jul 10 02:49:14.325702 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.325665 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/32d3e7bf-5f00-4488-9e3f-2687d0453377-hub\") pod \"cluster-proxy-proxy-agent-f959b7f6b-8jxpd\" (UID: \"32d3e7bf-5f00-4488-9e3f-2687d0453377\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f959b7f6b-8jxpd" Jul 10 02:49:14.325798 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.325698 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/32d3e7bf-5f00-4488-9e3f-2687d0453377-ca\") pod \"cluster-proxy-proxy-agent-f959b7f6b-8jxpd\" (UID: \"32d3e7bf-5f00-4488-9e3f-2687d0453377\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f959b7f6b-8jxpd" Jul 10 02:49:14.325798 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.325729 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c544c7ea-9123-4b52-9cd6-9865d8e552a0-tmp\") pod \"klusterlet-addon-workmgr-5b66749db6-zpm4b\" (UID: \"c544c7ea-9123-4b52-9cd6-9865d8e552a0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b66749db6-zpm4b" Jul 10 02:49:14.325798 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.325776 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/32d3e7bf-5f00-4488-9e3f-2687d0453377-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-f959b7f6b-8jxpd\" (UID: \"32d3e7bf-5f00-4488-9e3f-2687d0453377\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f959b7f6b-8jxpd" Jul 10 02:49:14.325960 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.325801 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzjtd\" (UniqueName: \"kubernetes.io/projected/32d3e7bf-5f00-4488-9e3f-2687d0453377-kube-api-access-wzjtd\") pod \"cluster-proxy-proxy-agent-f959b7f6b-8jxpd\" (UID: \"32d3e7bf-5f00-4488-9e3f-2687d0453377\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f959b7f6b-8jxpd" Jul 10 02:49:14.325960 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.325881 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hx6j\" (UniqueName: \"kubernetes.io/projected/3d71fd15-80c3-42ef-b772-f5af5436012d-kube-api-access-5hx6j\") pod \"managed-serviceaccount-addon-agent-7bdf5774c4-6c9wp\" (UID: \"3d71fd15-80c3-42ef-b772-f5af5436012d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7bdf5774c4-6c9wp" Jul 10 02:49:14.326028 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.325971 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/32d3e7bf-5f00-4488-9e3f-2687d0453377-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-f959b7f6b-8jxpd\" (UID: \"32d3e7bf-5f00-4488-9e3f-2687d0453377\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f959b7f6b-8jxpd" Jul 10 02:49:14.426751 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.426660 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/32d3e7bf-5f00-4488-9e3f-2687d0453377-additional-service-ca\") pod \"cluster-proxy-proxy-agent-f959b7f6b-8jxpd\" (UID: \"32d3e7bf-5f00-4488-9e3f-2687d0453377\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f959b7f6b-8jxpd" Jul 10 02:49:14.426751 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.426710 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prxjn\" (UniqueName: \"kubernetes.io/projected/c544c7ea-9123-4b52-9cd6-9865d8e552a0-kube-api-access-prxjn\") pod \"klusterlet-addon-workmgr-5b66749db6-zpm4b\" (UID: \"c544c7ea-9123-4b52-9cd6-9865d8e552a0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b66749db6-zpm4b" Jul 10 02:49:14.426751 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.426737 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c544c7ea-9123-4b52-9cd6-9865d8e552a0-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b66749db6-zpm4b\" (UID: \"c544c7ea-9123-4b52-9cd6-9865d8e552a0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b66749db6-zpm4b" Jul 10 02:49:14.427052 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.426766 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c6492dae-a18a-45ed-95be-93bc92198ea9-nginx-conf\") pod \"networking-console-plugin-6cf856758c-7v4dt\" (UID: \"c6492dae-a18a-45ed-95be-93bc92198ea9\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-7v4dt" Jul 10 02:49:14.427052 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.426803 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/32d3e7bf-5f00-4488-9e3f-2687d0453377-hub\") pod \"cluster-proxy-proxy-agent-f959b7f6b-8jxpd\" (UID: \"32d3e7bf-5f00-4488-9e3f-2687d0453377\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f959b7f6b-8jxpd" Jul 10 02:49:14.427166 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.427109 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/32d3e7bf-5f00-4488-9e3f-2687d0453377-ca\") pod \"cluster-proxy-proxy-agent-f959b7f6b-8jxpd\" (UID: \"32d3e7bf-5f00-4488-9e3f-2687d0453377\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f959b7f6b-8jxpd" Jul 10 02:49:14.427166 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.427157 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c544c7ea-9123-4b52-9cd6-9865d8e552a0-tmp\") pod \"klusterlet-addon-workmgr-5b66749db6-zpm4b\" (UID: \"c544c7ea-9123-4b52-9cd6-9865d8e552a0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b66749db6-zpm4b" Jul 10 02:49:14.427254 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.427195 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/32d3e7bf-5f00-4488-9e3f-2687d0453377-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-f959b7f6b-8jxpd\" (UID: \"32d3e7bf-5f00-4488-9e3f-2687d0453377\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f959b7f6b-8jxpd" Jul 10 02:49:14.427254 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.427220 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzjtd\" (UniqueName: \"kubernetes.io/projected/32d3e7bf-5f00-4488-9e3f-2687d0453377-kube-api-access-wzjtd\") pod \"cluster-proxy-proxy-agent-f959b7f6b-8jxpd\" (UID: \"32d3e7bf-5f00-4488-9e3f-2687d0453377\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f959b7f6b-8jxpd" Jul 10 02:49:14.427254 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.427250 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q79qr\" (UniqueName: \"kubernetes.io/projected/cbbc2bee-9865-401e-9bc3-23de969ff5eb-kube-api-access-q79qr\") pod \"downloads-68c9cbb497-nnk5x\" (UID: \"cbbc2bee-9865-401e-9bc3-23de969ff5eb\") " pod="openshift-console/downloads-68c9cbb497-nnk5x" Jul 10 02:49:14.427360 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.427282 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hx6j\" (UniqueName: \"kubernetes.io/projected/3d71fd15-80c3-42ef-b772-f5af5436012d-kube-api-access-5hx6j\") pod \"managed-serviceaccount-addon-agent-7bdf5774c4-6c9wp\" (UID: \"3d71fd15-80c3-42ef-b772-f5af5436012d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7bdf5774c4-6c9wp" Jul 10 02:49:14.427360 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.427336 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/32d3e7bf-5f00-4488-9e3f-2687d0453377-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-f959b7f6b-8jxpd\" (UID: \"32d3e7bf-5f00-4488-9e3f-2687d0453377\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f959b7f6b-8jxpd" Jul 10 02:49:14.427441 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.427374 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c6492dae-a18a-45ed-95be-93bc92198ea9-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-7v4dt\" (UID: \"c6492dae-a18a-45ed-95be-93bc92198ea9\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-7v4dt" Jul 10 02:49:14.427441 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.427406 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3d71fd15-80c3-42ef-b772-f5af5436012d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7bdf5774c4-6c9wp\" (UID: \"3d71fd15-80c3-42ef-b772-f5af5436012d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7bdf5774c4-6c9wp" Jul 10 02:49:14.427565 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.427540 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/32d3e7bf-5f00-4488-9e3f-2687d0453377-additional-service-ca\") pod \"cluster-proxy-proxy-agent-f959b7f6b-8jxpd\" (UID: \"32d3e7bf-5f00-4488-9e3f-2687d0453377\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f959b7f6b-8jxpd" Jul 10 02:49:14.427650 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.427547 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c6492dae-a18a-45ed-95be-93bc92198ea9-nginx-conf\") pod \"networking-console-plugin-6cf856758c-7v4dt\" (UID: \"c6492dae-a18a-45ed-95be-93bc92198ea9\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-7v4dt" Jul 10 02:49:14.428269 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.428222 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c544c7ea-9123-4b52-9cd6-9865d8e552a0-tmp\") pod \"klusterlet-addon-workmgr-5b66749db6-zpm4b\" (UID: \"c544c7ea-9123-4b52-9cd6-9865d8e552a0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b66749db6-zpm4b" Jul 10 02:49:14.430125 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.430054 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/32d3e7bf-5f00-4488-9e3f-2687d0453377-hub\") pod \"cluster-proxy-proxy-agent-f959b7f6b-8jxpd\" (UID: \"32d3e7bf-5f00-4488-9e3f-2687d0453377\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f959b7f6b-8jxpd" Jul 10 02:49:14.430923 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.430676 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/32d3e7bf-5f00-4488-9e3f-2687d0453377-ca\") pod \"cluster-proxy-proxy-agent-f959b7f6b-8jxpd\" (UID: \"32d3e7bf-5f00-4488-9e3f-2687d0453377\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f959b7f6b-8jxpd" Jul 10 02:49:14.431194 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.431168 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/32d3e7bf-5f00-4488-9e3f-2687d0453377-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-f959b7f6b-8jxpd\" (UID: \"32d3e7bf-5f00-4488-9e3f-2687d0453377\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f959b7f6b-8jxpd" Jul 10 02:49:14.431546 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.431525 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3d71fd15-80c3-42ef-b772-f5af5436012d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7bdf5774c4-6c9wp\" (UID: \"3d71fd15-80c3-42ef-b772-f5af5436012d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7bdf5774c4-6c9wp" Jul 10 02:49:14.431894 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.431872 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c544c7ea-9123-4b52-9cd6-9865d8e552a0-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b66749db6-zpm4b\" (UID: \"c544c7ea-9123-4b52-9cd6-9865d8e552a0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b66749db6-zpm4b" Jul 10 02:49:14.432030 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.432008 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c6492dae-a18a-45ed-95be-93bc92198ea9-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-7v4dt\" (UID: \"c6492dae-a18a-45ed-95be-93bc92198ea9\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-7v4dt" Jul 10 02:49:14.432146 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.432072 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/32d3e7bf-5f00-4488-9e3f-2687d0453377-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-f959b7f6b-8jxpd\" (UID: \"32d3e7bf-5f00-4488-9e3f-2687d0453377\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f959b7f6b-8jxpd" Jul 10 02:49:14.437794 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.437758 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prxjn\" (UniqueName: \"kubernetes.io/projected/c544c7ea-9123-4b52-9cd6-9865d8e552a0-kube-api-access-prxjn\") pod \"klusterlet-addon-workmgr-5b66749db6-zpm4b\" (UID: \"c544c7ea-9123-4b52-9cd6-9865d8e552a0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b66749db6-zpm4b" Jul 10 02:49:14.438039 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.437993 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hx6j\" (UniqueName: \"kubernetes.io/projected/3d71fd15-80c3-42ef-b772-f5af5436012d-kube-api-access-5hx6j\") pod \"managed-serviceaccount-addon-agent-7bdf5774c4-6c9wp\" (UID: \"3d71fd15-80c3-42ef-b772-f5af5436012d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7bdf5774c4-6c9wp" Jul 10 02:49:14.438139 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.438066 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzjtd\" (UniqueName: \"kubernetes.io/projected/32d3e7bf-5f00-4488-9e3f-2687d0453377-kube-api-access-wzjtd\") pod \"cluster-proxy-proxy-agent-f959b7f6b-8jxpd\" (UID: \"32d3e7bf-5f00-4488-9e3f-2687d0453377\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f959b7f6b-8jxpd" Jul 10 02:49:14.525056 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.525022 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f959b7f6b-8jxpd" Jul 10 02:49:14.528414 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.528382 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q79qr\" (UniqueName: \"kubernetes.io/projected/cbbc2bee-9865-401e-9bc3-23de969ff5eb-kube-api-access-q79qr\") pod \"downloads-68c9cbb497-nnk5x\" (UID: \"cbbc2bee-9865-401e-9bc3-23de969ff5eb\") " pod="openshift-console/downloads-68c9cbb497-nnk5x" Jul 10 02:49:14.534475 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.534453 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7bdf5774c4-6c9wp" Jul 10 02:49:14.543078 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.543051 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q79qr\" (UniqueName: \"kubernetes.io/projected/cbbc2bee-9865-401e-9bc3-23de969ff5eb-kube-api-access-q79qr\") pod \"downloads-68c9cbb497-nnk5x\" (UID: \"cbbc2bee-9865-401e-9bc3-23de969ff5eb\") " pod="openshift-console/downloads-68c9cbb497-nnk5x" Jul 10 02:49:14.554542 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.554506 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b66749db6-zpm4b" Jul 10 02:49:14.560327 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.560308 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-7v4dt" Jul 10 02:49:14.598677 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:14.598646 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-nnk5x" Jul 10 02:49:17.560656 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:49:17.560621 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d71fd15_80c3_42ef_b772_f5af5436012d.slice/crio-96d6ef5b783957bbd1c90e49023f66ca28cc6f8fa63fc77330104484cd43ec90 WatchSource:0}: Error finding container 96d6ef5b783957bbd1c90e49023f66ca28cc6f8fa63fc77330104484cd43ec90: Status 404 returned error can't find the container with id 96d6ef5b783957bbd1c90e49023f66ca28cc6f8fa63fc77330104484cd43ec90 Jul 10 02:49:17.561058 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:17.560755 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7bdf5774c4-6c9wp"] Jul 10 02:49:17.637322 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:17.637244 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f959b7f6b-8jxpd"] Jul 10 02:49:17.641110 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:49:17.641065 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod32d3e7bf_5f00_4488_9e3f_2687d0453377.slice/crio-621975b605884cb9d1939b8e22c135c788edf38515a50d61d9cebdc14465b446 WatchSource:0}: Error finding container 621975b605884cb9d1939b8e22c135c788edf38515a50d61d9cebdc14465b446: Status 404 returned error can't find the container with id 621975b605884cb9d1939b8e22c135c788edf38515a50d61d9cebdc14465b446 Jul 10 02:49:17.789385 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:17.789346 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-7v4dt"] Jul 10 02:49:17.795394 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:17.795336 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-nnk5x"] Jul 10 02:49:17.795394 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:17.795387 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b66749db6-zpm4b"] Jul 10 02:49:17.844860 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:17.844758 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-2rh2j" event={"ID":"bcda0e99-fdd8-498f-8e73-ecf4c279be71","Type":"ContainerStarted","Data":"48be3d85c6fe99048274de5997839bd4a5fe4e3c49cb20b2f4682f6076044563"} Jul 10 02:49:17.847321 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:17.847102 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-q44gx" event={"ID":"167ea7f3-1d8f-4892-bc0e-e9247912fef6","Type":"ContainerStarted","Data":"9a95893eb16690b65e8000938262329b01d77be24c7dad1263fde1d09c5d415b"} Jul 10 02:49:17.847321 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:17.847152 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-q44gx" event={"ID":"167ea7f3-1d8f-4892-bc0e-e9247912fef6","Type":"ContainerStarted","Data":"bd6b061809fa7d65a88dce8f2cd71f2c4c427214687b5c85876cc3aca336f28f"} Jul 10 02:49:17.848916 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:17.848858 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b66749db6-zpm4b" event={"ID":"c544c7ea-9123-4b52-9cd6-9865d8e552a0","Type":"ContainerStarted","Data":"6f99227ba3371ae0e35658ca2aec1542cd24a2aac09195c80e33bc3ca78f9979"} Jul 10 02:49:17.850199 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:17.850171 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7bdf5774c4-6c9wp" event={"ID":"3d71fd15-80c3-42ef-b772-f5af5436012d","Type":"ContainerStarted","Data":"96d6ef5b783957bbd1c90e49023f66ca28cc6f8fa63fc77330104484cd43ec90"} Jul 10 02:49:17.852557 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:17.852535 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w7npn" event={"ID":"b3661326-52c2-4c9b-b2db-420720fe4a86","Type":"ContainerStarted","Data":"e6262d2fd03ad4a78c9d2f831ea2d0acc236846d55c6d008264d60a9d1730026"} Jul 10 02:49:17.854922 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:17.854903 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-nnk5x" event={"ID":"cbbc2bee-9865-401e-9bc3-23de969ff5eb","Type":"ContainerStarted","Data":"e9ce55c30214ecd4c3e9d1edf152c828ad3f84272a1d11650ae0a77fdd7f35cc"} Jul 10 02:49:17.856485 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:17.856453 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-q28nr" event={"ID":"dc1f4b26-669e-42a0-abd4-f858ac6b1543","Type":"ContainerStarted","Data":"4713d5f5765206f9dec77516be48245983b2aac3b129c05bb13ce1b1dbfa3e70"} Jul 10 02:49:17.857842 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:17.857803 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f959b7f6b-8jxpd" event={"ID":"32d3e7bf-5f00-4488-9e3f-2687d0453377","Type":"ContainerStarted","Data":"621975b605884cb9d1939b8e22c135c788edf38515a50d61d9cebdc14465b446"} Jul 10 02:49:17.860220 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:17.860197 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cc72k" event={"ID":"5176df10-dfd5-4514-9a59-c854c4a0ec21","Type":"ContainerStarted","Data":"68ff6927116557a86913b2edd587dc5658884a1686c938c3072772584c0a338c"} Jul 10 02:49:17.863410 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:17.863385 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-7v4dt" event={"ID":"c6492dae-a18a-45ed-95be-93bc92198ea9","Type":"ContainerStarted","Data":"d9e65f8aa475ee12f74dd13f5cb4193ea20e3dea514ff1de8c84c6722d4acb77"} Jul 10 02:49:17.884627 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:17.884119 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-q44gx" podStartSLOduration=44.288304383 podStartE2EDuration="50.884098479s" totalImagesPullingTime="6.595794096s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:48:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:49:17.883042051 +0000 UTC m=+76.164158356" watchObservedRunningTime="2026-07-10 02:49:17.884098479 +0000 UTC m=+76.165214765" Jul 10 02:49:17.884627 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:17.884266 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-2rh2j" podStartSLOduration=43.188923333 podStartE2EDuration="49.884260216s" totalImagesPullingTime="6.695336883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:48:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:49:17.862900377 +0000 UTC m=+76.144016663" watchObservedRunningTime="2026-07-10 02:49:17.884260216 +0000 UTC m=+76.165376502" Jul 10 02:49:17.898021 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:17.897975 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-q28nr" podStartSLOduration=66.126675153 podStartE2EDuration="1m14.897956358s" totalImagesPullingTime="8.771281205s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 02:48:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:49:17.897081862 +0000 UTC m=+76.178198148" watchObservedRunningTime="2026-07-10 02:49:17.897956358 +0000 UTC m=+76.179072621" Jul 10 02:49:17.916238 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:17.916180 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-w7npn" podStartSLOduration=17.54579175 podStartE2EDuration="28.916158226s" totalImagesPullingTime="11.370366476s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 02:48:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:49:17.91452458 +0000 UTC m=+76.195640884" watchObservedRunningTime="2026-07-10 02:49:17.916158226 +0000 UTC m=+76.197274512" Jul 10 02:49:18.059602 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:18.059567 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-6fjb8"] Jul 10 02:49:18.081802 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:18.081776 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-6fjb8"] Jul 10 02:49:18.081965 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:18.081914 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-6fjb8" Jul 10 02:49:18.084097 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:18.084071 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 10 02:49:18.084262 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:18.084241 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-xnb2b\"" Jul 10 02:49:18.162730 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:18.162630 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9bd0b995-3a3a-47e8-b101-aaab0393550c-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-6fjb8\" (UID: \"9bd0b995-3a3a-47e8-b101-aaab0393550c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-6fjb8" Jul 10 02:49:18.263458 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:18.263421 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9bd0b995-3a3a-47e8-b101-aaab0393550c-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-6fjb8\" (UID: \"9bd0b995-3a3a-47e8-b101-aaab0393550c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-6fjb8" Jul 10 02:49:18.263654 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:49:18.263631 2575 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 10 02:49:18.263726 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:49:18.263714 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bd0b995-3a3a-47e8-b101-aaab0393550c-tls-certificates podName:9bd0b995-3a3a-47e8-b101-aaab0393550c nodeName:}" failed. No retries permitted until 2026-07-10 02:49:18.763691815 +0000 UTC m=+77.044808094 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/9bd0b995-3a3a-47e8-b101-aaab0393550c-tls-certificates") pod "prometheus-operator-admission-webhook-77d98bc786-6fjb8" (UID: "9bd0b995-3a3a-47e8-b101-aaab0393550c") : secret "prometheus-operator-admission-webhook-tls" not found Jul 10 02:49:18.769859 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:18.769300 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9bd0b995-3a3a-47e8-b101-aaab0393550c-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-6fjb8\" (UID: \"9bd0b995-3a3a-47e8-b101-aaab0393550c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-6fjb8" Jul 10 02:49:18.789421 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:18.789346 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9bd0b995-3a3a-47e8-b101-aaab0393550c-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-6fjb8\" (UID: \"9bd0b995-3a3a-47e8-b101-aaab0393550c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-6fjb8" Jul 10 02:49:18.874441 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:18.874107 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cc72k" event={"ID":"5176df10-dfd5-4514-9a59-c854c4a0ec21","Type":"ContainerStarted","Data":"8224295602cf2e5e2acf2bd561c0bac902e90828ea11bda7b4da862a03a4838a"} Jul 10 02:49:18.875650 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:18.874910 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-cc72k" Jul 10 02:49:18.895038 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:18.893763 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-cc72k" podStartSLOduration=34.394596167 podStartE2EDuration="40.893739885s" totalImagesPullingTime="6.499143718s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:48:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:49:18.891737422 +0000 UTC m=+77.172853728" watchObservedRunningTime="2026-07-10 02:49:18.893739885 +0000 UTC m=+77.174856170" Jul 10 02:49:18.992132 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:18.991830 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-6fjb8" Jul 10 02:49:19.215858 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:19.215823 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-6fjb8"] Jul 10 02:49:19.220225 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:49:19.220184 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bd0b995_3a3a_47e8_b101_aaab0393550c.slice/crio-8a04dd06c2e870d035fae80d3b75dd6b277bbb8fd6ca8e647c6331a99580bea0 WatchSource:0}: Error finding container 8a04dd06c2e870d035fae80d3b75dd6b277bbb8fd6ca8e647c6331a99580bea0: Status 404 returned error can't find the container with id 8a04dd06c2e870d035fae80d3b75dd6b277bbb8fd6ca8e647c6331a99580bea0 Jul 10 02:49:19.885978 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:19.885889 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-6fjb8" event={"ID":"9bd0b995-3a3a-47e8-b101-aaab0393550c","Type":"ContainerStarted","Data":"8a04dd06c2e870d035fae80d3b75dd6b277bbb8fd6ca8e647c6331a99580bea0"} Jul 10 02:49:25.922587 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:25.922494 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f959b7f6b-8jxpd" event={"ID":"32d3e7bf-5f00-4488-9e3f-2687d0453377","Type":"ContainerStarted","Data":"8b7594eba45bebf73b1d8cebdf37d33a4235988f5a5cafb08ccd8ad89eee9e8f"} Jul 10 02:49:25.925213 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:25.924684 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-7v4dt" event={"ID":"c6492dae-a18a-45ed-95be-93bc92198ea9","Type":"ContainerStarted","Data":"1b22e5efc90cacec80c5d5770dc4f085d314c68be52ad2f9b52cdb16ff5712a6"} Jul 10 02:49:25.926551 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:25.926483 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-6fjb8" event={"ID":"9bd0b995-3a3a-47e8-b101-aaab0393550c","Type":"ContainerStarted","Data":"fbb3de7f0f908eae2bec734bb11f88a909fb8b04e6166d729733cb4e99ef9ade"} Jul 10 02:49:25.926878 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:25.926854 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-6fjb8" Jul 10 02:49:25.929195 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:25.928971 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b66749db6-zpm4b" event={"ID":"c544c7ea-9123-4b52-9cd6-9865d8e552a0","Type":"ContainerStarted","Data":"a8211a7e20600b2c3d20cd269f90a468c3bdcc4434284fa8115fa57d54dc318e"} Jul 10 02:49:25.929195 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:25.929118 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b66749db6-zpm4b" Jul 10 02:49:25.931122 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:25.931100 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b66749db6-zpm4b" Jul 10 02:49:25.931715 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:25.931692 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7bdf5774c4-6c9wp" event={"ID":"3d71fd15-80c3-42ef-b772-f5af5436012d","Type":"ContainerStarted","Data":"16af3f8aa10dd9ee26e14510174380b309239854467d43ec1fc1d3e015fb262e"} Jul 10 02:49:25.934091 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:25.934043 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-6fjb8" Jul 10 02:49:25.941524 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:25.941433 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-7v4dt" podStartSLOduration=4.035527353 podStartE2EDuration="11.941417941s" totalImagesPullingTime="7.905890588s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:49:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:49:25.939268685 +0000 UTC m=+84.220384963" watchObservedRunningTime="2026-07-10 02:49:25.941417941 +0000 UTC m=+84.222534226" Jul 10 02:49:25.957401 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:25.956163 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7bdf5774c4-6c9wp" podStartSLOduration=3.797552822 podStartE2EDuration="11.956141991s" totalImagesPullingTime="8.158589169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:49:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:49:25.95490841 +0000 UTC m=+84.236024703" watchObservedRunningTime="2026-07-10 02:49:25.956141991 +0000 UTC m=+84.237258277" Jul 10 02:49:25.974423 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:25.974113 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-6fjb8" podStartSLOduration=1.471869787 podStartE2EDuration="7.974092733s" totalImagesPullingTime="6.502222946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:49:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:49:25.973013036 +0000 UTC m=+84.254129323" watchObservedRunningTime="2026-07-10 02:49:25.974092733 +0000 UTC m=+84.255209014" Jul 10 02:49:25.992404 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:25.992335 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b66749db6-zpm4b" podStartSLOduration=4.05106172 podStartE2EDuration="11.99231829s" totalImagesPullingTime="7.94125657s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:49:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:49:25.991015804 +0000 UTC m=+84.272132084" watchObservedRunningTime="2026-07-10 02:49:25.99231829 +0000 UTC m=+84.273434574" Jul 10 02:49:26.939439 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:26.939404 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f959b7f6b-8jxpd" event={"ID":"32d3e7bf-5f00-4488-9e3f-2687d0453377","Type":"ContainerStarted","Data":"2a2c67b44c492d30a95682f482015f186db423c4db41ca5bc9f0e445234f7442"} Jul 10 02:49:26.939439 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:26.939446 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f959b7f6b-8jxpd" event={"ID":"32d3e7bf-5f00-4488-9e3f-2687d0453377","Type":"ContainerStarted","Data":"7710da345e835379ded6e9f7c83646ba28fc330a7269b96680eff0b15ef9a70a"} Jul 10 02:49:26.958062 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:26.957997 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-f959b7f6b-8jxpd" podStartSLOduration=4.854765382 podStartE2EDuration="12.957975199s" totalImagesPullingTime="8.103209817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:49:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:49:26.957094823 +0000 UTC m=+85.238211111" watchObservedRunningTime="2026-07-10 02:49:26.957975199 +0000 UTC m=+85.239091485" Jul 10 02:49:29.897352 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:29.897313 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-cc72k" Jul 10 02:49:30.264470 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:30.264433 2575 patch_prober.go:28] interesting pod/image-registry-fbd7648c9-vtvdf container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 10 02:49:30.264652 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:30.264495 2575 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-fbd7648c9-vtvdf" podUID="f540bce4-0371-4500-8e77-3bb4291a2e3d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 02:49:32.488884 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:32.486649 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-jrfx9"] Jul 10 02:49:32.636571 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:32.636484 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jrfx9" Jul 10 02:49:32.641588 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:32.639421 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 10 02:49:32.641588 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:32.639431 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 10 02:49:32.641588 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:32.639674 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 10 02:49:32.641588 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:32.639974 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-z5cbw\"" Jul 10 02:49:32.641588 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:32.640260 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 10 02:49:32.704575 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:32.703583 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d66bd742-5ded-43b8-a8df-b202ed2fce86-root\") pod \"node-exporter-jrfx9\" (UID: \"d66bd742-5ded-43b8-a8df-b202ed2fce86\") " pod="openshift-monitoring/node-exporter-jrfx9" Jul 10 02:49:32.704575 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:32.703627 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d66bd742-5ded-43b8-a8df-b202ed2fce86-node-exporter-textfile\") pod \"node-exporter-jrfx9\" (UID: \"d66bd742-5ded-43b8-a8df-b202ed2fce86\") " pod="openshift-monitoring/node-exporter-jrfx9" Jul 10 02:49:32.704575 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:32.703657 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d66bd742-5ded-43b8-a8df-b202ed2fce86-metrics-client-ca\") pod \"node-exporter-jrfx9\" (UID: \"d66bd742-5ded-43b8-a8df-b202ed2fce86\") " pod="openshift-monitoring/node-exporter-jrfx9" Jul 10 02:49:32.704575 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:32.703683 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d66bd742-5ded-43b8-a8df-b202ed2fce86-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jrfx9\" (UID: \"d66bd742-5ded-43b8-a8df-b202ed2fce86\") " pod="openshift-monitoring/node-exporter-jrfx9" Jul 10 02:49:32.704575 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:32.703732 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d66bd742-5ded-43b8-a8df-b202ed2fce86-node-exporter-tls\") pod \"node-exporter-jrfx9\" (UID: \"d66bd742-5ded-43b8-a8df-b202ed2fce86\") " pod="openshift-monitoring/node-exporter-jrfx9" Jul 10 02:49:32.704575 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:32.703770 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d66bd742-5ded-43b8-a8df-b202ed2fce86-node-exporter-accelerators-collector-config\") pod \"node-exporter-jrfx9\" (UID: \"d66bd742-5ded-43b8-a8df-b202ed2fce86\") " pod="openshift-monitoring/node-exporter-jrfx9" Jul 10 02:49:32.704575 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:32.703829 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d66bd742-5ded-43b8-a8df-b202ed2fce86-sys\") pod \"node-exporter-jrfx9\" (UID: \"d66bd742-5ded-43b8-a8df-b202ed2fce86\") " pod="openshift-monitoring/node-exporter-jrfx9" Jul 10 02:49:32.704575 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:32.703857 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d66bd742-5ded-43b8-a8df-b202ed2fce86-node-exporter-wtmp\") pod \"node-exporter-jrfx9\" (UID: \"d66bd742-5ded-43b8-a8df-b202ed2fce86\") " pod="openshift-monitoring/node-exporter-jrfx9" Jul 10 02:49:32.704575 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:32.703882 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvtrh\" (UniqueName: \"kubernetes.io/projected/d66bd742-5ded-43b8-a8df-b202ed2fce86-kube-api-access-cvtrh\") pod \"node-exporter-jrfx9\" (UID: \"d66bd742-5ded-43b8-a8df-b202ed2fce86\") " pod="openshift-monitoring/node-exporter-jrfx9" Jul 10 02:49:32.805378 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:32.804919 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d66bd742-5ded-43b8-a8df-b202ed2fce86-root\") pod \"node-exporter-jrfx9\" (UID: \"d66bd742-5ded-43b8-a8df-b202ed2fce86\") " pod="openshift-monitoring/node-exporter-jrfx9" Jul 10 02:49:32.805378 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:32.804967 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d66bd742-5ded-43b8-a8df-b202ed2fce86-node-exporter-textfile\") pod \"node-exporter-jrfx9\" (UID: \"d66bd742-5ded-43b8-a8df-b202ed2fce86\") " pod="openshift-monitoring/node-exporter-jrfx9" Jul 10 02:49:32.805378 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:32.804991 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d66bd742-5ded-43b8-a8df-b202ed2fce86-metrics-client-ca\") pod \"node-exporter-jrfx9\" (UID: \"d66bd742-5ded-43b8-a8df-b202ed2fce86\") " pod="openshift-monitoring/node-exporter-jrfx9" Jul 10 02:49:32.805378 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:32.805016 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d66bd742-5ded-43b8-a8df-b202ed2fce86-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jrfx9\" (UID: \"d66bd742-5ded-43b8-a8df-b202ed2fce86\") " pod="openshift-monitoring/node-exporter-jrfx9" Jul 10 02:49:32.805378 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:32.805062 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d66bd742-5ded-43b8-a8df-b202ed2fce86-node-exporter-tls\") pod \"node-exporter-jrfx9\" (UID: \"d66bd742-5ded-43b8-a8df-b202ed2fce86\") " pod="openshift-monitoring/node-exporter-jrfx9" Jul 10 02:49:32.805378 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:32.805094 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d66bd742-5ded-43b8-a8df-b202ed2fce86-node-exporter-accelerators-collector-config\") pod \"node-exporter-jrfx9\" (UID: \"d66bd742-5ded-43b8-a8df-b202ed2fce86\") " pod="openshift-monitoring/node-exporter-jrfx9" Jul 10 02:49:32.805378 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:32.805126 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d66bd742-5ded-43b8-a8df-b202ed2fce86-sys\") pod \"node-exporter-jrfx9\" (UID: \"d66bd742-5ded-43b8-a8df-b202ed2fce86\") " pod="openshift-monitoring/node-exporter-jrfx9" Jul 10 02:49:32.805378 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:32.805151 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d66bd742-5ded-43b8-a8df-b202ed2fce86-node-exporter-wtmp\") pod \"node-exporter-jrfx9\" (UID: \"d66bd742-5ded-43b8-a8df-b202ed2fce86\") " pod="openshift-monitoring/node-exporter-jrfx9" Jul 10 02:49:32.805378 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:32.805175 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvtrh\" (UniqueName: \"kubernetes.io/projected/d66bd742-5ded-43b8-a8df-b202ed2fce86-kube-api-access-cvtrh\") pod \"node-exporter-jrfx9\" (UID: \"d66bd742-5ded-43b8-a8df-b202ed2fce86\") " pod="openshift-monitoring/node-exporter-jrfx9" Jul 10 02:49:32.816838 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:49:32.811973 2575 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 10 02:49:32.816838 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:49:32.812062 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d66bd742-5ded-43b8-a8df-b202ed2fce86-node-exporter-tls podName:d66bd742-5ded-43b8-a8df-b202ed2fce86 nodeName:}" failed. No retries permitted until 2026-07-10 02:49:33.312039637 +0000 UTC m=+91.593155904 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/d66bd742-5ded-43b8-a8df-b202ed2fce86-node-exporter-tls") pod "node-exporter-jrfx9" (UID: "d66bd742-5ded-43b8-a8df-b202ed2fce86") : secret "node-exporter-tls" not found Jul 10 02:49:32.816838 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:32.812387 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d66bd742-5ded-43b8-a8df-b202ed2fce86-root\") pod \"node-exporter-jrfx9\" (UID: \"d66bd742-5ded-43b8-a8df-b202ed2fce86\") " pod="openshift-monitoring/node-exporter-jrfx9" Jul 10 02:49:32.816838 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:32.812711 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d66bd742-5ded-43b8-a8df-b202ed2fce86-node-exporter-textfile\") pod \"node-exporter-jrfx9\" (UID: \"d66bd742-5ded-43b8-a8df-b202ed2fce86\") " pod="openshift-monitoring/node-exporter-jrfx9" Jul 10 02:49:32.816838 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:32.813328 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d66bd742-5ded-43b8-a8df-b202ed2fce86-node-exporter-wtmp\") pod \"node-exporter-jrfx9\" (UID: \"d66bd742-5ded-43b8-a8df-b202ed2fce86\") " pod="openshift-monitoring/node-exporter-jrfx9" Jul 10 02:49:32.816838 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:32.813883 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d66bd742-5ded-43b8-a8df-b202ed2fce86-sys\") pod \"node-exporter-jrfx9\" (UID: \"d66bd742-5ded-43b8-a8df-b202ed2fce86\") " pod="openshift-monitoring/node-exporter-jrfx9" Jul 10 02:49:32.816838 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:32.814070 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d66bd742-5ded-43b8-a8df-b202ed2fce86-node-exporter-accelerators-collector-config\") pod \"node-exporter-jrfx9\" (UID: \"d66bd742-5ded-43b8-a8df-b202ed2fce86\") " pod="openshift-monitoring/node-exporter-jrfx9" Jul 10 02:49:32.816838 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:32.814647 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d66bd742-5ded-43b8-a8df-b202ed2fce86-metrics-client-ca\") pod \"node-exporter-jrfx9\" (UID: \"d66bd742-5ded-43b8-a8df-b202ed2fce86\") " pod="openshift-monitoring/node-exporter-jrfx9" Jul 10 02:49:32.827892 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:32.827144 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvtrh\" (UniqueName: \"kubernetes.io/projected/d66bd742-5ded-43b8-a8df-b202ed2fce86-kube-api-access-cvtrh\") pod \"node-exporter-jrfx9\" (UID: \"d66bd742-5ded-43b8-a8df-b202ed2fce86\") " pod="openshift-monitoring/node-exporter-jrfx9" Jul 10 02:49:32.829729 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:32.829617 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d66bd742-5ded-43b8-a8df-b202ed2fce86-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jrfx9\" (UID: \"d66bd742-5ded-43b8-a8df-b202ed2fce86\") " pod="openshift-monitoring/node-exporter-jrfx9" Jul 10 02:49:32.833058 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:32.832281 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-fbd7648c9-vtvdf" Jul 10 02:49:33.410949 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:33.410872 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d66bd742-5ded-43b8-a8df-b202ed2fce86-node-exporter-tls\") pod \"node-exporter-jrfx9\" (UID: \"d66bd742-5ded-43b8-a8df-b202ed2fce86\") " pod="openshift-monitoring/node-exporter-jrfx9" Jul 10 02:49:33.413665 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:33.413633 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d66bd742-5ded-43b8-a8df-b202ed2fce86-node-exporter-tls\") pod \"node-exporter-jrfx9\" (UID: \"d66bd742-5ded-43b8-a8df-b202ed2fce86\") " pod="openshift-monitoring/node-exporter-jrfx9" Jul 10 02:49:33.551610 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:33.551571 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jrfx9" Jul 10 02:49:37.997905 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:37.997761 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-nnk5x" event={"ID":"cbbc2bee-9865-401e-9bc3-23de969ff5eb","Type":"ContainerStarted","Data":"d0b8caef9b7653797ec4dbf91de1855f75be9b90d2c54c1b7e7a230447e6de0c"} Jul 10 02:49:38.002859 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:37.998854 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-nnk5x" Jul 10 02:49:38.005193 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:38.005161 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jrfx9" event={"ID":"d66bd742-5ded-43b8-a8df-b202ed2fce86","Type":"ContainerStarted","Data":"6e3786fed42bdf0f4bd04631b7b5633ad7e070d84f97d082b2fbed330cc84ce9"} Jul 10 02:49:38.019307 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:38.019268 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-nnk5x" Jul 10 02:49:38.020530 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:38.020401 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-nnk5x" podStartSLOduration=4.253985436 podStartE2EDuration="24.02038222s" totalImagesPullingTime="19.766396784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:49:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:49:38.018685215 +0000 UTC m=+96.299801524" watchObservedRunningTime="2026-07-10 02:49:38.02038222 +0000 UTC m=+96.301498507" Jul 10 02:49:39.014262 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:39.014229 2575 generic.go:356] "Generic (PLEG): container finished" podID="d66bd742-5ded-43b8-a8df-b202ed2fce86" containerID="209dcdfe688b44163865bf54caee98bc9f8dfe51d3f5b737643b447076cf7416" exitCode=0 Jul 10 02:49:39.014789 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:39.014309 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jrfx9" event={"ID":"d66bd742-5ded-43b8-a8df-b202ed2fce86","Type":"ContainerDied","Data":"209dcdfe688b44163865bf54caee98bc9f8dfe51d3f5b737643b447076cf7416"} Jul 10 02:49:40.022287 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:40.022246 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jrfx9" event={"ID":"d66bd742-5ded-43b8-a8df-b202ed2fce86","Type":"ContainerStarted","Data":"5b5b540abb914acb78e8999add193b4d20ff0b2e5e0ccc758453fdccbce3e4c4"} Jul 10 02:49:40.022767 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:40.022314 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jrfx9" event={"ID":"d66bd742-5ded-43b8-a8df-b202ed2fce86","Type":"ContainerStarted","Data":"cf8df2164347c2a219fa9b802d9cec760b0f8e04e890f8f92253905ed9172a32"} Jul 10 02:49:40.042760 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:40.042707 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-jrfx9" podStartSLOduration=6.493500413 podStartE2EDuration="8.042685746s" totalImagesPullingTime="1.411546926s" totalInitContainerRuntime="137.638407ms" isStatefulPod=true podCreationTimestamp="2026-07-10 02:49:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:49:40.041482525 +0000 UTC m=+98.322598823" watchObservedRunningTime="2026-07-10 02:49:40.042685746 +0000 UTC m=+98.323802035" Jul 10 02:49:40.781670 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:40.781636 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-nf7q8" Jul 10 02:49:42.834979 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:42.834945 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-fbd7648c9-vtvdf"] Jul 10 02:49:55.089702 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:55.089667 2575 generic.go:356] "Generic (PLEG): container finished" podID="0cd9606f-fafb-4005-91fb-1579f4b0cc75" containerID="f5d6c83e4b4bb95a911cf9f38c6ed99ec3bb28fd5f81e5e6a5ef9023c273eda8" exitCode=0 Jul 10 02:49:55.090131 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:55.089740 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-28scv" event={"ID":"0cd9606f-fafb-4005-91fb-1579f4b0cc75","Type":"ContainerDied","Data":"f5d6c83e4b4bb95a911cf9f38c6ed99ec3bb28fd5f81e5e6a5ef9023c273eda8"} Jul 10 02:49:55.090131 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:55.090088 2575 scope.go:117] "RemoveContainer" containerID="f5d6c83e4b4bb95a911cf9f38c6ed99ec3bb28fd5f81e5e6a5ef9023c273eda8" Jul 10 02:49:56.095438 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:49:56.095406 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-28scv" event={"ID":"0cd9606f-fafb-4005-91fb-1579f4b0cc75","Type":"ContainerStarted","Data":"d4abb3859210475d77d4cc3f11d564eb1c5028fe09d76183c42c718b8c7e0ea7"} Jul 10 02:50:00.113382 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:50:00.113347 2575 generic.go:356] "Generic (PLEG): container finished" podID="64e37f35-938c-457a-a377-354c4bc12d73" containerID="f1aa012ec41c9cd7913e8bd040e9db4acb2e365c66afd0920c65d64874316458" exitCode=0 Jul 10 02:50:00.113879 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:50:00.113419 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-x22bq" event={"ID":"64e37f35-938c-457a-a377-354c4bc12d73","Type":"ContainerDied","Data":"f1aa012ec41c9cd7913e8bd040e9db4acb2e365c66afd0920c65d64874316458"} Jul 10 02:50:00.113879 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:50:00.113840 2575 scope.go:117] "RemoveContainer" containerID="f1aa012ec41c9cd7913e8bd040e9db4acb2e365c66afd0920c65d64874316458" Jul 10 02:50:00.114990 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:50:00.114972 2575 generic.go:356] "Generic (PLEG): container finished" podID="4094383d-2bb9-441b-8c30-94ff7551ce18" containerID="89fb07e085b038527a8b31c5a072286c88c3172ada0f1f95cc471a74fed8fe81" exitCode=0 Jul 10 02:50:00.115071 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:50:00.115031 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-gtqxb" event={"ID":"4094383d-2bb9-441b-8c30-94ff7551ce18","Type":"ContainerDied","Data":"89fb07e085b038527a8b31c5a072286c88c3172ada0f1f95cc471a74fed8fe81"} Jul 10 02:50:00.115273 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:50:00.115262 2575 scope.go:117] "RemoveContainer" containerID="89fb07e085b038527a8b31c5a072286c88c3172ada0f1f95cc471a74fed8fe81" Jul 10 02:50:01.124479 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:50:01.124447 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-gtqxb" event={"ID":"4094383d-2bb9-441b-8c30-94ff7551ce18","Type":"ContainerStarted","Data":"2c62179246ba008206de1c9da7ed7bb3d3b3c61fdd2ecfe4f9ce3409a28123b1"} Jul 10 02:50:01.126500 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:50:01.126475 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-x22bq" event={"ID":"64e37f35-938c-457a-a377-354c4bc12d73","Type":"ContainerStarted","Data":"ee700cce9d59c197139048375152e569d722bc527222366792d6725393a23847"} Jul 10 02:50:07.860733 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:50:07.860676 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-fbd7648c9-vtvdf" podUID="f540bce4-0371-4500-8e77-3bb4291a2e3d" containerName="registry" containerID="cri-o://73ff6532d45dd0f30864bb446dbae5fbc2e4352751afd7432965742f5269b75b" gracePeriod=30 Jul 10 02:50:08.153158 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:50:08.153132 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fbd7648c9-vtvdf" Jul 10 02:50:08.156925 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:50:08.156894 2575 generic.go:356] "Generic (PLEG): container finished" podID="f540bce4-0371-4500-8e77-3bb4291a2e3d" containerID="73ff6532d45dd0f30864bb446dbae5fbc2e4352751afd7432965742f5269b75b" exitCode=0 Jul 10 02:50:08.157049 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:50:08.156931 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fbd7648c9-vtvdf" Jul 10 02:50:08.157049 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:50:08.156954 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fbd7648c9-vtvdf" event={"ID":"f540bce4-0371-4500-8e77-3bb4291a2e3d","Type":"ContainerDied","Data":"73ff6532d45dd0f30864bb446dbae5fbc2e4352751afd7432965742f5269b75b"} Jul 10 02:50:08.157049 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:50:08.157026 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fbd7648c9-vtvdf" event={"ID":"f540bce4-0371-4500-8e77-3bb4291a2e3d","Type":"ContainerDied","Data":"6847b98461069af6b98c3b0610c1a328ae7e7195fabd1955d1770d18715f4668"} Jul 10 02:50:08.157216 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:50:08.157053 2575 scope.go:117] "RemoveContainer" containerID="73ff6532d45dd0f30864bb446dbae5fbc2e4352751afd7432965742f5269b75b" Jul 10 02:50:08.165005 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:50:08.164986 2575 scope.go:117] "RemoveContainer" containerID="73ff6532d45dd0f30864bb446dbae5fbc2e4352751afd7432965742f5269b75b" Jul 10 02:50:08.165258 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:50:08.165237 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73ff6532d45dd0f30864bb446dbae5fbc2e4352751afd7432965742f5269b75b\": container with ID starting with 73ff6532d45dd0f30864bb446dbae5fbc2e4352751afd7432965742f5269b75b not found: ID does not exist" containerID="73ff6532d45dd0f30864bb446dbae5fbc2e4352751afd7432965742f5269b75b" Jul 10 02:50:08.165323 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:50:08.165269 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73ff6532d45dd0f30864bb446dbae5fbc2e4352751afd7432965742f5269b75b"} err="failed to get container status \"73ff6532d45dd0f30864bb446dbae5fbc2e4352751afd7432965742f5269b75b\": rpc error: code = NotFound desc = could not find container \"73ff6532d45dd0f30864bb446dbae5fbc2e4352751afd7432965742f5269b75b\": container with ID starting with 73ff6532d45dd0f30864bb446dbae5fbc2e4352751afd7432965742f5269b75b not found: ID does not exist" Jul 10 02:50:08.337877 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:50:08.337845 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f540bce4-0371-4500-8e77-3bb4291a2e3d-image-registry-private-configuration\") pod \"f540bce4-0371-4500-8e77-3bb4291a2e3d\" (UID: \"f540bce4-0371-4500-8e77-3bb4291a2e3d\") " Jul 10 02:50:08.337877 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:50:08.337882 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f540bce4-0371-4500-8e77-3bb4291a2e3d-registry-certificates\") pod \"f540bce4-0371-4500-8e77-3bb4291a2e3d\" (UID: \"f540bce4-0371-4500-8e77-3bb4291a2e3d\") " Jul 10 02:50:08.338109 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:50:08.337913 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f540bce4-0371-4500-8e77-3bb4291a2e3d-ca-trust-extracted\") pod \"f540bce4-0371-4500-8e77-3bb4291a2e3d\" (UID: \"f540bce4-0371-4500-8e77-3bb4291a2e3d\") " Jul 10 02:50:08.338109 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:50:08.337931 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f540bce4-0371-4500-8e77-3bb4291a2e3d-trusted-ca\") pod \"f540bce4-0371-4500-8e77-3bb4291a2e3d\" (UID: \"f540bce4-0371-4500-8e77-3bb4291a2e3d\") " Jul 10 02:50:08.338109 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:50:08.337964 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f540bce4-0371-4500-8e77-3bb4291a2e3d-installation-pull-secrets\") pod \"f540bce4-0371-4500-8e77-3bb4291a2e3d\" (UID: \"f540bce4-0371-4500-8e77-3bb4291a2e3d\") " Jul 10 02:50:08.338109 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:50:08.338011 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vtqbp\" (UniqueName: \"kubernetes.io/projected/f540bce4-0371-4500-8e77-3bb4291a2e3d-kube-api-access-vtqbp\") pod \"f540bce4-0371-4500-8e77-3bb4291a2e3d\" (UID: \"f540bce4-0371-4500-8e77-3bb4291a2e3d\") " Jul 10 02:50:08.338109 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:50:08.338055 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f540bce4-0371-4500-8e77-3bb4291a2e3d-registry-tls\") pod \"f540bce4-0371-4500-8e77-3bb4291a2e3d\" (UID: \"f540bce4-0371-4500-8e77-3bb4291a2e3d\") " Jul 10 02:50:08.338109 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:50:08.338082 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f540bce4-0371-4500-8e77-3bb4291a2e3d-bound-sa-token\") pod \"f540bce4-0371-4500-8e77-3bb4291a2e3d\" (UID: \"f540bce4-0371-4500-8e77-3bb4291a2e3d\") " Jul 10 02:50:08.338469 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:50:08.338394 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f540bce4-0371-4500-8e77-3bb4291a2e3d-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "f540bce4-0371-4500-8e77-3bb4291a2e3d" (UID: "f540bce4-0371-4500-8e77-3bb4291a2e3d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 02:50:08.338753 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:50:08.338707 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f540bce4-0371-4500-8e77-3bb4291a2e3d-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "f540bce4-0371-4500-8e77-3bb4291a2e3d" (UID: "f540bce4-0371-4500-8e77-3bb4291a2e3d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 02:50:08.340737 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:50:08.340712 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f540bce4-0371-4500-8e77-3bb4291a2e3d-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "f540bce4-0371-4500-8e77-3bb4291a2e3d" (UID: "f540bce4-0371-4500-8e77-3bb4291a2e3d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 02:50:08.340873 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:50:08.340712 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f540bce4-0371-4500-8e77-3bb4291a2e3d-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "f540bce4-0371-4500-8e77-3bb4291a2e3d" (UID: "f540bce4-0371-4500-8e77-3bb4291a2e3d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 02:50:08.340944 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:50:08.340891 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f540bce4-0371-4500-8e77-3bb4291a2e3d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "f540bce4-0371-4500-8e77-3bb4291a2e3d" (UID: "f540bce4-0371-4500-8e77-3bb4291a2e3d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 02:50:08.340997 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:50:08.340954 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f540bce4-0371-4500-8e77-3bb4291a2e3d-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "f540bce4-0371-4500-8e77-3bb4291a2e3d" (UID: "f540bce4-0371-4500-8e77-3bb4291a2e3d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 02:50:08.340997 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:50:08.340982 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f540bce4-0371-4500-8e77-3bb4291a2e3d-kube-api-access-vtqbp" (OuterVolumeSpecName: "kube-api-access-vtqbp") pod "f540bce4-0371-4500-8e77-3bb4291a2e3d" (UID: "f540bce4-0371-4500-8e77-3bb4291a2e3d"). InnerVolumeSpecName "kube-api-access-vtqbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 02:50:08.349015 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:50:08.348989 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f540bce4-0371-4500-8e77-3bb4291a2e3d-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "f540bce4-0371-4500-8e77-3bb4291a2e3d" (UID: "f540bce4-0371-4500-8e77-3bb4291a2e3d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 02:50:08.439361 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:50:08.439272 2575 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f540bce4-0371-4500-8e77-3bb4291a2e3d-registry-tls\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 02:50:08.439361 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:50:08.439304 2575 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f540bce4-0371-4500-8e77-3bb4291a2e3d-bound-sa-token\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 02:50:08.439361 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:50:08.439315 2575 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f540bce4-0371-4500-8e77-3bb4291a2e3d-image-registry-private-configuration\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 02:50:08.439361 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:50:08.439327 2575 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f540bce4-0371-4500-8e77-3bb4291a2e3d-registry-certificates\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 02:50:08.439361 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:50:08.439337 2575 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f540bce4-0371-4500-8e77-3bb4291a2e3d-ca-trust-extracted\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 02:50:08.439361 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:50:08.439345 2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f540bce4-0371-4500-8e77-3bb4291a2e3d-trusted-ca\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 02:50:08.439361 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:50:08.439354 2575 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f540bce4-0371-4500-8e77-3bb4291a2e3d-installation-pull-secrets\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 02:50:08.439361 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:50:08.439363 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtqbp\" (UniqueName: \"kubernetes.io/projected/f540bce4-0371-4500-8e77-3bb4291a2e3d-kube-api-access-vtqbp\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 02:50:08.477801 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:50:08.477767 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-fbd7648c9-vtvdf"] Jul 10 02:50:08.481351 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:50:08.481320 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-fbd7648c9-vtvdf"] Jul 10 02:50:10.288115 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:50:10.288084 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f540bce4-0371-4500-8e77-3bb4291a2e3d" path="/var/lib/kubelet/pods/f540bce4-0371-4500-8e77-3bb4291a2e3d/volumes" Jul 10 02:53:02.184540 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:53:02.184517 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-xxlp6_372be3ce-a380-4c0e-bbee-49c2e3b80095/console-operator/1.log" Jul 10 02:53:02.189388 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:53:02.189364 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-xxlp6_372be3ce-a380-4c0e-bbee-49c2e3b80095/console-operator/1.log" Jul 10 02:53:02.200614 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:53:02.200588 2575 kubelet.go:1648] "Image garbage collection succeeded" Jul 10 02:54:27.796034 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:27.795960 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84dd56b898-rmcl2"] Jul 10 02:54:27.796462 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:27.796264 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f540bce4-0371-4500-8e77-3bb4291a2e3d" containerName="registry" Jul 10 02:54:27.796462 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:27.796275 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f540bce4-0371-4500-8e77-3bb4291a2e3d" containerName="registry" Jul 10 02:54:27.796462 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:27.796329 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f540bce4-0371-4500-8e77-3bb4291a2e3d" containerName="registry" Jul 10 02:54:27.799314 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:27.799292 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-84dd56b898-rmcl2" Jul 10 02:54:27.802554 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:27.802538 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 10 02:54:27.802920 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:27.802887 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 10 02:54:27.803104 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:27.803084 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-sgqkj\"" Jul 10 02:54:27.803233 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:27.803218 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 10 02:54:27.803385 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:27.803366 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 10 02:54:27.827487 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:27.827466 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84dd56b898-rmcl2"] Jul 10 02:54:27.829943 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:27.829919 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmz77\" (UniqueName: \"kubernetes.io/projected/9fa796aa-f442-43b7-886f-575c72f33603-kube-api-access-tmz77\") pod \"opendatahub-operator-controller-manager-84dd56b898-rmcl2\" (UID: \"9fa796aa-f442-43b7-886f-575c72f33603\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd56b898-rmcl2" Jul 10 02:54:27.830046 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:27.829981 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9fa796aa-f442-43b7-886f-575c72f33603-tmp\") pod \"opendatahub-operator-controller-manager-84dd56b898-rmcl2\" (UID: \"9fa796aa-f442-43b7-886f-575c72f33603\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd56b898-rmcl2" Jul 10 02:54:27.830046 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:27.830021 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9fa796aa-f442-43b7-886f-575c72f33603-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84dd56b898-rmcl2\" (UID: \"9fa796aa-f442-43b7-886f-575c72f33603\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd56b898-rmcl2" Jul 10 02:54:27.830046 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:27.830044 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9fa796aa-f442-43b7-886f-575c72f33603-webhook-cert\") pod \"opendatahub-operator-controller-manager-84dd56b898-rmcl2\" (UID: \"9fa796aa-f442-43b7-886f-575c72f33603\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd56b898-rmcl2" Jul 10 02:54:27.830191 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:27.830066 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9fa796aa-f442-43b7-886f-575c72f33603-manifests\") pod \"opendatahub-operator-controller-manager-84dd56b898-rmcl2\" (UID: \"9fa796aa-f442-43b7-886f-575c72f33603\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd56b898-rmcl2" Jul 10 02:54:27.930778 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:27.930743 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9fa796aa-f442-43b7-886f-575c72f33603-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84dd56b898-rmcl2\" (UID: \"9fa796aa-f442-43b7-886f-575c72f33603\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd56b898-rmcl2" Jul 10 02:54:27.931011 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:27.930787 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9fa796aa-f442-43b7-886f-575c72f33603-webhook-cert\") pod \"opendatahub-operator-controller-manager-84dd56b898-rmcl2\" (UID: \"9fa796aa-f442-43b7-886f-575c72f33603\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd56b898-rmcl2" Jul 10 02:54:27.931011 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:27.930981 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9fa796aa-f442-43b7-886f-575c72f33603-manifests\") pod \"opendatahub-operator-controller-manager-84dd56b898-rmcl2\" (UID: \"9fa796aa-f442-43b7-886f-575c72f33603\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd56b898-rmcl2" Jul 10 02:54:27.931122 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:27.931036 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmz77\" (UniqueName: \"kubernetes.io/projected/9fa796aa-f442-43b7-886f-575c72f33603-kube-api-access-tmz77\") pod \"opendatahub-operator-controller-manager-84dd56b898-rmcl2\" (UID: \"9fa796aa-f442-43b7-886f-575c72f33603\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd56b898-rmcl2" Jul 10 02:54:27.931174 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:27.931139 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9fa796aa-f442-43b7-886f-575c72f33603-tmp\") pod \"opendatahub-operator-controller-manager-84dd56b898-rmcl2\" (UID: \"9fa796aa-f442-43b7-886f-575c72f33603\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd56b898-rmcl2" Jul 10 02:54:27.931609 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:27.931582 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9fa796aa-f442-43b7-886f-575c72f33603-manifests\") pod \"opendatahub-operator-controller-manager-84dd56b898-rmcl2\" (UID: \"9fa796aa-f442-43b7-886f-575c72f33603\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd56b898-rmcl2" Jul 10 02:54:27.933354 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:27.933324 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9fa796aa-f442-43b7-886f-575c72f33603-tmp\") pod \"opendatahub-operator-controller-manager-84dd56b898-rmcl2\" (UID: \"9fa796aa-f442-43b7-886f-575c72f33603\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd56b898-rmcl2" Jul 10 02:54:27.933482 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:27.933454 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9fa796aa-f442-43b7-886f-575c72f33603-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84dd56b898-rmcl2\" (UID: \"9fa796aa-f442-43b7-886f-575c72f33603\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd56b898-rmcl2" Jul 10 02:54:27.933635 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:27.933614 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9fa796aa-f442-43b7-886f-575c72f33603-webhook-cert\") pod \"opendatahub-operator-controller-manager-84dd56b898-rmcl2\" (UID: \"9fa796aa-f442-43b7-886f-575c72f33603\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd56b898-rmcl2" Jul 10 02:54:27.942880 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:27.942827 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmz77\" (UniqueName: \"kubernetes.io/projected/9fa796aa-f442-43b7-886f-575c72f33603-kube-api-access-tmz77\") pod \"opendatahub-operator-controller-manager-84dd56b898-rmcl2\" (UID: \"9fa796aa-f442-43b7-886f-575c72f33603\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd56b898-rmcl2" Jul 10 02:54:28.109453 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:28.109359 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-84dd56b898-rmcl2" Jul 10 02:54:28.249832 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:28.249778 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84dd56b898-rmcl2"] Jul 10 02:54:28.253657 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:54:28.253623 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fa796aa_f442_43b7_886f_575c72f33603.slice/crio-a2318f3a63d4550e4bc67e3b9331a93ef8a1230f0294849ccc873afac0ff38df WatchSource:0}: Error finding container a2318f3a63d4550e4bc67e3b9331a93ef8a1230f0294849ccc873afac0ff38df: Status 404 returned error can't find the container with id a2318f3a63d4550e4bc67e3b9331a93ef8a1230f0294849ccc873afac0ff38df Jul 10 02:54:28.255369 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:28.255348 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 02:54:29.187828 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:29.187764 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84dd56b898-rmcl2" event={"ID":"9fa796aa-f442-43b7-886f-575c72f33603","Type":"ContainerStarted","Data":"a2318f3a63d4550e4bc67e3b9331a93ef8a1230f0294849ccc873afac0ff38df"} Jul 10 02:54:32.203439 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:32.203402 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84dd56b898-rmcl2" event={"ID":"9fa796aa-f442-43b7-886f-575c72f33603","Type":"ContainerStarted","Data":"602cfd56b03ab10465e19ee6400006ef5701fbcef7f660aee89f02543207af75"} Jul 10 02:54:33.211304 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:33.211258 2575 generic.go:356] "Generic (PLEG): container finished" podID="9fa796aa-f442-43b7-886f-575c72f33603" containerID="602cfd56b03ab10465e19ee6400006ef5701fbcef7f660aee89f02543207af75" exitCode=0 Jul 10 02:54:33.211304 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:33.211300 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84dd56b898-rmcl2" event={"ID":"9fa796aa-f442-43b7-886f-575c72f33603","Type":"ContainerDied","Data":"602cfd56b03ab10465e19ee6400006ef5701fbcef7f660aee89f02543207af75"} Jul 10 02:54:34.217328 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:34.217284 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84dd56b898-rmcl2" event={"ID":"9fa796aa-f442-43b7-886f-575c72f33603","Type":"ContainerStarted","Data":"4a241bc8ea20b4a621707b37542ca10bc72a4b3d5298e29c7c092b64d4adef76"} Jul 10 02:54:34.217752 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:34.217393 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-84dd56b898-rmcl2" Jul 10 02:54:34.238332 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:34.238283 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-84dd56b898-rmcl2" podStartSLOduration=2.816636151 podStartE2EDuration="7.238268789s" totalImagesPullingTime="3.403851983s" totalInitContainerRuntime="1.017780655s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:54:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:54:34.236504578 +0000 UTC m=+392.517620907" watchObservedRunningTime="2026-07-10 02:54:34.238268789 +0000 UTC m=+392.519385074" Jul 10 02:54:45.224200 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:45.224163 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-84dd56b898-rmcl2" Jul 10 02:54:47.610032 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:47.609999 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-864dd46fbc-vmnv7"] Jul 10 02:54:47.623461 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:47.623439 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-864dd46fbc-vmnv7" Jul 10 02:54:47.628382 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:47.628358 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 10 02:54:47.628536 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:47.628518 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-cmc46\"" Jul 10 02:54:47.628682 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:47.628659 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 10 02:54:47.637349 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:47.637326 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-864dd46fbc-vmnv7"] Jul 10 02:54:47.703171 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:47.703122 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74e98eb8-2d91-4e17-8be5-01a99f1d07b0-tmp\") pod \"kube-auth-proxy-864dd46fbc-vmnv7\" (UID: \"74e98eb8-2d91-4e17-8be5-01a99f1d07b0\") " pod="openshift-ingress/kube-auth-proxy-864dd46fbc-vmnv7" Jul 10 02:54:47.703334 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:47.703195 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87bcf\" (UniqueName: \"kubernetes.io/projected/74e98eb8-2d91-4e17-8be5-01a99f1d07b0-kube-api-access-87bcf\") pod \"kube-auth-proxy-864dd46fbc-vmnv7\" (UID: \"74e98eb8-2d91-4e17-8be5-01a99f1d07b0\") " pod="openshift-ingress/kube-auth-proxy-864dd46fbc-vmnv7" Jul 10 02:54:47.703334 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:47.703219 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74e98eb8-2d91-4e17-8be5-01a99f1d07b0-tls-certs\") pod \"kube-auth-proxy-864dd46fbc-vmnv7\" (UID: \"74e98eb8-2d91-4e17-8be5-01a99f1d07b0\") " pod="openshift-ingress/kube-auth-proxy-864dd46fbc-vmnv7" Jul 10 02:54:47.804128 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:47.804094 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74e98eb8-2d91-4e17-8be5-01a99f1d07b0-tmp\") pod \"kube-auth-proxy-864dd46fbc-vmnv7\" (UID: \"74e98eb8-2d91-4e17-8be5-01a99f1d07b0\") " pod="openshift-ingress/kube-auth-proxy-864dd46fbc-vmnv7" Jul 10 02:54:47.804272 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:47.804156 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87bcf\" (UniqueName: \"kubernetes.io/projected/74e98eb8-2d91-4e17-8be5-01a99f1d07b0-kube-api-access-87bcf\") pod \"kube-auth-proxy-864dd46fbc-vmnv7\" (UID: \"74e98eb8-2d91-4e17-8be5-01a99f1d07b0\") " pod="openshift-ingress/kube-auth-proxy-864dd46fbc-vmnv7" Jul 10 02:54:47.804272 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:47.804187 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74e98eb8-2d91-4e17-8be5-01a99f1d07b0-tls-certs\") pod \"kube-auth-proxy-864dd46fbc-vmnv7\" (UID: \"74e98eb8-2d91-4e17-8be5-01a99f1d07b0\") " pod="openshift-ingress/kube-auth-proxy-864dd46fbc-vmnv7" Jul 10 02:54:47.806290 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:47.806259 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74e98eb8-2d91-4e17-8be5-01a99f1d07b0-tmp\") pod \"kube-auth-proxy-864dd46fbc-vmnv7\" (UID: \"74e98eb8-2d91-4e17-8be5-01a99f1d07b0\") " pod="openshift-ingress/kube-auth-proxy-864dd46fbc-vmnv7" Jul 10 02:54:47.806607 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:47.806584 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74e98eb8-2d91-4e17-8be5-01a99f1d07b0-tls-certs\") pod \"kube-auth-proxy-864dd46fbc-vmnv7\" (UID: \"74e98eb8-2d91-4e17-8be5-01a99f1d07b0\") " pod="openshift-ingress/kube-auth-proxy-864dd46fbc-vmnv7" Jul 10 02:54:47.816753 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:47.816731 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87bcf\" (UniqueName: \"kubernetes.io/projected/74e98eb8-2d91-4e17-8be5-01a99f1d07b0-kube-api-access-87bcf\") pod \"kube-auth-proxy-864dd46fbc-vmnv7\" (UID: \"74e98eb8-2d91-4e17-8be5-01a99f1d07b0\") " pod="openshift-ingress/kube-auth-proxy-864dd46fbc-vmnv7" Jul 10 02:54:47.891167 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:47.891089 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-hkdfx"] Jul 10 02:54:47.894535 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:47.894519 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-hkdfx" Jul 10 02:54:47.897546 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:47.897523 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-lt964\"" Jul 10 02:54:47.897720 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:47.897705 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 10 02:54:47.907391 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:47.907369 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-hkdfx"] Jul 10 02:54:47.933111 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:47.933089 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-864dd46fbc-vmnv7" Jul 10 02:54:48.005637 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:48.005602 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwx8v\" (UniqueName: \"kubernetes.io/projected/4447dd2a-7547-4bbc-8738-7db73cc8c40f-kube-api-access-kwx8v\") pod \"odh-model-controller-7998c67dc4-hkdfx\" (UID: \"4447dd2a-7547-4bbc-8738-7db73cc8c40f\") " pod="opendatahub/odh-model-controller-7998c67dc4-hkdfx" Jul 10 02:54:48.005794 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:48.005660 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4447dd2a-7547-4bbc-8738-7db73cc8c40f-cert\") pod \"odh-model-controller-7998c67dc4-hkdfx\" (UID: \"4447dd2a-7547-4bbc-8738-7db73cc8c40f\") " pod="opendatahub/odh-model-controller-7998c67dc4-hkdfx" Jul 10 02:54:48.072034 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:48.071996 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-864dd46fbc-vmnv7"] Jul 10 02:54:48.075124 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:54:48.075098 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74e98eb8_2d91_4e17_8be5_01a99f1d07b0.slice/crio-9bbcf9e64332f42267c026ea4fa675e788cfd8d9fea0697417ddeca863becb99 WatchSource:0}: Error finding container 9bbcf9e64332f42267c026ea4fa675e788cfd8d9fea0697417ddeca863becb99: Status 404 returned error can't find the container with id 9bbcf9e64332f42267c026ea4fa675e788cfd8d9fea0697417ddeca863becb99 Jul 10 02:54:48.107149 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:48.107122 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwx8v\" (UniqueName: \"kubernetes.io/projected/4447dd2a-7547-4bbc-8738-7db73cc8c40f-kube-api-access-kwx8v\") pod \"odh-model-controller-7998c67dc4-hkdfx\" (UID: \"4447dd2a-7547-4bbc-8738-7db73cc8c40f\") " pod="opendatahub/odh-model-controller-7998c67dc4-hkdfx" Jul 10 02:54:48.107241 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:48.107164 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4447dd2a-7547-4bbc-8738-7db73cc8c40f-cert\") pod \"odh-model-controller-7998c67dc4-hkdfx\" (UID: \"4447dd2a-7547-4bbc-8738-7db73cc8c40f\") " pod="opendatahub/odh-model-controller-7998c67dc4-hkdfx" Jul 10 02:54:48.107289 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:54:48.107265 2575 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 10 02:54:48.107333 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:54:48.107314 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4447dd2a-7547-4bbc-8738-7db73cc8c40f-cert podName:4447dd2a-7547-4bbc-8738-7db73cc8c40f nodeName:}" failed. No retries permitted until 2026-07-10 02:54:48.607297096 +0000 UTC m=+406.888413359 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/4447dd2a-7547-4bbc-8738-7db73cc8c40f-cert") pod "odh-model-controller-7998c67dc4-hkdfx" (UID: "4447dd2a-7547-4bbc-8738-7db73cc8c40f") : secret "odh-model-controller-webhook-cert" not found Jul 10 02:54:48.121963 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:48.121933 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwx8v\" (UniqueName: \"kubernetes.io/projected/4447dd2a-7547-4bbc-8738-7db73cc8c40f-kube-api-access-kwx8v\") pod \"odh-model-controller-7998c67dc4-hkdfx\" (UID: \"4447dd2a-7547-4bbc-8738-7db73cc8c40f\") " pod="opendatahub/odh-model-controller-7998c67dc4-hkdfx" Jul 10 02:54:48.277905 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:48.277866 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-864dd46fbc-vmnv7" event={"ID":"74e98eb8-2d91-4e17-8be5-01a99f1d07b0","Type":"ContainerStarted","Data":"9bbcf9e64332f42267c026ea4fa675e788cfd8d9fea0697417ddeca863becb99"} Jul 10 02:54:48.611524 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:48.611441 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4447dd2a-7547-4bbc-8738-7db73cc8c40f-cert\") pod \"odh-model-controller-7998c67dc4-hkdfx\" (UID: \"4447dd2a-7547-4bbc-8738-7db73cc8c40f\") " pod="opendatahub/odh-model-controller-7998c67dc4-hkdfx" Jul 10 02:54:48.613915 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:48.613890 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4447dd2a-7547-4bbc-8738-7db73cc8c40f-cert\") pod \"odh-model-controller-7998c67dc4-hkdfx\" (UID: \"4447dd2a-7547-4bbc-8738-7db73cc8c40f\") " pod="opendatahub/odh-model-controller-7998c67dc4-hkdfx" Jul 10 02:54:48.805123 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:48.805070 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-hkdfx" Jul 10 02:54:48.881398 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:48.881362 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-6xkl7"] Jul 10 02:54:48.885337 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:48.885310 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-6xkl7" Jul 10 02:54:48.887952 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:48.887740 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 10 02:54:48.887952 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:48.887799 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-qzhzs\"" Jul 10 02:54:48.894246 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:48.894190 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-6xkl7"] Jul 10 02:54:48.961302 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:48.961034 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-hkdfx"] Jul 10 02:54:48.964254 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:54:48.964222 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4447dd2a_7547_4bbc_8738_7db73cc8c40f.slice/crio-d303cf33abe78aad5014a6d0d01868de62af47ea182ac807792769884c9e203a WatchSource:0}: Error finding container d303cf33abe78aad5014a6d0d01868de62af47ea182ac807792769884c9e203a: Status 404 returned error can't find the container with id d303cf33abe78aad5014a6d0d01868de62af47ea182ac807792769884c9e203a Jul 10 02:54:49.015007 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:49.014979 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfc95468-c766-4463-9b94-ad59662147e6-tls-certs\") pod \"model-serving-api-54d89bd5fc-6xkl7\" (UID: \"dfc95468-c766-4463-9b94-ad59662147e6\") " pod="opendatahub/model-serving-api-54d89bd5fc-6xkl7" Jul 10 02:54:49.015176 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:49.015021 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwpxq\" (UniqueName: \"kubernetes.io/projected/dfc95468-c766-4463-9b94-ad59662147e6-kube-api-access-vwpxq\") pod \"model-serving-api-54d89bd5fc-6xkl7\" (UID: \"dfc95468-c766-4463-9b94-ad59662147e6\") " pod="opendatahub/model-serving-api-54d89bd5fc-6xkl7" Jul 10 02:54:49.116019 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:49.115947 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfc95468-c766-4463-9b94-ad59662147e6-tls-certs\") pod \"model-serving-api-54d89bd5fc-6xkl7\" (UID: \"dfc95468-c766-4463-9b94-ad59662147e6\") " pod="opendatahub/model-serving-api-54d89bd5fc-6xkl7" Jul 10 02:54:49.116019 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:49.115988 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwpxq\" (UniqueName: \"kubernetes.io/projected/dfc95468-c766-4463-9b94-ad59662147e6-kube-api-access-vwpxq\") pod \"model-serving-api-54d89bd5fc-6xkl7\" (UID: \"dfc95468-c766-4463-9b94-ad59662147e6\") " pod="opendatahub/model-serving-api-54d89bd5fc-6xkl7" Jul 10 02:54:49.120449 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:49.118997 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfc95468-c766-4463-9b94-ad59662147e6-tls-certs\") pod \"model-serving-api-54d89bd5fc-6xkl7\" (UID: \"dfc95468-c766-4463-9b94-ad59662147e6\") " pod="opendatahub/model-serving-api-54d89bd5fc-6xkl7" Jul 10 02:54:49.124481 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:49.122233 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-kkjmc"] Jul 10 02:54:49.127094 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:49.127058 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kkjmc" Jul 10 02:54:49.127395 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:49.127373 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwpxq\" (UniqueName: \"kubernetes.io/projected/dfc95468-c766-4463-9b94-ad59662147e6-kube-api-access-vwpxq\") pod \"model-serving-api-54d89bd5fc-6xkl7\" (UID: \"dfc95468-c766-4463-9b94-ad59662147e6\") " pod="opendatahub/model-serving-api-54d89bd5fc-6xkl7" Jul 10 02:54:49.129648 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:49.129627 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 10 02:54:49.129648 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:49.129630 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-gg7fm\"" Jul 10 02:54:49.129822 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:49.129628 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 10 02:54:49.129822 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:49.129628 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 10 02:54:49.129822 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:49.129628 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 10 02:54:49.136459 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:49.136437 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-kkjmc"] Jul 10 02:54:49.200474 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:49.200444 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-6xkl7" Jul 10 02:54:49.217378 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:49.217347 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv9tf\" (UniqueName: \"kubernetes.io/projected/db3a1676-436b-4204-8c1f-c8717be0ec7a-kube-api-access-hv9tf\") pod \"istiod-openshift-gateway-7574459656-kkjmc\" (UID: \"db3a1676-436b-4204-8c1f-c8717be0ec7a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kkjmc" Jul 10 02:54:49.217504 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:49.217399 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/db3a1676-436b-4204-8c1f-c8717be0ec7a-local-certs\") pod \"istiod-openshift-gateway-7574459656-kkjmc\" (UID: \"db3a1676-436b-4204-8c1f-c8717be0ec7a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kkjmc" Jul 10 02:54:49.217504 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:49.217493 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/db3a1676-436b-4204-8c1f-c8717be0ec7a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-kkjmc\" (UID: \"db3a1676-436b-4204-8c1f-c8717be0ec7a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kkjmc" Jul 10 02:54:49.217611 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:49.217545 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/db3a1676-436b-4204-8c1f-c8717be0ec7a-cacerts\") pod \"istiod-openshift-gateway-7574459656-kkjmc\" (UID: \"db3a1676-436b-4204-8c1f-c8717be0ec7a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kkjmc" Jul 10 02:54:49.217611 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:49.217594 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/db3a1676-436b-4204-8c1f-c8717be0ec7a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-kkjmc\" (UID: \"db3a1676-436b-4204-8c1f-c8717be0ec7a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kkjmc" Jul 10 02:54:49.217719 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:49.217617 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/db3a1676-436b-4204-8c1f-c8717be0ec7a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-kkjmc\" (UID: \"db3a1676-436b-4204-8c1f-c8717be0ec7a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kkjmc" Jul 10 02:54:49.217719 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:49.217654 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db3a1676-436b-4204-8c1f-c8717be0ec7a-istio-token\") pod \"istiod-openshift-gateway-7574459656-kkjmc\" (UID: \"db3a1676-436b-4204-8c1f-c8717be0ec7a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kkjmc" Jul 10 02:54:49.282879 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:49.282837 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-hkdfx" event={"ID":"4447dd2a-7547-4bbc-8738-7db73cc8c40f","Type":"ContainerStarted","Data":"d303cf33abe78aad5014a6d0d01868de62af47ea182ac807792769884c9e203a"} Jul 10 02:54:49.318102 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:49.318066 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv9tf\" (UniqueName: \"kubernetes.io/projected/db3a1676-436b-4204-8c1f-c8717be0ec7a-kube-api-access-hv9tf\") pod \"istiod-openshift-gateway-7574459656-kkjmc\" (UID: \"db3a1676-436b-4204-8c1f-c8717be0ec7a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kkjmc" Jul 10 02:54:49.318271 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:49.318116 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/db3a1676-436b-4204-8c1f-c8717be0ec7a-local-certs\") pod \"istiod-openshift-gateway-7574459656-kkjmc\" (UID: \"db3a1676-436b-4204-8c1f-c8717be0ec7a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kkjmc" Jul 10 02:54:49.318271 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:49.318169 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/db3a1676-436b-4204-8c1f-c8717be0ec7a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-kkjmc\" (UID: \"db3a1676-436b-4204-8c1f-c8717be0ec7a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kkjmc" Jul 10 02:54:49.318379 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:49.318318 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/db3a1676-436b-4204-8c1f-c8717be0ec7a-cacerts\") pod \"istiod-openshift-gateway-7574459656-kkjmc\" (UID: \"db3a1676-436b-4204-8c1f-c8717be0ec7a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kkjmc" Jul 10 02:54:49.318433 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:49.318379 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/db3a1676-436b-4204-8c1f-c8717be0ec7a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-kkjmc\" (UID: \"db3a1676-436b-4204-8c1f-c8717be0ec7a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kkjmc" Jul 10 02:54:49.318433 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:49.318416 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/db3a1676-436b-4204-8c1f-c8717be0ec7a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-kkjmc\" (UID: \"db3a1676-436b-4204-8c1f-c8717be0ec7a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kkjmc" Jul 10 02:54:49.318538 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:49.318464 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db3a1676-436b-4204-8c1f-c8717be0ec7a-istio-token\") pod \"istiod-openshift-gateway-7574459656-kkjmc\" (UID: \"db3a1676-436b-4204-8c1f-c8717be0ec7a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kkjmc" Jul 10 02:54:49.319286 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:49.319231 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/db3a1676-436b-4204-8c1f-c8717be0ec7a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-kkjmc\" (UID: \"db3a1676-436b-4204-8c1f-c8717be0ec7a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kkjmc" Jul 10 02:54:49.321299 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:49.321266 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/db3a1676-436b-4204-8c1f-c8717be0ec7a-cacerts\") pod \"istiod-openshift-gateway-7574459656-kkjmc\" (UID: \"db3a1676-436b-4204-8c1f-c8717be0ec7a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kkjmc" Jul 10 02:54:49.321509 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:49.321485 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/db3a1676-436b-4204-8c1f-c8717be0ec7a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-kkjmc\" (UID: \"db3a1676-436b-4204-8c1f-c8717be0ec7a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kkjmc" Jul 10 02:54:49.321862 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:49.321837 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/db3a1676-436b-4204-8c1f-c8717be0ec7a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-kkjmc\" (UID: \"db3a1676-436b-4204-8c1f-c8717be0ec7a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kkjmc" Jul 10 02:54:49.322482 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:49.322460 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/db3a1676-436b-4204-8c1f-c8717be0ec7a-local-certs\") pod \"istiod-openshift-gateway-7574459656-kkjmc\" (UID: \"db3a1676-436b-4204-8c1f-c8717be0ec7a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kkjmc" Jul 10 02:54:49.327141 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:49.327117 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db3a1676-436b-4204-8c1f-c8717be0ec7a-istio-token\") pod \"istiod-openshift-gateway-7574459656-kkjmc\" (UID: \"db3a1676-436b-4204-8c1f-c8717be0ec7a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kkjmc" Jul 10 02:54:49.327714 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:49.327695 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv9tf\" (UniqueName: \"kubernetes.io/projected/db3a1676-436b-4204-8c1f-c8717be0ec7a-kube-api-access-hv9tf\") pod \"istiod-openshift-gateway-7574459656-kkjmc\" (UID: \"db3a1676-436b-4204-8c1f-c8717be0ec7a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-kkjmc" Jul 10 02:54:49.345735 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:49.345595 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-6xkl7"] Jul 10 02:54:49.348968 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:54:49.348789 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfc95468_c766_4463_9b94_ad59662147e6.slice/crio-69a1ac780bc61699ced052d90bca7ff3676b66da72d282cd0df786bcae8c9d35 WatchSource:0}: Error finding container 69a1ac780bc61699ced052d90bca7ff3676b66da72d282cd0df786bcae8c9d35: Status 404 returned error can't find the container with id 69a1ac780bc61699ced052d90bca7ff3676b66da72d282cd0df786bcae8c9d35 Jul 10 02:54:49.444339 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:49.444263 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kkjmc" Jul 10 02:54:49.587386 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:49.587341 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-kkjmc"] Jul 10 02:54:49.590776 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:54:49.590734 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb3a1676_436b_4204_8c1f_c8717be0ec7a.slice/crio-b02937ad1b7a940bfcb29e2f3bb9e54d922f425ea1721ddea71e8dfdf18817ee WatchSource:0}: Error finding container b02937ad1b7a940bfcb29e2f3bb9e54d922f425ea1721ddea71e8dfdf18817ee: Status 404 returned error can't find the container with id b02937ad1b7a940bfcb29e2f3bb9e54d922f425ea1721ddea71e8dfdf18817ee Jul 10 02:54:50.297145 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:50.297068 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kkjmc" event={"ID":"db3a1676-436b-4204-8c1f-c8717be0ec7a","Type":"ContainerStarted","Data":"b02937ad1b7a940bfcb29e2f3bb9e54d922f425ea1721ddea71e8dfdf18817ee"} Jul 10 02:54:50.299694 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:50.299642 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-6xkl7" event={"ID":"dfc95468-c766-4463-9b94-ad59662147e6","Type":"ContainerStarted","Data":"69a1ac780bc61699ced052d90bca7ff3676b66da72d282cd0df786bcae8c9d35"} Jul 10 02:54:53.887324 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:53.887289 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236236Ki","pods":"250"} Jul 10 02:54:53.887629 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:53.887364 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236236Ki","pods":"250"} Jul 10 02:54:54.321417 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:54.321378 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-hkdfx" event={"ID":"4447dd2a-7547-4bbc-8738-7db73cc8c40f","Type":"ContainerStarted","Data":"b9fd5c311b5c25600b2dbb0910b8467b88f69d7bdf7c63d6d69d7cbdd632c8d5"} Jul 10 02:54:54.321590 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:54.321572 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-hkdfx" Jul 10 02:54:54.322791 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:54.322762 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-864dd46fbc-vmnv7" event={"ID":"74e98eb8-2d91-4e17-8be5-01a99f1d07b0","Type":"ContainerStarted","Data":"db0fc39a2ad100d288b9932d9e3caaa41bd1b87d0ef4fe66fd841c64c53286bf"} Jul 10 02:54:54.324569 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:54.324539 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kkjmc" event={"ID":"db3a1676-436b-4204-8c1f-c8717be0ec7a","Type":"ContainerStarted","Data":"b348b313d2596fbbf2f630091c8dc37f871317c3b20fd01401113d69918d307e"} Jul 10 02:54:54.324719 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:54.324678 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kkjmc" Jul 10 02:54:54.326313 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:54.326256 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-6xkl7" event={"ID":"dfc95468-c766-4463-9b94-ad59662147e6","Type":"ContainerStarted","Data":"eac97209a785d7e25789e2e0f8de9fa5e94d602397d9fd44786956a39f490519"} Jul 10 02:54:54.326493 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:54.326385 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-6xkl7" Jul 10 02:54:54.326879 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:54.326854 2575 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-kkjmc container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 10 02:54:54.326964 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:54.326911 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kkjmc" podUID="db3a1676-436b-4204-8c1f-c8717be0ec7a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 02:54:54.340584 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:54.340540 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-hkdfx" podStartSLOduration=2.466609728 podStartE2EDuration="7.340524611s" totalImagesPullingTime="4.873914883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:54:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:54:54.338622365 +0000 UTC m=+412.619738677" watchObservedRunningTime="2026-07-10 02:54:54.340524611 +0000 UTC m=+412.621640898" Jul 10 02:54:54.360537 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:54.360465 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-864dd46fbc-vmnv7" podStartSLOduration=2.477166414 podStartE2EDuration="7.360448645s" totalImagesPullingTime="4.883282231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:54:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:54:54.35772742 +0000 UTC m=+412.638843723" watchObservedRunningTime="2026-07-10 02:54:54.360448645 +0000 UTC m=+412.641564930" Jul 10 02:54:54.381946 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:54.381851 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-6xkl7" podStartSLOduration=2.7727778450000002 podStartE2EDuration="6.381828467s" totalImagesPullingTime="3.609050622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:54:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:54:54.381048655 +0000 UTC m=+412.662164941" watchObservedRunningTime="2026-07-10 02:54:54.381828467 +0000 UTC m=+412.662944751" Jul 10 02:54:54.414459 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:54.414399 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kkjmc" podStartSLOduration=1.120697416 podStartE2EDuration="5.414382061s" totalImagesPullingTime="4.293684645s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:54:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:54:54.412518489 +0000 UTC m=+412.693634775" watchObservedRunningTime="2026-07-10 02:54:54.414382061 +0000 UTC m=+412.695498347" Jul 10 02:54:55.330937 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:55.330908 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kkjmc" Jul 10 02:54:58.053423 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:58.053387 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-r987c"] Jul 10 02:54:58.057163 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:58.057145 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-r987c" Jul 10 02:54:58.059964 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:58.059941 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 10 02:54:58.060093 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:58.060062 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-gqxnb\"" Jul 10 02:54:58.071648 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:58.071625 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-r987c"] Jul 10 02:54:58.206168 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:58.206122 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0fb37c8b-e397-4f28-8dee-2b7ee86976d2-cert\") pod \"kserve-controller-manager-d6f4b65b8-r987c\" (UID: \"0fb37c8b-e397-4f28-8dee-2b7ee86976d2\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-r987c" Jul 10 02:54:58.206368 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:58.206260 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8svf4\" (UniqueName: \"kubernetes.io/projected/0fb37c8b-e397-4f28-8dee-2b7ee86976d2-kube-api-access-8svf4\") pod \"kserve-controller-manager-d6f4b65b8-r987c\" (UID: \"0fb37c8b-e397-4f28-8dee-2b7ee86976d2\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-r987c" Jul 10 02:54:58.307141 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:58.307051 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8svf4\" (UniqueName: \"kubernetes.io/projected/0fb37c8b-e397-4f28-8dee-2b7ee86976d2-kube-api-access-8svf4\") pod \"kserve-controller-manager-d6f4b65b8-r987c\" (UID: \"0fb37c8b-e397-4f28-8dee-2b7ee86976d2\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-r987c" Jul 10 02:54:58.307141 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:58.307103 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0fb37c8b-e397-4f28-8dee-2b7ee86976d2-cert\") pod \"kserve-controller-manager-d6f4b65b8-r987c\" (UID: \"0fb37c8b-e397-4f28-8dee-2b7ee86976d2\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-r987c" Jul 10 02:54:58.309481 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:58.309459 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0fb37c8b-e397-4f28-8dee-2b7ee86976d2-cert\") pod \"kserve-controller-manager-d6f4b65b8-r987c\" (UID: \"0fb37c8b-e397-4f28-8dee-2b7ee86976d2\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-r987c" Jul 10 02:54:58.317904 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:58.317876 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8svf4\" (UniqueName: \"kubernetes.io/projected/0fb37c8b-e397-4f28-8dee-2b7ee86976d2-kube-api-access-8svf4\") pod \"kserve-controller-manager-d6f4b65b8-r987c\" (UID: \"0fb37c8b-e397-4f28-8dee-2b7ee86976d2\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-r987c" Jul 10 02:54:58.367536 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:58.367511 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-r987c" Jul 10 02:54:58.505605 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:58.505575 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-r987c"] Jul 10 02:54:58.508266 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:54:58.508235 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fb37c8b_e397_4f28_8dee_2b7ee86976d2.slice/crio-985a43547dd15d67553e7da6a451a75eb3a0f83f23f6de9ba226dd46b4367faf WatchSource:0}: Error finding container 985a43547dd15d67553e7da6a451a75eb3a0f83f23f6de9ba226dd46b4367faf: Status 404 returned error can't find the container with id 985a43547dd15d67553e7da6a451a75eb3a0f83f23f6de9ba226dd46b4367faf Jul 10 02:54:59.348478 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:54:59.348434 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-r987c" event={"ID":"0fb37c8b-e397-4f28-8dee-2b7ee86976d2","Type":"ContainerStarted","Data":"985a43547dd15d67553e7da6a451a75eb3a0f83f23f6de9ba226dd46b4367faf"} Jul 10 02:55:01.140413 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:01.140371 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-kkjmc"] Jul 10 02:55:01.140874 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:01.140609 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kkjmc" podUID="db3a1676-436b-4204-8c1f-c8717be0ec7a" containerName="discovery" containerID="cri-o://b348b313d2596fbbf2f630091c8dc37f871317c3b20fd01401113d69918d307e" gracePeriod=30 Jul 10 02:55:01.359869 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:01.359829 2575 generic.go:356] "Generic (PLEG): container finished" podID="db3a1676-436b-4204-8c1f-c8717be0ec7a" containerID="b348b313d2596fbbf2f630091c8dc37f871317c3b20fd01401113d69918d307e" exitCode=0 Jul 10 02:55:01.360049 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:01.359838 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kkjmc" event={"ID":"db3a1676-436b-4204-8c1f-c8717be0ec7a","Type":"ContainerDied","Data":"b348b313d2596fbbf2f630091c8dc37f871317c3b20fd01401113d69918d307e"} Jul 10 02:55:01.361446 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:01.361415 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-r987c" event={"ID":"0fb37c8b-e397-4f28-8dee-2b7ee86976d2","Type":"ContainerStarted","Data":"43dc6519c7c6632291970fb3cd709bf50e2e076624c20da034e5932ec8020bae"} Jul 10 02:55:01.361570 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:01.361508 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-r987c" Jul 10 02:55:01.379645 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:01.379602 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-r987c" podStartSLOduration=0.880660396 podStartE2EDuration="3.379579574s" totalImagesPullingTime="2.498919178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:54:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:55:01.377974622 +0000 UTC m=+419.659090909" watchObservedRunningTime="2026-07-10 02:55:01.379579574 +0000 UTC m=+419.660695859" Jul 10 02:55:01.395519 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:01.395491 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kkjmc" Jul 10 02:55:01.536033 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:01.536001 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/db3a1676-436b-4204-8c1f-c8717be0ec7a-cacerts\") pod \"db3a1676-436b-4204-8c1f-c8717be0ec7a\" (UID: \"db3a1676-436b-4204-8c1f-c8717be0ec7a\") " Jul 10 02:55:01.536206 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:01.536043 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db3a1676-436b-4204-8c1f-c8717be0ec7a-istio-token\") pod \"db3a1676-436b-4204-8c1f-c8717be0ec7a\" (UID: \"db3a1676-436b-4204-8c1f-c8717be0ec7a\") " Jul 10 02:55:01.536206 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:01.536085 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/db3a1676-436b-4204-8c1f-c8717be0ec7a-istio-csr-dns-cert\") pod \"db3a1676-436b-4204-8c1f-c8717be0ec7a\" (UID: \"db3a1676-436b-4204-8c1f-c8717be0ec7a\") " Jul 10 02:55:01.536206 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:01.536108 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/db3a1676-436b-4204-8c1f-c8717be0ec7a-local-certs\") pod \"db3a1676-436b-4204-8c1f-c8717be0ec7a\" (UID: \"db3a1676-436b-4204-8c1f-c8717be0ec7a\") " Jul 10 02:55:01.536206 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:01.536126 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/db3a1676-436b-4204-8c1f-c8717be0ec7a-istio-kubeconfig\") pod \"db3a1676-436b-4204-8c1f-c8717be0ec7a\" (UID: \"db3a1676-436b-4204-8c1f-c8717be0ec7a\") " Jul 10 02:55:01.536206 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:01.536171 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hv9tf\" (UniqueName: \"kubernetes.io/projected/db3a1676-436b-4204-8c1f-c8717be0ec7a-kube-api-access-hv9tf\") pod \"db3a1676-436b-4204-8c1f-c8717be0ec7a\" (UID: \"db3a1676-436b-4204-8c1f-c8717be0ec7a\") " Jul 10 02:55:01.536464 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:01.536219 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/db3a1676-436b-4204-8c1f-c8717be0ec7a-istio-csr-ca-configmap\") pod \"db3a1676-436b-4204-8c1f-c8717be0ec7a\" (UID: \"db3a1676-436b-4204-8c1f-c8717be0ec7a\") " Jul 10 02:55:01.536961 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:01.536882 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db3a1676-436b-4204-8c1f-c8717be0ec7a-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "db3a1676-436b-4204-8c1f-c8717be0ec7a" (UID: "db3a1676-436b-4204-8c1f-c8717be0ec7a"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 02:55:01.538508 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:01.538477 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db3a1676-436b-4204-8c1f-c8717be0ec7a-istio-token" (OuterVolumeSpecName: "istio-token") pod "db3a1676-436b-4204-8c1f-c8717be0ec7a" (UID: "db3a1676-436b-4204-8c1f-c8717be0ec7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 02:55:01.538628 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:01.538583 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db3a1676-436b-4204-8c1f-c8717be0ec7a-kube-api-access-hv9tf" (OuterVolumeSpecName: "kube-api-access-hv9tf") pod "db3a1676-436b-4204-8c1f-c8717be0ec7a" (UID: "db3a1676-436b-4204-8c1f-c8717be0ec7a"). InnerVolumeSpecName "kube-api-access-hv9tf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 02:55:01.538713 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:01.538688 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db3a1676-436b-4204-8c1f-c8717be0ec7a-cacerts" (OuterVolumeSpecName: "cacerts") pod "db3a1676-436b-4204-8c1f-c8717be0ec7a" (UID: "db3a1676-436b-4204-8c1f-c8717be0ec7a"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 02:55:01.538833 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:01.538708 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db3a1676-436b-4204-8c1f-c8717be0ec7a-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "db3a1676-436b-4204-8c1f-c8717be0ec7a" (UID: "db3a1676-436b-4204-8c1f-c8717be0ec7a"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 02:55:01.538833 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:01.538741 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db3a1676-436b-4204-8c1f-c8717be0ec7a-local-certs" (OuterVolumeSpecName: "local-certs") pod "db3a1676-436b-4204-8c1f-c8717be0ec7a" (UID: "db3a1676-436b-4204-8c1f-c8717be0ec7a"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 02:55:01.538948 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:01.538928 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db3a1676-436b-4204-8c1f-c8717be0ec7a-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "db3a1676-436b-4204-8c1f-c8717be0ec7a" (UID: "db3a1676-436b-4204-8c1f-c8717be0ec7a"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 02:55:01.637368 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:01.637288 2575 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/db3a1676-436b-4204-8c1f-c8717be0ec7a-istio-csr-ca-configmap\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 02:55:01.637368 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:01.637313 2575 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/db3a1676-436b-4204-8c1f-c8717be0ec7a-cacerts\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 02:55:01.637368 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:01.637324 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db3a1676-436b-4204-8c1f-c8717be0ec7a-istio-token\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 02:55:01.637368 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:01.637333 2575 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/db3a1676-436b-4204-8c1f-c8717be0ec7a-istio-csr-dns-cert\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 02:55:01.637368 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:01.637343 2575 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/db3a1676-436b-4204-8c1f-c8717be0ec7a-local-certs\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 02:55:01.637368 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:01.637352 2575 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/db3a1676-436b-4204-8c1f-c8717be0ec7a-istio-kubeconfig\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 02:55:01.637368 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:01.637360 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv9tf\" (UniqueName: \"kubernetes.io/projected/db3a1676-436b-4204-8c1f-c8717be0ec7a-kube-api-access-hv9tf\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 02:55:02.366399 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:02.366361 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kkjmc" Jul 10 02:55:02.366876 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:02.366397 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kkjmc" event={"ID":"db3a1676-436b-4204-8c1f-c8717be0ec7a","Type":"ContainerDied","Data":"b02937ad1b7a940bfcb29e2f3bb9e54d922f425ea1721ddea71e8dfdf18817ee"} Jul 10 02:55:02.366876 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:02.366452 2575 scope.go:117] "RemoveContainer" containerID="b348b313d2596fbbf2f630091c8dc37f871317c3b20fd01401113d69918d307e" Jul 10 02:55:02.389995 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:02.389962 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-kkjmc"] Jul 10 02:55:02.397463 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:02.397423 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-kkjmc"] Jul 10 02:55:04.288135 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:04.288100 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db3a1676-436b-4204-8c1f-c8717be0ec7a" path="/var/lib/kubelet/pods/db3a1676-436b-4204-8c1f-c8717be0ec7a/volumes" Jul 10 02:55:05.332784 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:05.332755 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-hkdfx" Jul 10 02:55:05.335555 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:05.335536 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-6xkl7" Jul 10 02:55:06.330988 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:06.330948 2575 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-kkjmc container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.31:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 10 02:55:06.331186 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:06.331013 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-kkjmc" podUID="db3a1676-436b-4204-8c1f-c8717be0ec7a" containerName="discovery" probeResult="failure" output="Get \"http://10.132.0.31:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 10 02:55:32.372193 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:32.372122 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-r987c" Jul 10 02:55:48.378033 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:48.377995 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lx7vl"] Jul 10 02:55:48.378458 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:48.378336 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db3a1676-436b-4204-8c1f-c8717be0ec7a" containerName="discovery" Jul 10 02:55:48.378458 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:48.378349 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="db3a1676-436b-4204-8c1f-c8717be0ec7a" containerName="discovery" Jul 10 02:55:48.378458 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:48.378411 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="db3a1676-436b-4204-8c1f-c8717be0ec7a" containerName="discovery" Jul 10 02:55:48.383448 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:48.383421 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-lx7vl" Jul 10 02:55:48.385710 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:48.385685 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 10 02:55:48.386562 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:48.386542 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 10 02:55:48.386685 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:48.386597 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-db9q9\"" Jul 10 02:55:48.390712 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:48.390464 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lx7vl"] Jul 10 02:55:48.493772 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:48.493745 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7b85\" (UniqueName: \"kubernetes.io/projected/26556ab8-9243-4a34-9466-8e5d048e3b84-kube-api-access-q7b85\") pod \"kuadrant-operator-catalog-lx7vl\" (UID: \"26556ab8-9243-4a34-9466-8e5d048e3b84\") " pod="kuadrant-system/kuadrant-operator-catalog-lx7vl" Jul 10 02:55:48.594494 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:48.594465 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7b85\" (UniqueName: \"kubernetes.io/projected/26556ab8-9243-4a34-9466-8e5d048e3b84-kube-api-access-q7b85\") pod \"kuadrant-operator-catalog-lx7vl\" (UID: \"26556ab8-9243-4a34-9466-8e5d048e3b84\") " pod="kuadrant-system/kuadrant-operator-catalog-lx7vl" Jul 10 02:55:48.602343 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:48.602314 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7b85\" (UniqueName: \"kubernetes.io/projected/26556ab8-9243-4a34-9466-8e5d048e3b84-kube-api-access-q7b85\") pod \"kuadrant-operator-catalog-lx7vl\" (UID: \"26556ab8-9243-4a34-9466-8e5d048e3b84\") " pod="kuadrant-system/kuadrant-operator-catalog-lx7vl" Jul 10 02:55:48.695367 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:48.695297 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-lx7vl" Jul 10 02:55:48.748822 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:48.748745 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lx7vl"] Jul 10 02:55:48.824543 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:48.824511 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lx7vl"] Jul 10 02:55:48.827491 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:55:48.827457 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26556ab8_9243_4a34_9466_8e5d048e3b84.slice/crio-502e12c156c755e5536a035309f5b784b32d289bacb409ec6c12e8d348f42df1 WatchSource:0}: Error finding container 502e12c156c755e5536a035309f5b784b32d289bacb409ec6c12e8d348f42df1: Status 404 returned error can't find the container with id 502e12c156c755e5536a035309f5b784b32d289bacb409ec6c12e8d348f42df1 Jul 10 02:55:48.971491 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:48.971414 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zzg8l"] Jul 10 02:55:48.976089 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:48.976071 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zzg8l" Jul 10 02:55:48.983067 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:48.983043 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zzg8l"] Jul 10 02:55:49.099920 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:49.099886 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d5mh\" (UniqueName: \"kubernetes.io/projected/4269dd60-fe03-48e4-8368-dd8e04a33335-kube-api-access-8d5mh\") pod \"kuadrant-operator-catalog-zzg8l\" (UID: \"4269dd60-fe03-48e4-8368-dd8e04a33335\") " pod="kuadrant-system/kuadrant-operator-catalog-zzg8l" Jul 10 02:55:49.201378 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:49.201336 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d5mh\" (UniqueName: \"kubernetes.io/projected/4269dd60-fe03-48e4-8368-dd8e04a33335-kube-api-access-8d5mh\") pod \"kuadrant-operator-catalog-zzg8l\" (UID: \"4269dd60-fe03-48e4-8368-dd8e04a33335\") " pod="kuadrant-system/kuadrant-operator-catalog-zzg8l" Jul 10 02:55:49.209347 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:49.209316 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d5mh\" (UniqueName: \"kubernetes.io/projected/4269dd60-fe03-48e4-8368-dd8e04a33335-kube-api-access-8d5mh\") pod \"kuadrant-operator-catalog-zzg8l\" (UID: \"4269dd60-fe03-48e4-8368-dd8e04a33335\") " pod="kuadrant-system/kuadrant-operator-catalog-zzg8l" Jul 10 02:55:49.285621 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:49.285594 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zzg8l" Jul 10 02:55:49.427113 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:49.427090 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zzg8l"] Jul 10 02:55:49.427922 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:55:49.427899 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4269dd60_fe03_48e4_8368_dd8e04a33335.slice/crio-7cf3aa88b57dcf95f785500cdfc405bb1214a509e9e960116bac252d5bbbc0b8 WatchSource:0}: Error finding container 7cf3aa88b57dcf95f785500cdfc405bb1214a509e9e960116bac252d5bbbc0b8: Status 404 returned error can't find the container with id 7cf3aa88b57dcf95f785500cdfc405bb1214a509e9e960116bac252d5bbbc0b8 Jul 10 02:55:49.577977 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:49.577898 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zzg8l" event={"ID":"4269dd60-fe03-48e4-8368-dd8e04a33335","Type":"ContainerStarted","Data":"7cf3aa88b57dcf95f785500cdfc405bb1214a509e9e960116bac252d5bbbc0b8"} Jul 10 02:55:49.579045 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:49.579016 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-lx7vl" event={"ID":"26556ab8-9243-4a34-9466-8e5d048e3b84","Type":"ContainerStarted","Data":"502e12c156c755e5536a035309f5b784b32d289bacb409ec6c12e8d348f42df1"} Jul 10 02:55:51.590357 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:51.590316 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zzg8l" event={"ID":"4269dd60-fe03-48e4-8368-dd8e04a33335","Type":"ContainerStarted","Data":"ebdd266c9b7d1edc02f0d280289e3755e84b8270c92d567b822870aa03278a46"} Jul 10 02:55:51.591886 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:51.591856 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-lx7vl" event={"ID":"26556ab8-9243-4a34-9466-8e5d048e3b84","Type":"ContainerStarted","Data":"fd34bbb0c3aea91f3f63423cf7bb4da4660f7fe341b2f99ded34132b05a2a07f"} Jul 10 02:55:51.592017 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:51.591881 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-lx7vl" podUID="26556ab8-9243-4a34-9466-8e5d048e3b84" containerName="registry-server" containerID="cri-o://fd34bbb0c3aea91f3f63423cf7bb4da4660f7fe341b2f99ded34132b05a2a07f" gracePeriod=2 Jul 10 02:55:51.606305 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:51.606261 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-zzg8l" podStartSLOduration=2.091655425 podStartE2EDuration="3.606247528s" totalImagesPullingTime="1.514592103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:55:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:55:51.60495271 +0000 UTC m=+469.886069033" watchObservedRunningTime="2026-07-10 02:55:51.606247528 +0000 UTC m=+469.887363838" Jul 10 02:55:51.621414 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:51.621369 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-lx7vl" podStartSLOduration=1.508812252 podStartE2EDuration="3.621351168s" totalImagesPullingTime="2.112538916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:55:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:55:51.620567206 +0000 UTC m=+469.901683491" watchObservedRunningTime="2026-07-10 02:55:51.621351168 +0000 UTC m=+469.902467456" Jul 10 02:55:51.834740 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:51.834712 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-lx7vl" Jul 10 02:55:51.925041 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:51.924972 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7b85\" (UniqueName: \"kubernetes.io/projected/26556ab8-9243-4a34-9466-8e5d048e3b84-kube-api-access-q7b85\") pod \"26556ab8-9243-4a34-9466-8e5d048e3b84\" (UID: \"26556ab8-9243-4a34-9466-8e5d048e3b84\") " Jul 10 02:55:51.927084 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:51.927065 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26556ab8-9243-4a34-9466-8e5d048e3b84-kube-api-access-q7b85" (OuterVolumeSpecName: "kube-api-access-q7b85") pod "26556ab8-9243-4a34-9466-8e5d048e3b84" (UID: "26556ab8-9243-4a34-9466-8e5d048e3b84"). InnerVolumeSpecName "kube-api-access-q7b85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 02:55:52.025838 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:52.025775 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7b85\" (UniqueName: \"kubernetes.io/projected/26556ab8-9243-4a34-9466-8e5d048e3b84-kube-api-access-q7b85\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 02:55:52.597912 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:52.597825 2575 generic.go:356] "Generic (PLEG): container finished" podID="26556ab8-9243-4a34-9466-8e5d048e3b84" containerID="fd34bbb0c3aea91f3f63423cf7bb4da4660f7fe341b2f99ded34132b05a2a07f" exitCode=0 Jul 10 02:55:52.597912 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:52.597876 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-lx7vl" Jul 10 02:55:52.597912 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:52.597902 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-lx7vl" event={"ID":"26556ab8-9243-4a34-9466-8e5d048e3b84","Type":"ContainerDied","Data":"fd34bbb0c3aea91f3f63423cf7bb4da4660f7fe341b2f99ded34132b05a2a07f"} Jul 10 02:55:52.598469 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:52.597951 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-lx7vl" event={"ID":"26556ab8-9243-4a34-9466-8e5d048e3b84","Type":"ContainerDied","Data":"502e12c156c755e5536a035309f5b784b32d289bacb409ec6c12e8d348f42df1"} Jul 10 02:55:52.598469 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:52.597972 2575 scope.go:117] "RemoveContainer" containerID="fd34bbb0c3aea91f3f63423cf7bb4da4660f7fe341b2f99ded34132b05a2a07f" Jul 10 02:55:52.609560 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:52.609531 2575 scope.go:117] "RemoveContainer" containerID="fd34bbb0c3aea91f3f63423cf7bb4da4660f7fe341b2f99ded34132b05a2a07f" Jul 10 02:55:52.609862 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:55:52.609846 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd34bbb0c3aea91f3f63423cf7bb4da4660f7fe341b2f99ded34132b05a2a07f\": container with ID starting with fd34bbb0c3aea91f3f63423cf7bb4da4660f7fe341b2f99ded34132b05a2a07f not found: ID does not exist" containerID="fd34bbb0c3aea91f3f63423cf7bb4da4660f7fe341b2f99ded34132b05a2a07f" Jul 10 02:55:52.609919 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:52.609871 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd34bbb0c3aea91f3f63423cf7bb4da4660f7fe341b2f99ded34132b05a2a07f"} err="failed to get container status \"fd34bbb0c3aea91f3f63423cf7bb4da4660f7fe341b2f99ded34132b05a2a07f\": rpc error: code = NotFound desc = could not find container \"fd34bbb0c3aea91f3f63423cf7bb4da4660f7fe341b2f99ded34132b05a2a07f\": container with ID starting with fd34bbb0c3aea91f3f63423cf7bb4da4660f7fe341b2f99ded34132b05a2a07f not found: ID does not exist" Jul 10 02:55:52.614366 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:52.614347 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lx7vl"] Jul 10 02:55:52.620623 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:52.620602 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lx7vl"] Jul 10 02:55:54.287957 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:54.287914 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26556ab8-9243-4a34-9466-8e5d048e3b84" path="/var/lib/kubelet/pods/26556ab8-9243-4a34-9466-8e5d048e3b84/volumes" Jul 10 02:55:59.285953 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:59.285917 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-zzg8l" Jul 10 02:55:59.286347 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:59.285970 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-zzg8l" Jul 10 02:55:59.308378 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:59.308351 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-zzg8l" Jul 10 02:55:59.650900 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:55:59.650830 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-zzg8l" Jul 10 02:56:21.478490 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:21.478456 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lszpx"] Jul 10 02:56:21.479017 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:21.478985 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26556ab8-9243-4a34-9466-8e5d048e3b84" containerName="registry-server" Jul 10 02:56:21.479017 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:21.479005 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="26556ab8-9243-4a34-9466-8e5d048e3b84" containerName="registry-server" Jul 10 02:56:21.479144 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:21.479129 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="26556ab8-9243-4a34-9466-8e5d048e3b84" containerName="registry-server" Jul 10 02:56:21.483477 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:21.483454 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lszpx" Jul 10 02:56:21.486084 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:21.486048 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-mks2k\"" Jul 10 02:56:21.494747 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:21.494726 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lszpx"] Jul 10 02:56:21.580659 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:21.580615 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npznw\" (UniqueName: \"kubernetes.io/projected/c65e0ad5-fb64-42c8-b1a9-172ab74ab15b-kube-api-access-npznw\") pod \"kuadrant-operator-controller-manager-8684c9ff49-lszpx\" (UID: \"c65e0ad5-fb64-42c8-b1a9-172ab74ab15b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lszpx" Jul 10 02:56:21.580659 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:21.580671 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c65e0ad5-fb64-42c8-b1a9-172ab74ab15b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-lszpx\" (UID: \"c65e0ad5-fb64-42c8-b1a9-172ab74ab15b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lszpx" Jul 10 02:56:21.681671 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:21.681636 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c65e0ad5-fb64-42c8-b1a9-172ab74ab15b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-lszpx\" (UID: \"c65e0ad5-fb64-42c8-b1a9-172ab74ab15b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lszpx" Jul 10 02:56:21.681855 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:21.681790 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npznw\" (UniqueName: \"kubernetes.io/projected/c65e0ad5-fb64-42c8-b1a9-172ab74ab15b-kube-api-access-npznw\") pod \"kuadrant-operator-controller-manager-8684c9ff49-lszpx\" (UID: \"c65e0ad5-fb64-42c8-b1a9-172ab74ab15b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lszpx" Jul 10 02:56:21.682106 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:21.682081 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c65e0ad5-fb64-42c8-b1a9-172ab74ab15b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-lszpx\" (UID: \"c65e0ad5-fb64-42c8-b1a9-172ab74ab15b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lszpx" Jul 10 02:56:21.692976 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:21.692949 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npznw\" (UniqueName: \"kubernetes.io/projected/c65e0ad5-fb64-42c8-b1a9-172ab74ab15b-kube-api-access-npznw\") pod \"kuadrant-operator-controller-manager-8684c9ff49-lszpx\" (UID: \"c65e0ad5-fb64-42c8-b1a9-172ab74ab15b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lszpx" Jul 10 02:56:21.794041 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:21.794005 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lszpx" Jul 10 02:56:21.924966 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:21.924941 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lszpx"] Jul 10 02:56:21.927303 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:56:21.927276 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc65e0ad5_fb64_42c8_b1a9_172ab74ab15b.slice/crio-12e02fc81403575d6d06344b76312120e44b251dd1f07cc7d3154edd9d7d46fb WatchSource:0}: Error finding container 12e02fc81403575d6d06344b76312120e44b251dd1f07cc7d3154edd9d7d46fb: Status 404 returned error can't find the container with id 12e02fc81403575d6d06344b76312120e44b251dd1f07cc7d3154edd9d7d46fb Jul 10 02:56:22.732267 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:22.732234 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lszpx" event={"ID":"c65e0ad5-fb64-42c8-b1a9-172ab74ab15b","Type":"ContainerStarted","Data":"12e02fc81403575d6d06344b76312120e44b251dd1f07cc7d3154edd9d7d46fb"} Jul 10 02:56:23.848040 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:23.848001 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-9zmcz"] Jul 10 02:56:23.851630 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:23.851606 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-9zmcz" Jul 10 02:56:23.854213 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:23.854193 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-phn4l\"" Jul 10 02:56:23.861177 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:23.861157 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-9zmcz"] Jul 10 02:56:23.900189 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:23.900161 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz5wk\" (UniqueName: \"kubernetes.io/projected/9ca0d756-cdbe-4517-b4d9-c3a6c96f44ed-kube-api-access-wz5wk\") pod \"authorino-operator-75cc8d5cb7-9zmcz\" (UID: \"9ca0d756-cdbe-4517-b4d9-c3a6c96f44ed\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-9zmcz" Jul 10 02:56:24.001234 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:24.001197 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz5wk\" (UniqueName: \"kubernetes.io/projected/9ca0d756-cdbe-4517-b4d9-c3a6c96f44ed-kube-api-access-wz5wk\") pod \"authorino-operator-75cc8d5cb7-9zmcz\" (UID: \"9ca0d756-cdbe-4517-b4d9-c3a6c96f44ed\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-9zmcz" Jul 10 02:56:24.009940 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:24.009910 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz5wk\" (UniqueName: \"kubernetes.io/projected/9ca0d756-cdbe-4517-b4d9-c3a6c96f44ed-kube-api-access-wz5wk\") pod \"authorino-operator-75cc8d5cb7-9zmcz\" (UID: \"9ca0d756-cdbe-4517-b4d9-c3a6c96f44ed\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-9zmcz" Jul 10 02:56:24.165473 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:24.165390 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-9zmcz" Jul 10 02:56:24.304047 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:24.303895 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-9zmcz"] Jul 10 02:56:24.306837 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:56:24.306788 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ca0d756_cdbe_4517_b4d9_c3a6c96f44ed.slice/crio-393a78faa0015430148dff90097f33fb6df798cd6de8acec4e3c1b9c57021915 WatchSource:0}: Error finding container 393a78faa0015430148dff90097f33fb6df798cd6de8acec4e3c1b9c57021915: Status 404 returned error can't find the container with id 393a78faa0015430148dff90097f33fb6df798cd6de8acec4e3c1b9c57021915 Jul 10 02:56:24.744477 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:24.744439 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-9zmcz" event={"ID":"9ca0d756-cdbe-4517-b4d9-c3a6c96f44ed","Type":"ContainerStarted","Data":"393a78faa0015430148dff90097f33fb6df798cd6de8acec4e3c1b9c57021915"} Jul 10 02:56:26.521995 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:26.521961 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zrxnt"] Jul 10 02:56:26.525834 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:26.525803 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zrxnt" Jul 10 02:56:26.528742 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:26.528722 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-bskzx\"" Jul 10 02:56:26.543374 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:26.543349 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zrxnt"] Jul 10 02:56:26.620263 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:26.620226 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx8tb\" (UniqueName: \"kubernetes.io/projected/10d8bf56-dd5d-4e0a-bec5-e6f3901b2347-kube-api-access-dx8tb\") pod \"limitador-operator-controller-manager-859d7b78bd-zrxnt\" (UID: \"10d8bf56-dd5d-4e0a-bec5-e6f3901b2347\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zrxnt" Jul 10 02:56:26.720930 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:26.720887 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx8tb\" (UniqueName: \"kubernetes.io/projected/10d8bf56-dd5d-4e0a-bec5-e6f3901b2347-kube-api-access-dx8tb\") pod \"limitador-operator-controller-manager-859d7b78bd-zrxnt\" (UID: \"10d8bf56-dd5d-4e0a-bec5-e6f3901b2347\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zrxnt" Jul 10 02:56:26.738688 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:26.738657 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx8tb\" (UniqueName: \"kubernetes.io/projected/10d8bf56-dd5d-4e0a-bec5-e6f3901b2347-kube-api-access-dx8tb\") pod \"limitador-operator-controller-manager-859d7b78bd-zrxnt\" (UID: \"10d8bf56-dd5d-4e0a-bec5-e6f3901b2347\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zrxnt" Jul 10 02:56:26.839621 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:26.839530 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zrxnt" Jul 10 02:56:28.061587 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:28.061554 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zrxnt"] Jul 10 02:56:28.065458 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:56:28.065428 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10d8bf56_dd5d_4e0a_bec5_e6f3901b2347.slice/crio-b9416477988c3657eff4d7594b9b4ed5f14760fea371f4fec437f13dd9ed5f50 WatchSource:0}: Error finding container b9416477988c3657eff4d7594b9b4ed5f14760fea371f4fec437f13dd9ed5f50: Status 404 returned error can't find the container with id b9416477988c3657eff4d7594b9b4ed5f14760fea371f4fec437f13dd9ed5f50 Jul 10 02:56:28.769849 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:28.769791 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-9zmcz" event={"ID":"9ca0d756-cdbe-4517-b4d9-c3a6c96f44ed","Type":"ContainerStarted","Data":"3bf52dab1622b795ecf8c5dcafa77126064adf4e9629ef4b8ad95e9271c6c16e"} Jul 10 02:56:28.770268 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:28.770175 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-9zmcz" Jul 10 02:56:28.772208 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:28.772170 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lszpx" event={"ID":"c65e0ad5-fb64-42c8-b1a9-172ab74ab15b","Type":"ContainerStarted","Data":"b20c05d648923507245ee4f2a64b6a1967690332541729f6448fcd907a8f9b5d"} Jul 10 02:56:28.772330 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:28.772304 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lszpx" Jul 10 02:56:28.773782 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:28.773746 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zrxnt" event={"ID":"10d8bf56-dd5d-4e0a-bec5-e6f3901b2347","Type":"ContainerStarted","Data":"b9416477988c3657eff4d7594b9b4ed5f14760fea371f4fec437f13dd9ed5f50"} Jul 10 02:56:28.789550 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:28.789495 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-9zmcz" podStartSLOduration=2.19200091 podStartE2EDuration="5.789474988s" totalImagesPullingTime="3.597474078s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:56:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:56:28.787531774 +0000 UTC m=+507.068648061" watchObservedRunningTime="2026-07-10 02:56:28.789474988 +0000 UTC m=+507.070591273" Jul 10 02:56:28.810364 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:28.810287 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lszpx" podStartSLOduration=1.817801359 podStartE2EDuration="7.810272286s" totalImagesPullingTime="5.992470927s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:56:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:56:28.806999107 +0000 UTC m=+507.088115392" watchObservedRunningTime="2026-07-10 02:56:28.810272286 +0000 UTC m=+507.091388571" Jul 10 02:56:29.453075 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:29.453045 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-46t5x"] Jul 10 02:56:29.456888 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:29.456867 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-46t5x" Jul 10 02:56:29.460000 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:29.459970 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-2q5ps\"" Jul 10 02:56:29.460298 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:29.460240 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 10 02:56:29.479737 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:29.479714 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-46t5x"] Jul 10 02:56:29.546770 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:29.546637 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvvhq\" (UniqueName: \"kubernetes.io/projected/fae1bef4-d254-4bed-8e45-4e65c35e78f8-kube-api-access-hvvhq\") pod \"dns-operator-controller-manager-864dd78c4c-46t5x\" (UID: \"fae1bef4-d254-4bed-8e45-4e65c35e78f8\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-46t5x" Jul 10 02:56:29.647616 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:29.647583 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvvhq\" (UniqueName: \"kubernetes.io/projected/fae1bef4-d254-4bed-8e45-4e65c35e78f8-kube-api-access-hvvhq\") pod \"dns-operator-controller-manager-864dd78c4c-46t5x\" (UID: \"fae1bef4-d254-4bed-8e45-4e65c35e78f8\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-46t5x" Jul 10 02:56:29.661328 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:29.661296 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvvhq\" (UniqueName: \"kubernetes.io/projected/fae1bef4-d254-4bed-8e45-4e65c35e78f8-kube-api-access-hvvhq\") pod \"dns-operator-controller-manager-864dd78c4c-46t5x\" (UID: \"fae1bef4-d254-4bed-8e45-4e65c35e78f8\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-46t5x" Jul 10 02:56:29.768462 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:29.768430 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-46t5x" Jul 10 02:56:29.904933 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:29.904912 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-46t5x"] Jul 10 02:56:29.907634 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:56:29.907608 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfae1bef4_d254_4bed_8e45_4e65c35e78f8.slice/crio-c81f104fc5ce124d42914749406eeaf361398bd162bb2251278f7bc18fb5923c WatchSource:0}: Error finding container c81f104fc5ce124d42914749406eeaf361398bd162bb2251278f7bc18fb5923c: Status 404 returned error can't find the container with id c81f104fc5ce124d42914749406eeaf361398bd162bb2251278f7bc18fb5923c Jul 10 02:56:30.785083 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:30.785032 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-46t5x" event={"ID":"fae1bef4-d254-4bed-8e45-4e65c35e78f8","Type":"ContainerStarted","Data":"c81f104fc5ce124d42914749406eeaf361398bd162bb2251278f7bc18fb5923c"} Jul 10 02:56:30.786845 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:30.786787 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zrxnt" event={"ID":"10d8bf56-dd5d-4e0a-bec5-e6f3901b2347","Type":"ContainerStarted","Data":"dab8707554aadb176d673a25a8bce0aa0b61ff7a768d8e74b8a32adba0700d33"} Jul 10 02:56:30.787135 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:30.787116 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zrxnt" Jul 10 02:56:30.804108 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:30.804061 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zrxnt" podStartSLOduration=3.100253969 podStartE2EDuration="4.804033862s" totalImagesPullingTime="1.703779893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:56:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:56:30.802057303 +0000 UTC m=+509.083173589" watchObservedRunningTime="2026-07-10 02:56:30.804033862 +0000 UTC m=+509.085150146" Jul 10 02:56:30.961730 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:30.961687 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-jtsd5"] Jul 10 02:56:30.966012 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:30.965981 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jtsd5" Jul 10 02:56:30.968388 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:30.968358 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 10 02:56:30.968388 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:30.968393 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-v4sst\"" Jul 10 02:56:30.968983 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:30.968731 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 10 02:56:30.976983 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:30.976960 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-jtsd5"] Jul 10 02:56:31.059679 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:31.059596 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg5dx\" (UniqueName: \"kubernetes.io/projected/47f6a75f-d956-4969-8b7b-991b752d06af-kube-api-access-pg5dx\") pod \"kuadrant-console-plugin-66b5774985-jtsd5\" (UID: \"47f6a75f-d956-4969-8b7b-991b752d06af\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jtsd5" Jul 10 02:56:31.059679 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:31.059672 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/47f6a75f-d956-4969-8b7b-991b752d06af-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-jtsd5\" (UID: \"47f6a75f-d956-4969-8b7b-991b752d06af\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jtsd5" Jul 10 02:56:31.059885 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:31.059701 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/47f6a75f-d956-4969-8b7b-991b752d06af-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-jtsd5\" (UID: \"47f6a75f-d956-4969-8b7b-991b752d06af\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jtsd5" Jul 10 02:56:31.160362 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:31.160324 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/47f6a75f-d956-4969-8b7b-991b752d06af-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-jtsd5\" (UID: \"47f6a75f-d956-4969-8b7b-991b752d06af\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jtsd5" Jul 10 02:56:31.160362 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:31.160363 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/47f6a75f-d956-4969-8b7b-991b752d06af-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-jtsd5\" (UID: \"47f6a75f-d956-4969-8b7b-991b752d06af\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jtsd5" Jul 10 02:56:31.160635 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:31.160432 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg5dx\" (UniqueName: \"kubernetes.io/projected/47f6a75f-d956-4969-8b7b-991b752d06af-kube-api-access-pg5dx\") pod \"kuadrant-console-plugin-66b5774985-jtsd5\" (UID: \"47f6a75f-d956-4969-8b7b-991b752d06af\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jtsd5" Jul 10 02:56:31.160635 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:56:31.160486 2575 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 10 02:56:31.160635 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:56:31.160581 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47f6a75f-d956-4969-8b7b-991b752d06af-plugin-serving-cert podName:47f6a75f-d956-4969-8b7b-991b752d06af nodeName:}" failed. No retries permitted until 2026-07-10 02:56:31.660560309 +0000 UTC m=+509.941676578 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/47f6a75f-d956-4969-8b7b-991b752d06af-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-jtsd5" (UID: "47f6a75f-d956-4969-8b7b-991b752d06af") : secret "plugin-serving-cert" not found Jul 10 02:56:31.161143 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:31.161127 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/47f6a75f-d956-4969-8b7b-991b752d06af-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-jtsd5\" (UID: \"47f6a75f-d956-4969-8b7b-991b752d06af\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jtsd5" Jul 10 02:56:31.171980 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:31.171950 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg5dx\" (UniqueName: \"kubernetes.io/projected/47f6a75f-d956-4969-8b7b-991b752d06af-kube-api-access-pg5dx\") pod \"kuadrant-console-plugin-66b5774985-jtsd5\" (UID: \"47f6a75f-d956-4969-8b7b-991b752d06af\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jtsd5" Jul 10 02:56:31.666154 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:31.666111 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/47f6a75f-d956-4969-8b7b-991b752d06af-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-jtsd5\" (UID: \"47f6a75f-d956-4969-8b7b-991b752d06af\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jtsd5" Jul 10 02:56:31.668654 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:31.668630 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/47f6a75f-d956-4969-8b7b-991b752d06af-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-jtsd5\" (UID: \"47f6a75f-d956-4969-8b7b-991b752d06af\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jtsd5" Jul 10 02:56:31.879881 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:31.879845 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jtsd5" Jul 10 02:56:32.210963 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:32.210939 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-jtsd5"] Jul 10 02:56:32.213322 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:56:32.213286 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47f6a75f_d956_4969_8b7b_991b752d06af.slice/crio-1d640a1baff09e6ddfd97833051449a0169a05e2afbe0cd645cfb3cae56f8c7d WatchSource:0}: Error finding container 1d640a1baff09e6ddfd97833051449a0169a05e2afbe0cd645cfb3cae56f8c7d: Status 404 returned error can't find the container with id 1d640a1baff09e6ddfd97833051449a0169a05e2afbe0cd645cfb3cae56f8c7d Jul 10 02:56:32.801445 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:32.801405 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-46t5x" event={"ID":"fae1bef4-d254-4bed-8e45-4e65c35e78f8","Type":"ContainerStarted","Data":"ca80a051a6e099c53b15914b4fd702f4cc97fab665892464823b667a951e07ab"} Jul 10 02:56:32.801632 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:32.801536 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-46t5x" Jul 10 02:56:32.802845 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:32.802782 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jtsd5" event={"ID":"47f6a75f-d956-4969-8b7b-991b752d06af","Type":"ContainerStarted","Data":"1d640a1baff09e6ddfd97833051449a0169a05e2afbe0cd645cfb3cae56f8c7d"} Jul 10 02:56:32.820067 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:32.820002 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-46t5x" podStartSLOduration=1.5985637110000002 podStartE2EDuration="3.81998602s" totalImagesPullingTime="2.221422309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:56:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:56:32.817609702 +0000 UTC m=+511.098726018" watchObservedRunningTime="2026-07-10 02:56:32.81998602 +0000 UTC m=+511.101102305" Jul 10 02:56:39.780447 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:39.780415 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lszpx" Jul 10 02:56:39.780824 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:39.780499 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-9zmcz" Jul 10 02:56:41.450585 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:41.450547 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lszpx"] Jul 10 02:56:41.453414 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:41.450792 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lszpx" podUID="c65e0ad5-fb64-42c8-b1a9-172ab74ab15b" containerName="manager" containerID="cri-o://b20c05d648923507245ee4f2a64b6a1967690332541729f6448fcd907a8f9b5d" gracePeriod=2 Jul 10 02:56:41.453414 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:56:41.452748 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-lszpx\" is forbidden: User \"system:node:ip-10-0-130-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-139.ec2.internal' and this object" podUID="c65e0ad5-fb64-42c8-b1a9-172ab74ab15b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lszpx" Jul 10 02:56:41.454099 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:41.454057 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lszpx"] Jul 10 02:56:41.467234 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:41.467208 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zrxnt"] Jul 10 02:56:41.467506 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:41.467450 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zrxnt" podUID="10d8bf56-dd5d-4e0a-bec5-e6f3901b2347" containerName="manager" containerID="cri-o://dab8707554aadb176d673a25a8bce0aa0b61ff7a768d8e74b8a32adba0700d33" gracePeriod=2 Jul 10 02:56:41.469574 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:56:41.469539 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-lszpx\" is forbidden: User \"system:node:ip-10-0-130-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-139.ec2.internal' and this object" podUID="c65e0ad5-fb64-42c8-b1a9-172ab74ab15b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lszpx" Jul 10 02:56:41.470351 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:41.470326 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zrxnt" Jul 10 02:56:41.470702 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:41.470678 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n9g7n"] Jul 10 02:56:41.471161 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:41.471140 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c65e0ad5-fb64-42c8-b1a9-172ab74ab15b" containerName="manager" Jul 10 02:56:41.471161 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:41.471161 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c65e0ad5-fb64-42c8-b1a9-172ab74ab15b" containerName="manager" Jul 10 02:56:41.471745 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:41.471279 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c65e0ad5-fb64-42c8-b1a9-172ab74ab15b" containerName="manager" Jul 10 02:56:41.475310 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:41.474919 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n9g7n" Jul 10 02:56:41.477060 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:56:41.477032 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-zrxnt\" is forbidden: User \"system:node:ip-10-0-130-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-139.ec2.internal' and this object" podUID="10d8bf56-dd5d-4e0a-bec5-e6f3901b2347" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zrxnt" Jul 10 02:56:41.477180 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:41.477107 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zrxnt"] Jul 10 02:56:41.492753 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:41.492704 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n9g7n"] Jul 10 02:56:41.494964 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:41.494936 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t2zbx"] Jul 10 02:56:41.495485 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:41.495463 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10d8bf56-dd5d-4e0a-bec5-e6f3901b2347" containerName="manager" Jul 10 02:56:41.495485 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:41.495488 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="10d8bf56-dd5d-4e0a-bec5-e6f3901b2347" containerName="manager" Jul 10 02:56:41.495651 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:41.495591 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="10d8bf56-dd5d-4e0a-bec5-e6f3901b2347" containerName="manager" Jul 10 02:56:41.499464 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:41.499444 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t2zbx" Jul 10 02:56:41.508368 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:41.508306 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t2zbx"] Jul 10 02:56:41.513665 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:56:41.513626 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-lszpx\" is forbidden: User \"system:node:ip-10-0-130-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-139.ec2.internal' and this object" podUID="c65e0ad5-fb64-42c8-b1a9-172ab74ab15b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lszpx" Jul 10 02:56:41.533893 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:56:41.533844 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-lszpx\" is forbidden: User \"system:node:ip-10-0-130-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-139.ec2.internal' and this object" podUID="c65e0ad5-fb64-42c8-b1a9-172ab74ab15b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lszpx" Jul 10 02:56:41.535612 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:56:41.535577 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-zrxnt\" is forbidden: User \"system:node:ip-10-0-130-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-139.ec2.internal' and this object" podUID="10d8bf56-dd5d-4e0a-bec5-e6f3901b2347" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zrxnt" Jul 10 02:56:41.557112 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:41.557078 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/60be55a4-238e-4b01-a659-3bbdb82878e7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-n9g7n\" (UID: \"60be55a4-238e-4b01-a659-3bbdb82878e7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n9g7n" Jul 10 02:56:41.557270 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:41.557126 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7t7h\" (UniqueName: \"kubernetes.io/projected/60be55a4-238e-4b01-a659-3bbdb82878e7-kube-api-access-s7t7h\") pod \"kuadrant-operator-controller-manager-8684c9ff49-n9g7n\" (UID: \"60be55a4-238e-4b01-a659-3bbdb82878e7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n9g7n" Jul 10 02:56:41.557270 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:41.557154 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkbg7\" (UniqueName: \"kubernetes.io/projected/127c755f-b34b-452a-a83d-855155192db2-kube-api-access-pkbg7\") pod \"limitador-operator-controller-manager-859d7b78bd-t2zbx\" (UID: \"127c755f-b34b-452a-a83d-855155192db2\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t2zbx" Jul 10 02:56:41.657972 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:41.657936 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/60be55a4-238e-4b01-a659-3bbdb82878e7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-n9g7n\" (UID: \"60be55a4-238e-4b01-a659-3bbdb82878e7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n9g7n" Jul 10 02:56:41.658163 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:41.657980 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7t7h\" (UniqueName: \"kubernetes.io/projected/60be55a4-238e-4b01-a659-3bbdb82878e7-kube-api-access-s7t7h\") pod \"kuadrant-operator-controller-manager-8684c9ff49-n9g7n\" (UID: \"60be55a4-238e-4b01-a659-3bbdb82878e7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n9g7n" Jul 10 02:56:41.658163 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:41.658104 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkbg7\" (UniqueName: \"kubernetes.io/projected/127c755f-b34b-452a-a83d-855155192db2-kube-api-access-pkbg7\") pod \"limitador-operator-controller-manager-859d7b78bd-t2zbx\" (UID: \"127c755f-b34b-452a-a83d-855155192db2\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t2zbx" Jul 10 02:56:41.658392 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:41.658358 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/60be55a4-238e-4b01-a659-3bbdb82878e7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-n9g7n\" (UID: \"60be55a4-238e-4b01-a659-3bbdb82878e7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n9g7n" Jul 10 02:56:41.667615 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:41.667589 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkbg7\" (UniqueName: \"kubernetes.io/projected/127c755f-b34b-452a-a83d-855155192db2-kube-api-access-pkbg7\") pod \"limitador-operator-controller-manager-859d7b78bd-t2zbx\" (UID: \"127c755f-b34b-452a-a83d-855155192db2\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t2zbx" Jul 10 02:56:41.670233 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:41.670205 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7t7h\" (UniqueName: \"kubernetes.io/projected/60be55a4-238e-4b01-a659-3bbdb82878e7-kube-api-access-s7t7h\") pod \"kuadrant-operator-controller-manager-8684c9ff49-n9g7n\" (UID: \"60be55a4-238e-4b01-a659-3bbdb82878e7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n9g7n" Jul 10 02:56:41.897365 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:41.897327 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n9g7n" Jul 10 02:56:41.904454 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:41.904421 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t2zbx" Jul 10 02:56:42.288399 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:56:42.288358 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-lszpx\" is forbidden: User \"system:node:ip-10-0-130-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-139.ec2.internal' and this object" podUID="c65e0ad5-fb64-42c8-b1a9-172ab74ab15b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lszpx" Jul 10 02:56:42.290426 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:56:42.290394 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-zrxnt\" is forbidden: User \"system:node:ip-10-0-130-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-139.ec2.internal' and this object" podUID="10d8bf56-dd5d-4e0a-bec5-e6f3901b2347" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zrxnt" Jul 10 02:56:43.812180 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:43.812151 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-46t5x" Jul 10 02:56:54.349775 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:54.349737 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n9g7n"] Jul 10 02:56:54.351314 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:56:54.351276 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60be55a4_238e_4b01_a659_3bbdb82878e7.slice/crio-be99906a3573211d5c150e5e7663c90daa977a9993a26e24bc79014d2bb29f79 WatchSource:0}: Error finding container be99906a3573211d5c150e5e7663c90daa977a9993a26e24bc79014d2bb29f79: Status 404 returned error can't find the container with id be99906a3573211d5c150e5e7663c90daa977a9993a26e24bc79014d2bb29f79 Jul 10 02:56:54.358802 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:54.358780 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zrxnt" Jul 10 02:56:54.361013 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:56:54.360980 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-zrxnt\" is forbidden: User \"system:node:ip-10-0-130-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-139.ec2.internal' and this object" podUID="10d8bf56-dd5d-4e0a-bec5-e6f3901b2347" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zrxnt" Jul 10 02:56:54.361441 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:54.361425 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lszpx" Jul 10 02:56:54.363149 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:56:54.363126 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-lszpx\" is forbidden: User \"system:node:ip-10-0-130-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-139.ec2.internal' and this object" podUID="c65e0ad5-fb64-42c8-b1a9-172ab74ab15b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lszpx" Jul 10 02:56:54.364729 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:56:54.364702 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-zrxnt\" is forbidden: User \"system:node:ip-10-0-130-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-139.ec2.internal' and this object" podUID="10d8bf56-dd5d-4e0a-bec5-e6f3901b2347" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zrxnt" Jul 10 02:56:54.377043 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:54.377024 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t2zbx"] Jul 10 02:56:54.387024 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:56:54.387002 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod127c755f_b34b_452a_a83d_855155192db2.slice/crio-8f494fceea8fc32d2c1d0c2e6beef4b221b2597a24fce0dd7afb37bc9b74dd10 WatchSource:0}: Error finding container 8f494fceea8fc32d2c1d0c2e6beef4b221b2597a24fce0dd7afb37bc9b74dd10: Status 404 returned error can't find the container with id 8f494fceea8fc32d2c1d0c2e6beef4b221b2597a24fce0dd7afb37bc9b74dd10 Jul 10 02:56:54.396474 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:56:54.396444 2575 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60be55a4_238e_4b01_a659_3bbdb82878e7.slice/crio-e24ac25542fb64baf5de130d1c2a239905f71e2ee4c886687cc927f18d4c60ce.scope\": RecentStats: unable to find data in memory cache]" Jul 10 02:56:54.479129 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:54.479108 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-npznw\" (UniqueName: \"kubernetes.io/projected/c65e0ad5-fb64-42c8-b1a9-172ab74ab15b-kube-api-access-npznw\") pod \"c65e0ad5-fb64-42c8-b1a9-172ab74ab15b\" (UID: \"c65e0ad5-fb64-42c8-b1a9-172ab74ab15b\") " Jul 10 02:56:54.479449 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:54.479144 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dx8tb\" (UniqueName: \"kubernetes.io/projected/10d8bf56-dd5d-4e0a-bec5-e6f3901b2347-kube-api-access-dx8tb\") pod \"10d8bf56-dd5d-4e0a-bec5-e6f3901b2347\" (UID: \"10d8bf56-dd5d-4e0a-bec5-e6f3901b2347\") " Jul 10 02:56:54.479449 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:54.479185 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c65e0ad5-fb64-42c8-b1a9-172ab74ab15b-extensions-socket-volume\") pod \"c65e0ad5-fb64-42c8-b1a9-172ab74ab15b\" (UID: \"c65e0ad5-fb64-42c8-b1a9-172ab74ab15b\") " Jul 10 02:56:54.479723 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:54.479694 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c65e0ad5-fb64-42c8-b1a9-172ab74ab15b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "c65e0ad5-fb64-42c8-b1a9-172ab74ab15b" (UID: "c65e0ad5-fb64-42c8-b1a9-172ab74ab15b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 02:56:54.481050 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:54.481031 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c65e0ad5-fb64-42c8-b1a9-172ab74ab15b-kube-api-access-npznw" (OuterVolumeSpecName: "kube-api-access-npznw") pod "c65e0ad5-fb64-42c8-b1a9-172ab74ab15b" (UID: "c65e0ad5-fb64-42c8-b1a9-172ab74ab15b"). InnerVolumeSpecName "kube-api-access-npznw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 02:56:54.481153 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:54.481131 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10d8bf56-dd5d-4e0a-bec5-e6f3901b2347-kube-api-access-dx8tb" (OuterVolumeSpecName: "kube-api-access-dx8tb") pod "10d8bf56-dd5d-4e0a-bec5-e6f3901b2347" (UID: "10d8bf56-dd5d-4e0a-bec5-e6f3901b2347"). InnerVolumeSpecName "kube-api-access-dx8tb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 02:56:54.580184 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:54.580154 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx8tb\" (UniqueName: \"kubernetes.io/projected/10d8bf56-dd5d-4e0a-bec5-e6f3901b2347-kube-api-access-dx8tb\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 02:56:54.580184 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:54.580184 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c65e0ad5-fb64-42c8-b1a9-172ab74ab15b-extensions-socket-volume\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 02:56:54.580377 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:54.580197 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npznw\" (UniqueName: \"kubernetes.io/projected/c65e0ad5-fb64-42c8-b1a9-172ab74ab15b-kube-api-access-npznw\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 02:56:54.929798 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:54.929760 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jtsd5" event={"ID":"47f6a75f-d956-4969-8b7b-991b752d06af","Type":"ContainerStarted","Data":"dee003a032236162448738b706e5bbd79d11cf086a28ace4995ab5194352b3e6"} Jul 10 02:56:54.931408 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:54.931378 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n9g7n" event={"ID":"60be55a4-238e-4b01-a659-3bbdb82878e7","Type":"ContainerStarted","Data":"e24ac25542fb64baf5de130d1c2a239905f71e2ee4c886687cc927f18d4c60ce"} Jul 10 02:56:54.931529 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:54.931437 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n9g7n" Jul 10 02:56:54.931529 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:54.931462 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n9g7n" event={"ID":"60be55a4-238e-4b01-a659-3bbdb82878e7","Type":"ContainerStarted","Data":"be99906a3573211d5c150e5e7663c90daa977a9993a26e24bc79014d2bb29f79"} Jul 10 02:56:54.931753 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:56:54.931723 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-lszpx\" is forbidden: User \"system:node:ip-10-0-130-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-139.ec2.internal' and this object" podUID="c65e0ad5-fb64-42c8-b1a9-172ab74ab15b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lszpx" Jul 10 02:56:54.933480 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:54.933389 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t2zbx" event={"ID":"127c755f-b34b-452a-a83d-855155192db2","Type":"ContainerStarted","Data":"c04850a7ac97c38d9302da1c6a2ed3af8615cf1775a1c708d908c65b7d314ff6"} Jul 10 02:56:54.933480 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:54.933443 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t2zbx" event={"ID":"127c755f-b34b-452a-a83d-855155192db2","Type":"ContainerStarted","Data":"8f494fceea8fc32d2c1d0c2e6beef4b221b2597a24fce0dd7afb37bc9b74dd10"} Jul 10 02:56:54.933653 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:56:54.933513 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-zrxnt\" is forbidden: User \"system:node:ip-10-0-130-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-139.ec2.internal' and this object" podUID="10d8bf56-dd5d-4e0a-bec5-e6f3901b2347" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zrxnt" Jul 10 02:56:54.933653 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:54.933630 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t2zbx" Jul 10 02:56:54.937178 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:54.937147 2575 generic.go:356] "Generic (PLEG): container finished" podID="c65e0ad5-fb64-42c8-b1a9-172ab74ab15b" containerID="b20c05d648923507245ee4f2a64b6a1967690332541729f6448fcd907a8f9b5d" exitCode=0 Jul 10 02:56:54.937300 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:54.937284 2575 scope.go:117] "RemoveContainer" containerID="b20c05d648923507245ee4f2a64b6a1967690332541729f6448fcd907a8f9b5d" Jul 10 02:56:54.937482 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:54.937468 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lszpx" Jul 10 02:56:54.945932 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:54.941513 2575 generic.go:356] "Generic (PLEG): container finished" podID="10d8bf56-dd5d-4e0a-bec5-e6f3901b2347" containerID="dab8707554aadb176d673a25a8bce0aa0b61ff7a768d8e74b8a32adba0700d33" exitCode=0 Jul 10 02:56:54.945932 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:54.941609 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zrxnt" Jul 10 02:56:54.951199 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:54.951067 2575 scope.go:117] "RemoveContainer" containerID="b20c05d648923507245ee4f2a64b6a1967690332541729f6448fcd907a8f9b5d" Jul 10 02:56:54.951649 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:56:54.951623 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b20c05d648923507245ee4f2a64b6a1967690332541729f6448fcd907a8f9b5d\": container with ID starting with b20c05d648923507245ee4f2a64b6a1967690332541729f6448fcd907a8f9b5d not found: ID does not exist" containerID="b20c05d648923507245ee4f2a64b6a1967690332541729f6448fcd907a8f9b5d" Jul 10 02:56:54.951744 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:54.951668 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b20c05d648923507245ee4f2a64b6a1967690332541729f6448fcd907a8f9b5d"} err="failed to get container status \"b20c05d648923507245ee4f2a64b6a1967690332541729f6448fcd907a8f9b5d\": rpc error: code = NotFound desc = could not find container \"b20c05d648923507245ee4f2a64b6a1967690332541729f6448fcd907a8f9b5d\": container with ID starting with b20c05d648923507245ee4f2a64b6a1967690332541729f6448fcd907a8f9b5d not found: ID does not exist" Jul 10 02:56:54.951744 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:56:54.951680 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-zrxnt\" is forbidden: User \"system:node:ip-10-0-130-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-139.ec2.internal' and this object" podUID="10d8bf56-dd5d-4e0a-bec5-e6f3901b2347" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zrxnt" Jul 10 02:56:54.951744 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:54.951675 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jtsd5" podStartSLOduration=2.887433208 podStartE2EDuration="24.951649393s" totalImagesPullingTime="22.064216185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:56:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:56:54.949845934 +0000 UTC m=+533.230962232" watchObservedRunningTime="2026-07-10 02:56:54.951649393 +0000 UTC m=+533.232765679" Jul 10 02:56:54.952023 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:54.951691 2575 scope.go:117] "RemoveContainer" containerID="dab8707554aadb176d673a25a8bce0aa0b61ff7a768d8e74b8a32adba0700d33" Jul 10 02:56:54.960701 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:54.960683 2575 scope.go:117] "RemoveContainer" containerID="dab8707554aadb176d673a25a8bce0aa0b61ff7a768d8e74b8a32adba0700d33" Jul 10 02:56:54.961083 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:56:54.961042 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dab8707554aadb176d673a25a8bce0aa0b61ff7a768d8e74b8a32adba0700d33\": container with ID starting with dab8707554aadb176d673a25a8bce0aa0b61ff7a768d8e74b8a32adba0700d33 not found: ID does not exist" containerID="dab8707554aadb176d673a25a8bce0aa0b61ff7a768d8e74b8a32adba0700d33" Jul 10 02:56:54.961185 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:54.961093 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dab8707554aadb176d673a25a8bce0aa0b61ff7a768d8e74b8a32adba0700d33"} err="failed to get container status \"dab8707554aadb176d673a25a8bce0aa0b61ff7a768d8e74b8a32adba0700d33\": rpc error: code = NotFound desc = could not find container \"dab8707554aadb176d673a25a8bce0aa0b61ff7a768d8e74b8a32adba0700d33\": container with ID starting with dab8707554aadb176d673a25a8bce0aa0b61ff7a768d8e74b8a32adba0700d33 not found: ID does not exist" Jul 10 02:56:54.976589 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:54.976292 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n9g7n" podStartSLOduration=13.976275577 podStartE2EDuration="13.976275577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:56:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:56:54.975159759 +0000 UTC m=+533.256276045" watchObservedRunningTime="2026-07-10 02:56:54.976275577 +0000 UTC m=+533.257391863" Jul 10 02:56:54.993490 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:54.993443 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t2zbx" podStartSLOduration=13.99342683 podStartE2EDuration="13.99342683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:56:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:56:54.992922935 +0000 UTC m=+533.274039221" watchObservedRunningTime="2026-07-10 02:56:54.99342683 +0000 UTC m=+533.274543133" Jul 10 02:56:54.994573 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:56:54.994548 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-lszpx\" is forbidden: User \"system:node:ip-10-0-130-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-139.ec2.internal' and this object" podUID="c65e0ad5-fb64-42c8-b1a9-172ab74ab15b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lszpx" Jul 10 02:56:54.996147 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:56:54.996121 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-lszpx\" is forbidden: User \"system:node:ip-10-0-130-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-139.ec2.internal' and this object" podUID="c65e0ad5-fb64-42c8-b1a9-172ab74ab15b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lszpx" Jul 10 02:56:54.997625 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:56:54.997597 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-zrxnt\" is forbidden: User \"system:node:ip-10-0-130-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-139.ec2.internal' and this object" podUID="10d8bf56-dd5d-4e0a-bec5-e6f3901b2347" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zrxnt" Jul 10 02:56:56.289193 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:56.289159 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10d8bf56-dd5d-4e0a-bec5-e6f3901b2347" path="/var/lib/kubelet/pods/10d8bf56-dd5d-4e0a-bec5-e6f3901b2347/volumes" Jul 10 02:56:56.289530 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:56.289507 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c65e0ad5-fb64-42c8-b1a9-172ab74ab15b" path="/var/lib/kubelet/pods/c65e0ad5-fb64-42c8-b1a9-172ab74ab15b/volumes" Jul 10 02:56:59.659386 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:59.659355 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n9g7n"] Jul 10 02:56:59.659830 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:59.659525 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n9g7n" podUID="60be55a4-238e-4b01-a659-3bbdb82878e7" containerName="manager" containerID="cri-o://e24ac25542fb64baf5de130d1c2a239905f71e2ee4c886687cc927f18d4c60ce" gracePeriod=10 Jul 10 02:56:59.661306 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:59.661281 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n9g7n" Jul 10 02:56:59.912233 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:59.912172 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n9g7n" Jul 10 02:56:59.970265 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:59.970236 2575 generic.go:356] "Generic (PLEG): container finished" podID="60be55a4-238e-4b01-a659-3bbdb82878e7" containerID="e24ac25542fb64baf5de130d1c2a239905f71e2ee4c886687cc927f18d4c60ce" exitCode=0 Jul 10 02:56:59.970423 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:59.970277 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n9g7n" Jul 10 02:56:59.970423 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:59.970285 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n9g7n" event={"ID":"60be55a4-238e-4b01-a659-3bbdb82878e7","Type":"ContainerDied","Data":"e24ac25542fb64baf5de130d1c2a239905f71e2ee4c886687cc927f18d4c60ce"} Jul 10 02:56:59.970423 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:59.970327 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n9g7n" event={"ID":"60be55a4-238e-4b01-a659-3bbdb82878e7","Type":"ContainerDied","Data":"be99906a3573211d5c150e5e7663c90daa977a9993a26e24bc79014d2bb29f79"} Jul 10 02:56:59.970423 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:59.970348 2575 scope.go:117] "RemoveContainer" containerID="e24ac25542fb64baf5de130d1c2a239905f71e2ee4c886687cc927f18d4c60ce" Jul 10 02:56:59.979240 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:59.979224 2575 scope.go:117] "RemoveContainer" containerID="e24ac25542fb64baf5de130d1c2a239905f71e2ee4c886687cc927f18d4c60ce" Jul 10 02:56:59.979498 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:56:59.979476 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e24ac25542fb64baf5de130d1c2a239905f71e2ee4c886687cc927f18d4c60ce\": container with ID starting with e24ac25542fb64baf5de130d1c2a239905f71e2ee4c886687cc927f18d4c60ce not found: ID does not exist" containerID="e24ac25542fb64baf5de130d1c2a239905f71e2ee4c886687cc927f18d4c60ce" Jul 10 02:56:59.979543 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:56:59.979505 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e24ac25542fb64baf5de130d1c2a239905f71e2ee4c886687cc927f18d4c60ce"} err="failed to get container status \"e24ac25542fb64baf5de130d1c2a239905f71e2ee4c886687cc927f18d4c60ce\": rpc error: code = NotFound desc = could not find container \"e24ac25542fb64baf5de130d1c2a239905f71e2ee4c886687cc927f18d4c60ce\": container with ID starting with e24ac25542fb64baf5de130d1c2a239905f71e2ee4c886687cc927f18d4c60ce not found: ID does not exist" Jul 10 02:57:00.025748 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:00.025723 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7t7h\" (UniqueName: \"kubernetes.io/projected/60be55a4-238e-4b01-a659-3bbdb82878e7-kube-api-access-s7t7h\") pod \"60be55a4-238e-4b01-a659-3bbdb82878e7\" (UID: \"60be55a4-238e-4b01-a659-3bbdb82878e7\") " Jul 10 02:57:00.025878 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:00.025828 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/60be55a4-238e-4b01-a659-3bbdb82878e7-extensions-socket-volume\") pod \"60be55a4-238e-4b01-a659-3bbdb82878e7\" (UID: \"60be55a4-238e-4b01-a659-3bbdb82878e7\") " Jul 10 02:57:00.026198 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:00.026175 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60be55a4-238e-4b01-a659-3bbdb82878e7-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "60be55a4-238e-4b01-a659-3bbdb82878e7" (UID: "60be55a4-238e-4b01-a659-3bbdb82878e7"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 02:57:00.027707 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:00.027688 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60be55a4-238e-4b01-a659-3bbdb82878e7-kube-api-access-s7t7h" (OuterVolumeSpecName: "kube-api-access-s7t7h") pod "60be55a4-238e-4b01-a659-3bbdb82878e7" (UID: "60be55a4-238e-4b01-a659-3bbdb82878e7"). InnerVolumeSpecName "kube-api-access-s7t7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 02:57:00.126929 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:00.126897 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7t7h\" (UniqueName: \"kubernetes.io/projected/60be55a4-238e-4b01-a659-3bbdb82878e7-kube-api-access-s7t7h\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 02:57:00.126929 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:00.126923 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/60be55a4-238e-4b01-a659-3bbdb82878e7-extensions-socket-volume\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 02:57:00.292339 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:00.292313 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n9g7n"] Jul 10 02:57:00.302205 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:00.297906 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n9g7n"] Jul 10 02:57:02.288917 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:02.288889 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60be55a4-238e-4b01-a659-3bbdb82878e7" path="/var/lib/kubelet/pods/60be55a4-238e-4b01-a659-3bbdb82878e7/volumes" Jul 10 02:57:05.952256 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:05.952217 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t2zbx" Jul 10 02:57:16.007073 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:16.007001 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6zr5k"] Jul 10 02:57:16.007620 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:16.007586 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60be55a4-238e-4b01-a659-3bbdb82878e7" containerName="manager" Jul 10 02:57:16.007620 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:16.007618 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="60be55a4-238e-4b01-a659-3bbdb82878e7" containerName="manager" Jul 10 02:57:16.007784 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:16.007759 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="60be55a4-238e-4b01-a659-3bbdb82878e7" containerName="manager" Jul 10 02:57:16.113201 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:16.113164 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6zr5k"] Jul 10 02:57:16.113351 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:16.113261 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6zr5k" Jul 10 02:57:16.115499 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:16.115477 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-9w9pw\"" Jul 10 02:57:16.115622 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:16.115479 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 10 02:57:16.161613 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:16.161583 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e5039b0-ee00-4541-8216-c4a7015b93d1-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-6zr5k\" (UID: \"1e5039b0-ee00-4541-8216-c4a7015b93d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6zr5k" Jul 10 02:57:16.161760 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:16.161635 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5039b0-ee00-4541-8216-c4a7015b93d1-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-6zr5k\" (UID: \"1e5039b0-ee00-4541-8216-c4a7015b93d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6zr5k" Jul 10 02:57:16.161760 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:16.161661 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e5039b0-ee00-4541-8216-c4a7015b93d1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-6zr5k\" (UID: \"1e5039b0-ee00-4541-8216-c4a7015b93d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6zr5k" Jul 10 02:57:16.161760 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:16.161715 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e5039b0-ee00-4541-8216-c4a7015b93d1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-6zr5k\" (UID: \"1e5039b0-ee00-4541-8216-c4a7015b93d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6zr5k" Jul 10 02:57:16.161913 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:16.161762 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5039b0-ee00-4541-8216-c4a7015b93d1-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-6zr5k\" (UID: \"1e5039b0-ee00-4541-8216-c4a7015b93d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6zr5k" Jul 10 02:57:16.161913 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:16.161792 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e5039b0-ee00-4541-8216-c4a7015b93d1-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-6zr5k\" (UID: \"1e5039b0-ee00-4541-8216-c4a7015b93d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6zr5k" Jul 10 02:57:16.161913 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:16.161849 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e5039b0-ee00-4541-8216-c4a7015b93d1-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-6zr5k\" (UID: \"1e5039b0-ee00-4541-8216-c4a7015b93d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6zr5k" Jul 10 02:57:16.161913 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:16.161877 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqmnr\" (UniqueName: \"kubernetes.io/projected/1e5039b0-ee00-4541-8216-c4a7015b93d1-kube-api-access-mqmnr\") pod \"maas-default-gateway-openshift-default-56654dcf7-6zr5k\" (UID: \"1e5039b0-ee00-4541-8216-c4a7015b93d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6zr5k" Jul 10 02:57:16.161913 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:16.161899 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e5039b0-ee00-4541-8216-c4a7015b93d1-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-6zr5k\" (UID: \"1e5039b0-ee00-4541-8216-c4a7015b93d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6zr5k" Jul 10 02:57:16.262689 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:16.262602 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5039b0-ee00-4541-8216-c4a7015b93d1-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-6zr5k\" (UID: \"1e5039b0-ee00-4541-8216-c4a7015b93d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6zr5k" Jul 10 02:57:16.262689 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:16.262639 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e5039b0-ee00-4541-8216-c4a7015b93d1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-6zr5k\" (UID: \"1e5039b0-ee00-4541-8216-c4a7015b93d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6zr5k" Jul 10 02:57:16.262689 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:16.262662 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e5039b0-ee00-4541-8216-c4a7015b93d1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-6zr5k\" (UID: \"1e5039b0-ee00-4541-8216-c4a7015b93d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6zr5k" Jul 10 02:57:16.262983 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:16.262780 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5039b0-ee00-4541-8216-c4a7015b93d1-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-6zr5k\" (UID: \"1e5039b0-ee00-4541-8216-c4a7015b93d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6zr5k" Jul 10 02:57:16.262983 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:16.262827 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e5039b0-ee00-4541-8216-c4a7015b93d1-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-6zr5k\" (UID: \"1e5039b0-ee00-4541-8216-c4a7015b93d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6zr5k" Jul 10 02:57:16.262983 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:16.262853 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e5039b0-ee00-4541-8216-c4a7015b93d1-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-6zr5k\" (UID: \"1e5039b0-ee00-4541-8216-c4a7015b93d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6zr5k" Jul 10 02:57:16.262983 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:16.262924 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqmnr\" (UniqueName: \"kubernetes.io/projected/1e5039b0-ee00-4541-8216-c4a7015b93d1-kube-api-access-mqmnr\") pod \"maas-default-gateway-openshift-default-56654dcf7-6zr5k\" (UID: \"1e5039b0-ee00-4541-8216-c4a7015b93d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6zr5k" Jul 10 02:57:16.262983 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:16.262965 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e5039b0-ee00-4541-8216-c4a7015b93d1-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-6zr5k\" (UID: \"1e5039b0-ee00-4541-8216-c4a7015b93d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6zr5k" Jul 10 02:57:16.263229 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:16.263050 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e5039b0-ee00-4541-8216-c4a7015b93d1-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-6zr5k\" (UID: \"1e5039b0-ee00-4541-8216-c4a7015b93d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6zr5k" Jul 10 02:57:16.263229 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:16.263166 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5039b0-ee00-4541-8216-c4a7015b93d1-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-6zr5k\" (UID: \"1e5039b0-ee00-4541-8216-c4a7015b93d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6zr5k" Jul 10 02:57:16.263229 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:16.263221 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e5039b0-ee00-4541-8216-c4a7015b93d1-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-6zr5k\" (UID: \"1e5039b0-ee00-4541-8216-c4a7015b93d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6zr5k" Jul 10 02:57:16.263547 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:16.263471 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e5039b0-ee00-4541-8216-c4a7015b93d1-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-6zr5k\" (UID: \"1e5039b0-ee00-4541-8216-c4a7015b93d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6zr5k" Jul 10 02:57:16.263547 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:16.263495 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e5039b0-ee00-4541-8216-c4a7015b93d1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-6zr5k\" (UID: \"1e5039b0-ee00-4541-8216-c4a7015b93d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6zr5k" Jul 10 02:57:16.263747 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:16.263599 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5039b0-ee00-4541-8216-c4a7015b93d1-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-6zr5k\" (UID: \"1e5039b0-ee00-4541-8216-c4a7015b93d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6zr5k" Jul 10 02:57:16.265254 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:16.265222 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e5039b0-ee00-4541-8216-c4a7015b93d1-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-6zr5k\" (UID: \"1e5039b0-ee00-4541-8216-c4a7015b93d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6zr5k" Jul 10 02:57:16.265538 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:16.265517 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e5039b0-ee00-4541-8216-c4a7015b93d1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-6zr5k\" (UID: \"1e5039b0-ee00-4541-8216-c4a7015b93d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6zr5k" Jul 10 02:57:16.270348 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:16.270323 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqmnr\" (UniqueName: \"kubernetes.io/projected/1e5039b0-ee00-4541-8216-c4a7015b93d1-kube-api-access-mqmnr\") pod \"maas-default-gateway-openshift-default-56654dcf7-6zr5k\" (UID: \"1e5039b0-ee00-4541-8216-c4a7015b93d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6zr5k" Jul 10 02:57:16.270451 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:16.270431 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e5039b0-ee00-4541-8216-c4a7015b93d1-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-6zr5k\" (UID: \"1e5039b0-ee00-4541-8216-c4a7015b93d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6zr5k" Jul 10 02:57:16.424158 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:16.424117 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6zr5k" Jul 10 02:57:16.610675 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:16.610648 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6zr5k"] Jul 10 02:57:16.613069 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:57:16.613041 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e5039b0_ee00_4541_8216_c4a7015b93d1.slice/crio-4e157fd6f5ddaea24688f426fa2ea2d45fcaea876158bd68d5e7b7589315305b WatchSource:0}: Error finding container 4e157fd6f5ddaea24688f426fa2ea2d45fcaea876158bd68d5e7b7589315305b: Status 404 returned error can't find the container with id 4e157fd6f5ddaea24688f426fa2ea2d45fcaea876158bd68d5e7b7589315305b Jul 10 02:57:17.054045 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:17.054003 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6zr5k" event={"ID":"1e5039b0-ee00-4541-8216-c4a7015b93d1","Type":"ContainerStarted","Data":"4e157fd6f5ddaea24688f426fa2ea2d45fcaea876158bd68d5e7b7589315305b"} Jul 10 02:57:20.184634 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:20.184596 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236236Ki","pods":"250"} Jul 10 02:57:20.184926 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:20.184685 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236236Ki","pods":"250"} Jul 10 02:57:20.184926 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:20.184724 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236236Ki","pods":"250"} Jul 10 02:57:21.077208 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:21.077174 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6zr5k" event={"ID":"1e5039b0-ee00-4541-8216-c4a7015b93d1","Type":"ContainerStarted","Data":"087423b36da81799884abc67feb5eb9f994fc83b30e14746f3b72a88a7d2330f"} Jul 10 02:57:21.097141 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:21.097094 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6zr5k" podStartSLOduration=2.527860359 podStartE2EDuration="6.097080119s" totalImagesPullingTime="3.56921976s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:57:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:57:21.094431672 +0000 UTC m=+559.375547958" watchObservedRunningTime="2026-07-10 02:57:21.097080119 +0000 UTC m=+559.378196403" Jul 10 02:57:21.424669 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:21.424583 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6zr5k" Jul 10 02:57:21.429525 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:21.429500 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6zr5k" Jul 10 02:57:22.082296 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:22.082269 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6zr5k" Jul 10 02:57:22.083466 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:22.083441 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6zr5k" Jul 10 02:57:22.171171 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:22.171133 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-qvgb6"] Jul 10 02:57:22.174919 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:22.174896 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-qvgb6" Jul 10 02:57:22.177105 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:22.177086 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 10 02:57:22.184146 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:22.184124 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-qvgb6"] Jul 10 02:57:22.204570 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:22.204536 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq7mc\" (UniqueName: \"kubernetes.io/projected/4ec42757-94b6-46ab-9caf-4e53c9de8730-kube-api-access-xq7mc\") pod \"limitador-limitador-64c94fd64f-qvgb6\" (UID: \"4ec42757-94b6-46ab-9caf-4e53c9de8730\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-qvgb6" Jul 10 02:57:22.204751 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:22.204612 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4ec42757-94b6-46ab-9caf-4e53c9de8730-config-file\") pod \"limitador-limitador-64c94fd64f-qvgb6\" (UID: \"4ec42757-94b6-46ab-9caf-4e53c9de8730\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-qvgb6" Jul 10 02:57:22.270596 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:22.270564 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-qvgb6"] Jul 10 02:57:22.305238 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:22.305210 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq7mc\" (UniqueName: \"kubernetes.io/projected/4ec42757-94b6-46ab-9caf-4e53c9de8730-kube-api-access-xq7mc\") pod \"limitador-limitador-64c94fd64f-qvgb6\" (UID: \"4ec42757-94b6-46ab-9caf-4e53c9de8730\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-qvgb6" Jul 10 02:57:22.305417 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:22.305294 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4ec42757-94b6-46ab-9caf-4e53c9de8730-config-file\") pod \"limitador-limitador-64c94fd64f-qvgb6\" (UID: \"4ec42757-94b6-46ab-9caf-4e53c9de8730\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-qvgb6" Jul 10 02:57:22.305936 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:22.305916 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4ec42757-94b6-46ab-9caf-4e53c9de8730-config-file\") pod \"limitador-limitador-64c94fd64f-qvgb6\" (UID: \"4ec42757-94b6-46ab-9caf-4e53c9de8730\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-qvgb6" Jul 10 02:57:22.313061 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:22.313039 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq7mc\" (UniqueName: \"kubernetes.io/projected/4ec42757-94b6-46ab-9caf-4e53c9de8730-kube-api-access-xq7mc\") pod \"limitador-limitador-64c94fd64f-qvgb6\" (UID: \"4ec42757-94b6-46ab-9caf-4e53c9de8730\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-qvgb6" Jul 10 02:57:22.486838 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:22.486745 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-qvgb6" Jul 10 02:57:22.622932 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:22.622908 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-qvgb6"] Jul 10 02:57:22.625297 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:57:22.625270 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ec42757_94b6_46ab_9caf_4e53c9de8730.slice/crio-929a20034205db5846039fc70ce60fcd3f0c5e6b1cd888998cc4b7e15917f442 WatchSource:0}: Error finding container 929a20034205db5846039fc70ce60fcd3f0c5e6b1cd888998cc4b7e15917f442: Status 404 returned error can't find the container with id 929a20034205db5846039fc70ce60fcd3f0c5e6b1cd888998cc4b7e15917f442 Jul 10 02:57:23.087410 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:23.087379 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-qvgb6" event={"ID":"4ec42757-94b6-46ab-9caf-4e53c9de8730","Type":"ContainerStarted","Data":"929a20034205db5846039fc70ce60fcd3f0c5e6b1cd888998cc4b7e15917f442"} Jul 10 02:57:23.129913 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:23.129884 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-tfsxg"] Jul 10 02:57:23.134550 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:23.134535 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-tfsxg" Jul 10 02:57:23.137154 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:23.137133 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-mhdwd\"" Jul 10 02:57:23.139411 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:23.139390 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-tfsxg"] Jul 10 02:57:23.212344 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:23.212309 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-576nc\" (UniqueName: \"kubernetes.io/projected/5157e52c-5417-4864-9622-a87b74d82728-kube-api-access-576nc\") pod \"authorino-75d556bd77-tfsxg\" (UID: \"5157e52c-5417-4864-9622-a87b74d82728\") " pod="kuadrant-system/authorino-75d556bd77-tfsxg" Jul 10 02:57:23.312665 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:23.312624 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-576nc\" (UniqueName: \"kubernetes.io/projected/5157e52c-5417-4864-9622-a87b74d82728-kube-api-access-576nc\") pod \"authorino-75d556bd77-tfsxg\" (UID: \"5157e52c-5417-4864-9622-a87b74d82728\") " pod="kuadrant-system/authorino-75d556bd77-tfsxg" Jul 10 02:57:23.320334 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:23.320304 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-576nc\" (UniqueName: \"kubernetes.io/projected/5157e52c-5417-4864-9622-a87b74d82728-kube-api-access-576nc\") pod \"authorino-75d556bd77-tfsxg\" (UID: \"5157e52c-5417-4864-9622-a87b74d82728\") " pod="kuadrant-system/authorino-75d556bd77-tfsxg" Jul 10 02:57:23.446193 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:23.445908 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-tfsxg" Jul 10 02:57:23.649433 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:23.648759 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-tfsxg"] Jul 10 02:57:24.093483 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:24.093444 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-tfsxg" event={"ID":"5157e52c-5417-4864-9622-a87b74d82728","Type":"ContainerStarted","Data":"ce67ebb49e35de1a5eede13f9be11598d22d99f8550635b989c836a69190c8f8"} Jul 10 02:57:28.126306 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:28.126199 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-qvgb6" event={"ID":"4ec42757-94b6-46ab-9caf-4e53c9de8730","Type":"ContainerStarted","Data":"12a1ac9dd51884258ec9c5b98679fe5e6c6b132d1a5b7fb20d448580e5cb8fe8"} Jul 10 02:57:28.126306 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:28.126267 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-qvgb6" Jul 10 02:57:28.127982 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:28.127952 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-tfsxg" event={"ID":"5157e52c-5417-4864-9622-a87b74d82728","Type":"ContainerStarted","Data":"0d7495a456082601377c818502d96fc1bd30fa75b7252dd953584a26274d3ca9"} Jul 10 02:57:28.145062 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:28.145012 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-qvgb6" podStartSLOduration=0.766817534 podStartE2EDuration="6.144996393s" totalImagesPullingTime="5.378178859s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:57:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:57:28.14188146 +0000 UTC m=+566.422997767" watchObservedRunningTime="2026-07-10 02:57:28.144996393 +0000 UTC m=+566.426112677" Jul 10 02:57:28.162467 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:28.162403 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-tfsxg" podStartSLOduration=0.809499786 podStartE2EDuration="5.162387875s" totalImagesPullingTime="4.352888089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:57:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:57:28.160995466 +0000 UTC m=+566.442111747" watchObservedRunningTime="2026-07-10 02:57:28.162387875 +0000 UTC m=+566.443504160" Jul 10 02:57:38.619495 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:38.619457 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-qvgb6"] Jul 10 02:57:38.619959 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:38.619661 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-qvgb6" podUID="4ec42757-94b6-46ab-9caf-4e53c9de8730" containerName="limitador" containerID="cri-o://12a1ac9dd51884258ec9c5b98679fe5e6c6b132d1a5b7fb20d448580e5cb8fe8" gracePeriod=30 Jul 10 02:57:38.621455 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:38.621436 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-qvgb6" Jul 10 02:57:39.171907 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:39.171885 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-qvgb6" Jul 10 02:57:39.183446 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:39.183387 2575 generic.go:356] "Generic (PLEG): container finished" podID="4ec42757-94b6-46ab-9caf-4e53c9de8730" containerID="12a1ac9dd51884258ec9c5b98679fe5e6c6b132d1a5b7fb20d448580e5cb8fe8" exitCode=0 Jul 10 02:57:39.183446 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:39.183426 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-qvgb6" Jul 10 02:57:39.183620 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:39.183460 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-qvgb6" event={"ID":"4ec42757-94b6-46ab-9caf-4e53c9de8730","Type":"ContainerDied","Data":"12a1ac9dd51884258ec9c5b98679fe5e6c6b132d1a5b7fb20d448580e5cb8fe8"} Jul 10 02:57:39.183620 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:39.183505 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-qvgb6" event={"ID":"4ec42757-94b6-46ab-9caf-4e53c9de8730","Type":"ContainerDied","Data":"929a20034205db5846039fc70ce60fcd3f0c5e6b1cd888998cc4b7e15917f442"} Jul 10 02:57:39.183620 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:39.183534 2575 scope.go:117] "RemoveContainer" containerID="12a1ac9dd51884258ec9c5b98679fe5e6c6b132d1a5b7fb20d448580e5cb8fe8" Jul 10 02:57:39.193002 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:39.192950 2575 scope.go:117] "RemoveContainer" containerID="12a1ac9dd51884258ec9c5b98679fe5e6c6b132d1a5b7fb20d448580e5cb8fe8" Jul 10 02:57:39.194171 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:57:39.194144 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12a1ac9dd51884258ec9c5b98679fe5e6c6b132d1a5b7fb20d448580e5cb8fe8\": container with ID starting with 12a1ac9dd51884258ec9c5b98679fe5e6c6b132d1a5b7fb20d448580e5cb8fe8 not found: ID does not exist" containerID="12a1ac9dd51884258ec9c5b98679fe5e6c6b132d1a5b7fb20d448580e5cb8fe8" Jul 10 02:57:39.194267 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:39.194179 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12a1ac9dd51884258ec9c5b98679fe5e6c6b132d1a5b7fb20d448580e5cb8fe8"} err="failed to get container status \"12a1ac9dd51884258ec9c5b98679fe5e6c6b132d1a5b7fb20d448580e5cb8fe8\": rpc error: code = NotFound desc = could not find container \"12a1ac9dd51884258ec9c5b98679fe5e6c6b132d1a5b7fb20d448580e5cb8fe8\": container with ID starting with 12a1ac9dd51884258ec9c5b98679fe5e6c6b132d1a5b7fb20d448580e5cb8fe8 not found: ID does not exist" Jul 10 02:57:39.252099 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:39.252069 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4ec42757-94b6-46ab-9caf-4e53c9de8730-config-file\") pod \"4ec42757-94b6-46ab-9caf-4e53c9de8730\" (UID: \"4ec42757-94b6-46ab-9caf-4e53c9de8730\") " Jul 10 02:57:39.252221 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:39.252148 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xq7mc\" (UniqueName: \"kubernetes.io/projected/4ec42757-94b6-46ab-9caf-4e53c9de8730-kube-api-access-xq7mc\") pod \"4ec42757-94b6-46ab-9caf-4e53c9de8730\" (UID: \"4ec42757-94b6-46ab-9caf-4e53c9de8730\") " Jul 10 02:57:39.252495 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:39.252465 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ec42757-94b6-46ab-9caf-4e53c9de8730-config-file" (OuterVolumeSpecName: "config-file") pod "4ec42757-94b6-46ab-9caf-4e53c9de8730" (UID: "4ec42757-94b6-46ab-9caf-4e53c9de8730"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 02:57:39.254188 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:39.254166 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ec42757-94b6-46ab-9caf-4e53c9de8730-kube-api-access-xq7mc" (OuterVolumeSpecName: "kube-api-access-xq7mc") pod "4ec42757-94b6-46ab-9caf-4e53c9de8730" (UID: "4ec42757-94b6-46ab-9caf-4e53c9de8730"). InnerVolumeSpecName "kube-api-access-xq7mc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 02:57:39.353656 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:39.353627 2575 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4ec42757-94b6-46ab-9caf-4e53c9de8730-config-file\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 02:57:39.353656 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:39.353653 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq7mc\" (UniqueName: \"kubernetes.io/projected/4ec42757-94b6-46ab-9caf-4e53c9de8730-kube-api-access-xq7mc\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 02:57:39.503773 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:39.503741 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-qvgb6"] Jul 10 02:57:39.507111 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:39.507086 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-qvgb6"] Jul 10 02:57:40.289222 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:40.289178 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ec42757-94b6-46ab-9caf-4e53c9de8730" path="/var/lib/kubelet/pods/4ec42757-94b6-46ab-9caf-4e53c9de8730/volumes" Jul 10 02:57:43.835777 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:43.835742 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-nrl67"] Jul 10 02:57:43.836260 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:43.836239 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ec42757-94b6-46ab-9caf-4e53c9de8730" containerName="limitador" Jul 10 02:57:43.836260 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:43.836258 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ec42757-94b6-46ab-9caf-4e53c9de8730" containerName="limitador" Jul 10 02:57:43.836391 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:43.836339 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ec42757-94b6-46ab-9caf-4e53c9de8730" containerName="limitador" Jul 10 02:57:43.840863 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:43.840841 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-nrl67" Jul 10 02:57:43.843399 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:43.843371 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-wbhgt\"" Jul 10 02:57:43.843399 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:43.843399 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 02:57:43.844162 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:43.844140 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 10 02:57:43.844162 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:43.844157 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 10 02:57:43.849710 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:43.849686 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-nrl67"] Jul 10 02:57:43.897550 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:43.897518 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b29e6d04-2b35-432e-9764-f047667337db-data\") pod \"postgres-69fb4594cf-nrl67\" (UID: \"b29e6d04-2b35-432e-9764-f047667337db\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-nrl67" Jul 10 02:57:43.897667 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:43.897575 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bzzh\" (UniqueName: \"kubernetes.io/projected/b29e6d04-2b35-432e-9764-f047667337db-kube-api-access-2bzzh\") pod \"postgres-69fb4594cf-nrl67\" (UID: \"b29e6d04-2b35-432e-9764-f047667337db\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-nrl67" Jul 10 02:57:43.998296 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:43.998269 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b29e6d04-2b35-432e-9764-f047667337db-data\") pod \"postgres-69fb4594cf-nrl67\" (UID: \"b29e6d04-2b35-432e-9764-f047667337db\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-nrl67" Jul 10 02:57:43.998446 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:43.998315 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bzzh\" (UniqueName: \"kubernetes.io/projected/b29e6d04-2b35-432e-9764-f047667337db-kube-api-access-2bzzh\") pod \"postgres-69fb4594cf-nrl67\" (UID: \"b29e6d04-2b35-432e-9764-f047667337db\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-nrl67" Jul 10 02:57:43.998678 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:43.998651 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b29e6d04-2b35-432e-9764-f047667337db-data\") pod \"postgres-69fb4594cf-nrl67\" (UID: \"b29e6d04-2b35-432e-9764-f047667337db\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-nrl67" Jul 10 02:57:44.010009 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:44.009984 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bzzh\" (UniqueName: \"kubernetes.io/projected/b29e6d04-2b35-432e-9764-f047667337db-kube-api-access-2bzzh\") pod \"postgres-69fb4594cf-nrl67\" (UID: \"b29e6d04-2b35-432e-9764-f047667337db\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-nrl67" Jul 10 02:57:44.154655 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:44.154577 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-nrl67" Jul 10 02:57:44.288736 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:44.288711 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-nrl67"] Jul 10 02:57:44.289548 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:57:44.289523 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb29e6d04_2b35_432e_9764_f047667337db.slice/crio-0b105e775d120c8b2bafc22075d9be90db86fa499b7653210e45b5bcbf0ee814 WatchSource:0}: Error finding container 0b105e775d120c8b2bafc22075d9be90db86fa499b7653210e45b5bcbf0ee814: Status 404 returned error can't find the container with id 0b105e775d120c8b2bafc22075d9be90db86fa499b7653210e45b5bcbf0ee814 Jul 10 02:57:45.221329 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:45.221276 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-nrl67" event={"ID":"b29e6d04-2b35-432e-9764-f047667337db","Type":"ContainerStarted","Data":"0b105e775d120c8b2bafc22075d9be90db86fa499b7653210e45b5bcbf0ee814"} Jul 10 02:57:50.248527 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:50.248455 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-nrl67" event={"ID":"b29e6d04-2b35-432e-9764-f047667337db","Type":"ContainerStarted","Data":"2647cb429a055e45bcb0c121013c93e3c13dd3a6f20c80edf1707beb833406c4"} Jul 10 02:57:50.249033 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:50.248700 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-nrl67" Jul 10 02:57:50.265687 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:50.265636 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-nrl67" podStartSLOduration=2.152566195 podStartE2EDuration="7.265619938s" totalImagesPullingTime="5.113053743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:57:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:57:50.263796994 +0000 UTC m=+588.544913298" watchObservedRunningTime="2026-07-10 02:57:50.265619938 +0000 UTC m=+588.546736263" Jul 10 02:57:56.289471 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:56.289443 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-nrl67" Jul 10 02:57:57.308990 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:57.308956 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-597dbc55fc-zdl7g"] Jul 10 02:57:57.312460 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:57.312441 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-597dbc55fc-zdl7g" Jul 10 02:57:57.314764 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:57.314744 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 10 02:57:57.319035 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:57.319013 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-597dbc55fc-zdl7g"] Jul 10 02:57:57.423137 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:57.423102 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96wzz\" (UniqueName: \"kubernetes.io/projected/9635f986-882e-4a5b-984f-0277e5440a03-kube-api-access-96wzz\") pod \"authorino-597dbc55fc-zdl7g\" (UID: \"9635f986-882e-4a5b-984f-0277e5440a03\") " pod="kuadrant-system/authorino-597dbc55fc-zdl7g" Jul 10 02:57:57.423310 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:57.423298 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9635f986-882e-4a5b-984f-0277e5440a03-tls-cert\") pod \"authorino-597dbc55fc-zdl7g\" (UID: \"9635f986-882e-4a5b-984f-0277e5440a03\") " pod="kuadrant-system/authorino-597dbc55fc-zdl7g" Jul 10 02:57:57.523965 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:57.523928 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9635f986-882e-4a5b-984f-0277e5440a03-tls-cert\") pod \"authorino-597dbc55fc-zdl7g\" (UID: \"9635f986-882e-4a5b-984f-0277e5440a03\") " pod="kuadrant-system/authorino-597dbc55fc-zdl7g" Jul 10 02:57:57.524134 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:57.524049 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96wzz\" (UniqueName: \"kubernetes.io/projected/9635f986-882e-4a5b-984f-0277e5440a03-kube-api-access-96wzz\") pod \"authorino-597dbc55fc-zdl7g\" (UID: \"9635f986-882e-4a5b-984f-0277e5440a03\") " pod="kuadrant-system/authorino-597dbc55fc-zdl7g" Jul 10 02:57:57.526458 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:57.526434 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9635f986-882e-4a5b-984f-0277e5440a03-tls-cert\") pod \"authorino-597dbc55fc-zdl7g\" (UID: \"9635f986-882e-4a5b-984f-0277e5440a03\") " pod="kuadrant-system/authorino-597dbc55fc-zdl7g" Jul 10 02:57:57.531763 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:57.531740 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96wzz\" (UniqueName: \"kubernetes.io/projected/9635f986-882e-4a5b-984f-0277e5440a03-kube-api-access-96wzz\") pod \"authorino-597dbc55fc-zdl7g\" (UID: \"9635f986-882e-4a5b-984f-0277e5440a03\") " pod="kuadrant-system/authorino-597dbc55fc-zdl7g" Jul 10 02:57:57.622843 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:57.622747 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-597dbc55fc-zdl7g" Jul 10 02:57:57.750382 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:57.750352 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-597dbc55fc-zdl7g"] Jul 10 02:57:57.754086 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:57:57.754056 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9635f986_882e_4a5b_984f_0277e5440a03.slice/crio-d4b63f1bcba839905e37571188878e34cda6d1859617bda69e0c690ef94bcdbf WatchSource:0}: Error finding container d4b63f1bcba839905e37571188878e34cda6d1859617bda69e0c690ef94bcdbf: Status 404 returned error can't find the container with id d4b63f1bcba839905e37571188878e34cda6d1859617bda69e0c690ef94bcdbf Jul 10 02:57:58.292997 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:58.292959 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-597dbc55fc-zdl7g" event={"ID":"9635f986-882e-4a5b-984f-0277e5440a03","Type":"ContainerStarted","Data":"bd882f9fe7fbacc8453891bf2a4da00664b0d9425fc487eff795305a45e8e7f4"} Jul 10 02:57:58.293311 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:58.293004 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-597dbc55fc-zdl7g" event={"ID":"9635f986-882e-4a5b-984f-0277e5440a03","Type":"ContainerStarted","Data":"d4b63f1bcba839905e37571188878e34cda6d1859617bda69e0c690ef94bcdbf"} Jul 10 02:57:58.309224 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:58.309170 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-597dbc55fc-zdl7g" podStartSLOduration=0.914065395 podStartE2EDuration="1.309155007s" totalImagesPullingTime="395.089612ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:57:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:57:58.308520362 +0000 UTC m=+596.589636669" watchObservedRunningTime="2026-07-10 02:57:58.309155007 +0000 UTC m=+596.590271292" Jul 10 02:57:58.334677 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:58.334646 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-tfsxg"] Jul 10 02:57:58.334847 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:58.334824 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-tfsxg" podUID="5157e52c-5417-4864-9622-a87b74d82728" containerName="authorino" containerID="cri-o://0d7495a456082601377c818502d96fc1bd30fa75b7252dd953584a26274d3ca9" gracePeriod=30 Jul 10 02:57:58.595061 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:58.595033 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-tfsxg" Jul 10 02:57:58.735559 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:58.735485 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-576nc\" (UniqueName: \"kubernetes.io/projected/5157e52c-5417-4864-9622-a87b74d82728-kube-api-access-576nc\") pod \"5157e52c-5417-4864-9622-a87b74d82728\" (UID: \"5157e52c-5417-4864-9622-a87b74d82728\") " Jul 10 02:57:58.737575 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:58.737549 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5157e52c-5417-4864-9622-a87b74d82728-kube-api-access-576nc" (OuterVolumeSpecName: "kube-api-access-576nc") pod "5157e52c-5417-4864-9622-a87b74d82728" (UID: "5157e52c-5417-4864-9622-a87b74d82728"). InnerVolumeSpecName "kube-api-access-576nc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 02:57:58.836246 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:58.836214 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-576nc\" (UniqueName: \"kubernetes.io/projected/5157e52c-5417-4864-9622-a87b74d82728-kube-api-access-576nc\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 02:57:59.305762 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:59.305727 2575 generic.go:356] "Generic (PLEG): container finished" podID="5157e52c-5417-4864-9622-a87b74d82728" containerID="0d7495a456082601377c818502d96fc1bd30fa75b7252dd953584a26274d3ca9" exitCode=0 Jul 10 02:57:59.306007 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:59.305784 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-tfsxg" Jul 10 02:57:59.306007 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:59.305824 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-tfsxg" event={"ID":"5157e52c-5417-4864-9622-a87b74d82728","Type":"ContainerDied","Data":"0d7495a456082601377c818502d96fc1bd30fa75b7252dd953584a26274d3ca9"} Jul 10 02:57:59.306007 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:59.305869 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-tfsxg" event={"ID":"5157e52c-5417-4864-9622-a87b74d82728","Type":"ContainerDied","Data":"ce67ebb49e35de1a5eede13f9be11598d22d99f8550635b989c836a69190c8f8"} Jul 10 02:57:59.306007 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:59.305891 2575 scope.go:117] "RemoveContainer" containerID="0d7495a456082601377c818502d96fc1bd30fa75b7252dd953584a26274d3ca9" Jul 10 02:57:59.316062 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:59.316046 2575 scope.go:117] "RemoveContainer" containerID="0d7495a456082601377c818502d96fc1bd30fa75b7252dd953584a26274d3ca9" Jul 10 02:57:59.316354 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:57:59.316294 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d7495a456082601377c818502d96fc1bd30fa75b7252dd953584a26274d3ca9\": container with ID starting with 0d7495a456082601377c818502d96fc1bd30fa75b7252dd953584a26274d3ca9 not found: ID does not exist" containerID="0d7495a456082601377c818502d96fc1bd30fa75b7252dd953584a26274d3ca9" Jul 10 02:57:59.316354 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:59.316319 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d7495a456082601377c818502d96fc1bd30fa75b7252dd953584a26274d3ca9"} err="failed to get container status \"0d7495a456082601377c818502d96fc1bd30fa75b7252dd953584a26274d3ca9\": rpc error: code = NotFound desc = could not find container \"0d7495a456082601377c818502d96fc1bd30fa75b7252dd953584a26274d3ca9\": container with ID starting with 0d7495a456082601377c818502d96fc1bd30fa75b7252dd953584a26274d3ca9 not found: ID does not exist" Jul 10 02:57:59.329229 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:59.329205 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-tfsxg"] Jul 10 02:57:59.333559 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:57:59.333541 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-tfsxg"] Jul 10 02:58:00.289252 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:00.289218 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5157e52c-5417-4864-9622-a87b74d82728" path="/var/lib/kubelet/pods/5157e52c-5417-4864-9622-a87b74d82728/volumes" Jul 10 02:58:02.222943 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:02.222917 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-xxlp6_372be3ce-a380-4c0e-bbee-49c2e3b80095/console-operator/1.log" Jul 10 02:58:02.228312 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:02.228289 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-xxlp6_372be3ce-a380-4c0e-bbee-49c2e3b80095/console-operator/1.log" Jul 10 02:58:02.734720 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:02.734679 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-cb79cdddf-7s2vt"] Jul 10 02:58:02.735333 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:02.735308 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5157e52c-5417-4864-9622-a87b74d82728" containerName="authorino" Jul 10 02:58:02.735333 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:02.735329 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5157e52c-5417-4864-9622-a87b74d82728" containerName="authorino" Jul 10 02:58:02.735479 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:02.735469 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5157e52c-5417-4864-9622-a87b74d82728" containerName="authorino" Jul 10 02:58:02.741630 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:02.741603 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cb79cdddf-7s2vt" Jul 10 02:58:02.744385 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:02.744347 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 10 02:58:02.744662 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:02.744640 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-64bvh\"" Jul 10 02:58:02.744779 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:02.744697 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 10 02:58:02.746287 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:02.746261 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-cb79cdddf-7s2vt"] Jul 10 02:58:02.872422 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:02.872384 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/70c67a25-a4c6-47bf-9209-9cfcab6cfca0-webhook-cert\") pod \"maas-controller-cb79cdddf-7s2vt\" (UID: \"70c67a25-a4c6-47bf-9209-9cfcab6cfca0\") " pod="opendatahub/maas-controller-cb79cdddf-7s2vt" Jul 10 02:58:02.872422 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:02.872421 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnmwn\" (UniqueName: \"kubernetes.io/projected/70c67a25-a4c6-47bf-9209-9cfcab6cfca0-kube-api-access-dnmwn\") pod \"maas-controller-cb79cdddf-7s2vt\" (UID: \"70c67a25-a4c6-47bf-9209-9cfcab6cfca0\") " pod="opendatahub/maas-controller-cb79cdddf-7s2vt" Jul 10 02:58:02.894599 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:02.894560 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5dcc8799c9-b6c5r"] Jul 10 02:58:02.898247 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:02.898224 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5dcc8799c9-b6c5r" Jul 10 02:58:02.909302 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:02.909279 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5dcc8799c9-b6c5r"] Jul 10 02:58:02.973950 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:02.973918 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/70c67a25-a4c6-47bf-9209-9cfcab6cfca0-webhook-cert\") pod \"maas-controller-cb79cdddf-7s2vt\" (UID: \"70c67a25-a4c6-47bf-9209-9cfcab6cfca0\") " pod="opendatahub/maas-controller-cb79cdddf-7s2vt" Jul 10 02:58:02.973950 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:02.973951 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnmwn\" (UniqueName: \"kubernetes.io/projected/70c67a25-a4c6-47bf-9209-9cfcab6cfca0-kube-api-access-dnmwn\") pod \"maas-controller-cb79cdddf-7s2vt\" (UID: \"70c67a25-a4c6-47bf-9209-9cfcab6cfca0\") " pod="opendatahub/maas-controller-cb79cdddf-7s2vt" Jul 10 02:58:02.974133 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:58:02.974078 2575 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 10 02:58:02.974173 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:58:02.974162 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70c67a25-a4c6-47bf-9209-9cfcab6cfca0-webhook-cert podName:70c67a25-a4c6-47bf-9209-9cfcab6cfca0 nodeName:}" failed. No retries permitted until 2026-07-10 02:58:03.474140967 +0000 UTC m=+601.755257238 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/70c67a25-a4c6-47bf-9209-9cfcab6cfca0-webhook-cert") pod "maas-controller-cb79cdddf-7s2vt" (UID: "70c67a25-a4c6-47bf-9209-9cfcab6cfca0") : secret "maas-controller-webhook-cert" not found Jul 10 02:58:02.982399 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:02.982371 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnmwn\" (UniqueName: \"kubernetes.io/projected/70c67a25-a4c6-47bf-9209-9cfcab6cfca0-kube-api-access-dnmwn\") pod \"maas-controller-cb79cdddf-7s2vt\" (UID: \"70c67a25-a4c6-47bf-9209-9cfcab6cfca0\") " pod="opendatahub/maas-controller-cb79cdddf-7s2vt" Jul 10 02:58:03.074662 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:03.074630 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j5mn\" (UniqueName: \"kubernetes.io/projected/81164ee1-cc95-4f0f-9acd-07982b980ee4-kube-api-access-9j5mn\") pod \"maas-controller-5dcc8799c9-b6c5r\" (UID: \"81164ee1-cc95-4f0f-9acd-07982b980ee4\") " pod="opendatahub/maas-controller-5dcc8799c9-b6c5r" Jul 10 02:58:03.074845 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:03.074678 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/81164ee1-cc95-4f0f-9acd-07982b980ee4-webhook-cert\") pod \"maas-controller-5dcc8799c9-b6c5r\" (UID: \"81164ee1-cc95-4f0f-9acd-07982b980ee4\") " pod="opendatahub/maas-controller-5dcc8799c9-b6c5r" Jul 10 02:58:03.175722 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:03.175682 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j5mn\" (UniqueName: \"kubernetes.io/projected/81164ee1-cc95-4f0f-9acd-07982b980ee4-kube-api-access-9j5mn\") pod \"maas-controller-5dcc8799c9-b6c5r\" (UID: \"81164ee1-cc95-4f0f-9acd-07982b980ee4\") " pod="opendatahub/maas-controller-5dcc8799c9-b6c5r" Jul 10 02:58:03.175923 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:03.175731 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/81164ee1-cc95-4f0f-9acd-07982b980ee4-webhook-cert\") pod \"maas-controller-5dcc8799c9-b6c5r\" (UID: \"81164ee1-cc95-4f0f-9acd-07982b980ee4\") " pod="opendatahub/maas-controller-5dcc8799c9-b6c5r" Jul 10 02:58:03.175923 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:58:03.175852 2575 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 10 02:58:03.175923 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:58:03.175910 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81164ee1-cc95-4f0f-9acd-07982b980ee4-webhook-cert podName:81164ee1-cc95-4f0f-9acd-07982b980ee4 nodeName:}" failed. No retries permitted until 2026-07-10 02:58:03.675894317 +0000 UTC m=+601.957010580 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/81164ee1-cc95-4f0f-9acd-07982b980ee4-webhook-cert") pod "maas-controller-5dcc8799c9-b6c5r" (UID: "81164ee1-cc95-4f0f-9acd-07982b980ee4") : secret "maas-controller-webhook-cert" not found Jul 10 02:58:03.183974 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:03.183945 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j5mn\" (UniqueName: \"kubernetes.io/projected/81164ee1-cc95-4f0f-9acd-07982b980ee4-kube-api-access-9j5mn\") pod \"maas-controller-5dcc8799c9-b6c5r\" (UID: \"81164ee1-cc95-4f0f-9acd-07982b980ee4\") " pod="opendatahub/maas-controller-5dcc8799c9-b6c5r" Jul 10 02:58:03.479306 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:03.479225 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/70c67a25-a4c6-47bf-9209-9cfcab6cfca0-webhook-cert\") pod \"maas-controller-cb79cdddf-7s2vt\" (UID: \"70c67a25-a4c6-47bf-9209-9cfcab6cfca0\") " pod="opendatahub/maas-controller-cb79cdddf-7s2vt" Jul 10 02:58:03.479679 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:58:03.479402 2575 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 10 02:58:03.479679 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:58:03.479476 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70c67a25-a4c6-47bf-9209-9cfcab6cfca0-webhook-cert podName:70c67a25-a4c6-47bf-9209-9cfcab6cfca0 nodeName:}" failed. No retries permitted until 2026-07-10 02:58:04.479456802 +0000 UTC m=+602.760573078 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/70c67a25-a4c6-47bf-9209-9cfcab6cfca0-webhook-cert") pod "maas-controller-cb79cdddf-7s2vt" (UID: "70c67a25-a4c6-47bf-9209-9cfcab6cfca0") : secret "maas-controller-webhook-cert" not found Jul 10 02:58:03.680548 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:03.680513 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/81164ee1-cc95-4f0f-9acd-07982b980ee4-webhook-cert\") pod \"maas-controller-5dcc8799c9-b6c5r\" (UID: \"81164ee1-cc95-4f0f-9acd-07982b980ee4\") " pod="opendatahub/maas-controller-5dcc8799c9-b6c5r" Jul 10 02:58:03.682907 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:03.682882 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/81164ee1-cc95-4f0f-9acd-07982b980ee4-webhook-cert\") pod \"maas-controller-5dcc8799c9-b6c5r\" (UID: \"81164ee1-cc95-4f0f-9acd-07982b980ee4\") " pod="opendatahub/maas-controller-5dcc8799c9-b6c5r" Jul 10 02:58:03.809796 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:03.809756 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5dcc8799c9-b6c5r" Jul 10 02:58:03.940354 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:03.940328 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5dcc8799c9-b6c5r"] Jul 10 02:58:03.942011 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:58:03.941981 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81164ee1_cc95_4f0f_9acd_07982b980ee4.slice/crio-9f4bf9e7b3d349a1af6a2345950c7bde837450d269a646786fbf583d295adb8c WatchSource:0}: Error finding container 9f4bf9e7b3d349a1af6a2345950c7bde837450d269a646786fbf583d295adb8c: Status 404 returned error can't find the container with id 9f4bf9e7b3d349a1af6a2345950c7bde837450d269a646786fbf583d295adb8c Jul 10 02:58:04.336620 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:04.336588 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5dcc8799c9-b6c5r" event={"ID":"81164ee1-cc95-4f0f-9acd-07982b980ee4","Type":"ContainerStarted","Data":"9f4bf9e7b3d349a1af6a2345950c7bde837450d269a646786fbf583d295adb8c"} Jul 10 02:58:04.488207 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:04.488166 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/70c67a25-a4c6-47bf-9209-9cfcab6cfca0-webhook-cert\") pod \"maas-controller-cb79cdddf-7s2vt\" (UID: \"70c67a25-a4c6-47bf-9209-9cfcab6cfca0\") " pod="opendatahub/maas-controller-cb79cdddf-7s2vt" Jul 10 02:58:04.491013 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:04.490985 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/70c67a25-a4c6-47bf-9209-9cfcab6cfca0-webhook-cert\") pod \"maas-controller-cb79cdddf-7s2vt\" (UID: \"70c67a25-a4c6-47bf-9209-9cfcab6cfca0\") " pod="opendatahub/maas-controller-cb79cdddf-7s2vt" Jul 10 02:58:04.555035 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:04.554964 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cb79cdddf-7s2vt" Jul 10 02:58:04.716534 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:04.716499 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-cb79cdddf-7s2vt"] Jul 10 02:58:04.721595 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:58:04.720907 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70c67a25_a4c6_47bf_9209_9cfcab6cfca0.slice/crio-865049d741cd66d256ed90940ef2760ffad469836d57db8b9a12798fd151ce26 WatchSource:0}: Error finding container 865049d741cd66d256ed90940ef2760ffad469836d57db8b9a12798fd151ce26: Status 404 returned error can't find the container with id 865049d741cd66d256ed90940ef2760ffad469836d57db8b9a12798fd151ce26 Jul 10 02:58:05.349001 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:05.348954 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cb79cdddf-7s2vt" event={"ID":"70c67a25-a4c6-47bf-9209-9cfcab6cfca0","Type":"ContainerStarted","Data":"865049d741cd66d256ed90940ef2760ffad469836d57db8b9a12798fd151ce26"} Jul 10 02:58:07.362637 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:07.362592 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cb79cdddf-7s2vt" event={"ID":"70c67a25-a4c6-47bf-9209-9cfcab6cfca0","Type":"ContainerStarted","Data":"b5c68fe73f46ddbf2459d35a2c4c0154af12fbc167da34dfe5806af9ef404661"} Jul 10 02:58:07.363128 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:07.362773 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-cb79cdddf-7s2vt" Jul 10 02:58:07.364180 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:07.364158 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5dcc8799c9-b6c5r" event={"ID":"81164ee1-cc95-4f0f-9acd-07982b980ee4","Type":"ContainerStarted","Data":"16d5450e3b2a1bdcbff0efadb7fe45f9edbf23773062315e06ed11b4cc638cdd"} Jul 10 02:58:07.364309 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:07.364297 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5dcc8799c9-b6c5r" Jul 10 02:58:07.380568 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:07.380520 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-cb79cdddf-7s2vt" podStartSLOduration=3.490654004 podStartE2EDuration="5.380504034s" totalImagesPullingTime="1.88985003s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:58:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:58:07.379517673 +0000 UTC m=+605.660633958" watchObservedRunningTime="2026-07-10 02:58:07.380504034 +0000 UTC m=+605.661620321" Jul 10 02:58:07.397426 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:07.397390 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5dcc8799c9-b6c5r" podStartSLOduration=2.736201266 podStartE2EDuration="5.397377601s" totalImagesPullingTime="2.661176335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:58:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:58:07.395524257 +0000 UTC m=+605.676640567" watchObservedRunningTime="2026-07-10 02:58:07.397377601 +0000 UTC m=+605.678493885" Jul 10 02:58:18.374618 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:18.374587 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-cb79cdddf-7s2vt" Jul 10 02:58:18.375024 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:18.374659 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5dcc8799c9-b6c5r" Jul 10 02:58:18.435288 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:18.435259 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-cb79cdddf-7s2vt"] Jul 10 02:58:18.435478 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:18.435427 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-cb79cdddf-7s2vt" podUID="70c67a25-a4c6-47bf-9209-9cfcab6cfca0" containerName="manager" containerID="cri-o://b5c68fe73f46ddbf2459d35a2c4c0154af12fbc167da34dfe5806af9ef404661" gracePeriod=30 Jul 10 02:58:18.690464 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:18.690439 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cb79cdddf-7s2vt" Jul 10 02:58:18.714914 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:18.714886 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/70c67a25-a4c6-47bf-9209-9cfcab6cfca0-webhook-cert\") pod \"70c67a25-a4c6-47bf-9209-9cfcab6cfca0\" (UID: \"70c67a25-a4c6-47bf-9209-9cfcab6cfca0\") " Jul 10 02:58:18.715061 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:18.715031 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dnmwn\" (UniqueName: \"kubernetes.io/projected/70c67a25-a4c6-47bf-9209-9cfcab6cfca0-kube-api-access-dnmwn\") pod \"70c67a25-a4c6-47bf-9209-9cfcab6cfca0\" (UID: \"70c67a25-a4c6-47bf-9209-9cfcab6cfca0\") " Jul 10 02:58:18.717176 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:18.717148 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70c67a25-a4c6-47bf-9209-9cfcab6cfca0-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "70c67a25-a4c6-47bf-9209-9cfcab6cfca0" (UID: "70c67a25-a4c6-47bf-9209-9cfcab6cfca0"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 02:58:18.717176 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:18.717160 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70c67a25-a4c6-47bf-9209-9cfcab6cfca0-kube-api-access-dnmwn" (OuterVolumeSpecName: "kube-api-access-dnmwn") pod "70c67a25-a4c6-47bf-9209-9cfcab6cfca0" (UID: "70c67a25-a4c6-47bf-9209-9cfcab6cfca0"). InnerVolumeSpecName "kube-api-access-dnmwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 02:58:18.816214 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:18.816177 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnmwn\" (UniqueName: \"kubernetes.io/projected/70c67a25-a4c6-47bf-9209-9cfcab6cfca0-kube-api-access-dnmwn\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 02:58:18.816214 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:18.816206 2575 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/70c67a25-a4c6-47bf-9209-9cfcab6cfca0-webhook-cert\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 02:58:19.427052 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:19.427013 2575 generic.go:356] "Generic (PLEG): container finished" podID="70c67a25-a4c6-47bf-9209-9cfcab6cfca0" containerID="b5c68fe73f46ddbf2459d35a2c4c0154af12fbc167da34dfe5806af9ef404661" exitCode=0 Jul 10 02:58:19.427052 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:19.427056 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cb79cdddf-7s2vt" Jul 10 02:58:19.427507 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:19.427105 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cb79cdddf-7s2vt" event={"ID":"70c67a25-a4c6-47bf-9209-9cfcab6cfca0","Type":"ContainerDied","Data":"b5c68fe73f46ddbf2459d35a2c4c0154af12fbc167da34dfe5806af9ef404661"} Jul 10 02:58:19.427507 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:19.427174 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cb79cdddf-7s2vt" event={"ID":"70c67a25-a4c6-47bf-9209-9cfcab6cfca0","Type":"ContainerDied","Data":"865049d741cd66d256ed90940ef2760ffad469836d57db8b9a12798fd151ce26"} Jul 10 02:58:19.427507 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:19.427209 2575 scope.go:117] "RemoveContainer" containerID="b5c68fe73f46ddbf2459d35a2c4c0154af12fbc167da34dfe5806af9ef404661" Jul 10 02:58:19.436854 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:19.436664 2575 scope.go:117] "RemoveContainer" containerID="b5c68fe73f46ddbf2459d35a2c4c0154af12fbc167da34dfe5806af9ef404661" Jul 10 02:58:19.437014 ip-10-0-130-139 kubenswrapper[2575]: E0710 02:58:19.436993 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5c68fe73f46ddbf2459d35a2c4c0154af12fbc167da34dfe5806af9ef404661\": container with ID starting with b5c68fe73f46ddbf2459d35a2c4c0154af12fbc167da34dfe5806af9ef404661 not found: ID does not exist" containerID="b5c68fe73f46ddbf2459d35a2c4c0154af12fbc167da34dfe5806af9ef404661" Jul 10 02:58:19.437056 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:19.437024 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5c68fe73f46ddbf2459d35a2c4c0154af12fbc167da34dfe5806af9ef404661"} err="failed to get container status \"b5c68fe73f46ddbf2459d35a2c4c0154af12fbc167da34dfe5806af9ef404661\": rpc error: code = NotFound desc = could not find container \"b5c68fe73f46ddbf2459d35a2c4c0154af12fbc167da34dfe5806af9ef404661\": container with ID starting with b5c68fe73f46ddbf2459d35a2c4c0154af12fbc167da34dfe5806af9ef404661 not found: ID does not exist" Jul 10 02:58:19.447783 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:19.447758 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-cb79cdddf-7s2vt"] Jul 10 02:58:19.451967 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:19.451944 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-cb79cdddf-7s2vt"] Jul 10 02:58:20.288682 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:20.288649 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70c67a25-a4c6-47bf-9209-9cfcab6cfca0" path="/var/lib/kubelet/pods/70c67a25-a4c6-47bf-9209-9cfcab6cfca0/volumes" Jul 10 02:58:42.471574 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:42.471537 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5cbf495c54-x76lb"] Jul 10 02:58:42.472068 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:42.471933 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70c67a25-a4c6-47bf-9209-9cfcab6cfca0" containerName="manager" Jul 10 02:58:42.472068 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:42.471946 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="70c67a25-a4c6-47bf-9209-9cfcab6cfca0" containerName="manager" Jul 10 02:58:42.472068 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:42.472023 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="70c67a25-a4c6-47bf-9209-9cfcab6cfca0" containerName="manager" Jul 10 02:58:42.477875 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:42.477853 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5cbf495c54-x76lb" Jul 10 02:58:42.480532 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:42.480507 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 10 02:58:42.480642 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:42.480515 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-6lpn4\"" Jul 10 02:58:42.487542 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:42.487518 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cbf495c54-x76lb"] Jul 10 02:58:42.627330 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:42.627298 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9054c73-a611-4b55-8f91-bbc9b08201f4-openshift-service-ca\") pod \"maas-api-5cbf495c54-x76lb\" (UID: \"d9054c73-a611-4b55-8f91-bbc9b08201f4\") " pod="odh-ai-gateway-infra/maas-api-5cbf495c54-x76lb" Jul 10 02:58:42.627515 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:42.627350 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmxz2\" (UniqueName: \"kubernetes.io/projected/d9054c73-a611-4b55-8f91-bbc9b08201f4-kube-api-access-mmxz2\") pod \"maas-api-5cbf495c54-x76lb\" (UID: \"d9054c73-a611-4b55-8f91-bbc9b08201f4\") " pod="odh-ai-gateway-infra/maas-api-5cbf495c54-x76lb" Jul 10 02:58:42.627515 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:42.627412 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9054c73-a611-4b55-8f91-bbc9b08201f4-maas-api-tls\") pod \"maas-api-5cbf495c54-x76lb\" (UID: \"d9054c73-a611-4b55-8f91-bbc9b08201f4\") " pod="odh-ai-gateway-infra/maas-api-5cbf495c54-x76lb" Jul 10 02:58:42.728959 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:42.728870 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9054c73-a611-4b55-8f91-bbc9b08201f4-openshift-service-ca\") pod \"maas-api-5cbf495c54-x76lb\" (UID: \"d9054c73-a611-4b55-8f91-bbc9b08201f4\") " pod="odh-ai-gateway-infra/maas-api-5cbf495c54-x76lb" Jul 10 02:58:42.728959 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:42.728921 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmxz2\" (UniqueName: \"kubernetes.io/projected/d9054c73-a611-4b55-8f91-bbc9b08201f4-kube-api-access-mmxz2\") pod \"maas-api-5cbf495c54-x76lb\" (UID: \"d9054c73-a611-4b55-8f91-bbc9b08201f4\") " pod="odh-ai-gateway-infra/maas-api-5cbf495c54-x76lb" Jul 10 02:58:42.728959 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:42.728943 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9054c73-a611-4b55-8f91-bbc9b08201f4-maas-api-tls\") pod \"maas-api-5cbf495c54-x76lb\" (UID: \"d9054c73-a611-4b55-8f91-bbc9b08201f4\") " pod="odh-ai-gateway-infra/maas-api-5cbf495c54-x76lb" Jul 10 02:58:42.729588 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:42.729564 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9054c73-a611-4b55-8f91-bbc9b08201f4-openshift-service-ca\") pod \"maas-api-5cbf495c54-x76lb\" (UID: \"d9054c73-a611-4b55-8f91-bbc9b08201f4\") " pod="odh-ai-gateway-infra/maas-api-5cbf495c54-x76lb" Jul 10 02:58:42.731303 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:42.731285 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9054c73-a611-4b55-8f91-bbc9b08201f4-maas-api-tls\") pod \"maas-api-5cbf495c54-x76lb\" (UID: \"d9054c73-a611-4b55-8f91-bbc9b08201f4\") " pod="odh-ai-gateway-infra/maas-api-5cbf495c54-x76lb" Jul 10 02:58:42.737188 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:42.737165 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmxz2\" (UniqueName: \"kubernetes.io/projected/d9054c73-a611-4b55-8f91-bbc9b08201f4-kube-api-access-mmxz2\") pod \"maas-api-5cbf495c54-x76lb\" (UID: \"d9054c73-a611-4b55-8f91-bbc9b08201f4\") " pod="odh-ai-gateway-infra/maas-api-5cbf495c54-x76lb" Jul 10 02:58:42.790025 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:42.789993 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5cbf495c54-x76lb" Jul 10 02:58:42.929277 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:42.929250 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cbf495c54-x76lb"] Jul 10 02:58:42.931787 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:58:42.931761 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9054c73_a611_4b55_8f91_bbc9b08201f4.slice/crio-611dd70d7d7ebfbad19e02c404f2489d6325c2fb48b35f4210b54c57c9d3d11f WatchSource:0}: Error finding container 611dd70d7d7ebfbad19e02c404f2489d6325c2fb48b35f4210b54c57c9d3d11f: Status 404 returned error can't find the container with id 611dd70d7d7ebfbad19e02c404f2489d6325c2fb48b35f4210b54c57c9d3d11f Jul 10 02:58:43.101202 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:43.101171 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-vhjvc"] Jul 10 02:58:43.105742 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:43.105720 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-vhjvc" Jul 10 02:58:43.108292 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:43.108270 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 10 02:58:43.108409 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:43.108270 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-nnw8h\"" Jul 10 02:58:43.112376 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:43.112353 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-vhjvc"] Jul 10 02:58:43.233713 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:43.233680 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npgtf\" (UniqueName: \"kubernetes.io/projected/17135a96-76bf-434d-a6c1-0edd00d7901c-kube-api-access-npgtf\") pod \"payload-processing-58f86c7798-vhjvc\" (UID: \"17135a96-76bf-434d-a6c1-0edd00d7901c\") " pod="openshift-ingress/payload-processing-58f86c7798-vhjvc" Jul 10 02:58:43.233891 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:43.233748 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17135a96-76bf-434d-a6c1-0edd00d7901c-plugins-config-volume\") pod \"payload-processing-58f86c7798-vhjvc\" (UID: \"17135a96-76bf-434d-a6c1-0edd00d7901c\") " pod="openshift-ingress/payload-processing-58f86c7798-vhjvc" Jul 10 02:58:43.335168 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:43.335139 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17135a96-76bf-434d-a6c1-0edd00d7901c-plugins-config-volume\") pod \"payload-processing-58f86c7798-vhjvc\" (UID: \"17135a96-76bf-434d-a6c1-0edd00d7901c\") " pod="openshift-ingress/payload-processing-58f86c7798-vhjvc" Jul 10 02:58:43.335320 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:43.335202 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npgtf\" (UniqueName: \"kubernetes.io/projected/17135a96-76bf-434d-a6c1-0edd00d7901c-kube-api-access-npgtf\") pod \"payload-processing-58f86c7798-vhjvc\" (UID: \"17135a96-76bf-434d-a6c1-0edd00d7901c\") " pod="openshift-ingress/payload-processing-58f86c7798-vhjvc" Jul 10 02:58:43.335920 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:43.335902 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17135a96-76bf-434d-a6c1-0edd00d7901c-plugins-config-volume\") pod \"payload-processing-58f86c7798-vhjvc\" (UID: \"17135a96-76bf-434d-a6c1-0edd00d7901c\") " pod="openshift-ingress/payload-processing-58f86c7798-vhjvc" Jul 10 02:58:43.347116 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:43.347091 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npgtf\" (UniqueName: \"kubernetes.io/projected/17135a96-76bf-434d-a6c1-0edd00d7901c-kube-api-access-npgtf\") pod \"payload-processing-58f86c7798-vhjvc\" (UID: \"17135a96-76bf-434d-a6c1-0edd00d7901c\") " pod="openshift-ingress/payload-processing-58f86c7798-vhjvc" Jul 10 02:58:43.418236 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:43.418169 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-vhjvc" Jul 10 02:58:43.548993 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:43.548959 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5cbf495c54-x76lb" event={"ID":"d9054c73-a611-4b55-8f91-bbc9b08201f4","Type":"ContainerStarted","Data":"611dd70d7d7ebfbad19e02c404f2489d6325c2fb48b35f4210b54c57c9d3d11f"} Jul 10 02:58:43.555901 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:43.555875 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-vhjvc"] Jul 10 02:58:43.557777 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:58:43.557749 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17135a96_76bf_434d_a6c1_0edd00d7901c.slice/crio-a304d655de78d73240df2034a8693ee06bf22f78d144be52511d983cac7f267e WatchSource:0}: Error finding container a304d655de78d73240df2034a8693ee06bf22f78d144be52511d983cac7f267e: Status 404 returned error can't find the container with id a304d655de78d73240df2034a8693ee06bf22f78d144be52511d983cac7f267e Jul 10 02:58:44.566749 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:44.566704 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-vhjvc" event={"ID":"17135a96-76bf-434d-a6c1-0edd00d7901c","Type":"ContainerStarted","Data":"a304d655de78d73240df2034a8693ee06bf22f78d144be52511d983cac7f267e"} Jul 10 02:58:45.577997 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:45.577964 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5cbf495c54-x76lb" event={"ID":"d9054c73-a611-4b55-8f91-bbc9b08201f4","Type":"ContainerStarted","Data":"6f342dddf215d3dd92424d57500bcdcc9dcb3d4f09be839ff0041500e02d5872"} Jul 10 02:58:45.578441 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:45.578081 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5cbf495c54-x76lb" Jul 10 02:58:45.579692 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:45.579655 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-vhjvc" event={"ID":"17135a96-76bf-434d-a6c1-0edd00d7901c","Type":"ContainerStarted","Data":"df5e88d2851a48e8b2b6f69701fe8431fc9b9d41b2bbd46bce63bf2845317144"} Jul 10 02:58:45.579847 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:45.579713 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-vhjvc" Jul 10 02:58:45.595071 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:45.595026 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5cbf495c54-x76lb" podStartSLOduration=1.397201822 podStartE2EDuration="3.595009271s" totalImagesPullingTime="2.197807449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:58:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:58:45.594520134 +0000 UTC m=+643.875636420" watchObservedRunningTime="2026-07-10 02:58:45.595009271 +0000 UTC m=+643.876125557" Jul 10 02:58:45.610771 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:45.610728 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-vhjvc" podStartSLOduration=1.024612557 podStartE2EDuration="2.610715791s" totalImagesPullingTime="1.586103234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:58:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:58:45.608504227 +0000 UTC m=+643.889620532" watchObservedRunningTime="2026-07-10 02:58:45.610715791 +0000 UTC m=+643.891832075" Jul 10 02:58:50.606444 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:50.606346 2575 generic.go:356] "Generic (PLEG): container finished" podID="1e5039b0-ee00-4541-8216-c4a7015b93d1" containerID="087423b36da81799884abc67feb5eb9f994fc83b30e14746f3b72a88a7d2330f" exitCode=0 Jul 10 02:58:50.606444 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:50.606423 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6zr5k" event={"ID":"1e5039b0-ee00-4541-8216-c4a7015b93d1","Type":"ContainerDied","Data":"087423b36da81799884abc67feb5eb9f994fc83b30e14746f3b72a88a7d2330f"} Jul 10 02:58:50.606891 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:50.606804 2575 scope.go:117] "RemoveContainer" containerID="087423b36da81799884abc67feb5eb9f994fc83b30e14746f3b72a88a7d2330f" Jul 10 02:58:50.607875 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:50.607850 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236236Ki","pods":"250"} Jul 10 02:58:50.607964 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:50.607914 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236236Ki","pods":"250"} Jul 10 02:58:50.607964 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:50.607945 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236236Ki","pods":"250"} Jul 10 02:58:51.591310 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:51.591279 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5cbf495c54-x76lb" Jul 10 02:58:51.613510 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:51.613473 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6zr5k" event={"ID":"1e5039b0-ee00-4541-8216-c4a7015b93d1","Type":"ContainerStarted","Data":"02c36dfd5b3b2167fc893db6b738cc7f8ef5c1cc84f7c9d1fbe7440b376384a6"} Jul 10 02:58:52.082738 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:52.082704 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6zr5k" Jul 10 02:58:52.424800 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:52.424722 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6zr5k" Jul 10 02:58:52.425984 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:52.425960 2575 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-6zr5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Jul 10 02:58:52.426127 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:52.426012 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6zr5k" podUID="1e5039b0-ee00-4541-8216-c4a7015b93d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Jul 10 02:58:53.425509 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:53.425469 2575 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-6zr5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Jul 10 02:58:53.426078 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:53.425533 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6zr5k" podUID="1e5039b0-ee00-4541-8216-c4a7015b93d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Jul 10 02:58:54.424485 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:54.424449 2575 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-6zr5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Jul 10 02:58:54.424654 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:54.424507 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6zr5k" podUID="1e5039b0-ee00-4541-8216-c4a7015b93d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Jul 10 02:58:55.464122 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:55.464087 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6zr5k" Jul 10 02:58:55.465235 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:55.465216 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6zr5k" Jul 10 02:58:56.588214 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:58:56.588180 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-vhjvc" Jul 10 02:59:14.453419 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:14.453386 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-vnxvn"] Jul 10 02:59:14.458552 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:14.458528 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vnxvn" Jul 10 02:59:14.461018 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:14.460996 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-4m722\"" Jul 10 02:59:14.461146 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:14.461125 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 10 02:59:14.461940 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:14.461920 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 10 02:59:14.462028 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:14.461971 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 10 02:59:14.465246 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:14.465225 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-vnxvn"] Jul 10 02:59:14.545901 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:14.545862 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79ab8a96-b91b-4a79-87e1-222a409a29e4-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vnxvn\" (UID: \"79ab8a96-b91b-4a79-87e1-222a409a29e4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vnxvn" Jul 10 02:59:14.545901 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:14.545909 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79ab8a96-b91b-4a79-87e1-222a409a29e4-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vnxvn\" (UID: \"79ab8a96-b91b-4a79-87e1-222a409a29e4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vnxvn" Jul 10 02:59:14.546152 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:14.545955 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79ab8a96-b91b-4a79-87e1-222a409a29e4-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vnxvn\" (UID: \"79ab8a96-b91b-4a79-87e1-222a409a29e4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vnxvn" Jul 10 02:59:14.546152 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:14.546025 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79ab8a96-b91b-4a79-87e1-222a409a29e4-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vnxvn\" (UID: \"79ab8a96-b91b-4a79-87e1-222a409a29e4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vnxvn" Jul 10 02:59:14.546152 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:14.546048 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79ab8a96-b91b-4a79-87e1-222a409a29e4-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vnxvn\" (UID: \"79ab8a96-b91b-4a79-87e1-222a409a29e4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vnxvn" Jul 10 02:59:14.546152 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:14.546065 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd62f\" (UniqueName: \"kubernetes.io/projected/79ab8a96-b91b-4a79-87e1-222a409a29e4-kube-api-access-sd62f\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vnxvn\" (UID: \"79ab8a96-b91b-4a79-87e1-222a409a29e4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vnxvn" Jul 10 02:59:14.647484 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:14.647453 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79ab8a96-b91b-4a79-87e1-222a409a29e4-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vnxvn\" (UID: \"79ab8a96-b91b-4a79-87e1-222a409a29e4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vnxvn" Jul 10 02:59:14.647484 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:14.647486 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79ab8a96-b91b-4a79-87e1-222a409a29e4-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vnxvn\" (UID: \"79ab8a96-b91b-4a79-87e1-222a409a29e4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vnxvn" Jul 10 02:59:14.647730 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:14.647537 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79ab8a96-b91b-4a79-87e1-222a409a29e4-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vnxvn\" (UID: \"79ab8a96-b91b-4a79-87e1-222a409a29e4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vnxvn" Jul 10 02:59:14.647730 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:14.647557 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79ab8a96-b91b-4a79-87e1-222a409a29e4-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vnxvn\" (UID: \"79ab8a96-b91b-4a79-87e1-222a409a29e4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vnxvn" Jul 10 02:59:14.647730 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:14.647573 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd62f\" (UniqueName: \"kubernetes.io/projected/79ab8a96-b91b-4a79-87e1-222a409a29e4-kube-api-access-sd62f\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vnxvn\" (UID: \"79ab8a96-b91b-4a79-87e1-222a409a29e4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vnxvn" Jul 10 02:59:14.647730 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:14.647643 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79ab8a96-b91b-4a79-87e1-222a409a29e4-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vnxvn\" (UID: \"79ab8a96-b91b-4a79-87e1-222a409a29e4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vnxvn" Jul 10 02:59:14.647982 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:14.647944 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79ab8a96-b91b-4a79-87e1-222a409a29e4-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vnxvn\" (UID: \"79ab8a96-b91b-4a79-87e1-222a409a29e4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vnxvn" Jul 10 02:59:14.648038 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:14.647980 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79ab8a96-b91b-4a79-87e1-222a409a29e4-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vnxvn\" (UID: \"79ab8a96-b91b-4a79-87e1-222a409a29e4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vnxvn" Jul 10 02:59:14.648038 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:14.648023 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79ab8a96-b91b-4a79-87e1-222a409a29e4-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vnxvn\" (UID: \"79ab8a96-b91b-4a79-87e1-222a409a29e4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vnxvn" Jul 10 02:59:14.649924 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:14.649891 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79ab8a96-b91b-4a79-87e1-222a409a29e4-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vnxvn\" (UID: \"79ab8a96-b91b-4a79-87e1-222a409a29e4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vnxvn" Jul 10 02:59:14.650090 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:14.650073 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79ab8a96-b91b-4a79-87e1-222a409a29e4-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vnxvn\" (UID: \"79ab8a96-b91b-4a79-87e1-222a409a29e4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vnxvn" Jul 10 02:59:14.659967 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:14.659946 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd62f\" (UniqueName: \"kubernetes.io/projected/79ab8a96-b91b-4a79-87e1-222a409a29e4-kube-api-access-sd62f\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vnxvn\" (UID: \"79ab8a96-b91b-4a79-87e1-222a409a29e4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vnxvn" Jul 10 02:59:14.773619 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:14.773595 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vnxvn" Jul 10 02:59:14.932405 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:14.932382 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-vnxvn"] Jul 10 02:59:14.934513 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:59:14.934486 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79ab8a96_b91b_4a79_87e1_222a409a29e4.slice/crio-aabf821000f3e54d478a90a43421446dc0c06a21f393e80d8727126082902cb9 WatchSource:0}: Error finding container aabf821000f3e54d478a90a43421446dc0c06a21f393e80d8727126082902cb9: Status 404 returned error can't find the container with id aabf821000f3e54d478a90a43421446dc0c06a21f393e80d8727126082902cb9 Jul 10 02:59:15.739960 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:15.739923 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vnxvn" event={"ID":"79ab8a96-b91b-4a79-87e1-222a409a29e4","Type":"ContainerStarted","Data":"aabf821000f3e54d478a90a43421446dc0c06a21f393e80d8727126082902cb9"} Jul 10 02:59:16.746964 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:16.746926 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vnxvn" event={"ID":"79ab8a96-b91b-4a79-87e1-222a409a29e4","Type":"ContainerStarted","Data":"0ddcc8cb28ff4ecde0bd81bc6839eaf4a0747f556646987d68ce6752ed2e7b31"} Jul 10 02:59:16.765347 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:16.765295 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vnxvn" podStartSLOduration=1.612143633 podStartE2EDuration="2.765280068s" totalImagesPullingTime="1.153136435s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:59:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:59:16.762202782 +0000 UTC m=+675.043319067" watchObservedRunningTime="2026-07-10 02:59:16.765280068 +0000 UTC m=+675.046396352" Jul 10 02:59:24.774359 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:24.774318 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vnxvn" Jul 10 02:59:24.774767 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:24.774390 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vnxvn" Jul 10 02:59:24.787216 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:24.787182 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vnxvn" Jul 10 02:59:24.806567 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:24.806542 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vnxvn" Jul 10 02:59:31.461871 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:31.461838 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-rmrxw"] Jul 10 02:59:31.465753 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:31.465731 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rmrxw" Jul 10 02:59:31.474038 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:31.474013 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 10 02:59:31.476427 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:31.476405 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-rmrxw"] Jul 10 02:59:31.611594 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:31.611558 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82b3cdab-2d8b-46a2-ba7c-c9d319b418a9-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rmrxw\" (UID: \"82b3cdab-2d8b-46a2-ba7c-c9d319b418a9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rmrxw" Jul 10 02:59:31.611594 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:31.611594 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82b3cdab-2d8b-46a2-ba7c-c9d319b418a9-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rmrxw\" (UID: \"82b3cdab-2d8b-46a2-ba7c-c9d319b418a9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rmrxw" Jul 10 02:59:31.612102 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:31.611670 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82b3cdab-2d8b-46a2-ba7c-c9d319b418a9-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rmrxw\" (UID: \"82b3cdab-2d8b-46a2-ba7c-c9d319b418a9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rmrxw" Jul 10 02:59:31.612102 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:31.611750 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82b3cdab-2d8b-46a2-ba7c-c9d319b418a9-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rmrxw\" (UID: \"82b3cdab-2d8b-46a2-ba7c-c9d319b418a9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rmrxw" Jul 10 02:59:31.612102 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:31.611790 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82b3cdab-2d8b-46a2-ba7c-c9d319b418a9-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rmrxw\" (UID: \"82b3cdab-2d8b-46a2-ba7c-c9d319b418a9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rmrxw" Jul 10 02:59:31.612102 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:31.611834 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgn84\" (UniqueName: \"kubernetes.io/projected/82b3cdab-2d8b-46a2-ba7c-c9d319b418a9-kube-api-access-lgn84\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rmrxw\" (UID: \"82b3cdab-2d8b-46a2-ba7c-c9d319b418a9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rmrxw" Jul 10 02:59:31.712912 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:31.712800 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgn84\" (UniqueName: \"kubernetes.io/projected/82b3cdab-2d8b-46a2-ba7c-c9d319b418a9-kube-api-access-lgn84\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rmrxw\" (UID: \"82b3cdab-2d8b-46a2-ba7c-c9d319b418a9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rmrxw" Jul 10 02:59:31.712912 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:31.712884 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82b3cdab-2d8b-46a2-ba7c-c9d319b418a9-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rmrxw\" (UID: \"82b3cdab-2d8b-46a2-ba7c-c9d319b418a9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rmrxw" Jul 10 02:59:31.712912 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:31.712908 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82b3cdab-2d8b-46a2-ba7c-c9d319b418a9-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rmrxw\" (UID: \"82b3cdab-2d8b-46a2-ba7c-c9d319b418a9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rmrxw" Jul 10 02:59:31.713570 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:31.712977 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82b3cdab-2d8b-46a2-ba7c-c9d319b418a9-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rmrxw\" (UID: \"82b3cdab-2d8b-46a2-ba7c-c9d319b418a9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rmrxw" Jul 10 02:59:31.713570 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:31.713049 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82b3cdab-2d8b-46a2-ba7c-c9d319b418a9-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rmrxw\" (UID: \"82b3cdab-2d8b-46a2-ba7c-c9d319b418a9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rmrxw" Jul 10 02:59:31.713570 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:31.713082 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82b3cdab-2d8b-46a2-ba7c-c9d319b418a9-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rmrxw\" (UID: \"82b3cdab-2d8b-46a2-ba7c-c9d319b418a9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rmrxw" Jul 10 02:59:31.713570 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:31.713358 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82b3cdab-2d8b-46a2-ba7c-c9d319b418a9-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rmrxw\" (UID: \"82b3cdab-2d8b-46a2-ba7c-c9d319b418a9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rmrxw" Jul 10 02:59:31.713570 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:31.713417 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82b3cdab-2d8b-46a2-ba7c-c9d319b418a9-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rmrxw\" (UID: \"82b3cdab-2d8b-46a2-ba7c-c9d319b418a9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rmrxw" Jul 10 02:59:31.713570 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:31.713539 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82b3cdab-2d8b-46a2-ba7c-c9d319b418a9-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rmrxw\" (UID: \"82b3cdab-2d8b-46a2-ba7c-c9d319b418a9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rmrxw" Jul 10 02:59:31.715134 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:31.715111 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82b3cdab-2d8b-46a2-ba7c-c9d319b418a9-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rmrxw\" (UID: \"82b3cdab-2d8b-46a2-ba7c-c9d319b418a9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rmrxw" Jul 10 02:59:31.715426 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:31.715408 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82b3cdab-2d8b-46a2-ba7c-c9d319b418a9-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rmrxw\" (UID: \"82b3cdab-2d8b-46a2-ba7c-c9d319b418a9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rmrxw" Jul 10 02:59:31.720308 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:31.720284 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgn84\" (UniqueName: \"kubernetes.io/projected/82b3cdab-2d8b-46a2-ba7c-c9d319b418a9-kube-api-access-lgn84\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rmrxw\" (UID: \"82b3cdab-2d8b-46a2-ba7c-c9d319b418a9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rmrxw" Jul 10 02:59:31.782232 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:31.782200 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rmrxw" Jul 10 02:59:31.920747 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:31.920723 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-rmrxw"] Jul 10 02:59:31.923196 ip-10-0-130-139 kubenswrapper[2575]: W0710 02:59:31.923173 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82b3cdab_2d8b_46a2_ba7c_c9d319b418a9.slice/crio-5c2c268133b85133a78927e5af50d99122bdf58160168d4b89ca74653e0f0782 WatchSource:0}: Error finding container 5c2c268133b85133a78927e5af50d99122bdf58160168d4b89ca74653e0f0782: Status 404 returned error can't find the container with id 5c2c268133b85133a78927e5af50d99122bdf58160168d4b89ca74653e0f0782 Jul 10 02:59:31.924844 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:31.924823 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 02:59:32.836825 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:32.836772 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rmrxw" event={"ID":"82b3cdab-2d8b-46a2-ba7c-c9d319b418a9","Type":"ContainerStarted","Data":"07be348f86424e40add691b620f5359e6a3aa9829f365599920f4d288b00aaab"} Jul 10 02:59:32.837206 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:32.836862 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rmrxw" event={"ID":"82b3cdab-2d8b-46a2-ba7c-c9d319b418a9","Type":"ContainerStarted","Data":"5c2c268133b85133a78927e5af50d99122bdf58160168d4b89ca74653e0f0782"} Jul 10 02:59:32.855081 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:32.855024 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rmrxw" podStartSLOduration=1.689680144 podStartE2EDuration="1.855007268s" totalImagesPullingTime="165.327124ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 02:59:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 02:59:32.852344263 +0000 UTC m=+691.133460550" watchObservedRunningTime="2026-07-10 02:59:32.855007268 +0000 UTC m=+691.136123553" Jul 10 02:59:41.782453 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:41.782410 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rmrxw" Jul 10 02:59:41.782883 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:41.782490 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rmrxw" Jul 10 02:59:41.794945 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:41.794920 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rmrxw" Jul 10 02:59:41.894207 ip-10-0-130-139 kubenswrapper[2575]: I0710 02:59:41.894174 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rmrxw" Jul 10 03:00:28.414181 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:00:28.414095 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-597dbc55fc-zdl7g"] Jul 10 03:00:28.414737 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:00:28.414285 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-597dbc55fc-zdl7g" podUID="9635f986-882e-4a5b-984f-0277e5440a03" containerName="authorino" containerID="cri-o://bd882f9fe7fbacc8453891bf2a4da00664b0d9425fc487eff795305a45e8e7f4" gracePeriod=30 Jul 10 03:00:28.677708 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:00:28.677645 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-597dbc55fc-zdl7g" Jul 10 03:00:28.722262 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:00:28.722213 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-96wzz\" (UniqueName: \"kubernetes.io/projected/9635f986-882e-4a5b-984f-0277e5440a03-kube-api-access-96wzz\") pod \"9635f986-882e-4a5b-984f-0277e5440a03\" (UID: \"9635f986-882e-4a5b-984f-0277e5440a03\") " Jul 10 03:00:28.722454 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:00:28.722339 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9635f986-882e-4a5b-984f-0277e5440a03-tls-cert\") pod \"9635f986-882e-4a5b-984f-0277e5440a03\" (UID: \"9635f986-882e-4a5b-984f-0277e5440a03\") " Jul 10 03:00:28.724590 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:00:28.724559 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9635f986-882e-4a5b-984f-0277e5440a03-kube-api-access-96wzz" (OuterVolumeSpecName: "kube-api-access-96wzz") pod "9635f986-882e-4a5b-984f-0277e5440a03" (UID: "9635f986-882e-4a5b-984f-0277e5440a03"). InnerVolumeSpecName "kube-api-access-96wzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 03:00:28.733886 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:00:28.733865 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9635f986-882e-4a5b-984f-0277e5440a03-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "9635f986-882e-4a5b-984f-0277e5440a03" (UID: "9635f986-882e-4a5b-984f-0277e5440a03"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 03:00:28.823062 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:00:28.823028 2575 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9635f986-882e-4a5b-984f-0277e5440a03-tls-cert\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 03:00:28.823062 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:00:28.823055 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96wzz\" (UniqueName: \"kubernetes.io/projected/9635f986-882e-4a5b-984f-0277e5440a03-kube-api-access-96wzz\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 03:00:29.127568 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:00:29.127534 2575 generic.go:356] "Generic (PLEG): container finished" podID="9635f986-882e-4a5b-984f-0277e5440a03" containerID="bd882f9fe7fbacc8453891bf2a4da00664b0d9425fc487eff795305a45e8e7f4" exitCode=0 Jul 10 03:00:29.127762 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:00:29.127574 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-597dbc55fc-zdl7g" Jul 10 03:00:29.127762 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:00:29.127609 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-597dbc55fc-zdl7g" event={"ID":"9635f986-882e-4a5b-984f-0277e5440a03","Type":"ContainerDied","Data":"bd882f9fe7fbacc8453891bf2a4da00664b0d9425fc487eff795305a45e8e7f4"} Jul 10 03:00:29.127762 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:00:29.127640 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-597dbc55fc-zdl7g" event={"ID":"9635f986-882e-4a5b-984f-0277e5440a03","Type":"ContainerDied","Data":"d4b63f1bcba839905e37571188878e34cda6d1859617bda69e0c690ef94bcdbf"} Jul 10 03:00:29.127762 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:00:29.127665 2575 scope.go:117] "RemoveContainer" containerID="bd882f9fe7fbacc8453891bf2a4da00664b0d9425fc487eff795305a45e8e7f4" Jul 10 03:00:29.136721 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:00:29.136703 2575 scope.go:117] "RemoveContainer" containerID="bd882f9fe7fbacc8453891bf2a4da00664b0d9425fc487eff795305a45e8e7f4" Jul 10 03:00:29.137022 ip-10-0-130-139 kubenswrapper[2575]: E0710 03:00:29.137005 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd882f9fe7fbacc8453891bf2a4da00664b0d9425fc487eff795305a45e8e7f4\": container with ID starting with bd882f9fe7fbacc8453891bf2a4da00664b0d9425fc487eff795305a45e8e7f4 not found: ID does not exist" containerID="bd882f9fe7fbacc8453891bf2a4da00664b0d9425fc487eff795305a45e8e7f4" Jul 10 03:00:29.137072 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:00:29.137032 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd882f9fe7fbacc8453891bf2a4da00664b0d9425fc487eff795305a45e8e7f4"} err="failed to get container status \"bd882f9fe7fbacc8453891bf2a4da00664b0d9425fc487eff795305a45e8e7f4\": rpc error: code = NotFound desc = could not find container \"bd882f9fe7fbacc8453891bf2a4da00664b0d9425fc487eff795305a45e8e7f4\": container with ID starting with bd882f9fe7fbacc8453891bf2a4da00664b0d9425fc487eff795305a45e8e7f4 not found: ID does not exist" Jul 10 03:00:29.147844 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:00:29.147801 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-597dbc55fc-zdl7g"] Jul 10 03:00:29.156612 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:00:29.156587 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-597dbc55fc-zdl7g"] Jul 10 03:00:30.288964 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:00:30.288934 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9635f986-882e-4a5b-984f-0277e5440a03" path="/var/lib/kubelet/pods/9635f986-882e-4a5b-984f-0277e5440a03/volumes" Jul 10 03:01:39.322138 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:01:39.322059 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5dcc8799c9-b6c5r"] Jul 10 03:01:39.322648 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:01:39.322247 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5dcc8799c9-b6c5r" podUID="81164ee1-cc95-4f0f-9acd-07982b980ee4" containerName="manager" containerID="cri-o://16d5450e3b2a1bdcbff0efadb7fe45f9edbf23773062315e06ed11b4cc638cdd" gracePeriod=30 Jul 10 03:01:39.499663 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:01:39.499438 2575 generic.go:356] "Generic (PLEG): container finished" podID="81164ee1-cc95-4f0f-9acd-07982b980ee4" containerID="16d5450e3b2a1bdcbff0efadb7fe45f9edbf23773062315e06ed11b4cc638cdd" exitCode=0 Jul 10 03:01:39.499663 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:01:39.499580 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5dcc8799c9-b6c5r" event={"ID":"81164ee1-cc95-4f0f-9acd-07982b980ee4","Type":"ContainerDied","Data":"16d5450e3b2a1bdcbff0efadb7fe45f9edbf23773062315e06ed11b4cc638cdd"} Jul 10 03:01:39.581519 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:01:39.581457 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5dcc8799c9-b6c5r" Jul 10 03:01:39.644854 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:01:39.644804 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/81164ee1-cc95-4f0f-9acd-07982b980ee4-webhook-cert\") pod \"81164ee1-cc95-4f0f-9acd-07982b980ee4\" (UID: \"81164ee1-cc95-4f0f-9acd-07982b980ee4\") " Jul 10 03:01:39.645010 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:01:39.644906 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9j5mn\" (UniqueName: \"kubernetes.io/projected/81164ee1-cc95-4f0f-9acd-07982b980ee4-kube-api-access-9j5mn\") pod \"81164ee1-cc95-4f0f-9acd-07982b980ee4\" (UID: \"81164ee1-cc95-4f0f-9acd-07982b980ee4\") " Jul 10 03:01:39.647153 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:01:39.647121 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81164ee1-cc95-4f0f-9acd-07982b980ee4-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "81164ee1-cc95-4f0f-9acd-07982b980ee4" (UID: "81164ee1-cc95-4f0f-9acd-07982b980ee4"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 03:01:39.647274 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:01:39.647184 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81164ee1-cc95-4f0f-9acd-07982b980ee4-kube-api-access-9j5mn" (OuterVolumeSpecName: "kube-api-access-9j5mn") pod "81164ee1-cc95-4f0f-9acd-07982b980ee4" (UID: "81164ee1-cc95-4f0f-9acd-07982b980ee4"). InnerVolumeSpecName "kube-api-access-9j5mn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 03:01:39.746112 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:01:39.746078 2575 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/81164ee1-cc95-4f0f-9acd-07982b980ee4-webhook-cert\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 03:01:39.746112 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:01:39.746108 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9j5mn\" (UniqueName: \"kubernetes.io/projected/81164ee1-cc95-4f0f-9acd-07982b980ee4-kube-api-access-9j5mn\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 03:01:40.505587 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:01:40.505560 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5dcc8799c9-b6c5r" Jul 10 03:01:40.506037 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:01:40.505558 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5dcc8799c9-b6c5r" event={"ID":"81164ee1-cc95-4f0f-9acd-07982b980ee4","Type":"ContainerDied","Data":"9f4bf9e7b3d349a1af6a2345950c7bde837450d269a646786fbf583d295adb8c"} Jul 10 03:01:40.506037 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:01:40.505699 2575 scope.go:117] "RemoveContainer" containerID="16d5450e3b2a1bdcbff0efadb7fe45f9edbf23773062315e06ed11b4cc638cdd" Jul 10 03:01:40.522653 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:01:40.522626 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5dcc8799c9-b6c5r"] Jul 10 03:01:40.526551 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:01:40.526530 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5dcc8799c9-b6c5r"] Jul 10 03:01:41.291018 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:01:41.290989 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5dcc8799c9-nwbqh"] Jul 10 03:01:41.291384 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:01:41.291373 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81164ee1-cc95-4f0f-9acd-07982b980ee4" containerName="manager" Jul 10 03:01:41.291435 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:01:41.291386 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="81164ee1-cc95-4f0f-9acd-07982b980ee4" containerName="manager" Jul 10 03:01:41.291435 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:01:41.291411 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9635f986-882e-4a5b-984f-0277e5440a03" containerName="authorino" Jul 10 03:01:41.291435 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:01:41.291417 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9635f986-882e-4a5b-984f-0277e5440a03" containerName="authorino" Jul 10 03:01:41.291526 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:01:41.291471 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9635f986-882e-4a5b-984f-0277e5440a03" containerName="authorino" Jul 10 03:01:41.291526 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:01:41.291483 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="81164ee1-cc95-4f0f-9acd-07982b980ee4" containerName="manager" Jul 10 03:01:41.295893 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:01:41.295875 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5dcc8799c9-nwbqh" Jul 10 03:01:41.298194 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:01:41.298174 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 10 03:01:41.299132 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:01:41.299116 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-64bvh\"" Jul 10 03:01:41.299246 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:01:41.299153 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 10 03:01:41.304677 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:01:41.304657 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5dcc8799c9-nwbqh"] Jul 10 03:01:41.363091 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:01:41.363059 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48lzg\" (UniqueName: \"kubernetes.io/projected/8e7713f9-b289-4dd3-9545-5362a8bdbd4e-kube-api-access-48lzg\") pod \"maas-controller-5dcc8799c9-nwbqh\" (UID: \"8e7713f9-b289-4dd3-9545-5362a8bdbd4e\") " pod="opendatahub/maas-controller-5dcc8799c9-nwbqh" Jul 10 03:01:41.363293 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:01:41.363164 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8e7713f9-b289-4dd3-9545-5362a8bdbd4e-webhook-cert\") pod \"maas-controller-5dcc8799c9-nwbqh\" (UID: \"8e7713f9-b289-4dd3-9545-5362a8bdbd4e\") " pod="opendatahub/maas-controller-5dcc8799c9-nwbqh" Jul 10 03:01:41.464612 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:01:41.464578 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8e7713f9-b289-4dd3-9545-5362a8bdbd4e-webhook-cert\") pod \"maas-controller-5dcc8799c9-nwbqh\" (UID: \"8e7713f9-b289-4dd3-9545-5362a8bdbd4e\") " pod="opendatahub/maas-controller-5dcc8799c9-nwbqh" Jul 10 03:01:41.464802 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:01:41.464668 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48lzg\" (UniqueName: \"kubernetes.io/projected/8e7713f9-b289-4dd3-9545-5362a8bdbd4e-kube-api-access-48lzg\") pod \"maas-controller-5dcc8799c9-nwbqh\" (UID: \"8e7713f9-b289-4dd3-9545-5362a8bdbd4e\") " pod="opendatahub/maas-controller-5dcc8799c9-nwbqh" Jul 10 03:01:41.466973 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:01:41.466953 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8e7713f9-b289-4dd3-9545-5362a8bdbd4e-webhook-cert\") pod \"maas-controller-5dcc8799c9-nwbqh\" (UID: \"8e7713f9-b289-4dd3-9545-5362a8bdbd4e\") " pod="opendatahub/maas-controller-5dcc8799c9-nwbqh" Jul 10 03:01:41.473153 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:01:41.473130 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48lzg\" (UniqueName: \"kubernetes.io/projected/8e7713f9-b289-4dd3-9545-5362a8bdbd4e-kube-api-access-48lzg\") pod \"maas-controller-5dcc8799c9-nwbqh\" (UID: \"8e7713f9-b289-4dd3-9545-5362a8bdbd4e\") " pod="opendatahub/maas-controller-5dcc8799c9-nwbqh" Jul 10 03:01:41.607349 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:01:41.607260 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5dcc8799c9-nwbqh" Jul 10 03:01:41.743404 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:01:41.743380 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5dcc8799c9-nwbqh"] Jul 10 03:01:41.745022 ip-10-0-130-139 kubenswrapper[2575]: W0710 03:01:41.744994 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e7713f9_b289_4dd3_9545_5362a8bdbd4e.slice/crio-5186b72850e6cd6964fb57533824f12092d0257a1dd113840fdae2f17ea9d65d WatchSource:0}: Error finding container 5186b72850e6cd6964fb57533824f12092d0257a1dd113840fdae2f17ea9d65d: Status 404 returned error can't find the container with id 5186b72850e6cd6964fb57533824f12092d0257a1dd113840fdae2f17ea9d65d Jul 10 03:01:42.289153 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:01:42.289118 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81164ee1-cc95-4f0f-9acd-07982b980ee4" path="/var/lib/kubelet/pods/81164ee1-cc95-4f0f-9acd-07982b980ee4/volumes" Jul 10 03:01:42.519572 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:01:42.519534 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5dcc8799c9-nwbqh" event={"ID":"8e7713f9-b289-4dd3-9545-5362a8bdbd4e","Type":"ContainerStarted","Data":"a7f38bf67d62a8a5a316d609585c765ac3b9a5fd0d11c701d390a3f8517e9c9a"} Jul 10 03:01:42.519756 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:01:42.519601 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5dcc8799c9-nwbqh" Jul 10 03:01:42.519756 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:01:42.519627 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5dcc8799c9-nwbqh" event={"ID":"8e7713f9-b289-4dd3-9545-5362a8bdbd4e","Type":"ContainerStarted","Data":"5186b72850e6cd6964fb57533824f12092d0257a1dd113840fdae2f17ea9d65d"} Jul 10 03:01:42.537873 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:01:42.537797 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5dcc8799c9-nwbqh" podStartSLOduration=1.09877792 podStartE2EDuration="1.537779687s" totalImagesPullingTime="439.001767ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 03:01:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 03:01:42.536235708 +0000 UTC m=+820.817351998" watchObservedRunningTime="2026-07-10 03:01:42.537779687 +0000 UTC m=+820.818895971" Jul 10 03:01:53.531425 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:01:53.531391 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5dcc8799c9-nwbqh" Jul 10 03:03:02.257924 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:03:02.257842 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-xxlp6_372be3ce-a380-4c0e-bbee-49c2e3b80095/console-operator/1.log" Jul 10 03:03:02.260120 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:03:02.260096 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-xxlp6_372be3ce-a380-4c0e-bbee-49c2e3b80095/console-operator/1.log" Jul 10 03:08:02.298824 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:08:02.298783 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-xxlp6_372be3ce-a380-4c0e-bbee-49c2e3b80095/console-operator/1.log" Jul 10 03:08:02.299284 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:08:02.298847 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-xxlp6_372be3ce-a380-4c0e-bbee-49c2e3b80095/console-operator/1.log" Jul 10 03:13:02.328575 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:13:02.328542 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-xxlp6_372be3ce-a380-4c0e-bbee-49c2e3b80095/console-operator/1.log" Jul 10 03:13:02.330211 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:13:02.330185 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-xxlp6_372be3ce-a380-4c0e-bbee-49c2e3b80095/console-operator/1.log" Jul 10 03:13:58.346078 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:13:58.346044 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-363fbef7-5df9f4d97-5djl9"] Jul 10 03:13:58.349872 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:13:58.349850 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-363fbef7-5df9f4d97-5djl9" Jul 10 03:13:58.352314 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:13:58.352291 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-363fbef7-serving-cert\"" Jul 10 03:13:58.359639 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:13:58.359615 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-363fbef7-5df9f4d97-5djl9"] Jul 10 03:13:58.430621 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:13:58.430576 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61a01719-6e66-4d0e-a787-cf7bdab99bc5-openshift-service-ca\") pod \"maas-api-e2e-ait-363fbef7-5df9f4d97-5djl9\" (UID: \"61a01719-6e66-4d0e-a787-cf7bdab99bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-363fbef7-5df9f4d97-5djl9" Jul 10 03:13:58.430791 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:13:58.430673 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61a01719-6e66-4d0e-a787-cf7bdab99bc5-maas-api-tls\") pod \"maas-api-e2e-ait-363fbef7-5df9f4d97-5djl9\" (UID: \"61a01719-6e66-4d0e-a787-cf7bdab99bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-363fbef7-5df9f4d97-5djl9" Jul 10 03:13:58.430791 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:13:58.430700 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkbmp\" (UniqueName: \"kubernetes.io/projected/61a01719-6e66-4d0e-a787-cf7bdab99bc5-kube-api-access-wkbmp\") pod \"maas-api-e2e-ait-363fbef7-5df9f4d97-5djl9\" (UID: \"61a01719-6e66-4d0e-a787-cf7bdab99bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-363fbef7-5df9f4d97-5djl9" Jul 10 03:13:58.531409 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:13:58.531376 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61a01719-6e66-4d0e-a787-cf7bdab99bc5-openshift-service-ca\") pod \"maas-api-e2e-ait-363fbef7-5df9f4d97-5djl9\" (UID: \"61a01719-6e66-4d0e-a787-cf7bdab99bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-363fbef7-5df9f4d97-5djl9" Jul 10 03:13:58.531611 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:13:58.531433 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61a01719-6e66-4d0e-a787-cf7bdab99bc5-maas-api-tls\") pod \"maas-api-e2e-ait-363fbef7-5df9f4d97-5djl9\" (UID: \"61a01719-6e66-4d0e-a787-cf7bdab99bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-363fbef7-5df9f4d97-5djl9" Jul 10 03:13:58.531611 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:13:58.531457 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkbmp\" (UniqueName: \"kubernetes.io/projected/61a01719-6e66-4d0e-a787-cf7bdab99bc5-kube-api-access-wkbmp\") pod \"maas-api-e2e-ait-363fbef7-5df9f4d97-5djl9\" (UID: \"61a01719-6e66-4d0e-a787-cf7bdab99bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-363fbef7-5df9f4d97-5djl9" Jul 10 03:13:58.531611 ip-10-0-130-139 kubenswrapper[2575]: E0710 03:13:58.531584 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-363fbef7-serving-cert: secret "maas-api-e2e-ait-363fbef7-serving-cert" not found Jul 10 03:13:58.531787 ip-10-0-130-139 kubenswrapper[2575]: E0710 03:13:58.531648 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61a01719-6e66-4d0e-a787-cf7bdab99bc5-maas-api-tls podName:61a01719-6e66-4d0e-a787-cf7bdab99bc5 nodeName:}" failed. No retries permitted until 2026-07-10 03:13:59.031631459 +0000 UTC m=+1557.312747722 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/61a01719-6e66-4d0e-a787-cf7bdab99bc5-maas-api-tls") pod "maas-api-e2e-ait-363fbef7-5df9f4d97-5djl9" (UID: "61a01719-6e66-4d0e-a787-cf7bdab99bc5") : secret "maas-api-e2e-ait-363fbef7-serving-cert" not found Jul 10 03:13:58.532088 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:13:58.532069 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61a01719-6e66-4d0e-a787-cf7bdab99bc5-openshift-service-ca\") pod \"maas-api-e2e-ait-363fbef7-5df9f4d97-5djl9\" (UID: \"61a01719-6e66-4d0e-a787-cf7bdab99bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-363fbef7-5df9f4d97-5djl9" Jul 10 03:13:58.540156 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:13:58.540124 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkbmp\" (UniqueName: \"kubernetes.io/projected/61a01719-6e66-4d0e-a787-cf7bdab99bc5-kube-api-access-wkbmp\") pod \"maas-api-e2e-ait-363fbef7-5df9f4d97-5djl9\" (UID: \"61a01719-6e66-4d0e-a787-cf7bdab99bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-363fbef7-5df9f4d97-5djl9" Jul 10 03:13:59.035914 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:13:59.035879 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61a01719-6e66-4d0e-a787-cf7bdab99bc5-maas-api-tls\") pod \"maas-api-e2e-ait-363fbef7-5df9f4d97-5djl9\" (UID: \"61a01719-6e66-4d0e-a787-cf7bdab99bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-363fbef7-5df9f4d97-5djl9" Jul 10 03:13:59.038420 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:13:59.038394 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61a01719-6e66-4d0e-a787-cf7bdab99bc5-maas-api-tls\") pod \"maas-api-e2e-ait-363fbef7-5df9f4d97-5djl9\" (UID: \"61a01719-6e66-4d0e-a787-cf7bdab99bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-363fbef7-5df9f4d97-5djl9" Jul 10 03:13:59.261591 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:13:59.261551 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-363fbef7-5df9f4d97-5djl9" Jul 10 03:13:59.413448 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:13:59.413418 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-363fbef7-5df9f4d97-5djl9"] Jul 10 03:13:59.414789 ip-10-0-130-139 kubenswrapper[2575]: W0710 03:13:59.414765 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61a01719_6e66_4d0e_a787_cf7bdab99bc5.slice/crio-6229a73c075fad7b3d0d6e23515d0855650ca13102f7f47fcb8eb3540280e29c WatchSource:0}: Error finding container 6229a73c075fad7b3d0d6e23515d0855650ca13102f7f47fcb8eb3540280e29c: Status 404 returned error can't find the container with id 6229a73c075fad7b3d0d6e23515d0855650ca13102f7f47fcb8eb3540280e29c Jul 10 03:13:59.416440 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:13:59.416420 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 03:14:00.260107 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:00.260071 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-363fbef7-5df9f4d97-5djl9" event={"ID":"61a01719-6e66-4d0e-a787-cf7bdab99bc5","Type":"ContainerStarted","Data":"1c9ede8d54187b5ed5baae1dc16e52a0295de2893449ff8d7e05191d991fa7d3"} Jul 10 03:14:00.260287 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:00.260127 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-363fbef7-5df9f4d97-5djl9" event={"ID":"61a01719-6e66-4d0e-a787-cf7bdab99bc5","Type":"ContainerStarted","Data":"6229a73c075fad7b3d0d6e23515d0855650ca13102f7f47fcb8eb3540280e29c"} Jul 10 03:14:00.260287 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:00.260168 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-363fbef7-5df9f4d97-5djl9" Jul 10 03:14:00.277297 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:00.277244 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-363fbef7-5df9f4d97-5djl9" podStartSLOduration=1.786680655 podStartE2EDuration="2.277226166s" totalImagesPullingTime="490.545511ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 03:13:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 03:14:00.27497045 +0000 UTC m=+1558.556086761" watchObservedRunningTime="2026-07-10 03:14:00.277226166 +0000 UTC m=+1558.558342453" Jul 10 03:14:03.679464 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:03.679429 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-363fbef7-5df9f4d97-5djl9"] Jul 10 03:14:03.679944 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:03.679690 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-363fbef7-5df9f4d97-5djl9" podUID="61a01719-6e66-4d0e-a787-cf7bdab99bc5" containerName="maas-api" containerID="cri-o://1c9ede8d54187b5ed5baae1dc16e52a0295de2893449ff8d7e05191d991fa7d3" gracePeriod=30 Jul 10 03:14:03.937907 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:03.937845 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-363fbef7-5df9f4d97-5djl9" Jul 10 03:14:04.080636 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:04.080603 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wkbmp\" (UniqueName: \"kubernetes.io/projected/61a01719-6e66-4d0e-a787-cf7bdab99bc5-kube-api-access-wkbmp\") pod \"61a01719-6e66-4d0e-a787-cf7bdab99bc5\" (UID: \"61a01719-6e66-4d0e-a787-cf7bdab99bc5\") " Jul 10 03:14:04.080826 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:04.080663 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61a01719-6e66-4d0e-a787-cf7bdab99bc5-maas-api-tls\") pod \"61a01719-6e66-4d0e-a787-cf7bdab99bc5\" (UID: \"61a01719-6e66-4d0e-a787-cf7bdab99bc5\") " Jul 10 03:14:04.080826 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:04.080725 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61a01719-6e66-4d0e-a787-cf7bdab99bc5-openshift-service-ca\") pod \"61a01719-6e66-4d0e-a787-cf7bdab99bc5\" (UID: \"61a01719-6e66-4d0e-a787-cf7bdab99bc5\") " Jul 10 03:14:04.081179 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:04.081143 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61a01719-6e66-4d0e-a787-cf7bdab99bc5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "61a01719-6e66-4d0e-a787-cf7bdab99bc5" (UID: "61a01719-6e66-4d0e-a787-cf7bdab99bc5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 03:14:04.082831 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:04.082792 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61a01719-6e66-4d0e-a787-cf7bdab99bc5-kube-api-access-wkbmp" (OuterVolumeSpecName: "kube-api-access-wkbmp") pod "61a01719-6e66-4d0e-a787-cf7bdab99bc5" (UID: "61a01719-6e66-4d0e-a787-cf7bdab99bc5"). InnerVolumeSpecName "kube-api-access-wkbmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 03:14:04.082896 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:04.082839 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61a01719-6e66-4d0e-a787-cf7bdab99bc5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "61a01719-6e66-4d0e-a787-cf7bdab99bc5" (UID: "61a01719-6e66-4d0e-a787-cf7bdab99bc5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 03:14:04.182133 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:04.182104 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkbmp\" (UniqueName: \"kubernetes.io/projected/61a01719-6e66-4d0e-a787-cf7bdab99bc5-kube-api-access-wkbmp\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 03:14:04.182133 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:04.182129 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61a01719-6e66-4d0e-a787-cf7bdab99bc5-maas-api-tls\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 03:14:04.182133 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:04.182138 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61a01719-6e66-4d0e-a787-cf7bdab99bc5-openshift-service-ca\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 03:14:04.281358 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:04.281322 2575 generic.go:356] "Generic (PLEG): container finished" podID="61a01719-6e66-4d0e-a787-cf7bdab99bc5" containerID="1c9ede8d54187b5ed5baae1dc16e52a0295de2893449ff8d7e05191d991fa7d3" exitCode=0 Jul 10 03:14:04.281509 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:04.281369 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-363fbef7-5df9f4d97-5djl9" Jul 10 03:14:04.281509 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:04.281407 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-363fbef7-5df9f4d97-5djl9" event={"ID":"61a01719-6e66-4d0e-a787-cf7bdab99bc5","Type":"ContainerDied","Data":"1c9ede8d54187b5ed5baae1dc16e52a0295de2893449ff8d7e05191d991fa7d3"} Jul 10 03:14:04.281509 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:04.281463 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-363fbef7-5df9f4d97-5djl9" event={"ID":"61a01719-6e66-4d0e-a787-cf7bdab99bc5","Type":"ContainerDied","Data":"6229a73c075fad7b3d0d6e23515d0855650ca13102f7f47fcb8eb3540280e29c"} Jul 10 03:14:04.281509 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:04.281486 2575 scope.go:117] "RemoveContainer" containerID="1c9ede8d54187b5ed5baae1dc16e52a0295de2893449ff8d7e05191d991fa7d3" Jul 10 03:14:04.291673 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:04.291652 2575 scope.go:117] "RemoveContainer" containerID="1c9ede8d54187b5ed5baae1dc16e52a0295de2893449ff8d7e05191d991fa7d3" Jul 10 03:14:04.292061 ip-10-0-130-139 kubenswrapper[2575]: E0710 03:14:04.292039 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c9ede8d54187b5ed5baae1dc16e52a0295de2893449ff8d7e05191d991fa7d3\": container with ID starting with 1c9ede8d54187b5ed5baae1dc16e52a0295de2893449ff8d7e05191d991fa7d3 not found: ID does not exist" containerID="1c9ede8d54187b5ed5baae1dc16e52a0295de2893449ff8d7e05191d991fa7d3" Jul 10 03:14:04.292155 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:04.292073 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c9ede8d54187b5ed5baae1dc16e52a0295de2893449ff8d7e05191d991fa7d3"} err="failed to get container status \"1c9ede8d54187b5ed5baae1dc16e52a0295de2893449ff8d7e05191d991fa7d3\": rpc error: code = NotFound desc = could not find container \"1c9ede8d54187b5ed5baae1dc16e52a0295de2893449ff8d7e05191d991fa7d3\": container with ID starting with 1c9ede8d54187b5ed5baae1dc16e52a0295de2893449ff8d7e05191d991fa7d3 not found: ID does not exist" Jul 10 03:14:04.305546 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:04.305265 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-363fbef7-5df9f4d97-5djl9"] Jul 10 03:14:04.307446 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:04.307426 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-363fbef7-5df9f4d97-5djl9"] Jul 10 03:14:06.289480 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:06.289447 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61a01719-6e66-4d0e-a787-cf7bdab99bc5" path="/var/lib/kubelet/pods/61a01719-6e66-4d0e-a787-cf7bdab99bc5/volumes" Jul 10 03:14:10.814369 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:10.814320 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-df84dc3f-58fc64596f-bjq2v"] Jul 10 03:14:10.814861 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:10.814765 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61a01719-6e66-4d0e-a787-cf7bdab99bc5" containerName="maas-api" Jul 10 03:14:10.814861 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:10.814777 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="61a01719-6e66-4d0e-a787-cf7bdab99bc5" containerName="maas-api" Jul 10 03:14:10.814986 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:10.814890 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="61a01719-6e66-4d0e-a787-cf7bdab99bc5" containerName="maas-api" Jul 10 03:14:10.819372 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:10.819353 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df84dc3f-58fc64596f-bjq2v" Jul 10 03:14:10.821758 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:10.821734 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-df84dc3f-serving-cert\"" Jul 10 03:14:10.828842 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:10.827920 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-df84dc3f-58fc64596f-bjq2v"] Jul 10 03:14:10.947238 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:10.947204 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a560d3b-d568-4dca-81fe-6a70cf1a3ebe-maas-api-tls\") pod \"maas-api-e2e-ait-df84dc3f-58fc64596f-bjq2v\" (UID: \"9a560d3b-d568-4dca-81fe-6a70cf1a3ebe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df84dc3f-58fc64596f-bjq2v" Jul 10 03:14:10.947414 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:10.947269 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a560d3b-d568-4dca-81fe-6a70cf1a3ebe-openshift-service-ca\") pod \"maas-api-e2e-ait-df84dc3f-58fc64596f-bjq2v\" (UID: \"9a560d3b-d568-4dca-81fe-6a70cf1a3ebe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df84dc3f-58fc64596f-bjq2v" Jul 10 03:14:10.947414 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:10.947373 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txjn9\" (UniqueName: \"kubernetes.io/projected/9a560d3b-d568-4dca-81fe-6a70cf1a3ebe-kube-api-access-txjn9\") pod \"maas-api-e2e-ait-df84dc3f-58fc64596f-bjq2v\" (UID: \"9a560d3b-d568-4dca-81fe-6a70cf1a3ebe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df84dc3f-58fc64596f-bjq2v" Jul 10 03:14:11.048154 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:11.048120 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txjn9\" (UniqueName: \"kubernetes.io/projected/9a560d3b-d568-4dca-81fe-6a70cf1a3ebe-kube-api-access-txjn9\") pod \"maas-api-e2e-ait-df84dc3f-58fc64596f-bjq2v\" (UID: \"9a560d3b-d568-4dca-81fe-6a70cf1a3ebe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df84dc3f-58fc64596f-bjq2v" Jul 10 03:14:11.048154 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:11.048164 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a560d3b-d568-4dca-81fe-6a70cf1a3ebe-maas-api-tls\") pod \"maas-api-e2e-ait-df84dc3f-58fc64596f-bjq2v\" (UID: \"9a560d3b-d568-4dca-81fe-6a70cf1a3ebe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df84dc3f-58fc64596f-bjq2v" Jul 10 03:14:11.048422 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:11.048198 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a560d3b-d568-4dca-81fe-6a70cf1a3ebe-openshift-service-ca\") pod \"maas-api-e2e-ait-df84dc3f-58fc64596f-bjq2v\" (UID: \"9a560d3b-d568-4dca-81fe-6a70cf1a3ebe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df84dc3f-58fc64596f-bjq2v" Jul 10 03:14:11.048422 ip-10-0-130-139 kubenswrapper[2575]: E0710 03:14:11.048343 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-df84dc3f-serving-cert: secret "maas-api-e2e-ait-df84dc3f-serving-cert" not found Jul 10 03:14:11.048422 ip-10-0-130-139 kubenswrapper[2575]: E0710 03:14:11.048422 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a560d3b-d568-4dca-81fe-6a70cf1a3ebe-maas-api-tls podName:9a560d3b-d568-4dca-81fe-6a70cf1a3ebe nodeName:}" failed. No retries permitted until 2026-07-10 03:14:11.548399776 +0000 UTC m=+1569.829516051 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9a560d3b-d568-4dca-81fe-6a70cf1a3ebe-maas-api-tls") pod "maas-api-e2e-ait-df84dc3f-58fc64596f-bjq2v" (UID: "9a560d3b-d568-4dca-81fe-6a70cf1a3ebe") : secret "maas-api-e2e-ait-df84dc3f-serving-cert" not found Jul 10 03:14:11.048803 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:11.048785 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a560d3b-d568-4dca-81fe-6a70cf1a3ebe-openshift-service-ca\") pod \"maas-api-e2e-ait-df84dc3f-58fc64596f-bjq2v\" (UID: \"9a560d3b-d568-4dca-81fe-6a70cf1a3ebe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df84dc3f-58fc64596f-bjq2v" Jul 10 03:14:11.059428 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:11.059400 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txjn9\" (UniqueName: \"kubernetes.io/projected/9a560d3b-d568-4dca-81fe-6a70cf1a3ebe-kube-api-access-txjn9\") pod \"maas-api-e2e-ait-df84dc3f-58fc64596f-bjq2v\" (UID: \"9a560d3b-d568-4dca-81fe-6a70cf1a3ebe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df84dc3f-58fc64596f-bjq2v" Jul 10 03:14:11.552724 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:11.552685 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a560d3b-d568-4dca-81fe-6a70cf1a3ebe-maas-api-tls\") pod \"maas-api-e2e-ait-df84dc3f-58fc64596f-bjq2v\" (UID: \"9a560d3b-d568-4dca-81fe-6a70cf1a3ebe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df84dc3f-58fc64596f-bjq2v" Jul 10 03:14:11.555059 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:11.555031 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a560d3b-d568-4dca-81fe-6a70cf1a3ebe-maas-api-tls\") pod \"maas-api-e2e-ait-df84dc3f-58fc64596f-bjq2v\" (UID: \"9a560d3b-d568-4dca-81fe-6a70cf1a3ebe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-df84dc3f-58fc64596f-bjq2v" Jul 10 03:14:11.733273 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:11.733238 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df84dc3f-58fc64596f-bjq2v" Jul 10 03:14:11.884140 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:11.883991 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-df84dc3f-58fc64596f-bjq2v"] Jul 10 03:14:11.886998 ip-10-0-130-139 kubenswrapper[2575]: W0710 03:14:11.886969 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a560d3b_d568_4dca_81fe_6a70cf1a3ebe.slice/crio-0f1e8d9da146f9726a3764ba81264383b00148d42d0042ec93c1c8ac52a9efc3 WatchSource:0}: Error finding container 0f1e8d9da146f9726a3764ba81264383b00148d42d0042ec93c1c8ac52a9efc3: Status 404 returned error can't find the container with id 0f1e8d9da146f9726a3764ba81264383b00148d42d0042ec93c1c8ac52a9efc3 Jul 10 03:14:12.324980 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:12.324945 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df84dc3f-58fc64596f-bjq2v" event={"ID":"9a560d3b-d568-4dca-81fe-6a70cf1a3ebe","Type":"ContainerStarted","Data":"0f1e8d9da146f9726a3764ba81264383b00148d42d0042ec93c1c8ac52a9efc3"} Jul 10 03:14:13.331562 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:13.331525 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df84dc3f-58fc64596f-bjq2v" event={"ID":"9a560d3b-d568-4dca-81fe-6a70cf1a3ebe","Type":"ContainerStarted","Data":"f0bd6a47cbb1910995d0626ae7351648dbb1bb0ef6dcbfef8f416a07da5ec300"} Jul 10 03:14:13.332129 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:13.331643 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df84dc3f-58fc64596f-bjq2v" Jul 10 03:14:13.346950 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:13.346906 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df84dc3f-58fc64596f-bjq2v" podStartSLOduration=2.86168397 podStartE2EDuration="3.34689383s" totalImagesPullingTime="485.20986ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 03:14:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 03:14:13.345835914 +0000 UTC m=+1571.626952218" watchObservedRunningTime="2026-07-10 03:14:13.34689383 +0000 UTC m=+1571.628010114" Jul 10 03:14:16.172887 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:16.172845 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-df84dc3f-58fc64596f-bjq2v"] Jul 10 03:14:16.173356 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:16.173086 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df84dc3f-58fc64596f-bjq2v" podUID="9a560d3b-d568-4dca-81fe-6a70cf1a3ebe" containerName="maas-api" containerID="cri-o://f0bd6a47cbb1910995d0626ae7351648dbb1bb0ef6dcbfef8f416a07da5ec300" gracePeriod=30 Jul 10 03:14:16.350983 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:16.350955 2575 generic.go:356] "Generic (PLEG): container finished" podID="9a560d3b-d568-4dca-81fe-6a70cf1a3ebe" containerID="f0bd6a47cbb1910995d0626ae7351648dbb1bb0ef6dcbfef8f416a07da5ec300" exitCode=0 Jul 10 03:14:16.351103 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:16.351014 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df84dc3f-58fc64596f-bjq2v" event={"ID":"9a560d3b-d568-4dca-81fe-6a70cf1a3ebe","Type":"ContainerDied","Data":"f0bd6a47cbb1910995d0626ae7351648dbb1bb0ef6dcbfef8f416a07da5ec300"} Jul 10 03:14:16.449580 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:16.449556 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df84dc3f-58fc64596f-bjq2v" Jul 10 03:14:16.599213 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:16.599183 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a560d3b-d568-4dca-81fe-6a70cf1a3ebe-openshift-service-ca\") pod \"9a560d3b-d568-4dca-81fe-6a70cf1a3ebe\" (UID: \"9a560d3b-d568-4dca-81fe-6a70cf1a3ebe\") " Jul 10 03:14:16.599399 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:16.599221 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-txjn9\" (UniqueName: \"kubernetes.io/projected/9a560d3b-d568-4dca-81fe-6a70cf1a3ebe-kube-api-access-txjn9\") pod \"9a560d3b-d568-4dca-81fe-6a70cf1a3ebe\" (UID: \"9a560d3b-d568-4dca-81fe-6a70cf1a3ebe\") " Jul 10 03:14:16.599399 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:16.599369 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a560d3b-d568-4dca-81fe-6a70cf1a3ebe-maas-api-tls\") pod \"9a560d3b-d568-4dca-81fe-6a70cf1a3ebe\" (UID: \"9a560d3b-d568-4dca-81fe-6a70cf1a3ebe\") " Jul 10 03:14:16.599552 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:16.599523 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a560d3b-d568-4dca-81fe-6a70cf1a3ebe-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9a560d3b-d568-4dca-81fe-6a70cf1a3ebe" (UID: "9a560d3b-d568-4dca-81fe-6a70cf1a3ebe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 03:14:16.601330 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:16.601296 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a560d3b-d568-4dca-81fe-6a70cf1a3ebe-kube-api-access-txjn9" (OuterVolumeSpecName: "kube-api-access-txjn9") pod "9a560d3b-d568-4dca-81fe-6a70cf1a3ebe" (UID: "9a560d3b-d568-4dca-81fe-6a70cf1a3ebe"). InnerVolumeSpecName "kube-api-access-txjn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 03:14:16.601442 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:16.601395 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a560d3b-d568-4dca-81fe-6a70cf1a3ebe-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9a560d3b-d568-4dca-81fe-6a70cf1a3ebe" (UID: "9a560d3b-d568-4dca-81fe-6a70cf1a3ebe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 03:14:16.700323 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:16.700236 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a560d3b-d568-4dca-81fe-6a70cf1a3ebe-maas-api-tls\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 03:14:16.700323 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:16.700267 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a560d3b-d568-4dca-81fe-6a70cf1a3ebe-openshift-service-ca\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 03:14:16.700323 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:16.700278 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txjn9\" (UniqueName: \"kubernetes.io/projected/9a560d3b-d568-4dca-81fe-6a70cf1a3ebe-kube-api-access-txjn9\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 03:14:17.364673 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:17.364634 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df84dc3f-58fc64596f-bjq2v" event={"ID":"9a560d3b-d568-4dca-81fe-6a70cf1a3ebe","Type":"ContainerDied","Data":"0f1e8d9da146f9726a3764ba81264383b00148d42d0042ec93c1c8ac52a9efc3"} Jul 10 03:14:17.364673 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:17.364678 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-df84dc3f-58fc64596f-bjq2v" Jul 10 03:14:17.365187 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:17.364737 2575 scope.go:117] "RemoveContainer" containerID="f0bd6a47cbb1910995d0626ae7351648dbb1bb0ef6dcbfef8f416a07da5ec300" Jul 10 03:14:17.385397 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:17.385373 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-df84dc3f-58fc64596f-bjq2v"] Jul 10 03:14:17.388616 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:17.388593 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-df84dc3f-58fc64596f-bjq2v"] Jul 10 03:14:18.289783 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:18.289747 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a560d3b-d568-4dca-81fe-6a70cf1a3ebe" path="/var/lib/kubelet/pods/9a560d3b-d568-4dca-81fe-6a70cf1a3ebe/volumes" Jul 10 03:14:29.047500 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:29.047463 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-df842841-7c4f5dc8f5-x9fzv"] Jul 10 03:14:29.048150 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:29.048129 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a560d3b-d568-4dca-81fe-6a70cf1a3ebe" containerName="maas-api" Jul 10 03:14:29.048217 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:29.048154 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a560d3b-d568-4dca-81fe-6a70cf1a3ebe" containerName="maas-api" Jul 10 03:14:29.048324 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:29.048310 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a560d3b-d568-4dca-81fe-6a70cf1a3ebe" containerName="maas-api" Jul 10 03:14:29.052015 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:29.051992 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-df842841-7c4f5dc8f5-x9fzv" Jul 10 03:14:29.054403 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:29.054379 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-df842841-serving-cert\"" Jul 10 03:14:29.063443 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:29.063421 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-df842841-7c4f5dc8f5-x9fzv"] Jul 10 03:14:29.113821 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:29.113782 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43d5bbe6-41fc-4dec-bc12-a6306a9468f3-maas-api-tls\") pod \"maas-api-e2e-derive-df842841-7c4f5dc8f5-x9fzv\" (UID: \"43d5bbe6-41fc-4dec-bc12-a6306a9468f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-df842841-7c4f5dc8f5-x9fzv" Jul 10 03:14:29.113935 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:29.113851 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43d5bbe6-41fc-4dec-bc12-a6306a9468f3-openshift-service-ca\") pod \"maas-api-e2e-derive-df842841-7c4f5dc8f5-x9fzv\" (UID: \"43d5bbe6-41fc-4dec-bc12-a6306a9468f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-df842841-7c4f5dc8f5-x9fzv" Jul 10 03:14:29.113935 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:29.113925 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgf5q\" (UniqueName: \"kubernetes.io/projected/43d5bbe6-41fc-4dec-bc12-a6306a9468f3-kube-api-access-fgf5q\") pod \"maas-api-e2e-derive-df842841-7c4f5dc8f5-x9fzv\" (UID: \"43d5bbe6-41fc-4dec-bc12-a6306a9468f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-df842841-7c4f5dc8f5-x9fzv" Jul 10 03:14:29.215083 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:29.215054 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgf5q\" (UniqueName: \"kubernetes.io/projected/43d5bbe6-41fc-4dec-bc12-a6306a9468f3-kube-api-access-fgf5q\") pod \"maas-api-e2e-derive-df842841-7c4f5dc8f5-x9fzv\" (UID: \"43d5bbe6-41fc-4dec-bc12-a6306a9468f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-df842841-7c4f5dc8f5-x9fzv" Jul 10 03:14:29.215245 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:29.215120 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43d5bbe6-41fc-4dec-bc12-a6306a9468f3-maas-api-tls\") pod \"maas-api-e2e-derive-df842841-7c4f5dc8f5-x9fzv\" (UID: \"43d5bbe6-41fc-4dec-bc12-a6306a9468f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-df842841-7c4f5dc8f5-x9fzv" Jul 10 03:14:29.215245 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:29.215137 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43d5bbe6-41fc-4dec-bc12-a6306a9468f3-openshift-service-ca\") pod \"maas-api-e2e-derive-df842841-7c4f5dc8f5-x9fzv\" (UID: \"43d5bbe6-41fc-4dec-bc12-a6306a9468f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-df842841-7c4f5dc8f5-x9fzv" Jul 10 03:14:29.215335 ip-10-0-130-139 kubenswrapper[2575]: E0710 03:14:29.215272 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-df842841-serving-cert: secret "maas-api-e2e-derive-df842841-serving-cert" not found Jul 10 03:14:29.215375 ip-10-0-130-139 kubenswrapper[2575]: E0710 03:14:29.215349 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43d5bbe6-41fc-4dec-bc12-a6306a9468f3-maas-api-tls podName:43d5bbe6-41fc-4dec-bc12-a6306a9468f3 nodeName:}" failed. No retries permitted until 2026-07-10 03:14:29.715326812 +0000 UTC m=+1587.996443079 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/43d5bbe6-41fc-4dec-bc12-a6306a9468f3-maas-api-tls") pod "maas-api-e2e-derive-df842841-7c4f5dc8f5-x9fzv" (UID: "43d5bbe6-41fc-4dec-bc12-a6306a9468f3") : secret "maas-api-e2e-derive-df842841-serving-cert" not found Jul 10 03:14:29.215719 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:29.215700 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43d5bbe6-41fc-4dec-bc12-a6306a9468f3-openshift-service-ca\") pod \"maas-api-e2e-derive-df842841-7c4f5dc8f5-x9fzv\" (UID: \"43d5bbe6-41fc-4dec-bc12-a6306a9468f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-df842841-7c4f5dc8f5-x9fzv" Jul 10 03:14:29.223127 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:29.223093 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgf5q\" (UniqueName: \"kubernetes.io/projected/43d5bbe6-41fc-4dec-bc12-a6306a9468f3-kube-api-access-fgf5q\") pod \"maas-api-e2e-derive-df842841-7c4f5dc8f5-x9fzv\" (UID: \"43d5bbe6-41fc-4dec-bc12-a6306a9468f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-df842841-7c4f5dc8f5-x9fzv" Jul 10 03:14:29.719111 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:29.719079 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43d5bbe6-41fc-4dec-bc12-a6306a9468f3-maas-api-tls\") pod \"maas-api-e2e-derive-df842841-7c4f5dc8f5-x9fzv\" (UID: \"43d5bbe6-41fc-4dec-bc12-a6306a9468f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-df842841-7c4f5dc8f5-x9fzv" Jul 10 03:14:29.721452 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:29.721424 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43d5bbe6-41fc-4dec-bc12-a6306a9468f3-maas-api-tls\") pod \"maas-api-e2e-derive-df842841-7c4f5dc8f5-x9fzv\" (UID: \"43d5bbe6-41fc-4dec-bc12-a6306a9468f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-df842841-7c4f5dc8f5-x9fzv" Jul 10 03:14:29.966181 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:29.966148 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-df842841-7c4f5dc8f5-x9fzv" Jul 10 03:14:30.132858 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:30.132819 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-df842841-7c4f5dc8f5-x9fzv"] Jul 10 03:14:30.135358 ip-10-0-130-139 kubenswrapper[2575]: W0710 03:14:30.135317 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43d5bbe6_41fc_4dec_bc12_a6306a9468f3.slice/crio-b606031e96bd13e1ea31f47ff74c0ddc6c70690b91b0505cdd47320cbf3286d8 WatchSource:0}: Error finding container b606031e96bd13e1ea31f47ff74c0ddc6c70690b91b0505cdd47320cbf3286d8: Status 404 returned error can't find the container with id b606031e96bd13e1ea31f47ff74c0ddc6c70690b91b0505cdd47320cbf3286d8 Jul 10 03:14:30.433576 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:30.433543 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-df842841-7c4f5dc8f5-x9fzv" event={"ID":"43d5bbe6-41fc-4dec-bc12-a6306a9468f3","Type":"ContainerStarted","Data":"b606031e96bd13e1ea31f47ff74c0ddc6c70690b91b0505cdd47320cbf3286d8"} Jul 10 03:14:31.440004 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:31.439959 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-df842841-7c4f5dc8f5-x9fzv" event={"ID":"43d5bbe6-41fc-4dec-bc12-a6306a9468f3","Type":"ContainerStarted","Data":"3a4197214fc1c2de8998e714e196496dfdfb9fcc9bf7a6778b0db29426e013a8"} Jul 10 03:14:31.440399 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:31.440084 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-df842841-7c4f5dc8f5-x9fzv" Jul 10 03:14:31.455588 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:31.455537 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-df842841-7c4f5dc8f5-x9fzv" podStartSLOduration=2.06586956 podStartE2EDuration="2.455520678s" totalImagesPullingTime="389.651118ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 03:14:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 03:14:31.454697421 +0000 UTC m=+1589.735813733" watchObservedRunningTime="2026-07-10 03:14:31.455520678 +0000 UTC m=+1589.736636964" Jul 10 03:14:33.746307 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:33.746268 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-df842841-7c4f5dc8f5-x9fzv"] Jul 10 03:14:33.746747 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:33.746498 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-df842841-7c4f5dc8f5-x9fzv" podUID="43d5bbe6-41fc-4dec-bc12-a6306a9468f3" containerName="maas-api" containerID="cri-o://3a4197214fc1c2de8998e714e196496dfdfb9fcc9bf7a6778b0db29426e013a8" gracePeriod=30 Jul 10 03:14:34.006049 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:34.006023 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-df842841-7c4f5dc8f5-x9fzv" Jul 10 03:14:34.065443 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:34.065413 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43d5bbe6-41fc-4dec-bc12-a6306a9468f3-openshift-service-ca\") pod \"43d5bbe6-41fc-4dec-bc12-a6306a9468f3\" (UID: \"43d5bbe6-41fc-4dec-bc12-a6306a9468f3\") " Jul 10 03:14:34.065630 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:34.065472 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43d5bbe6-41fc-4dec-bc12-a6306a9468f3-maas-api-tls\") pod \"43d5bbe6-41fc-4dec-bc12-a6306a9468f3\" (UID: \"43d5bbe6-41fc-4dec-bc12-a6306a9468f3\") " Jul 10 03:14:34.065630 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:34.065569 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fgf5q\" (UniqueName: \"kubernetes.io/projected/43d5bbe6-41fc-4dec-bc12-a6306a9468f3-kube-api-access-fgf5q\") pod \"43d5bbe6-41fc-4dec-bc12-a6306a9468f3\" (UID: \"43d5bbe6-41fc-4dec-bc12-a6306a9468f3\") " Jul 10 03:14:34.065876 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:34.065850 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43d5bbe6-41fc-4dec-bc12-a6306a9468f3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "43d5bbe6-41fc-4dec-bc12-a6306a9468f3" (UID: "43d5bbe6-41fc-4dec-bc12-a6306a9468f3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 03:14:34.067609 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:34.067585 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43d5bbe6-41fc-4dec-bc12-a6306a9468f3-kube-api-access-fgf5q" (OuterVolumeSpecName: "kube-api-access-fgf5q") pod "43d5bbe6-41fc-4dec-bc12-a6306a9468f3" (UID: "43d5bbe6-41fc-4dec-bc12-a6306a9468f3"). InnerVolumeSpecName "kube-api-access-fgf5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 03:14:34.067683 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:34.067612 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43d5bbe6-41fc-4dec-bc12-a6306a9468f3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "43d5bbe6-41fc-4dec-bc12-a6306a9468f3" (UID: "43d5bbe6-41fc-4dec-bc12-a6306a9468f3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 03:14:34.167080 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:34.167051 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgf5q\" (UniqueName: \"kubernetes.io/projected/43d5bbe6-41fc-4dec-bc12-a6306a9468f3-kube-api-access-fgf5q\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 03:14:34.167080 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:34.167076 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43d5bbe6-41fc-4dec-bc12-a6306a9468f3-openshift-service-ca\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 03:14:34.167080 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:34.167086 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43d5bbe6-41fc-4dec-bc12-a6306a9468f3-maas-api-tls\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 03:14:34.456726 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:34.456693 2575 generic.go:356] "Generic (PLEG): container finished" podID="43d5bbe6-41fc-4dec-bc12-a6306a9468f3" containerID="3a4197214fc1c2de8998e714e196496dfdfb9fcc9bf7a6778b0db29426e013a8" exitCode=0 Jul 10 03:14:34.456917 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:34.456754 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-df842841-7c4f5dc8f5-x9fzv" Jul 10 03:14:34.456917 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:34.456770 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-df842841-7c4f5dc8f5-x9fzv" event={"ID":"43d5bbe6-41fc-4dec-bc12-a6306a9468f3","Type":"ContainerDied","Data":"3a4197214fc1c2de8998e714e196496dfdfb9fcc9bf7a6778b0db29426e013a8"} Jul 10 03:14:34.456917 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:34.456837 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-df842841-7c4f5dc8f5-x9fzv" event={"ID":"43d5bbe6-41fc-4dec-bc12-a6306a9468f3","Type":"ContainerDied","Data":"b606031e96bd13e1ea31f47ff74c0ddc6c70690b91b0505cdd47320cbf3286d8"} Jul 10 03:14:34.456917 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:34.456860 2575 scope.go:117] "RemoveContainer" containerID="3a4197214fc1c2de8998e714e196496dfdfb9fcc9bf7a6778b0db29426e013a8" Jul 10 03:14:34.466049 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:34.466036 2575 scope.go:117] "RemoveContainer" containerID="3a4197214fc1c2de8998e714e196496dfdfb9fcc9bf7a6778b0db29426e013a8" Jul 10 03:14:34.466304 ip-10-0-130-139 kubenswrapper[2575]: E0710 03:14:34.466286 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a4197214fc1c2de8998e714e196496dfdfb9fcc9bf7a6778b0db29426e013a8\": container with ID starting with 3a4197214fc1c2de8998e714e196496dfdfb9fcc9bf7a6778b0db29426e013a8 not found: ID does not exist" containerID="3a4197214fc1c2de8998e714e196496dfdfb9fcc9bf7a6778b0db29426e013a8" Jul 10 03:14:34.466347 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:34.466313 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a4197214fc1c2de8998e714e196496dfdfb9fcc9bf7a6778b0db29426e013a8"} err="failed to get container status \"3a4197214fc1c2de8998e714e196496dfdfb9fcc9bf7a6778b0db29426e013a8\": rpc error: code = NotFound desc = could not find container \"3a4197214fc1c2de8998e714e196496dfdfb9fcc9bf7a6778b0db29426e013a8\": container with ID starting with 3a4197214fc1c2de8998e714e196496dfdfb9fcc9bf7a6778b0db29426e013a8 not found: ID does not exist" Jul 10 03:14:34.473518 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:34.473496 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-df842841-7c4f5dc8f5-x9fzv"] Jul 10 03:14:34.477015 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:34.476996 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-df842841-7c4f5dc8f5-x9fzv"] Jul 10 03:14:36.289110 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:14:36.289078 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43d5bbe6-41fc-4dec-bc12-a6306a9468f3" path="/var/lib/kubelet/pods/43d5bbe6-41fc-4dec-bc12-a6306a9468f3/volumes" Jul 10 03:15:00.136417 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:15:00.136327 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727555-bfljk"] Jul 10 03:15:00.137008 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:15:00.136960 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43d5bbe6-41fc-4dec-bc12-a6306a9468f3" containerName="maas-api" Jul 10 03:15:00.137008 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:15:00.136981 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="43d5bbe6-41fc-4dec-bc12-a6306a9468f3" containerName="maas-api" Jul 10 03:15:00.137139 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:15:00.137114 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="43d5bbe6-41fc-4dec-bc12-a6306a9468f3" containerName="maas-api" Jul 10 03:15:00.141168 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:15:00.141146 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727555-bfljk" Jul 10 03:15:00.144254 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:15:00.144230 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cns2h\"" Jul 10 03:15:00.146861 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:15:00.146840 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727555-bfljk"] Jul 10 03:15:00.152965 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:15:00.152949 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727555-bfljk" Jul 10 03:15:00.290310 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:15:00.290285 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727555-bfljk"] Jul 10 03:15:00.292545 ip-10-0-130-139 kubenswrapper[2575]: W0710 03:15:00.292519 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda085aafd_a0a0_467a_b169_c35e56849d09.slice/crio-6580283498f26bd2e653bbf800238b88bb49d993bc9c6e677a94b1c94f629685 WatchSource:0}: Error finding container 6580283498f26bd2e653bbf800238b88bb49d993bc9c6e677a94b1c94f629685: Status 404 returned error can't find the container with id 6580283498f26bd2e653bbf800238b88bb49d993bc9c6e677a94b1c94f629685 Jul 10 03:15:00.598524 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:15:00.598488 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727555-bfljk" event={"ID":"a085aafd-a0a0-467a-b169-c35e56849d09","Type":"ContainerStarted","Data":"6580283498f26bd2e653bbf800238b88bb49d993bc9c6e677a94b1c94f629685"} Jul 10 03:15:03.616518 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:15:03.616476 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727555-bfljk" event={"ID":"a085aafd-a0a0-467a-b169-c35e56849d09","Type":"ContainerStarted","Data":"f304b76c5496e0b4b1765331dc03b062feb139cc8f0865349b0f7900d8c9c5e5"} Jul 10 03:15:03.629969 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:15:03.629921 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727555-bfljk" podStartSLOduration=1.358103955 podStartE2EDuration="3.629904597s" totalImagesPullingTime="2.271800642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 03:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 03:15:03.629354696 +0000 UTC m=+1621.910470982" watchObservedRunningTime="2026-07-10 03:15:03.629904597 +0000 UTC m=+1621.911020882" Jul 10 03:16:00.021114 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:00.021075 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727555-bfljk"] Jul 10 03:16:00.021680 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:00.021276 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727555-bfljk" podUID="a085aafd-a0a0-467a-b169-c35e56849d09" containerName="cleanup" containerID="cri-o://f304b76c5496e0b4b1765331dc03b062feb139cc8f0865349b0f7900d8c9c5e5" gracePeriod=30 Jul 10 03:16:17.251592 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:17.251560 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-66c0aa35-599896975-k2x45"] Jul 10 03:16:17.255402 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:17.255380 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-66c0aa35-599896975-k2x45" Jul 10 03:16:17.257760 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:17.257736 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-66c0aa35-serving-cert\"" Jul 10 03:16:17.264788 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:17.264703 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-66c0aa35-599896975-k2x45"] Jul 10 03:16:17.292033 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:17.291999 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fd7ec3b-612a-4916-b4f2-562a34da02e8-openshift-service-ca\") pod \"maas-api-e2e-mt-66c0aa35-599896975-k2x45\" (UID: \"5fd7ec3b-612a-4916-b4f2-562a34da02e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-66c0aa35-599896975-k2x45" Jul 10 03:16:17.292186 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:17.292088 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsch7\" (UniqueName: \"kubernetes.io/projected/5fd7ec3b-612a-4916-b4f2-562a34da02e8-kube-api-access-rsch7\") pod \"maas-api-e2e-mt-66c0aa35-599896975-k2x45\" (UID: \"5fd7ec3b-612a-4916-b4f2-562a34da02e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-66c0aa35-599896975-k2x45" Jul 10 03:16:17.292186 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:17.292151 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fd7ec3b-612a-4916-b4f2-562a34da02e8-maas-api-tls\") pod \"maas-api-e2e-mt-66c0aa35-599896975-k2x45\" (UID: \"5fd7ec3b-612a-4916-b4f2-562a34da02e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-66c0aa35-599896975-k2x45" Jul 10 03:16:17.393203 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:17.393173 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fd7ec3b-612a-4916-b4f2-562a34da02e8-maas-api-tls\") pod \"maas-api-e2e-mt-66c0aa35-599896975-k2x45\" (UID: \"5fd7ec3b-612a-4916-b4f2-562a34da02e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-66c0aa35-599896975-k2x45" Jul 10 03:16:17.393418 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:17.393223 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fd7ec3b-612a-4916-b4f2-562a34da02e8-openshift-service-ca\") pod \"maas-api-e2e-mt-66c0aa35-599896975-k2x45\" (UID: \"5fd7ec3b-612a-4916-b4f2-562a34da02e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-66c0aa35-599896975-k2x45" Jul 10 03:16:17.393418 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:17.393277 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsch7\" (UniqueName: \"kubernetes.io/projected/5fd7ec3b-612a-4916-b4f2-562a34da02e8-kube-api-access-rsch7\") pod \"maas-api-e2e-mt-66c0aa35-599896975-k2x45\" (UID: \"5fd7ec3b-612a-4916-b4f2-562a34da02e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-66c0aa35-599896975-k2x45" Jul 10 03:16:17.393418 ip-10-0-130-139 kubenswrapper[2575]: E0710 03:16:17.393329 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-66c0aa35-serving-cert: secret "maas-api-e2e-mt-66c0aa35-serving-cert" not found Jul 10 03:16:17.393418 ip-10-0-130-139 kubenswrapper[2575]: E0710 03:16:17.393398 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fd7ec3b-612a-4916-b4f2-562a34da02e8-maas-api-tls podName:5fd7ec3b-612a-4916-b4f2-562a34da02e8 nodeName:}" failed. No retries permitted until 2026-07-10 03:16:17.893380602 +0000 UTC m=+1696.174496866 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5fd7ec3b-612a-4916-b4f2-562a34da02e8-maas-api-tls") pod "maas-api-e2e-mt-66c0aa35-599896975-k2x45" (UID: "5fd7ec3b-612a-4916-b4f2-562a34da02e8") : secret "maas-api-e2e-mt-66c0aa35-serving-cert" not found Jul 10 03:16:17.393952 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:17.393934 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fd7ec3b-612a-4916-b4f2-562a34da02e8-openshift-service-ca\") pod \"maas-api-e2e-mt-66c0aa35-599896975-k2x45\" (UID: \"5fd7ec3b-612a-4916-b4f2-562a34da02e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-66c0aa35-599896975-k2x45" Jul 10 03:16:17.403804 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:17.403785 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsch7\" (UniqueName: \"kubernetes.io/projected/5fd7ec3b-612a-4916-b4f2-562a34da02e8-kube-api-access-rsch7\") pod \"maas-api-e2e-mt-66c0aa35-599896975-k2x45\" (UID: \"5fd7ec3b-612a-4916-b4f2-562a34da02e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-66c0aa35-599896975-k2x45" Jul 10 03:16:17.897529 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:17.897494 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fd7ec3b-612a-4916-b4f2-562a34da02e8-maas-api-tls\") pod \"maas-api-e2e-mt-66c0aa35-599896975-k2x45\" (UID: \"5fd7ec3b-612a-4916-b4f2-562a34da02e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-66c0aa35-599896975-k2x45" Jul 10 03:16:17.899939 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:17.899921 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fd7ec3b-612a-4916-b4f2-562a34da02e8-maas-api-tls\") pod \"maas-api-e2e-mt-66c0aa35-599896975-k2x45\" (UID: \"5fd7ec3b-612a-4916-b4f2-562a34da02e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-66c0aa35-599896975-k2x45" Jul 10 03:16:18.169380 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:18.169300 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-66c0aa35-599896975-k2x45" Jul 10 03:16:18.308752 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:18.308727 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-66c0aa35-599896975-k2x45"] Jul 10 03:16:18.311454 ip-10-0-130-139 kubenswrapper[2575]: W0710 03:16:18.311424 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fd7ec3b_612a_4916_b4f2_562a34da02e8.slice/crio-9ec11a0421d8134939304a421856ea7b56982381f4e2c537a87d1a152373cbaf WatchSource:0}: Error finding container 9ec11a0421d8134939304a421856ea7b56982381f4e2c537a87d1a152373cbaf: Status 404 returned error can't find the container with id 9ec11a0421d8134939304a421856ea7b56982381f4e2c537a87d1a152373cbaf Jul 10 03:16:19.003558 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:19.003526 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-66c0aa35-599896975-k2x45" event={"ID":"5fd7ec3b-612a-4916-b4f2-562a34da02e8","Type":"ContainerStarted","Data":"5790e0b469a517d54a834bd8d4afd1914745642c27e25e6ffe026b1b979f4d88"} Jul 10 03:16:19.003726 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:19.003576 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-66c0aa35-599896975-k2x45" event={"ID":"5fd7ec3b-612a-4916-b4f2-562a34da02e8","Type":"ContainerStarted","Data":"9ec11a0421d8134939304a421856ea7b56982381f4e2c537a87d1a152373cbaf"} Jul 10 03:16:19.003726 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:19.003627 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-66c0aa35-599896975-k2x45" Jul 10 03:16:19.022735 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:19.022688 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-66c0aa35-599896975-k2x45" podStartSLOduration=1.483333648 podStartE2EDuration="2.02267365s" totalImagesPullingTime="539.340002ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 03:16:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 03:16:19.020339297 +0000 UTC m=+1697.301455600" watchObservedRunningTime="2026-07-10 03:16:19.02267365 +0000 UTC m=+1697.303789963" Jul 10 03:16:25.015429 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:25.015341 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-66c0aa35-599896975-k2x45" Jul 10 03:16:29.161849 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:29.161797 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-66c0aa35-599896975-k2x45"] Jul 10 03:16:29.162286 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:29.162081 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-66c0aa35-599896975-k2x45" podUID="5fd7ec3b-612a-4916-b4f2-562a34da02e8" containerName="maas-api" containerID="cri-o://5790e0b469a517d54a834bd8d4afd1914745642c27e25e6ffe026b1b979f4d88" gracePeriod=30 Jul 10 03:16:29.412616 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:29.412560 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-66c0aa35-599896975-k2x45" Jul 10 03:16:29.500995 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:29.500963 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rsch7\" (UniqueName: \"kubernetes.io/projected/5fd7ec3b-612a-4916-b4f2-562a34da02e8-kube-api-access-rsch7\") pod \"5fd7ec3b-612a-4916-b4f2-562a34da02e8\" (UID: \"5fd7ec3b-612a-4916-b4f2-562a34da02e8\") " Jul 10 03:16:29.501153 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:29.501025 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fd7ec3b-612a-4916-b4f2-562a34da02e8-openshift-service-ca\") pod \"5fd7ec3b-612a-4916-b4f2-562a34da02e8\" (UID: \"5fd7ec3b-612a-4916-b4f2-562a34da02e8\") " Jul 10 03:16:29.501153 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:29.501086 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fd7ec3b-612a-4916-b4f2-562a34da02e8-maas-api-tls\") pod \"5fd7ec3b-612a-4916-b4f2-562a34da02e8\" (UID: \"5fd7ec3b-612a-4916-b4f2-562a34da02e8\") " Jul 10 03:16:29.501373 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:29.501348 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fd7ec3b-612a-4916-b4f2-562a34da02e8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5fd7ec3b-612a-4916-b4f2-562a34da02e8" (UID: "5fd7ec3b-612a-4916-b4f2-562a34da02e8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 03:16:29.503084 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:29.503053 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fd7ec3b-612a-4916-b4f2-562a34da02e8-kube-api-access-rsch7" (OuterVolumeSpecName: "kube-api-access-rsch7") pod "5fd7ec3b-612a-4916-b4f2-562a34da02e8" (UID: "5fd7ec3b-612a-4916-b4f2-562a34da02e8"). InnerVolumeSpecName "kube-api-access-rsch7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 03:16:29.503177 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:29.503134 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fd7ec3b-612a-4916-b4f2-562a34da02e8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5fd7ec3b-612a-4916-b4f2-562a34da02e8" (UID: "5fd7ec3b-612a-4916-b4f2-562a34da02e8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 03:16:29.601889 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:29.601858 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsch7\" (UniqueName: \"kubernetes.io/projected/5fd7ec3b-612a-4916-b4f2-562a34da02e8-kube-api-access-rsch7\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 03:16:29.601889 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:29.601883 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fd7ec3b-612a-4916-b4f2-562a34da02e8-openshift-service-ca\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 03:16:29.601889 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:29.601892 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fd7ec3b-612a-4916-b4f2-562a34da02e8-maas-api-tls\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 03:16:30.067695 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:30.067666 2575 generic.go:356] "Generic (PLEG): container finished" podID="5fd7ec3b-612a-4916-b4f2-562a34da02e8" containerID="5790e0b469a517d54a834bd8d4afd1914745642c27e25e6ffe026b1b979f4d88" exitCode=0 Jul 10 03:16:30.067876 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:30.067709 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-66c0aa35-599896975-k2x45" Jul 10 03:16:30.067876 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:30.067757 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-66c0aa35-599896975-k2x45" event={"ID":"5fd7ec3b-612a-4916-b4f2-562a34da02e8","Type":"ContainerDied","Data":"5790e0b469a517d54a834bd8d4afd1914745642c27e25e6ffe026b1b979f4d88"} Jul 10 03:16:30.067876 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:30.067840 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-66c0aa35-599896975-k2x45" event={"ID":"5fd7ec3b-612a-4916-b4f2-562a34da02e8","Type":"ContainerDied","Data":"9ec11a0421d8134939304a421856ea7b56982381f4e2c537a87d1a152373cbaf"} Jul 10 03:16:30.067876 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:30.067877 2575 scope.go:117] "RemoveContainer" containerID="5790e0b469a517d54a834bd8d4afd1914745642c27e25e6ffe026b1b979f4d88" Jul 10 03:16:30.085696 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:30.085675 2575 scope.go:117] "RemoveContainer" containerID="5790e0b469a517d54a834bd8d4afd1914745642c27e25e6ffe026b1b979f4d88" Jul 10 03:16:30.086165 ip-10-0-130-139 kubenswrapper[2575]: E0710 03:16:30.086132 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5790e0b469a517d54a834bd8d4afd1914745642c27e25e6ffe026b1b979f4d88\": container with ID starting with 5790e0b469a517d54a834bd8d4afd1914745642c27e25e6ffe026b1b979f4d88 not found: ID does not exist" containerID="5790e0b469a517d54a834bd8d4afd1914745642c27e25e6ffe026b1b979f4d88" Jul 10 03:16:30.086275 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:30.086175 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5790e0b469a517d54a834bd8d4afd1914745642c27e25e6ffe026b1b979f4d88"} err="failed to get container status \"5790e0b469a517d54a834bd8d4afd1914745642c27e25e6ffe026b1b979f4d88\": rpc error: code = NotFound desc = could not find container \"5790e0b469a517d54a834bd8d4afd1914745642c27e25e6ffe026b1b979f4d88\": container with ID starting with 5790e0b469a517d54a834bd8d4afd1914745642c27e25e6ffe026b1b979f4d88 not found: ID does not exist" Jul 10 03:16:30.097079 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:30.097047 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-66c0aa35-599896975-k2x45"] Jul 10 03:16:30.103611 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:30.103581 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-66c0aa35-599896975-k2x45"] Jul 10 03:16:30.231280 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:30.231254 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727555-bfljk" Jul 10 03:16:30.290074 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:30.289999 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fd7ec3b-612a-4916-b4f2-562a34da02e8" path="/var/lib/kubelet/pods/5fd7ec3b-612a-4916-b4f2-562a34da02e8/volumes" Jul 10 03:16:31.075422 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:31.075382 2575 generic.go:356] "Generic (PLEG): container finished" podID="a085aafd-a0a0-467a-b169-c35e56849d09" containerID="f304b76c5496e0b4b1765331dc03b062feb139cc8f0865349b0f7900d8c9c5e5" exitCode=137 Jul 10 03:16:31.075422 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:31.075420 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727555-bfljk" Jul 10 03:16:31.075692 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:31.075472 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727555-bfljk" event={"ID":"a085aafd-a0a0-467a-b169-c35e56849d09","Type":"ContainerDied","Data":"f304b76c5496e0b4b1765331dc03b062feb139cc8f0865349b0f7900d8c9c5e5"} Jul 10 03:16:31.075692 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:31.075521 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727555-bfljk" event={"ID":"a085aafd-a0a0-467a-b169-c35e56849d09","Type":"ContainerDied","Data":"6580283498f26bd2e653bbf800238b88bb49d993bc9c6e677a94b1c94f629685"} Jul 10 03:16:31.075692 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:31.075546 2575 scope.go:117] "RemoveContainer" containerID="f304b76c5496e0b4b1765331dc03b062feb139cc8f0865349b0f7900d8c9c5e5" Jul 10 03:16:31.084151 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:31.084133 2575 scope.go:117] "RemoveContainer" containerID="f304b76c5496e0b4b1765331dc03b062feb139cc8f0865349b0f7900d8c9c5e5" Jul 10 03:16:31.084431 ip-10-0-130-139 kubenswrapper[2575]: E0710 03:16:31.084410 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f304b76c5496e0b4b1765331dc03b062feb139cc8f0865349b0f7900d8c9c5e5\": container with ID starting with f304b76c5496e0b4b1765331dc03b062feb139cc8f0865349b0f7900d8c9c5e5 not found: ID does not exist" containerID="f304b76c5496e0b4b1765331dc03b062feb139cc8f0865349b0f7900d8c9c5e5" Jul 10 03:16:31.084471 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:31.084442 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f304b76c5496e0b4b1765331dc03b062feb139cc8f0865349b0f7900d8c9c5e5"} err="failed to get container status \"f304b76c5496e0b4b1765331dc03b062feb139cc8f0865349b0f7900d8c9c5e5\": rpc error: code = NotFound desc = could not find container \"f304b76c5496e0b4b1765331dc03b062feb139cc8f0865349b0f7900d8c9c5e5\": container with ID starting with f304b76c5496e0b4b1765331dc03b062feb139cc8f0865349b0f7900d8c9c5e5 not found: ID does not exist" Jul 10 03:16:31.090260 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:31.090236 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727555-bfljk"] Jul 10 03:16:31.094052 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:31.094033 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727555-bfljk"] Jul 10 03:16:32.288827 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:32.288765 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a085aafd-a0a0-467a-b169-c35e56849d09" path="/var/lib/kubelet/pods/a085aafd-a0a0-467a-b169-c35e56849d09/volumes" Jul 10 03:16:57.594070 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:57.594036 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0212b4b5-5899bcb85d-965k8"] Jul 10 03:16:57.596650 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:57.594684 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a085aafd-a0a0-467a-b169-c35e56849d09" containerName="cleanup" Jul 10 03:16:57.596650 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:57.594702 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a085aafd-a0a0-467a-b169-c35e56849d09" containerName="cleanup" Jul 10 03:16:57.596650 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:57.594731 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fd7ec3b-612a-4916-b4f2-562a34da02e8" containerName="maas-api" Jul 10 03:16:57.596650 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:57.594740 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fd7ec3b-612a-4916-b4f2-562a34da02e8" containerName="maas-api" Jul 10 03:16:57.596650 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:57.594873 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a085aafd-a0a0-467a-b169-c35e56849d09" containerName="cleanup" Jul 10 03:16:57.596650 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:57.594887 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fd7ec3b-612a-4916-b4f2-562a34da02e8" containerName="maas-api" Jul 10 03:16:57.598306 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:57.598288 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0212b4b5-5899bcb85d-965k8" Jul 10 03:16:57.600431 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:57.600412 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-0212b4b5-serving-cert\"" Jul 10 03:16:57.606661 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:57.606640 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0212b4b5-5899bcb85d-965k8"] Jul 10 03:16:57.656577 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:57.656539 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rln22\" (UniqueName: \"kubernetes.io/projected/0adde78f-695f-409d-912b-443f08db780f-kube-api-access-rln22\") pod \"maas-api-e2e-mt-0212b4b5-5899bcb85d-965k8\" (UID: \"0adde78f-695f-409d-912b-443f08db780f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0212b4b5-5899bcb85d-965k8" Jul 10 03:16:57.656730 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:57.656595 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0adde78f-695f-409d-912b-443f08db780f-openshift-service-ca\") pod \"maas-api-e2e-mt-0212b4b5-5899bcb85d-965k8\" (UID: \"0adde78f-695f-409d-912b-443f08db780f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0212b4b5-5899bcb85d-965k8" Jul 10 03:16:57.656730 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:57.656620 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0adde78f-695f-409d-912b-443f08db780f-maas-api-tls\") pod \"maas-api-e2e-mt-0212b4b5-5899bcb85d-965k8\" (UID: \"0adde78f-695f-409d-912b-443f08db780f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0212b4b5-5899bcb85d-965k8" Jul 10 03:16:57.758020 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:57.757984 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rln22\" (UniqueName: \"kubernetes.io/projected/0adde78f-695f-409d-912b-443f08db780f-kube-api-access-rln22\") pod \"maas-api-e2e-mt-0212b4b5-5899bcb85d-965k8\" (UID: \"0adde78f-695f-409d-912b-443f08db780f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0212b4b5-5899bcb85d-965k8" Jul 10 03:16:57.758190 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:57.758037 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0adde78f-695f-409d-912b-443f08db780f-openshift-service-ca\") pod \"maas-api-e2e-mt-0212b4b5-5899bcb85d-965k8\" (UID: \"0adde78f-695f-409d-912b-443f08db780f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0212b4b5-5899bcb85d-965k8" Jul 10 03:16:57.758190 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:57.758056 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0adde78f-695f-409d-912b-443f08db780f-maas-api-tls\") pod \"maas-api-e2e-mt-0212b4b5-5899bcb85d-965k8\" (UID: \"0adde78f-695f-409d-912b-443f08db780f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0212b4b5-5899bcb85d-965k8" Jul 10 03:16:57.758190 ip-10-0-130-139 kubenswrapper[2575]: E0710 03:16:57.758144 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-0212b4b5-serving-cert: secret "maas-api-e2e-mt-0212b4b5-serving-cert" not found Jul 10 03:16:57.758311 ip-10-0-130-139 kubenswrapper[2575]: E0710 03:16:57.758206 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0adde78f-695f-409d-912b-443f08db780f-maas-api-tls podName:0adde78f-695f-409d-912b-443f08db780f nodeName:}" failed. No retries permitted until 2026-07-10 03:16:58.25819066 +0000 UTC m=+1736.539306927 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0adde78f-695f-409d-912b-443f08db780f-maas-api-tls") pod "maas-api-e2e-mt-0212b4b5-5899bcb85d-965k8" (UID: "0adde78f-695f-409d-912b-443f08db780f") : secret "maas-api-e2e-mt-0212b4b5-serving-cert" not found Jul 10 03:16:57.758856 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:57.758836 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0adde78f-695f-409d-912b-443f08db780f-openshift-service-ca\") pod \"maas-api-e2e-mt-0212b4b5-5899bcb85d-965k8\" (UID: \"0adde78f-695f-409d-912b-443f08db780f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0212b4b5-5899bcb85d-965k8" Jul 10 03:16:57.767896 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:57.767869 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rln22\" (UniqueName: \"kubernetes.io/projected/0adde78f-695f-409d-912b-443f08db780f-kube-api-access-rln22\") pod \"maas-api-e2e-mt-0212b4b5-5899bcb85d-965k8\" (UID: \"0adde78f-695f-409d-912b-443f08db780f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0212b4b5-5899bcb85d-965k8" Jul 10 03:16:58.261329 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:58.261288 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0adde78f-695f-409d-912b-443f08db780f-maas-api-tls\") pod \"maas-api-e2e-mt-0212b4b5-5899bcb85d-965k8\" (UID: \"0adde78f-695f-409d-912b-443f08db780f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0212b4b5-5899bcb85d-965k8" Jul 10 03:16:58.263709 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:58.263688 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0adde78f-695f-409d-912b-443f08db780f-maas-api-tls\") pod \"maas-api-e2e-mt-0212b4b5-5899bcb85d-965k8\" (UID: \"0adde78f-695f-409d-912b-443f08db780f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0212b4b5-5899bcb85d-965k8" Jul 10 03:16:58.512190 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:58.512110 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0212b4b5-5899bcb85d-965k8" Jul 10 03:16:58.650555 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:58.650527 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0212b4b5-5899bcb85d-965k8"] Jul 10 03:16:58.652978 ip-10-0-130-139 kubenswrapper[2575]: W0710 03:16:58.652945 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0adde78f_695f_409d_912b_443f08db780f.slice/crio-ce62dfea6525b5015f83a4590f3a09909c3f4953e04b1073e0771dd68b4f655d WatchSource:0}: Error finding container ce62dfea6525b5015f83a4590f3a09909c3f4953e04b1073e0771dd68b4f655d: Status 404 returned error can't find the container with id ce62dfea6525b5015f83a4590f3a09909c3f4953e04b1073e0771dd68b4f655d Jul 10 03:16:59.226496 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:59.226452 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0212b4b5-5899bcb85d-965k8" event={"ID":"0adde78f-695f-409d-912b-443f08db780f","Type":"ContainerStarted","Data":"ff719a469917efe9707e4e903a772900cbe88188e5f6a77bdab3aff02574fd44"} Jul 10 03:16:59.226670 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:59.226509 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0212b4b5-5899bcb85d-965k8" event={"ID":"0adde78f-695f-409d-912b-443f08db780f","Type":"ContainerStarted","Data":"ce62dfea6525b5015f83a4590f3a09909c3f4953e04b1073e0771dd68b4f655d"} Jul 10 03:16:59.226670 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:59.226547 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0212b4b5-5899bcb85d-965k8" Jul 10 03:16:59.242736 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:16:59.242680 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0212b4b5-5899bcb85d-965k8" podStartSLOduration=1.881167328 podStartE2EDuration="2.242664069s" totalImagesPullingTime="361.496741ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 03:16:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 03:16:59.241840814 +0000 UTC m=+1737.522957111" watchObservedRunningTime="2026-07-10 03:16:59.242664069 +0000 UTC m=+1737.523780354" Jul 10 03:17:05.236667 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:17:05.236637 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0212b4b5-5899bcb85d-965k8" Jul 10 03:17:44.916889 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:17:44.916859 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0212b4b5-5899bcb85d-965k8"] Jul 10 03:17:44.917409 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:17:44.917125 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0212b4b5-5899bcb85d-965k8" podUID="0adde78f-695f-409d-912b-443f08db780f" containerName="maas-api" containerID="cri-o://ff719a469917efe9707e4e903a772900cbe88188e5f6a77bdab3aff02574fd44" gracePeriod=30 Jul 10 03:17:45.173728 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:17:45.173667 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0212b4b5-5899bcb85d-965k8" Jul 10 03:17:45.251334 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:17:45.251304 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0adde78f-695f-409d-912b-443f08db780f-openshift-service-ca\") pod \"0adde78f-695f-409d-912b-443f08db780f\" (UID: \"0adde78f-695f-409d-912b-443f08db780f\") " Jul 10 03:17:45.251468 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:17:45.251382 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0adde78f-695f-409d-912b-443f08db780f-maas-api-tls\") pod \"0adde78f-695f-409d-912b-443f08db780f\" (UID: \"0adde78f-695f-409d-912b-443f08db780f\") " Jul 10 03:17:45.251468 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:17:45.251405 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rln22\" (UniqueName: \"kubernetes.io/projected/0adde78f-695f-409d-912b-443f08db780f-kube-api-access-rln22\") pod \"0adde78f-695f-409d-912b-443f08db780f\" (UID: \"0adde78f-695f-409d-912b-443f08db780f\") " Jul 10 03:17:45.251715 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:17:45.251689 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0adde78f-695f-409d-912b-443f08db780f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0adde78f-695f-409d-912b-443f08db780f" (UID: "0adde78f-695f-409d-912b-443f08db780f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 03:17:45.253466 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:17:45.253435 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0adde78f-695f-409d-912b-443f08db780f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0adde78f-695f-409d-912b-443f08db780f" (UID: "0adde78f-695f-409d-912b-443f08db780f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 03:17:45.253579 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:17:45.253497 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0adde78f-695f-409d-912b-443f08db780f-kube-api-access-rln22" (OuterVolumeSpecName: "kube-api-access-rln22") pod "0adde78f-695f-409d-912b-443f08db780f" (UID: "0adde78f-695f-409d-912b-443f08db780f"). InnerVolumeSpecName "kube-api-access-rln22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 03:17:45.352794 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:17:45.352761 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0adde78f-695f-409d-912b-443f08db780f-openshift-service-ca\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 03:17:45.352794 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:17:45.352789 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0adde78f-695f-409d-912b-443f08db780f-maas-api-tls\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 03:17:45.352794 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:17:45.352798 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rln22\" (UniqueName: \"kubernetes.io/projected/0adde78f-695f-409d-912b-443f08db780f-kube-api-access-rln22\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 03:17:45.475877 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:17:45.475782 2575 generic.go:356] "Generic (PLEG): container finished" podID="0adde78f-695f-409d-912b-443f08db780f" containerID="ff719a469917efe9707e4e903a772900cbe88188e5f6a77bdab3aff02574fd44" exitCode=0 Jul 10 03:17:45.475877 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:17:45.475854 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0212b4b5-5899bcb85d-965k8" Jul 10 03:17:45.476061 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:17:45.475870 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0212b4b5-5899bcb85d-965k8" event={"ID":"0adde78f-695f-409d-912b-443f08db780f","Type":"ContainerDied","Data":"ff719a469917efe9707e4e903a772900cbe88188e5f6a77bdab3aff02574fd44"} Jul 10 03:17:45.476061 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:17:45.475928 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0212b4b5-5899bcb85d-965k8" event={"ID":"0adde78f-695f-409d-912b-443f08db780f","Type":"ContainerDied","Data":"ce62dfea6525b5015f83a4590f3a09909c3f4953e04b1073e0771dd68b4f655d"} Jul 10 03:17:45.476061 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:17:45.475950 2575 scope.go:117] "RemoveContainer" containerID="ff719a469917efe9707e4e903a772900cbe88188e5f6a77bdab3aff02574fd44" Jul 10 03:17:45.488242 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:17:45.488212 2575 scope.go:117] "RemoveContainer" containerID="ff719a469917efe9707e4e903a772900cbe88188e5f6a77bdab3aff02574fd44" Jul 10 03:17:45.488941 ip-10-0-130-139 kubenswrapper[2575]: E0710 03:17:45.488920 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff719a469917efe9707e4e903a772900cbe88188e5f6a77bdab3aff02574fd44\": container with ID starting with ff719a469917efe9707e4e903a772900cbe88188e5f6a77bdab3aff02574fd44 not found: ID does not exist" containerID="ff719a469917efe9707e4e903a772900cbe88188e5f6a77bdab3aff02574fd44" Jul 10 03:17:45.489030 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:17:45.488953 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff719a469917efe9707e4e903a772900cbe88188e5f6a77bdab3aff02574fd44"} err="failed to get container status \"ff719a469917efe9707e4e903a772900cbe88188e5f6a77bdab3aff02574fd44\": rpc error: code = NotFound desc = could not find container \"ff719a469917efe9707e4e903a772900cbe88188e5f6a77bdab3aff02574fd44\": container with ID starting with ff719a469917efe9707e4e903a772900cbe88188e5f6a77bdab3aff02574fd44 not found: ID does not exist" Jul 10 03:17:45.497620 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:17:45.497595 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0212b4b5-5899bcb85d-965k8"] Jul 10 03:17:45.499735 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:17:45.499712 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0212b4b5-5899bcb85d-965k8"] Jul 10 03:17:46.289241 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:17:46.289212 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0adde78f-695f-409d-912b-443f08db780f" path="/var/lib/kubelet/pods/0adde78f-695f-409d-912b-443f08db780f/volumes" Jul 10 03:18:02.358903 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:18:02.358874 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-xxlp6_372be3ce-a380-4c0e-bbee-49c2e3b80095/console-operator/1.log" Jul 10 03:18:02.361743 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:18:02.361719 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-xxlp6_372be3ce-a380-4c0e-bbee-49c2e3b80095/console-operator/1.log" Jul 10 03:19:35.297308 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:19:35.297228 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-58a4b7-6f47c7746f-n84b9"] Jul 10 03:19:35.297866 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:19:35.297732 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0adde78f-695f-409d-912b-443f08db780f" containerName="maas-api" Jul 10 03:19:35.297866 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:19:35.297759 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0adde78f-695f-409d-912b-443f08db780f" containerName="maas-api" Jul 10 03:19:35.298023 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:19:35.297897 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0adde78f-695f-409d-912b-443f08db780f" containerName="maas-api" Jul 10 03:19:35.301055 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:19:35.301038 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-58a4b7-6f47c7746f-n84b9" Jul 10 03:19:35.303319 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:19:35.303300 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-58a4b7-serving-cert\"" Jul 10 03:19:35.310103 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:19:35.310084 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-58a4b7-6f47c7746f-n84b9"] Jul 10 03:19:35.424333 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:19:35.424296 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb6ce01c-abd2-4ee1-ac81-6b3e525ab9e0-openshift-service-ca\") pod \"maas-api-e2e-inf-a-58a4b7-6f47c7746f-n84b9\" (UID: \"fb6ce01c-abd2-4ee1-ac81-6b3e525ab9e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-58a4b7-6f47c7746f-n84b9" Jul 10 03:19:35.424533 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:19:35.424340 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb6ce01c-abd2-4ee1-ac81-6b3e525ab9e0-maas-api-tls\") pod \"maas-api-e2e-inf-a-58a4b7-6f47c7746f-n84b9\" (UID: \"fb6ce01c-abd2-4ee1-ac81-6b3e525ab9e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-58a4b7-6f47c7746f-n84b9" Jul 10 03:19:35.424606 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:19:35.424546 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drz2g\" (UniqueName: \"kubernetes.io/projected/fb6ce01c-abd2-4ee1-ac81-6b3e525ab9e0-kube-api-access-drz2g\") pod \"maas-api-e2e-inf-a-58a4b7-6f47c7746f-n84b9\" (UID: \"fb6ce01c-abd2-4ee1-ac81-6b3e525ab9e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-58a4b7-6f47c7746f-n84b9" Jul 10 03:19:35.525736 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:19:35.525696 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb6ce01c-abd2-4ee1-ac81-6b3e525ab9e0-openshift-service-ca\") pod \"maas-api-e2e-inf-a-58a4b7-6f47c7746f-n84b9\" (UID: \"fb6ce01c-abd2-4ee1-ac81-6b3e525ab9e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-58a4b7-6f47c7746f-n84b9" Jul 10 03:19:35.525736 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:19:35.525738 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb6ce01c-abd2-4ee1-ac81-6b3e525ab9e0-maas-api-tls\") pod \"maas-api-e2e-inf-a-58a4b7-6f47c7746f-n84b9\" (UID: \"fb6ce01c-abd2-4ee1-ac81-6b3e525ab9e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-58a4b7-6f47c7746f-n84b9" Jul 10 03:19:35.526005 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:19:35.525839 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drz2g\" (UniqueName: \"kubernetes.io/projected/fb6ce01c-abd2-4ee1-ac81-6b3e525ab9e0-kube-api-access-drz2g\") pod \"maas-api-e2e-inf-a-58a4b7-6f47c7746f-n84b9\" (UID: \"fb6ce01c-abd2-4ee1-ac81-6b3e525ab9e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-58a4b7-6f47c7746f-n84b9" Jul 10 03:19:35.526005 ip-10-0-130-139 kubenswrapper[2575]: E0710 03:19:35.525924 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-58a4b7-serving-cert: secret "maas-api-e2e-inf-a-58a4b7-serving-cert" not found Jul 10 03:19:35.526005 ip-10-0-130-139 kubenswrapper[2575]: E0710 03:19:35.526004 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb6ce01c-abd2-4ee1-ac81-6b3e525ab9e0-maas-api-tls podName:fb6ce01c-abd2-4ee1-ac81-6b3e525ab9e0 nodeName:}" failed. No retries permitted until 2026-07-10 03:19:36.025985524 +0000 UTC m=+1894.307101790 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fb6ce01c-abd2-4ee1-ac81-6b3e525ab9e0-maas-api-tls") pod "maas-api-e2e-inf-a-58a4b7-6f47c7746f-n84b9" (UID: "fb6ce01c-abd2-4ee1-ac81-6b3e525ab9e0") : secret "maas-api-e2e-inf-a-58a4b7-serving-cert" not found Jul 10 03:19:35.526377 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:19:35.526359 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb6ce01c-abd2-4ee1-ac81-6b3e525ab9e0-openshift-service-ca\") pod \"maas-api-e2e-inf-a-58a4b7-6f47c7746f-n84b9\" (UID: \"fb6ce01c-abd2-4ee1-ac81-6b3e525ab9e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-58a4b7-6f47c7746f-n84b9" Jul 10 03:19:35.540588 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:19:35.540565 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drz2g\" (UniqueName: \"kubernetes.io/projected/fb6ce01c-abd2-4ee1-ac81-6b3e525ab9e0-kube-api-access-drz2g\") pod \"maas-api-e2e-inf-a-58a4b7-6f47c7746f-n84b9\" (UID: \"fb6ce01c-abd2-4ee1-ac81-6b3e525ab9e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-58a4b7-6f47c7746f-n84b9" Jul 10 03:19:36.031220 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:19:36.031184 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb6ce01c-abd2-4ee1-ac81-6b3e525ab9e0-maas-api-tls\") pod \"maas-api-e2e-inf-a-58a4b7-6f47c7746f-n84b9\" (UID: \"fb6ce01c-abd2-4ee1-ac81-6b3e525ab9e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-58a4b7-6f47c7746f-n84b9" Jul 10 03:19:36.033669 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:19:36.033641 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb6ce01c-abd2-4ee1-ac81-6b3e525ab9e0-maas-api-tls\") pod \"maas-api-e2e-inf-a-58a4b7-6f47c7746f-n84b9\" (UID: \"fb6ce01c-abd2-4ee1-ac81-6b3e525ab9e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-58a4b7-6f47c7746f-n84b9" Jul 10 03:19:36.212160 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:19:36.212124 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-58a4b7-6f47c7746f-n84b9" Jul 10 03:19:36.365858 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:19:36.365832 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-58a4b7-6f47c7746f-n84b9"] Jul 10 03:19:36.368561 ip-10-0-130-139 kubenswrapper[2575]: W0710 03:19:36.368528 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb6ce01c_abd2_4ee1_ac81_6b3e525ab9e0.slice/crio-3a2ff5da5bdda704e7194fd137274855772af832dd2cc131748d5043a6de4008 WatchSource:0}: Error finding container 3a2ff5da5bdda704e7194fd137274855772af832dd2cc131748d5043a6de4008: Status 404 returned error can't find the container with id 3a2ff5da5bdda704e7194fd137274855772af832dd2cc131748d5043a6de4008 Jul 10 03:19:36.370075 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:19:36.370055 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 03:19:37.062091 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:19:37.062054 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-58a4b7-6f47c7746f-n84b9" event={"ID":"fb6ce01c-abd2-4ee1-ac81-6b3e525ab9e0","Type":"ContainerStarted","Data":"3efbdb51fd5bb55a95d6c37a150fbb6e8a52e8e18aa6c08fca9b3ca2850830e4"} Jul 10 03:19:37.062277 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:19:37.062118 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-58a4b7-6f47c7746f-n84b9" event={"ID":"fb6ce01c-abd2-4ee1-ac81-6b3e525ab9e0","Type":"ContainerStarted","Data":"3a2ff5da5bdda704e7194fd137274855772af832dd2cc131748d5043a6de4008"} Jul 10 03:19:37.062277 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:19:37.062148 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-58a4b7-6f47c7746f-n84b9" Jul 10 03:19:37.078551 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:19:37.078498 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-58a4b7-6f47c7746f-n84b9" podStartSLOduration=1.537844281 podStartE2EDuration="2.078480672s" totalImagesPullingTime="540.636391ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 03:19:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 03:19:37.075765133 +0000 UTC m=+1895.356881414" watchObservedRunningTime="2026-07-10 03:19:37.078480672 +0000 UTC m=+1895.359596957" Jul 10 03:19:43.073726 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:19:43.073697 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-58a4b7-6f47c7746f-n84b9" Jul 10 03:19:43.344622 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:19:43.344537 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5597fe-75fd7dc6c6-b8l2s"] Jul 10 03:19:43.349194 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:19:43.349168 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5597fe-75fd7dc6c6-b8l2s" Jul 10 03:19:43.352579 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:19:43.352545 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-5597fe-serving-cert\"" Jul 10 03:19:43.364244 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:19:43.364217 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5597fe-75fd7dc6c6-b8l2s"] Jul 10 03:19:43.500089 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:19:43.500056 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2762ccc-37ad-4303-9a12-cfd658a65bc9-maas-api-tls\") pod \"maas-api-e2e-inf-b-5597fe-75fd7dc6c6-b8l2s\" (UID: \"e2762ccc-37ad-4303-9a12-cfd658a65bc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5597fe-75fd7dc6c6-b8l2s" Jul 10 03:19:43.500268 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:19:43.500116 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qrvq\" (UniqueName: \"kubernetes.io/projected/e2762ccc-37ad-4303-9a12-cfd658a65bc9-kube-api-access-6qrvq\") pod \"maas-api-e2e-inf-b-5597fe-75fd7dc6c6-b8l2s\" (UID: \"e2762ccc-37ad-4303-9a12-cfd658a65bc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5597fe-75fd7dc6c6-b8l2s" Jul 10 03:19:43.500268 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:19:43.500176 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2762ccc-37ad-4303-9a12-cfd658a65bc9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5597fe-75fd7dc6c6-b8l2s\" (UID: \"e2762ccc-37ad-4303-9a12-cfd658a65bc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5597fe-75fd7dc6c6-b8l2s" Jul 10 03:19:43.601295 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:19:43.601203 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2762ccc-37ad-4303-9a12-cfd658a65bc9-maas-api-tls\") pod \"maas-api-e2e-inf-b-5597fe-75fd7dc6c6-b8l2s\" (UID: \"e2762ccc-37ad-4303-9a12-cfd658a65bc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5597fe-75fd7dc6c6-b8l2s" Jul 10 03:19:43.601295 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:19:43.601272 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qrvq\" (UniqueName: \"kubernetes.io/projected/e2762ccc-37ad-4303-9a12-cfd658a65bc9-kube-api-access-6qrvq\") pod \"maas-api-e2e-inf-b-5597fe-75fd7dc6c6-b8l2s\" (UID: \"e2762ccc-37ad-4303-9a12-cfd658a65bc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5597fe-75fd7dc6c6-b8l2s" Jul 10 03:19:43.601295 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:19:43.601298 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2762ccc-37ad-4303-9a12-cfd658a65bc9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5597fe-75fd7dc6c6-b8l2s\" (UID: \"e2762ccc-37ad-4303-9a12-cfd658a65bc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5597fe-75fd7dc6c6-b8l2s" Jul 10 03:19:43.601978 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:19:43.601960 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2762ccc-37ad-4303-9a12-cfd658a65bc9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5597fe-75fd7dc6c6-b8l2s\" (UID: \"e2762ccc-37ad-4303-9a12-cfd658a65bc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5597fe-75fd7dc6c6-b8l2s" Jul 10 03:19:43.603844 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:19:43.603793 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2762ccc-37ad-4303-9a12-cfd658a65bc9-maas-api-tls\") pod \"maas-api-e2e-inf-b-5597fe-75fd7dc6c6-b8l2s\" (UID: \"e2762ccc-37ad-4303-9a12-cfd658a65bc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5597fe-75fd7dc6c6-b8l2s" Jul 10 03:19:43.613263 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:19:43.613233 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qrvq\" (UniqueName: \"kubernetes.io/projected/e2762ccc-37ad-4303-9a12-cfd658a65bc9-kube-api-access-6qrvq\") pod \"maas-api-e2e-inf-b-5597fe-75fd7dc6c6-b8l2s\" (UID: \"e2762ccc-37ad-4303-9a12-cfd658a65bc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5597fe-75fd7dc6c6-b8l2s" Jul 10 03:19:43.663284 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:19:43.663250 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5597fe-75fd7dc6c6-b8l2s" Jul 10 03:19:43.808606 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:19:43.808553 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5597fe-75fd7dc6c6-b8l2s"] Jul 10 03:19:43.810691 ip-10-0-130-139 kubenswrapper[2575]: W0710 03:19:43.810639 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2762ccc_37ad_4303_9a12_cfd658a65bc9.slice/crio-f10a5bed697f4d1f35c13c21b3d8b76da8b47554bf8d188cc0f2a74348f38243 WatchSource:0}: Error finding container f10a5bed697f4d1f35c13c21b3d8b76da8b47554bf8d188cc0f2a74348f38243: Status 404 returned error can't find the container with id f10a5bed697f4d1f35c13c21b3d8b76da8b47554bf8d188cc0f2a74348f38243 Jul 10 03:19:44.102425 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:19:44.102390 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5597fe-75fd7dc6c6-b8l2s" event={"ID":"e2762ccc-37ad-4303-9a12-cfd658a65bc9","Type":"ContainerStarted","Data":"f10a5bed697f4d1f35c13c21b3d8b76da8b47554bf8d188cc0f2a74348f38243"} Jul 10 03:19:45.109982 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:19:45.109943 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5597fe-75fd7dc6c6-b8l2s" event={"ID":"e2762ccc-37ad-4303-9a12-cfd658a65bc9","Type":"ContainerStarted","Data":"5db810cb1c942728f397c6e57fe785e64cc1383b950ddb252c6f87c86d6013dc"} Jul 10 03:19:45.110435 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:19:45.110039 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5597fe-75fd7dc6c6-b8l2s" Jul 10 03:19:45.125659 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:19:45.125614 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5597fe-75fd7dc6c6-b8l2s" podStartSLOduration=1.816656369 podStartE2EDuration="2.125596858s" totalImagesPullingTime="308.940489ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 03:19:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 03:19:45.124470181 +0000 UTC m=+1903.405586465" watchObservedRunningTime="2026-07-10 03:19:45.125596858 +0000 UTC m=+1903.406713142" Jul 10 03:19:51.121108 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:19:51.121074 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5597fe-75fd7dc6c6-b8l2s" Jul 10 03:20:32.629682 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:20:32.629652 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-58a4b7-6f47c7746f-n84b9"] Jul 10 03:20:32.630249 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:20:32.629923 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-58a4b7-6f47c7746f-n84b9" podUID="fb6ce01c-abd2-4ee1-ac81-6b3e525ab9e0" containerName="maas-api" containerID="cri-o://3efbdb51fd5bb55a95d6c37a150fbb6e8a52e8e18aa6c08fca9b3ca2850830e4" gracePeriod=30 Jul 10 03:20:32.892059 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:20:32.891999 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-58a4b7-6f47c7746f-n84b9" Jul 10 03:20:33.051725 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:20:33.051690 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb6ce01c-abd2-4ee1-ac81-6b3e525ab9e0-openshift-service-ca\") pod \"fb6ce01c-abd2-4ee1-ac81-6b3e525ab9e0\" (UID: \"fb6ce01c-abd2-4ee1-ac81-6b3e525ab9e0\") " Jul 10 03:20:33.051949 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:20:33.051755 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb6ce01c-abd2-4ee1-ac81-6b3e525ab9e0-maas-api-tls\") pod \"fb6ce01c-abd2-4ee1-ac81-6b3e525ab9e0\" (UID: \"fb6ce01c-abd2-4ee1-ac81-6b3e525ab9e0\") " Jul 10 03:20:33.051949 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:20:33.051851 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drz2g\" (UniqueName: \"kubernetes.io/projected/fb6ce01c-abd2-4ee1-ac81-6b3e525ab9e0-kube-api-access-drz2g\") pod \"fb6ce01c-abd2-4ee1-ac81-6b3e525ab9e0\" (UID: \"fb6ce01c-abd2-4ee1-ac81-6b3e525ab9e0\") " Jul 10 03:20:33.052153 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:20:33.052128 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb6ce01c-abd2-4ee1-ac81-6b3e525ab9e0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fb6ce01c-abd2-4ee1-ac81-6b3e525ab9e0" (UID: "fb6ce01c-abd2-4ee1-ac81-6b3e525ab9e0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 03:20:33.053854 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:20:33.053822 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb6ce01c-abd2-4ee1-ac81-6b3e525ab9e0-kube-api-access-drz2g" (OuterVolumeSpecName: "kube-api-access-drz2g") pod "fb6ce01c-abd2-4ee1-ac81-6b3e525ab9e0" (UID: "fb6ce01c-abd2-4ee1-ac81-6b3e525ab9e0"). InnerVolumeSpecName "kube-api-access-drz2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 03:20:33.053969 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:20:33.053914 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb6ce01c-abd2-4ee1-ac81-6b3e525ab9e0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fb6ce01c-abd2-4ee1-ac81-6b3e525ab9e0" (UID: "fb6ce01c-abd2-4ee1-ac81-6b3e525ab9e0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 03:20:33.153648 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:20:33.153568 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb6ce01c-abd2-4ee1-ac81-6b3e525ab9e0-openshift-service-ca\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 03:20:33.153648 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:20:33.153594 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb6ce01c-abd2-4ee1-ac81-6b3e525ab9e0-maas-api-tls\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 03:20:33.153648 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:20:33.153604 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drz2g\" (UniqueName: \"kubernetes.io/projected/fb6ce01c-abd2-4ee1-ac81-6b3e525ab9e0-kube-api-access-drz2g\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 03:20:33.357857 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:20:33.357822 2575 generic.go:356] "Generic (PLEG): container finished" podID="fb6ce01c-abd2-4ee1-ac81-6b3e525ab9e0" containerID="3efbdb51fd5bb55a95d6c37a150fbb6e8a52e8e18aa6c08fca9b3ca2850830e4" exitCode=0 Jul 10 03:20:33.358038 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:20:33.357868 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-58a4b7-6f47c7746f-n84b9" Jul 10 03:20:33.358038 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:20:33.357910 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-58a4b7-6f47c7746f-n84b9" event={"ID":"fb6ce01c-abd2-4ee1-ac81-6b3e525ab9e0","Type":"ContainerDied","Data":"3efbdb51fd5bb55a95d6c37a150fbb6e8a52e8e18aa6c08fca9b3ca2850830e4"} Jul 10 03:20:33.358038 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:20:33.357968 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-58a4b7-6f47c7746f-n84b9" event={"ID":"fb6ce01c-abd2-4ee1-ac81-6b3e525ab9e0","Type":"ContainerDied","Data":"3a2ff5da5bdda704e7194fd137274855772af832dd2cc131748d5043a6de4008"} Jul 10 03:20:33.358038 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:20:33.357994 2575 scope.go:117] "RemoveContainer" containerID="3efbdb51fd5bb55a95d6c37a150fbb6e8a52e8e18aa6c08fca9b3ca2850830e4" Jul 10 03:20:33.367929 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:20:33.367908 2575 scope.go:117] "RemoveContainer" containerID="3efbdb51fd5bb55a95d6c37a150fbb6e8a52e8e18aa6c08fca9b3ca2850830e4" Jul 10 03:20:33.368187 ip-10-0-130-139 kubenswrapper[2575]: E0710 03:20:33.368168 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3efbdb51fd5bb55a95d6c37a150fbb6e8a52e8e18aa6c08fca9b3ca2850830e4\": container with ID starting with 3efbdb51fd5bb55a95d6c37a150fbb6e8a52e8e18aa6c08fca9b3ca2850830e4 not found: ID does not exist" containerID="3efbdb51fd5bb55a95d6c37a150fbb6e8a52e8e18aa6c08fca9b3ca2850830e4" Jul 10 03:20:33.368237 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:20:33.368196 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3efbdb51fd5bb55a95d6c37a150fbb6e8a52e8e18aa6c08fca9b3ca2850830e4"} err="failed to get container status \"3efbdb51fd5bb55a95d6c37a150fbb6e8a52e8e18aa6c08fca9b3ca2850830e4\": rpc error: code = NotFound desc = could not find container \"3efbdb51fd5bb55a95d6c37a150fbb6e8a52e8e18aa6c08fca9b3ca2850830e4\": container with ID starting with 3efbdb51fd5bb55a95d6c37a150fbb6e8a52e8e18aa6c08fca9b3ca2850830e4 not found: ID does not exist" Jul 10 03:20:33.378734 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:20:33.378708 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-58a4b7-6f47c7746f-n84b9"] Jul 10 03:20:33.385081 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:20:33.385062 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-58a4b7-6f47c7746f-n84b9"] Jul 10 03:20:34.290660 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:20:34.290624 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb6ce01c-abd2-4ee1-ac81-6b3e525ab9e0" path="/var/lib/kubelet/pods/fb6ce01c-abd2-4ee1-ac81-6b3e525ab9e0/volumes" Jul 10 03:21:15.612542 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:15.612502 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5597fe-75fd7dc6c6-b8l2s"] Jul 10 03:21:15.613162 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:15.612748 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5597fe-75fd7dc6c6-b8l2s" podUID="e2762ccc-37ad-4303-9a12-cfd658a65bc9" containerName="maas-api" containerID="cri-o://5db810cb1c942728f397c6e57fe785e64cc1383b950ddb252c6f87c86d6013dc" gracePeriod=30 Jul 10 03:21:15.876408 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:15.876345 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5597fe-75fd7dc6c6-b8l2s" Jul 10 03:21:16.057868 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:16.057835 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2762ccc-37ad-4303-9a12-cfd658a65bc9-maas-api-tls\") pod \"e2762ccc-37ad-4303-9a12-cfd658a65bc9\" (UID: \"e2762ccc-37ad-4303-9a12-cfd658a65bc9\") " Jul 10 03:21:16.057868 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:16.057868 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2762ccc-37ad-4303-9a12-cfd658a65bc9-openshift-service-ca\") pod \"e2762ccc-37ad-4303-9a12-cfd658a65bc9\" (UID: \"e2762ccc-37ad-4303-9a12-cfd658a65bc9\") " Jul 10 03:21:16.058086 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:16.057896 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6qrvq\" (UniqueName: \"kubernetes.io/projected/e2762ccc-37ad-4303-9a12-cfd658a65bc9-kube-api-access-6qrvq\") pod \"e2762ccc-37ad-4303-9a12-cfd658a65bc9\" (UID: \"e2762ccc-37ad-4303-9a12-cfd658a65bc9\") " Jul 10 03:21:16.058335 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:16.058304 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2762ccc-37ad-4303-9a12-cfd658a65bc9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e2762ccc-37ad-4303-9a12-cfd658a65bc9" (UID: "e2762ccc-37ad-4303-9a12-cfd658a65bc9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 03:21:16.060095 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:16.060069 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2762ccc-37ad-4303-9a12-cfd658a65bc9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e2762ccc-37ad-4303-9a12-cfd658a65bc9" (UID: "e2762ccc-37ad-4303-9a12-cfd658a65bc9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 03:21:16.060095 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:16.060079 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2762ccc-37ad-4303-9a12-cfd658a65bc9-kube-api-access-6qrvq" (OuterVolumeSpecName: "kube-api-access-6qrvq") pod "e2762ccc-37ad-4303-9a12-cfd658a65bc9" (UID: "e2762ccc-37ad-4303-9a12-cfd658a65bc9"). InnerVolumeSpecName "kube-api-access-6qrvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 03:21:16.159600 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:16.159518 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2762ccc-37ad-4303-9a12-cfd658a65bc9-maas-api-tls\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 03:21:16.159600 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:16.159545 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2762ccc-37ad-4303-9a12-cfd658a65bc9-openshift-service-ca\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 03:21:16.159600 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:16.159556 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qrvq\" (UniqueName: \"kubernetes.io/projected/e2762ccc-37ad-4303-9a12-cfd658a65bc9-kube-api-access-6qrvq\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 03:21:16.577304 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:16.577265 2575 generic.go:356] "Generic (PLEG): container finished" podID="e2762ccc-37ad-4303-9a12-cfd658a65bc9" containerID="5db810cb1c942728f397c6e57fe785e64cc1383b950ddb252c6f87c86d6013dc" exitCode=0 Jul 10 03:21:16.577480 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:16.577308 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5597fe-75fd7dc6c6-b8l2s" Jul 10 03:21:16.577480 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:16.577355 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5597fe-75fd7dc6c6-b8l2s" event={"ID":"e2762ccc-37ad-4303-9a12-cfd658a65bc9","Type":"ContainerDied","Data":"5db810cb1c942728f397c6e57fe785e64cc1383b950ddb252c6f87c86d6013dc"} Jul 10 03:21:16.577480 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:16.577402 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5597fe-75fd7dc6c6-b8l2s" event={"ID":"e2762ccc-37ad-4303-9a12-cfd658a65bc9","Type":"ContainerDied","Data":"f10a5bed697f4d1f35c13c21b3d8b76da8b47554bf8d188cc0f2a74348f38243"} Jul 10 03:21:16.577480 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:16.577425 2575 scope.go:117] "RemoveContainer" containerID="5db810cb1c942728f397c6e57fe785e64cc1383b950ddb252c6f87c86d6013dc" Jul 10 03:21:16.586237 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:16.586218 2575 scope.go:117] "RemoveContainer" containerID="5db810cb1c942728f397c6e57fe785e64cc1383b950ddb252c6f87c86d6013dc" Jul 10 03:21:16.586577 ip-10-0-130-139 kubenswrapper[2575]: E0710 03:21:16.586549 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5db810cb1c942728f397c6e57fe785e64cc1383b950ddb252c6f87c86d6013dc\": container with ID starting with 5db810cb1c942728f397c6e57fe785e64cc1383b950ddb252c6f87c86d6013dc not found: ID does not exist" containerID="5db810cb1c942728f397c6e57fe785e64cc1383b950ddb252c6f87c86d6013dc" Jul 10 03:21:16.586678 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:16.586588 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5db810cb1c942728f397c6e57fe785e64cc1383b950ddb252c6f87c86d6013dc"} err="failed to get container status \"5db810cb1c942728f397c6e57fe785e64cc1383b950ddb252c6f87c86d6013dc\": rpc error: code = NotFound desc = could not find container \"5db810cb1c942728f397c6e57fe785e64cc1383b950ddb252c6f87c86d6013dc\": container with ID starting with 5db810cb1c942728f397c6e57fe785e64cc1383b950ddb252c6f87c86d6013dc not found: ID does not exist" Jul 10 03:21:16.600345 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:16.600319 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5597fe-75fd7dc6c6-b8l2s"] Jul 10 03:21:16.602373 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:16.602348 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5597fe-75fd7dc6c6-b8l2s"] Jul 10 03:21:18.289196 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:18.289157 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2762ccc-37ad-4303-9a12-cfd658a65bc9" path="/var/lib/kubelet/pods/e2762ccc-37ad-4303-9a12-cfd658a65bc9/volumes" Jul 10 03:21:24.839789 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:24.839757 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-54ccca-646bf47478-kt57f"] Jul 10 03:21:24.840257 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:24.840212 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb6ce01c-abd2-4ee1-ac81-6b3e525ab9e0" containerName="maas-api" Jul 10 03:21:24.840257 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:24.840229 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb6ce01c-abd2-4ee1-ac81-6b3e525ab9e0" containerName="maas-api" Jul 10 03:21:24.840257 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:24.840240 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2762ccc-37ad-4303-9a12-cfd658a65bc9" containerName="maas-api" Jul 10 03:21:24.840257 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:24.840246 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2762ccc-37ad-4303-9a12-cfd658a65bc9" containerName="maas-api" Jul 10 03:21:24.840477 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:24.840319 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2762ccc-37ad-4303-9a12-cfd658a65bc9" containerName="maas-api" Jul 10 03:21:24.840477 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:24.840329 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb6ce01c-abd2-4ee1-ac81-6b3e525ab9e0" containerName="maas-api" Jul 10 03:21:24.844857 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:24.844829 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54ccca-646bf47478-kt57f" Jul 10 03:21:24.847229 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:24.847207 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-54ccca-serving-cert\"" Jul 10 03:21:24.853819 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:24.853783 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-54ccca-646bf47478-kt57f"] Jul 10 03:21:24.927421 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:24.927384 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6dc862d-c455-4818-8590-ddfabcb8548d-openshift-service-ca\") pod \"maas-api-e2e-api-a-54ccca-646bf47478-kt57f\" (UID: \"a6dc862d-c455-4818-8590-ddfabcb8548d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54ccca-646bf47478-kt57f" Jul 10 03:21:24.927421 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:24.927427 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6dc862d-c455-4818-8590-ddfabcb8548d-maas-api-tls\") pod \"maas-api-e2e-api-a-54ccca-646bf47478-kt57f\" (UID: \"a6dc862d-c455-4818-8590-ddfabcb8548d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54ccca-646bf47478-kt57f" Jul 10 03:21:24.927673 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:24.927526 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8c2c\" (UniqueName: \"kubernetes.io/projected/a6dc862d-c455-4818-8590-ddfabcb8548d-kube-api-access-q8c2c\") pod \"maas-api-e2e-api-a-54ccca-646bf47478-kt57f\" (UID: \"a6dc862d-c455-4818-8590-ddfabcb8548d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54ccca-646bf47478-kt57f" Jul 10 03:21:25.028070 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:25.028031 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6dc862d-c455-4818-8590-ddfabcb8548d-openshift-service-ca\") pod \"maas-api-e2e-api-a-54ccca-646bf47478-kt57f\" (UID: \"a6dc862d-c455-4818-8590-ddfabcb8548d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54ccca-646bf47478-kt57f" Jul 10 03:21:25.028266 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:25.028084 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6dc862d-c455-4818-8590-ddfabcb8548d-maas-api-tls\") pod \"maas-api-e2e-api-a-54ccca-646bf47478-kt57f\" (UID: \"a6dc862d-c455-4818-8590-ddfabcb8548d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54ccca-646bf47478-kt57f" Jul 10 03:21:25.028266 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:25.028153 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8c2c\" (UniqueName: \"kubernetes.io/projected/a6dc862d-c455-4818-8590-ddfabcb8548d-kube-api-access-q8c2c\") pod \"maas-api-e2e-api-a-54ccca-646bf47478-kt57f\" (UID: \"a6dc862d-c455-4818-8590-ddfabcb8548d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54ccca-646bf47478-kt57f" Jul 10 03:21:25.028384 ip-10-0-130-139 kubenswrapper[2575]: E0710 03:21:25.028336 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-54ccca-serving-cert: secret "maas-api-e2e-api-a-54ccca-serving-cert" not found Jul 10 03:21:25.028536 ip-10-0-130-139 kubenswrapper[2575]: E0710 03:21:25.028504 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6dc862d-c455-4818-8590-ddfabcb8548d-maas-api-tls podName:a6dc862d-c455-4818-8590-ddfabcb8548d nodeName:}" failed. No retries permitted until 2026-07-10 03:21:25.528463066 +0000 UTC m=+2003.809579331 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a6dc862d-c455-4818-8590-ddfabcb8548d-maas-api-tls") pod "maas-api-e2e-api-a-54ccca-646bf47478-kt57f" (UID: "a6dc862d-c455-4818-8590-ddfabcb8548d") : secret "maas-api-e2e-api-a-54ccca-serving-cert" not found Jul 10 03:21:25.028636 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:25.028617 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6dc862d-c455-4818-8590-ddfabcb8548d-openshift-service-ca\") pod \"maas-api-e2e-api-a-54ccca-646bf47478-kt57f\" (UID: \"a6dc862d-c455-4818-8590-ddfabcb8548d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54ccca-646bf47478-kt57f" Jul 10 03:21:25.038418 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:25.038392 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8c2c\" (UniqueName: \"kubernetes.io/projected/a6dc862d-c455-4818-8590-ddfabcb8548d-kube-api-access-q8c2c\") pod \"maas-api-e2e-api-a-54ccca-646bf47478-kt57f\" (UID: \"a6dc862d-c455-4818-8590-ddfabcb8548d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54ccca-646bf47478-kt57f" Jul 10 03:21:25.531898 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:25.531860 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6dc862d-c455-4818-8590-ddfabcb8548d-maas-api-tls\") pod \"maas-api-e2e-api-a-54ccca-646bf47478-kt57f\" (UID: \"a6dc862d-c455-4818-8590-ddfabcb8548d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54ccca-646bf47478-kt57f" Jul 10 03:21:25.532065 ip-10-0-130-139 kubenswrapper[2575]: E0710 03:21:25.532011 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-54ccca-serving-cert: secret "maas-api-e2e-api-a-54ccca-serving-cert" not found Jul 10 03:21:25.532109 ip-10-0-130-139 kubenswrapper[2575]: E0710 03:21:25.532079 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6dc862d-c455-4818-8590-ddfabcb8548d-maas-api-tls podName:a6dc862d-c455-4818-8590-ddfabcb8548d nodeName:}" failed. No retries permitted until 2026-07-10 03:21:26.532063588 +0000 UTC m=+2004.813179859 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a6dc862d-c455-4818-8590-ddfabcb8548d-maas-api-tls") pod "maas-api-e2e-api-a-54ccca-646bf47478-kt57f" (UID: "a6dc862d-c455-4818-8590-ddfabcb8548d") : secret "maas-api-e2e-api-a-54ccca-serving-cert" not found Jul 10 03:21:26.540162 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:26.540130 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6dc862d-c455-4818-8590-ddfabcb8548d-maas-api-tls\") pod \"maas-api-e2e-api-a-54ccca-646bf47478-kt57f\" (UID: \"a6dc862d-c455-4818-8590-ddfabcb8548d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54ccca-646bf47478-kt57f" Jul 10 03:21:26.542529 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:26.542504 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6dc862d-c455-4818-8590-ddfabcb8548d-maas-api-tls\") pod \"maas-api-e2e-api-a-54ccca-646bf47478-kt57f\" (UID: \"a6dc862d-c455-4818-8590-ddfabcb8548d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54ccca-646bf47478-kt57f" Jul 10 03:21:26.657991 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:26.657958 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54ccca-646bf47478-kt57f" Jul 10 03:21:26.795138 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:26.795082 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-54ccca-646bf47478-kt57f"] Jul 10 03:21:26.797570 ip-10-0-130-139 kubenswrapper[2575]: W0710 03:21:26.797540 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6dc862d_c455_4818_8590_ddfabcb8548d.slice/crio-56651bc18af288c2dcc22f1761201b4c673d5a53d1986ca2816ac6429e8eeb8e WatchSource:0}: Error finding container 56651bc18af288c2dcc22f1761201b4c673d5a53d1986ca2816ac6429e8eeb8e: Status 404 returned error can't find the container with id 56651bc18af288c2dcc22f1761201b4c673d5a53d1986ca2816ac6429e8eeb8e Jul 10 03:21:27.633153 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:27.633059 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54ccca-646bf47478-kt57f" event={"ID":"a6dc862d-c455-4818-8590-ddfabcb8548d","Type":"ContainerStarted","Data":"684755db753ba45f38aaad8eecf014e9493f4e1cc497795ce0508a4efd9d5ec6"} Jul 10 03:21:27.633153 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:27.633117 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54ccca-646bf47478-kt57f" event={"ID":"a6dc862d-c455-4818-8590-ddfabcb8548d","Type":"ContainerStarted","Data":"56651bc18af288c2dcc22f1761201b4c673d5a53d1986ca2816ac6429e8eeb8e"} Jul 10 03:21:27.633153 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:27.633139 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54ccca-646bf47478-kt57f" Jul 10 03:21:27.647689 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:27.647642 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54ccca-646bf47478-kt57f" podStartSLOduration=3.188597986 podStartE2EDuration="3.647627297s" totalImagesPullingTime="459.029311ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 03:21:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 03:21:27.645781074 +0000 UTC m=+2005.926897384" watchObservedRunningTime="2026-07-10 03:21:27.647627297 +0000 UTC m=+2005.928743582" Jul 10 03:21:32.886354 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:32.879057 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8504b1-59b48c795b-n85hr"] Jul 10 03:21:32.897663 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:32.897628 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8504b1-59b48c795b-n85hr"] Jul 10 03:21:32.897823 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:32.897741 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8504b1-59b48c795b-n85hr" Jul 10 03:21:32.900037 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:32.900014 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-8504b1-serving-cert\"" Jul 10 03:21:32.994126 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:32.994093 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54f6d4ee-e5c1-4acf-8fb5-654668f8191e-maas-api-tls\") pod \"maas-api-e2e-api-b-8504b1-59b48c795b-n85hr\" (UID: \"54f6d4ee-e5c1-4acf-8fb5-654668f8191e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8504b1-59b48c795b-n85hr" Jul 10 03:21:32.994126 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:32.994131 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54f6d4ee-e5c1-4acf-8fb5-654668f8191e-openshift-service-ca\") pod \"maas-api-e2e-api-b-8504b1-59b48c795b-n85hr\" (UID: \"54f6d4ee-e5c1-4acf-8fb5-654668f8191e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8504b1-59b48c795b-n85hr" Jul 10 03:21:32.994364 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:32.994167 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfg6z\" (UniqueName: \"kubernetes.io/projected/54f6d4ee-e5c1-4acf-8fb5-654668f8191e-kube-api-access-rfg6z\") pod \"maas-api-e2e-api-b-8504b1-59b48c795b-n85hr\" (UID: \"54f6d4ee-e5c1-4acf-8fb5-654668f8191e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8504b1-59b48c795b-n85hr" Jul 10 03:21:33.095173 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:33.095138 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54f6d4ee-e5c1-4acf-8fb5-654668f8191e-maas-api-tls\") pod \"maas-api-e2e-api-b-8504b1-59b48c795b-n85hr\" (UID: \"54f6d4ee-e5c1-4acf-8fb5-654668f8191e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8504b1-59b48c795b-n85hr" Jul 10 03:21:33.095173 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:33.095176 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54f6d4ee-e5c1-4acf-8fb5-654668f8191e-openshift-service-ca\") pod \"maas-api-e2e-api-b-8504b1-59b48c795b-n85hr\" (UID: \"54f6d4ee-e5c1-4acf-8fb5-654668f8191e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8504b1-59b48c795b-n85hr" Jul 10 03:21:33.095437 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:33.095203 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfg6z\" (UniqueName: \"kubernetes.io/projected/54f6d4ee-e5c1-4acf-8fb5-654668f8191e-kube-api-access-rfg6z\") pod \"maas-api-e2e-api-b-8504b1-59b48c795b-n85hr\" (UID: \"54f6d4ee-e5c1-4acf-8fb5-654668f8191e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8504b1-59b48c795b-n85hr" Jul 10 03:21:33.095437 ip-10-0-130-139 kubenswrapper[2575]: E0710 03:21:33.095310 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-8504b1-serving-cert: secret "maas-api-e2e-api-b-8504b1-serving-cert" not found Jul 10 03:21:33.095437 ip-10-0-130-139 kubenswrapper[2575]: E0710 03:21:33.095403 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54f6d4ee-e5c1-4acf-8fb5-654668f8191e-maas-api-tls podName:54f6d4ee-e5c1-4acf-8fb5-654668f8191e nodeName:}" failed. No retries permitted until 2026-07-10 03:21:33.595379172 +0000 UTC m=+2011.876495447 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/54f6d4ee-e5c1-4acf-8fb5-654668f8191e-maas-api-tls") pod "maas-api-e2e-api-b-8504b1-59b48c795b-n85hr" (UID: "54f6d4ee-e5c1-4acf-8fb5-654668f8191e") : secret "maas-api-e2e-api-b-8504b1-serving-cert" not found Jul 10 03:21:33.095904 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:33.095885 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54f6d4ee-e5c1-4acf-8fb5-654668f8191e-openshift-service-ca\") pod \"maas-api-e2e-api-b-8504b1-59b48c795b-n85hr\" (UID: \"54f6d4ee-e5c1-4acf-8fb5-654668f8191e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8504b1-59b48c795b-n85hr" Jul 10 03:21:33.106317 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:33.106291 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfg6z\" (UniqueName: \"kubernetes.io/projected/54f6d4ee-e5c1-4acf-8fb5-654668f8191e-kube-api-access-rfg6z\") pod \"maas-api-e2e-api-b-8504b1-59b48c795b-n85hr\" (UID: \"54f6d4ee-e5c1-4acf-8fb5-654668f8191e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8504b1-59b48c795b-n85hr" Jul 10 03:21:33.600467 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:33.600435 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54f6d4ee-e5c1-4acf-8fb5-654668f8191e-maas-api-tls\") pod \"maas-api-e2e-api-b-8504b1-59b48c795b-n85hr\" (UID: \"54f6d4ee-e5c1-4acf-8fb5-654668f8191e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8504b1-59b48c795b-n85hr" Jul 10 03:21:33.603063 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:33.603042 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54f6d4ee-e5c1-4acf-8fb5-654668f8191e-maas-api-tls\") pod \"maas-api-e2e-api-b-8504b1-59b48c795b-n85hr\" (UID: \"54f6d4ee-e5c1-4acf-8fb5-654668f8191e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8504b1-59b48c795b-n85hr" Jul 10 03:21:33.645479 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:33.645441 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54ccca-646bf47478-kt57f" Jul 10 03:21:33.811485 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:33.811452 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8504b1-59b48c795b-n85hr" Jul 10 03:21:33.950983 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:33.950946 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8504b1-59b48c795b-n85hr"] Jul 10 03:21:33.954750 ip-10-0-130-139 kubenswrapper[2575]: W0710 03:21:33.954718 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54f6d4ee_e5c1_4acf_8fb5_654668f8191e.slice/crio-a6bb1a4ec3e0b20f5cae3f5c556eb78c0ae61e3b82151913cb53b4085def7fb5 WatchSource:0}: Error finding container a6bb1a4ec3e0b20f5cae3f5c556eb78c0ae61e3b82151913cb53b4085def7fb5: Status 404 returned error can't find the container with id a6bb1a4ec3e0b20f5cae3f5c556eb78c0ae61e3b82151913cb53b4085def7fb5 Jul 10 03:21:34.676442 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:34.676358 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8504b1-59b48c795b-n85hr" event={"ID":"54f6d4ee-e5c1-4acf-8fb5-654668f8191e","Type":"ContainerStarted","Data":"002e42d0823f316c2dea5c860a3ded8b5a97708f2a240b13c746335f56358c1e"} Jul 10 03:21:34.676442 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:34.676423 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8504b1-59b48c795b-n85hr" event={"ID":"54f6d4ee-e5c1-4acf-8fb5-654668f8191e","Type":"ContainerStarted","Data":"a6bb1a4ec3e0b20f5cae3f5c556eb78c0ae61e3b82151913cb53b4085def7fb5"} Jul 10 03:21:34.676646 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:34.676579 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8504b1-59b48c795b-n85hr" Jul 10 03:21:34.691397 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:34.691345 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8504b1-59b48c795b-n85hr" podStartSLOduration=2.348194348 podStartE2EDuration="2.691332452s" totalImagesPullingTime="343.138104ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 03:21:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 03:21:34.690903044 +0000 UTC m=+2012.972019329" watchObservedRunningTime="2026-07-10 03:21:34.691332452 +0000 UTC m=+2012.972448743" Jul 10 03:21:40.688110 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:40.688079 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8504b1-59b48c795b-n85hr" Jul 10 03:21:46.069937 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:46.069898 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-54ccca-646bf47478-kt57f"] Jul 10 03:21:46.070425 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:46.070140 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54ccca-646bf47478-kt57f" podUID="a6dc862d-c455-4818-8590-ddfabcb8548d" containerName="maas-api" containerID="cri-o://684755db753ba45f38aaad8eecf014e9493f4e1cc497795ce0508a4efd9d5ec6" gracePeriod=30 Jul 10 03:21:46.318983 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:46.318958 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54ccca-646bf47478-kt57f" Jul 10 03:21:46.422868 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:46.422761 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6dc862d-c455-4818-8590-ddfabcb8548d-maas-api-tls\") pod \"a6dc862d-c455-4818-8590-ddfabcb8548d\" (UID: \"a6dc862d-c455-4818-8590-ddfabcb8548d\") " Jul 10 03:21:46.422868 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:46.422799 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8c2c\" (UniqueName: \"kubernetes.io/projected/a6dc862d-c455-4818-8590-ddfabcb8548d-kube-api-access-q8c2c\") pod \"a6dc862d-c455-4818-8590-ddfabcb8548d\" (UID: \"a6dc862d-c455-4818-8590-ddfabcb8548d\") " Jul 10 03:21:46.423056 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:46.422877 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6dc862d-c455-4818-8590-ddfabcb8548d-openshift-service-ca\") pod \"a6dc862d-c455-4818-8590-ddfabcb8548d\" (UID: \"a6dc862d-c455-4818-8590-ddfabcb8548d\") " Jul 10 03:21:46.423301 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:46.423276 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6dc862d-c455-4818-8590-ddfabcb8548d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a6dc862d-c455-4818-8590-ddfabcb8548d" (UID: "a6dc862d-c455-4818-8590-ddfabcb8548d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 03:21:46.424958 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:46.424937 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6dc862d-c455-4818-8590-ddfabcb8548d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a6dc862d-c455-4818-8590-ddfabcb8548d" (UID: "a6dc862d-c455-4818-8590-ddfabcb8548d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 03:21:46.425064 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:46.424998 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6dc862d-c455-4818-8590-ddfabcb8548d-kube-api-access-q8c2c" (OuterVolumeSpecName: "kube-api-access-q8c2c") pod "a6dc862d-c455-4818-8590-ddfabcb8548d" (UID: "a6dc862d-c455-4818-8590-ddfabcb8548d"). InnerVolumeSpecName "kube-api-access-q8c2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 03:21:46.524094 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:46.524062 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6dc862d-c455-4818-8590-ddfabcb8548d-maas-api-tls\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 03:21:46.524094 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:46.524089 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8c2c\" (UniqueName: \"kubernetes.io/projected/a6dc862d-c455-4818-8590-ddfabcb8548d-kube-api-access-q8c2c\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 03:21:46.524094 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:46.524098 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6dc862d-c455-4818-8590-ddfabcb8548d-openshift-service-ca\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 03:21:46.739264 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:46.739175 2575 generic.go:356] "Generic (PLEG): container finished" podID="a6dc862d-c455-4818-8590-ddfabcb8548d" containerID="684755db753ba45f38aaad8eecf014e9493f4e1cc497795ce0508a4efd9d5ec6" exitCode=0 Jul 10 03:21:46.739264 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:46.739214 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54ccca-646bf47478-kt57f" Jul 10 03:21:46.739495 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:46.739256 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54ccca-646bf47478-kt57f" event={"ID":"a6dc862d-c455-4818-8590-ddfabcb8548d","Type":"ContainerDied","Data":"684755db753ba45f38aaad8eecf014e9493f4e1cc497795ce0508a4efd9d5ec6"} Jul 10 03:21:46.739495 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:46.739313 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-54ccca-646bf47478-kt57f" event={"ID":"a6dc862d-c455-4818-8590-ddfabcb8548d","Type":"ContainerDied","Data":"56651bc18af288c2dcc22f1761201b4c673d5a53d1986ca2816ac6429e8eeb8e"} Jul 10 03:21:46.739495 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:46.739345 2575 scope.go:117] "RemoveContainer" containerID="684755db753ba45f38aaad8eecf014e9493f4e1cc497795ce0508a4efd9d5ec6" Jul 10 03:21:46.752407 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:46.752388 2575 scope.go:117] "RemoveContainer" containerID="684755db753ba45f38aaad8eecf014e9493f4e1cc497795ce0508a4efd9d5ec6" Jul 10 03:21:46.752672 ip-10-0-130-139 kubenswrapper[2575]: E0710 03:21:46.752656 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"684755db753ba45f38aaad8eecf014e9493f4e1cc497795ce0508a4efd9d5ec6\": container with ID starting with 684755db753ba45f38aaad8eecf014e9493f4e1cc497795ce0508a4efd9d5ec6 not found: ID does not exist" containerID="684755db753ba45f38aaad8eecf014e9493f4e1cc497795ce0508a4efd9d5ec6" Jul 10 03:21:46.752750 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:46.752681 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"684755db753ba45f38aaad8eecf014e9493f4e1cc497795ce0508a4efd9d5ec6"} err="failed to get container status \"684755db753ba45f38aaad8eecf014e9493f4e1cc497795ce0508a4efd9d5ec6\": rpc error: code = NotFound desc = could not find container \"684755db753ba45f38aaad8eecf014e9493f4e1cc497795ce0508a4efd9d5ec6\": container with ID starting with 684755db753ba45f38aaad8eecf014e9493f4e1cc497795ce0508a4efd9d5ec6 not found: ID does not exist" Jul 10 03:21:46.762687 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:46.762655 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-54ccca-646bf47478-kt57f"] Jul 10 03:21:46.767953 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:46.767927 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-54ccca-646bf47478-kt57f"] Jul 10 03:21:48.289096 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:48.289061 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6dc862d-c455-4818-8590-ddfabcb8548d" path="/var/lib/kubelet/pods/a6dc862d-c455-4818-8590-ddfabcb8548d/volumes" Jul 10 03:21:53.056132 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:53.056101 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8504b1-59b48c795b-n85hr"] Jul 10 03:21:53.056633 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:53.056309 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8504b1-59b48c795b-n85hr" podUID="54f6d4ee-e5c1-4acf-8fb5-654668f8191e" containerName="maas-api" containerID="cri-o://002e42d0823f316c2dea5c860a3ded8b5a97708f2a240b13c746335f56358c1e" gracePeriod=30 Jul 10 03:21:53.328882 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:53.328858 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8504b1-59b48c795b-n85hr" Jul 10 03:21:53.385074 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:53.385043 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54f6d4ee-e5c1-4acf-8fb5-654668f8191e-maas-api-tls\") pod \"54f6d4ee-e5c1-4acf-8fb5-654668f8191e\" (UID: \"54f6d4ee-e5c1-4acf-8fb5-654668f8191e\") " Jul 10 03:21:53.385253 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:53.385153 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54f6d4ee-e5c1-4acf-8fb5-654668f8191e-openshift-service-ca\") pod \"54f6d4ee-e5c1-4acf-8fb5-654668f8191e\" (UID: \"54f6d4ee-e5c1-4acf-8fb5-654668f8191e\") " Jul 10 03:21:53.385253 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:53.385211 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rfg6z\" (UniqueName: \"kubernetes.io/projected/54f6d4ee-e5c1-4acf-8fb5-654668f8191e-kube-api-access-rfg6z\") pod \"54f6d4ee-e5c1-4acf-8fb5-654668f8191e\" (UID: \"54f6d4ee-e5c1-4acf-8fb5-654668f8191e\") " Jul 10 03:21:53.385534 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:53.385513 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54f6d4ee-e5c1-4acf-8fb5-654668f8191e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "54f6d4ee-e5c1-4acf-8fb5-654668f8191e" (UID: "54f6d4ee-e5c1-4acf-8fb5-654668f8191e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 03:21:53.387244 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:53.387226 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54f6d4ee-e5c1-4acf-8fb5-654668f8191e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "54f6d4ee-e5c1-4acf-8fb5-654668f8191e" (UID: "54f6d4ee-e5c1-4acf-8fb5-654668f8191e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 03:21:53.387338 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:53.387288 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54f6d4ee-e5c1-4acf-8fb5-654668f8191e-kube-api-access-rfg6z" (OuterVolumeSpecName: "kube-api-access-rfg6z") pod "54f6d4ee-e5c1-4acf-8fb5-654668f8191e" (UID: "54f6d4ee-e5c1-4acf-8fb5-654668f8191e"). InnerVolumeSpecName "kube-api-access-rfg6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 03:21:53.486123 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:53.486089 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54f6d4ee-e5c1-4acf-8fb5-654668f8191e-openshift-service-ca\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 03:21:53.486123 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:53.486118 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfg6z\" (UniqueName: \"kubernetes.io/projected/54f6d4ee-e5c1-4acf-8fb5-654668f8191e-kube-api-access-rfg6z\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 03:21:53.486123 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:53.486131 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54f6d4ee-e5c1-4acf-8fb5-654668f8191e-maas-api-tls\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 03:21:53.777241 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:53.777210 2575 generic.go:356] "Generic (PLEG): container finished" podID="54f6d4ee-e5c1-4acf-8fb5-654668f8191e" containerID="002e42d0823f316c2dea5c860a3ded8b5a97708f2a240b13c746335f56358c1e" exitCode=0 Jul 10 03:21:53.777435 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:53.777255 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8504b1-59b48c795b-n85hr" Jul 10 03:21:53.777435 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:53.777297 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8504b1-59b48c795b-n85hr" event={"ID":"54f6d4ee-e5c1-4acf-8fb5-654668f8191e","Type":"ContainerDied","Data":"002e42d0823f316c2dea5c860a3ded8b5a97708f2a240b13c746335f56358c1e"} Jul 10 03:21:53.777435 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:53.777351 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8504b1-59b48c795b-n85hr" event={"ID":"54f6d4ee-e5c1-4acf-8fb5-654668f8191e","Type":"ContainerDied","Data":"a6bb1a4ec3e0b20f5cae3f5c556eb78c0ae61e3b82151913cb53b4085def7fb5"} Jul 10 03:21:53.777435 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:53.777380 2575 scope.go:117] "RemoveContainer" containerID="002e42d0823f316c2dea5c860a3ded8b5a97708f2a240b13c746335f56358c1e" Jul 10 03:21:53.787303 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:53.787289 2575 scope.go:117] "RemoveContainer" containerID="002e42d0823f316c2dea5c860a3ded8b5a97708f2a240b13c746335f56358c1e" Jul 10 03:21:53.787519 ip-10-0-130-139 kubenswrapper[2575]: E0710 03:21:53.787504 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"002e42d0823f316c2dea5c860a3ded8b5a97708f2a240b13c746335f56358c1e\": container with ID starting with 002e42d0823f316c2dea5c860a3ded8b5a97708f2a240b13c746335f56358c1e not found: ID does not exist" containerID="002e42d0823f316c2dea5c860a3ded8b5a97708f2a240b13c746335f56358c1e" Jul 10 03:21:53.787574 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:53.787527 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"002e42d0823f316c2dea5c860a3ded8b5a97708f2a240b13c746335f56358c1e"} err="failed to get container status \"002e42d0823f316c2dea5c860a3ded8b5a97708f2a240b13c746335f56358c1e\": rpc error: code = NotFound desc = could not find container \"002e42d0823f316c2dea5c860a3ded8b5a97708f2a240b13c746335f56358c1e\": container with ID starting with 002e42d0823f316c2dea5c860a3ded8b5a97708f2a240b13c746335f56358c1e not found: ID does not exist" Jul 10 03:21:53.800057 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:53.800031 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8504b1-59b48c795b-n85hr"] Jul 10 03:21:53.803901 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:53.803880 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8504b1-59b48c795b-n85hr"] Jul 10 03:21:54.289618 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:21:54.289586 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54f6d4ee-e5c1-4acf-8fb5-654668f8191e" path="/var/lib/kubelet/pods/54f6d4ee-e5c1-4acf-8fb5-654668f8191e/volumes" Jul 10 03:22:03.058747 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:22:03.058712 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-f42020-66d5d4b868-hz9vq"] Jul 10 03:22:03.059298 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:22:03.059148 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6dc862d-c455-4818-8590-ddfabcb8548d" containerName="maas-api" Jul 10 03:22:03.059298 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:22:03.059165 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6dc862d-c455-4818-8590-ddfabcb8548d" containerName="maas-api" Jul 10 03:22:03.059298 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:22:03.059188 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54f6d4ee-e5c1-4acf-8fb5-654668f8191e" containerName="maas-api" Jul 10 03:22:03.059298 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:22:03.059194 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="54f6d4ee-e5c1-4acf-8fb5-654668f8191e" containerName="maas-api" Jul 10 03:22:03.059298 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:22:03.059296 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="54f6d4ee-e5c1-4acf-8fb5-654668f8191e" containerName="maas-api" Jul 10 03:22:03.059577 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:22:03.059310 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6dc862d-c455-4818-8590-ddfabcb8548d" containerName="maas-api" Jul 10 03:22:03.064595 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:22:03.064564 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f42020-66d5d4b868-hz9vq" Jul 10 03:22:03.066860 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:22:03.066832 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-f42020-serving-cert\"" Jul 10 03:22:03.071786 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:22:03.071751 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-f42020-66d5d4b868-hz9vq"] Jul 10 03:22:03.179243 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:22:03.179192 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/008c1c15-3410-420a-894c-5c37b33c3273-openshift-service-ca\") pod \"maas-api-e2e-shared-a-f42020-66d5d4b868-hz9vq\" (UID: \"008c1c15-3410-420a-894c-5c37b33c3273\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f42020-66d5d4b868-hz9vq" Jul 10 03:22:03.179243 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:22:03.179239 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86zpc\" (UniqueName: \"kubernetes.io/projected/008c1c15-3410-420a-894c-5c37b33c3273-kube-api-access-86zpc\") pod \"maas-api-e2e-shared-a-f42020-66d5d4b868-hz9vq\" (UID: \"008c1c15-3410-420a-894c-5c37b33c3273\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f42020-66d5d4b868-hz9vq" Jul 10 03:22:03.179510 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:22:03.179305 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/008c1c15-3410-420a-894c-5c37b33c3273-maas-api-tls\") pod \"maas-api-e2e-shared-a-f42020-66d5d4b868-hz9vq\" (UID: \"008c1c15-3410-420a-894c-5c37b33c3273\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f42020-66d5d4b868-hz9vq" Jul 10 03:22:03.280089 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:22:03.280041 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/008c1c15-3410-420a-894c-5c37b33c3273-openshift-service-ca\") pod \"maas-api-e2e-shared-a-f42020-66d5d4b868-hz9vq\" (UID: \"008c1c15-3410-420a-894c-5c37b33c3273\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f42020-66d5d4b868-hz9vq" Jul 10 03:22:03.280306 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:22:03.280099 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86zpc\" (UniqueName: \"kubernetes.io/projected/008c1c15-3410-420a-894c-5c37b33c3273-kube-api-access-86zpc\") pod \"maas-api-e2e-shared-a-f42020-66d5d4b868-hz9vq\" (UID: \"008c1c15-3410-420a-894c-5c37b33c3273\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f42020-66d5d4b868-hz9vq" Jul 10 03:22:03.280306 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:22:03.280131 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/008c1c15-3410-420a-894c-5c37b33c3273-maas-api-tls\") pod \"maas-api-e2e-shared-a-f42020-66d5d4b868-hz9vq\" (UID: \"008c1c15-3410-420a-894c-5c37b33c3273\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f42020-66d5d4b868-hz9vq" Jul 10 03:22:03.280306 ip-10-0-130-139 kubenswrapper[2575]: E0710 03:22:03.280283 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-f42020-serving-cert: secret "maas-api-e2e-shared-a-f42020-serving-cert" not found Jul 10 03:22:03.280475 ip-10-0-130-139 kubenswrapper[2575]: E0710 03:22:03.280360 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/008c1c15-3410-420a-894c-5c37b33c3273-maas-api-tls podName:008c1c15-3410-420a-894c-5c37b33c3273 nodeName:}" failed. No retries permitted until 2026-07-10 03:22:03.78033835 +0000 UTC m=+2042.061454617 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/008c1c15-3410-420a-894c-5c37b33c3273-maas-api-tls") pod "maas-api-e2e-shared-a-f42020-66d5d4b868-hz9vq" (UID: "008c1c15-3410-420a-894c-5c37b33c3273") : secret "maas-api-e2e-shared-a-f42020-serving-cert" not found Jul 10 03:22:03.280770 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:22:03.280750 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/008c1c15-3410-420a-894c-5c37b33c3273-openshift-service-ca\") pod \"maas-api-e2e-shared-a-f42020-66d5d4b868-hz9vq\" (UID: \"008c1c15-3410-420a-894c-5c37b33c3273\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f42020-66d5d4b868-hz9vq" Jul 10 03:22:03.289291 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:22:03.289263 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86zpc\" (UniqueName: \"kubernetes.io/projected/008c1c15-3410-420a-894c-5c37b33c3273-kube-api-access-86zpc\") pod \"maas-api-e2e-shared-a-f42020-66d5d4b868-hz9vq\" (UID: \"008c1c15-3410-420a-894c-5c37b33c3273\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f42020-66d5d4b868-hz9vq" Jul 10 03:22:03.785066 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:22:03.785034 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/008c1c15-3410-420a-894c-5c37b33c3273-maas-api-tls\") pod \"maas-api-e2e-shared-a-f42020-66d5d4b868-hz9vq\" (UID: \"008c1c15-3410-420a-894c-5c37b33c3273\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f42020-66d5d4b868-hz9vq" Jul 10 03:22:03.787620 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:22:03.787591 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/008c1c15-3410-420a-894c-5c37b33c3273-maas-api-tls\") pod \"maas-api-e2e-shared-a-f42020-66d5d4b868-hz9vq\" (UID: \"008c1c15-3410-420a-894c-5c37b33c3273\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f42020-66d5d4b868-hz9vq" Jul 10 03:22:03.980061 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:22:03.980024 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f42020-66d5d4b868-hz9vq" Jul 10 03:22:04.127845 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:22:04.127802 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-f42020-66d5d4b868-hz9vq"] Jul 10 03:22:04.130630 ip-10-0-130-139 kubenswrapper[2575]: W0710 03:22:04.130600 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod008c1c15_3410_420a_894c_5c37b33c3273.slice/crio-80243a860c81c19e061504963dcc2fbb56e87dc79c2c464fcf17c97f5e28d065 WatchSource:0}: Error finding container 80243a860c81c19e061504963dcc2fbb56e87dc79c2c464fcf17c97f5e28d065: Status 404 returned error can't find the container with id 80243a860c81c19e061504963dcc2fbb56e87dc79c2c464fcf17c97f5e28d065 Jul 10 03:22:04.835267 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:22:04.835229 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f42020-66d5d4b868-hz9vq" event={"ID":"008c1c15-3410-420a-894c-5c37b33c3273","Type":"ContainerStarted","Data":"7ff81a5ee7f905b9f221c556062aec9ca93120f1b77d957dcc9853e5b8709d44"} Jul 10 03:22:04.835489 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:22:04.835288 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f42020-66d5d4b868-hz9vq" event={"ID":"008c1c15-3410-420a-894c-5c37b33c3273","Type":"ContainerStarted","Data":"80243a860c81c19e061504963dcc2fbb56e87dc79c2c464fcf17c97f5e28d065"} Jul 10 03:22:04.835489 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:22:04.835344 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f42020-66d5d4b868-hz9vq" Jul 10 03:22:04.850855 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:22:04.850788 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f42020-66d5d4b868-hz9vq" podStartSLOduration=1.515311708 podStartE2EDuration="1.850773431s" totalImagesPullingTime="335.461723ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 03:22:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 03:22:04.848650518 +0000 UTC m=+2043.129766804" watchObservedRunningTime="2026-07-10 03:22:04.850773431 +0000 UTC m=+2043.131889715" Jul 10 03:22:10.845632 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:22:10.845597 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f42020-66d5d4b868-hz9vq" Jul 10 03:22:11.059279 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:22:11.059234 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-45a5a5-5bd65885d-w672q"] Jul 10 03:22:11.066233 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:22:11.066204 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-45a5a5-5bd65885d-w672q" Jul 10 03:22:11.068161 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:22:11.068143 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-45a5a5-serving-cert\"" Jul 10 03:22:11.072929 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:22:11.072901 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-45a5a5-5bd65885d-w672q"] Jul 10 03:22:11.257075 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:22:11.257041 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d659ebd2-bb90-4d2b-9676-0a14f9627e7b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-45a5a5-5bd65885d-w672q\" (UID: \"d659ebd2-bb90-4d2b-9676-0a14f9627e7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-45a5a5-5bd65885d-w672q" Jul 10 03:22:11.257252 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:22:11.257144 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d659ebd2-bb90-4d2b-9676-0a14f9627e7b-maas-api-tls\") pod \"maas-api-e2e-shared-b-45a5a5-5bd65885d-w672q\" (UID: \"d659ebd2-bb90-4d2b-9676-0a14f9627e7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-45a5a5-5bd65885d-w672q" Jul 10 03:22:11.257252 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:22:11.257195 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8bhl\" (UniqueName: \"kubernetes.io/projected/d659ebd2-bb90-4d2b-9676-0a14f9627e7b-kube-api-access-s8bhl\") pod \"maas-api-e2e-shared-b-45a5a5-5bd65885d-w672q\" (UID: \"d659ebd2-bb90-4d2b-9676-0a14f9627e7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-45a5a5-5bd65885d-w672q" Jul 10 03:22:11.358135 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:22:11.358102 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d659ebd2-bb90-4d2b-9676-0a14f9627e7b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-45a5a5-5bd65885d-w672q\" (UID: \"d659ebd2-bb90-4d2b-9676-0a14f9627e7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-45a5a5-5bd65885d-w672q" Jul 10 03:22:11.358309 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:22:11.358178 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d659ebd2-bb90-4d2b-9676-0a14f9627e7b-maas-api-tls\") pod \"maas-api-e2e-shared-b-45a5a5-5bd65885d-w672q\" (UID: \"d659ebd2-bb90-4d2b-9676-0a14f9627e7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-45a5a5-5bd65885d-w672q" Jul 10 03:22:11.358309 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:22:11.358214 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8bhl\" (UniqueName: \"kubernetes.io/projected/d659ebd2-bb90-4d2b-9676-0a14f9627e7b-kube-api-access-s8bhl\") pod \"maas-api-e2e-shared-b-45a5a5-5bd65885d-w672q\" (UID: \"d659ebd2-bb90-4d2b-9676-0a14f9627e7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-45a5a5-5bd65885d-w672q" Jul 10 03:22:11.358403 ip-10-0-130-139 kubenswrapper[2575]: E0710 03:22:11.358336 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-45a5a5-serving-cert: secret "maas-api-e2e-shared-b-45a5a5-serving-cert" not found Jul 10 03:22:11.358441 ip-10-0-130-139 kubenswrapper[2575]: E0710 03:22:11.358419 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d659ebd2-bb90-4d2b-9676-0a14f9627e7b-maas-api-tls podName:d659ebd2-bb90-4d2b-9676-0a14f9627e7b nodeName:}" failed. No retries permitted until 2026-07-10 03:22:11.858394904 +0000 UTC m=+2050.139511180 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d659ebd2-bb90-4d2b-9676-0a14f9627e7b-maas-api-tls") pod "maas-api-e2e-shared-b-45a5a5-5bd65885d-w672q" (UID: "d659ebd2-bb90-4d2b-9676-0a14f9627e7b") : secret "maas-api-e2e-shared-b-45a5a5-serving-cert" not found Jul 10 03:22:11.358853 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:22:11.358835 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d659ebd2-bb90-4d2b-9676-0a14f9627e7b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-45a5a5-5bd65885d-w672q\" (UID: \"d659ebd2-bb90-4d2b-9676-0a14f9627e7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-45a5a5-5bd65885d-w672q" Jul 10 03:22:11.368353 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:22:11.368328 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8bhl\" (UniqueName: \"kubernetes.io/projected/d659ebd2-bb90-4d2b-9676-0a14f9627e7b-kube-api-access-s8bhl\") pod \"maas-api-e2e-shared-b-45a5a5-5bd65885d-w672q\" (UID: \"d659ebd2-bb90-4d2b-9676-0a14f9627e7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-45a5a5-5bd65885d-w672q" Jul 10 03:22:11.863211 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:22:11.863164 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d659ebd2-bb90-4d2b-9676-0a14f9627e7b-maas-api-tls\") pod \"maas-api-e2e-shared-b-45a5a5-5bd65885d-w672q\" (UID: \"d659ebd2-bb90-4d2b-9676-0a14f9627e7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-45a5a5-5bd65885d-w672q" Jul 10 03:22:11.866130 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:22:11.866103 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d659ebd2-bb90-4d2b-9676-0a14f9627e7b-maas-api-tls\") pod \"maas-api-e2e-shared-b-45a5a5-5bd65885d-w672q\" (UID: \"d659ebd2-bb90-4d2b-9676-0a14f9627e7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-45a5a5-5bd65885d-w672q" Jul 10 03:22:11.980261 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:22:11.980224 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-45a5a5-5bd65885d-w672q" Jul 10 03:22:12.124396 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:22:12.124369 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-45a5a5-5bd65885d-w672q"] Jul 10 03:22:12.126354 ip-10-0-130-139 kubenswrapper[2575]: W0710 03:22:12.126324 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd659ebd2_bb90_4d2b_9676_0a14f9627e7b.slice/crio-f652037108e3388d42eb6941137b59fb9bcddbd70deae19fffe6647ff5275268 WatchSource:0}: Error finding container f652037108e3388d42eb6941137b59fb9bcddbd70deae19fffe6647ff5275268: Status 404 returned error can't find the container with id f652037108e3388d42eb6941137b59fb9bcddbd70deae19fffe6647ff5275268 Jul 10 03:22:12.877340 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:22:12.877304 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-45a5a5-5bd65885d-w672q" event={"ID":"d659ebd2-bb90-4d2b-9676-0a14f9627e7b","Type":"ContainerStarted","Data":"267ef2cd97a5774d5adba82e6c0324e2f888cfcc2ce3ba5fdcb454c088313de1"} Jul 10 03:22:12.877756 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:22:12.877391 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-45a5a5-5bd65885d-w672q" event={"ID":"d659ebd2-bb90-4d2b-9676-0a14f9627e7b","Type":"ContainerStarted","Data":"f652037108e3388d42eb6941137b59fb9bcddbd70deae19fffe6647ff5275268"} Jul 10 03:22:12.877756 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:22:12.877416 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-45a5a5-5bd65885d-w672q" Jul 10 03:22:12.909028 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:22:12.908976 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-45a5a5-5bd65885d-w672q" podStartSLOduration=1.5649437910000001 podStartE2EDuration="1.908960389s" totalImagesPullingTime="344.016598ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 03:22:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 03:22:12.907297108 +0000 UTC m=+2051.188413392" watchObservedRunningTime="2026-07-10 03:22:12.908960389 +0000 UTC m=+2051.190076675" Jul 10 03:22:18.888310 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:22:18.888276 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-45a5a5-5bd65885d-w672q" Jul 10 03:23:02.393511 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:23:02.393475 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-xxlp6_372be3ce-a380-4c0e-bbee-49c2e3b80095/console-operator/1.log" Jul 10 03:23:02.397061 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:23:02.397037 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-xxlp6_372be3ce-a380-4c0e-bbee-49c2e3b80095/console-operator/1.log" Jul 10 03:27:38.258058 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:38.258024 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7146-140688457342256"] Jul 10 03:27:38.261488 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:38.261471 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7146-140688457342256" Jul 10 03:27:38.263720 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:38.263700 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-4tp4q\"" Jul 10 03:27:38.268898 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:38.268867 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7146-140688457342256"] Jul 10 03:27:38.429756 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:38.429724 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7tt5\" (UniqueName: \"kubernetes.io/projected/31035172-4e21-4ca5-99e5-87b88533eccb-kube-api-access-p7tt5\") pod \"test-curl-7146-140688457342256\" (UID: \"31035172-4e21-4ca5-99e5-87b88533eccb\") " pod="openshift-ingress/test-curl-7146-140688457342256" Jul 10 03:27:38.530907 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:38.530827 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7tt5\" (UniqueName: \"kubernetes.io/projected/31035172-4e21-4ca5-99e5-87b88533eccb-kube-api-access-p7tt5\") pod \"test-curl-7146-140688457342256\" (UID: \"31035172-4e21-4ca5-99e5-87b88533eccb\") " pod="openshift-ingress/test-curl-7146-140688457342256" Jul 10 03:27:38.537975 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:38.537952 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7tt5\" (UniqueName: \"kubernetes.io/projected/31035172-4e21-4ca5-99e5-87b88533eccb-kube-api-access-p7tt5\") pod \"test-curl-7146-140688457342256\" (UID: \"31035172-4e21-4ca5-99e5-87b88533eccb\") " pod="openshift-ingress/test-curl-7146-140688457342256" Jul 10 03:27:38.572732 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:38.572709 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7146-140688457342256" Jul 10 03:27:38.706123 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:38.706100 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7146-140688457342256"] Jul 10 03:27:38.708379 ip-10-0-130-139 kubenswrapper[2575]: W0710 03:27:38.708352 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31035172_4e21_4ca5_99e5_87b88533eccb.slice/crio-3707f29f7a17ff3c84022e31ea0aa54700161c527edfe95867d8c00179cd2851 WatchSource:0}: Error finding container 3707f29f7a17ff3c84022e31ea0aa54700161c527edfe95867d8c00179cd2851: Status 404 returned error can't find the container with id 3707f29f7a17ff3c84022e31ea0aa54700161c527edfe95867d8c00179cd2851 Jul 10 03:27:38.709750 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:38.709732 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 03:27:39.573011 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:39.572982 2575 generic.go:356] "Generic (PLEG): container finished" podID="31035172-4e21-4ca5-99e5-87b88533eccb" containerID="625f6221f85a1557af92070236a0d41e70e50ab4327a7b1c477a44a6f8211d9d" exitCode=0 Jul 10 03:27:39.573342 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:39.573044 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7146-140688457342256" event={"ID":"31035172-4e21-4ca5-99e5-87b88533eccb","Type":"ContainerDied","Data":"625f6221f85a1557af92070236a0d41e70e50ab4327a7b1c477a44a6f8211d9d"} Jul 10 03:27:39.573342 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:39.573071 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7146-140688457342256" event={"ID":"31035172-4e21-4ca5-99e5-87b88533eccb","Type":"ContainerStarted","Data":"3707f29f7a17ff3c84022e31ea0aa54700161c527edfe95867d8c00179cd2851"} Jul 10 03:27:40.725549 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:40.725525 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7146-140688457342256" Jul 10 03:27:40.759103 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:40.759081 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7146-140688457342256_31035172-4e21-4ca5-99e5-87b88533eccb/test-curl-7146-140688457342256/0.log" Jul 10 03:27:40.786203 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:40.786177 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7146-140688457342256"] Jul 10 03:27:40.788995 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:40.788972 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7146-140688457342256"] Jul 10 03:27:40.852646 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:40.852575 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p7tt5\" (UniqueName: \"kubernetes.io/projected/31035172-4e21-4ca5-99e5-87b88533eccb-kube-api-access-p7tt5\") pod \"31035172-4e21-4ca5-99e5-87b88533eccb\" (UID: \"31035172-4e21-4ca5-99e5-87b88533eccb\") " Jul 10 03:27:40.854554 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:40.854529 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31035172-4e21-4ca5-99e5-87b88533eccb-kube-api-access-p7tt5" (OuterVolumeSpecName: "kube-api-access-p7tt5") pod "31035172-4e21-4ca5-99e5-87b88533eccb" (UID: "31035172-4e21-4ca5-99e5-87b88533eccb"). InnerVolumeSpecName "kube-api-access-p7tt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 03:27:40.913855 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:40.913825 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7146-140688457340464"] Jul 10 03:27:40.914266 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:40.914253 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31035172-4e21-4ca5-99e5-87b88533eccb" containerName="test-curl-7146-140688457342256" Jul 10 03:27:40.914311 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:40.914267 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="31035172-4e21-4ca5-99e5-87b88533eccb" containerName="test-curl-7146-140688457342256" Jul 10 03:27:40.914353 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:40.914344 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="31035172-4e21-4ca5-99e5-87b88533eccb" containerName="test-curl-7146-140688457342256" Jul 10 03:27:40.917416 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:40.917400 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7146-140688457340464" Jul 10 03:27:40.923960 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:40.923937 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7146-140688457340464"] Jul 10 03:27:40.953661 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:40.953631 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7tt5\" (UniqueName: \"kubernetes.io/projected/31035172-4e21-4ca5-99e5-87b88533eccb-kube-api-access-p7tt5\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 03:27:41.054732 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:41.054693 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjvrm\" (UniqueName: \"kubernetes.io/projected/7a1bdfcc-da18-4cda-90b0-bb01d8860bbe-kube-api-access-mjvrm\") pod \"test-curl-7146-140688457340464\" (UID: \"7a1bdfcc-da18-4cda-90b0-bb01d8860bbe\") " pod="openshift-ingress/test-curl-7146-140688457340464" Jul 10 03:27:41.155967 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:41.155872 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjvrm\" (UniqueName: \"kubernetes.io/projected/7a1bdfcc-da18-4cda-90b0-bb01d8860bbe-kube-api-access-mjvrm\") pod \"test-curl-7146-140688457340464\" (UID: \"7a1bdfcc-da18-4cda-90b0-bb01d8860bbe\") " pod="openshift-ingress/test-curl-7146-140688457340464" Jul 10 03:27:41.164733 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:41.164700 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjvrm\" (UniqueName: \"kubernetes.io/projected/7a1bdfcc-da18-4cda-90b0-bb01d8860bbe-kube-api-access-mjvrm\") pod \"test-curl-7146-140688457340464\" (UID: \"7a1bdfcc-da18-4cda-90b0-bb01d8860bbe\") " pod="openshift-ingress/test-curl-7146-140688457340464" Jul 10 03:27:41.227859 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:41.227782 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7146-140688457340464" Jul 10 03:27:41.368637 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:41.368615 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7146-140688457340464"] Jul 10 03:27:41.370322 ip-10-0-130-139 kubenswrapper[2575]: W0710 03:27:41.370297 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a1bdfcc_da18_4cda_90b0_bb01d8860bbe.slice/crio-a8973625ebaec6c21e18a4e3ce81567f48b6ac563c15ce2d5b7303a981b31b84 WatchSource:0}: Error finding container a8973625ebaec6c21e18a4e3ce81567f48b6ac563c15ce2d5b7303a981b31b84: Status 404 returned error can't find the container with id a8973625ebaec6c21e18a4e3ce81567f48b6ac563c15ce2d5b7303a981b31b84 Jul 10 03:27:41.584426 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:41.584392 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7146-140688457340464" event={"ID":"7a1bdfcc-da18-4cda-90b0-bb01d8860bbe","Type":"ContainerStarted","Data":"a8973625ebaec6c21e18a4e3ce81567f48b6ac563c15ce2d5b7303a981b31b84"} Jul 10 03:27:41.585635 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:41.585616 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3707f29f7a17ff3c84022e31ea0aa54700161c527edfe95867d8c00179cd2851" Jul 10 03:27:41.585770 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:41.585654 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7146-140688457342256" Jul 10 03:27:41.598274 ip-10-0-130-139 kubenswrapper[2575]: E0710 03:27:41.598247 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7146-140688457342256\" is forbidden: User \"system:node:ip-10-0-130-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-130-139.ec2.internal' and this object" podUID="31035172-4e21-4ca5-99e5-87b88533eccb" pod="openshift-ingress/test-curl-7146-140688457342256" Jul 10 03:27:42.287065 ip-10-0-130-139 kubenswrapper[2575]: E0710 03:27:42.287019 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7146-140688457342256\" is forbidden: User \"system:node:ip-10-0-130-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-130-139.ec2.internal' and this object" podUID="31035172-4e21-4ca5-99e5-87b88533eccb" pod="openshift-ingress/test-curl-7146-140688457342256" Jul 10 03:27:42.289377 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:42.289355 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31035172-4e21-4ca5-99e5-87b88533eccb" path="/var/lib/kubelet/pods/31035172-4e21-4ca5-99e5-87b88533eccb/volumes" Jul 10 03:27:42.592404 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:42.592316 2575 generic.go:356] "Generic (PLEG): container finished" podID="7a1bdfcc-da18-4cda-90b0-bb01d8860bbe" containerID="c2e6f27920a52b9bca124bf0fa26e61feb22834dec4f3863be483fc359f0645e" exitCode=0 Jul 10 03:27:42.592404 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:42.592362 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7146-140688457340464" event={"ID":"7a1bdfcc-da18-4cda-90b0-bb01d8860bbe","Type":"ContainerDied","Data":"c2e6f27920a52b9bca124bf0fa26e61feb22834dec4f3863be483fc359f0645e"} Jul 10 03:27:43.744347 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:43.744318 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7146-140688457340464" Jul 10 03:27:43.765329 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:43.765300 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7146-140688457340464_7a1bdfcc-da18-4cda-90b0-bb01d8860bbe/test-curl-7146-140688457340464/0.log" Jul 10 03:27:43.792667 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:43.792629 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7146-140688457340464"] Jul 10 03:27:43.798121 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:43.798083 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7146-140688457340464"] Jul 10 03:27:43.883078 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:43.882990 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mjvrm\" (UniqueName: \"kubernetes.io/projected/7a1bdfcc-da18-4cda-90b0-bb01d8860bbe-kube-api-access-mjvrm\") pod \"7a1bdfcc-da18-4cda-90b0-bb01d8860bbe\" (UID: \"7a1bdfcc-da18-4cda-90b0-bb01d8860bbe\") " Jul 10 03:27:43.885199 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:43.885167 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a1bdfcc-da18-4cda-90b0-bb01d8860bbe-kube-api-access-mjvrm" (OuterVolumeSpecName: "kube-api-access-mjvrm") pod "7a1bdfcc-da18-4cda-90b0-bb01d8860bbe" (UID: "7a1bdfcc-da18-4cda-90b0-bb01d8860bbe"). InnerVolumeSpecName "kube-api-access-mjvrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 03:27:43.965409 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:43.965378 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7146-140688457619328"] Jul 10 03:27:43.965851 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:43.965837 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a1bdfcc-da18-4cda-90b0-bb01d8860bbe" containerName="test-curl-7146-140688457340464" Jul 10 03:27:43.965907 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:43.965852 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a1bdfcc-da18-4cda-90b0-bb01d8860bbe" containerName="test-curl-7146-140688457340464" Jul 10 03:27:43.965943 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:43.965939 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a1bdfcc-da18-4cda-90b0-bb01d8860bbe" containerName="test-curl-7146-140688457340464" Jul 10 03:27:43.968992 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:43.968964 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7146-140688457619328" Jul 10 03:27:43.972734 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:43.972715 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7146-140688457619328"] Jul 10 03:27:43.984150 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:43.984120 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjvrm\" (UniqueName: \"kubernetes.io/projected/7a1bdfcc-da18-4cda-90b0-bb01d8860bbe-kube-api-access-mjvrm\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 03:27:44.085575 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:44.085539 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbtfw\" (UniqueName: \"kubernetes.io/projected/b5a9aee5-872a-4cc6-b048-e023f59d5a9b-kube-api-access-qbtfw\") pod \"test-curl-7146-140688457619328\" (UID: \"b5a9aee5-872a-4cc6-b048-e023f59d5a9b\") " pod="openshift-ingress/test-curl-7146-140688457619328" Jul 10 03:27:44.186671 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:44.186577 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbtfw\" (UniqueName: \"kubernetes.io/projected/b5a9aee5-872a-4cc6-b048-e023f59d5a9b-kube-api-access-qbtfw\") pod \"test-curl-7146-140688457619328\" (UID: \"b5a9aee5-872a-4cc6-b048-e023f59d5a9b\") " pod="openshift-ingress/test-curl-7146-140688457619328" Jul 10 03:27:44.195297 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:44.195267 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbtfw\" (UniqueName: \"kubernetes.io/projected/b5a9aee5-872a-4cc6-b048-e023f59d5a9b-kube-api-access-qbtfw\") pod \"test-curl-7146-140688457619328\" (UID: \"b5a9aee5-872a-4cc6-b048-e023f59d5a9b\") " pod="openshift-ingress/test-curl-7146-140688457619328" Jul 10 03:27:44.278962 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:44.278919 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7146-140688457619328" Jul 10 03:27:44.290164 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:44.290133 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a1bdfcc-da18-4cda-90b0-bb01d8860bbe" path="/var/lib/kubelet/pods/7a1bdfcc-da18-4cda-90b0-bb01d8860bbe/volumes" Jul 10 03:27:44.421694 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:44.421669 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7146-140688457619328"] Jul 10 03:27:44.424172 ip-10-0-130-139 kubenswrapper[2575]: W0710 03:27:44.424145 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb5a9aee5_872a_4cc6_b048_e023f59d5a9b.slice/crio-50b805ab65acc01b3e31825f9621b0f31b2c50271bc1b2bc4a5fa00a36793678 WatchSource:0}: Error finding container 50b805ab65acc01b3e31825f9621b0f31b2c50271bc1b2bc4a5fa00a36793678: Status 404 returned error can't find the container with id 50b805ab65acc01b3e31825f9621b0f31b2c50271bc1b2bc4a5fa00a36793678 Jul 10 03:27:44.603706 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:44.603666 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7146-140688457619328" event={"ID":"b5a9aee5-872a-4cc6-b048-e023f59d5a9b","Type":"ContainerStarted","Data":"50b805ab65acc01b3e31825f9621b0f31b2c50271bc1b2bc4a5fa00a36793678"} Jul 10 03:27:44.604857 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:44.604840 2575 scope.go:117] "RemoveContainer" containerID="c2e6f27920a52b9bca124bf0fa26e61feb22834dec4f3863be483fc359f0645e" Jul 10 03:27:44.604944 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:44.604875 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7146-140688457340464" Jul 10 03:27:45.612109 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:45.612077 2575 generic.go:356] "Generic (PLEG): container finished" podID="b5a9aee5-872a-4cc6-b048-e023f59d5a9b" containerID="b53a29c346138cfce1618847053338c8ecb0920d4448efcdf9a33bc1cbe7a335" exitCode=0 Jul 10 03:27:45.612567 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:45.612161 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7146-140688457619328" event={"ID":"b5a9aee5-872a-4cc6-b048-e023f59d5a9b","Type":"ContainerDied","Data":"b53a29c346138cfce1618847053338c8ecb0920d4448efcdf9a33bc1cbe7a335"} Jul 10 03:27:46.760470 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:46.760445 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7146-140688457619328" Jul 10 03:27:46.781325 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:46.781300 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7146-140688457619328_b5a9aee5-872a-4cc6-b048-e023f59d5a9b/test-curl-7146-140688457619328/0.log" Jul 10 03:27:46.807597 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:46.807565 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7146-140688457619328"] Jul 10 03:27:46.809494 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:46.809472 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7146-140688457619328"] Jul 10 03:27:46.913567 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:46.913491 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qbtfw\" (UniqueName: \"kubernetes.io/projected/b5a9aee5-872a-4cc6-b048-e023f59d5a9b-kube-api-access-qbtfw\") pod \"b5a9aee5-872a-4cc6-b048-e023f59d5a9b\" (UID: \"b5a9aee5-872a-4cc6-b048-e023f59d5a9b\") " Jul 10 03:27:46.915665 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:46.915641 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5a9aee5-872a-4cc6-b048-e023f59d5a9b-kube-api-access-qbtfw" (OuterVolumeSpecName: "kube-api-access-qbtfw") pod "b5a9aee5-872a-4cc6-b048-e023f59d5a9b" (UID: "b5a9aee5-872a-4cc6-b048-e023f59d5a9b"). InnerVolumeSpecName "kube-api-access-qbtfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 03:27:46.933413 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:46.933388 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7146-140688457619328"] Jul 10 03:27:46.933856 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:46.933842 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5a9aee5-872a-4cc6-b048-e023f59d5a9b" containerName="test-curl-7146-140688457619328" Jul 10 03:27:46.933904 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:46.933857 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5a9aee5-872a-4cc6-b048-e023f59d5a9b" containerName="test-curl-7146-140688457619328" Jul 10 03:27:46.933944 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:46.933934 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5a9aee5-872a-4cc6-b048-e023f59d5a9b" containerName="test-curl-7146-140688457619328" Jul 10 03:27:46.936929 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:46.936914 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7146-140688457619328" Jul 10 03:27:46.940095 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:46.940073 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7146-140688457619328"] Jul 10 03:27:47.014783 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:47.014750 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbtfw\" (UniqueName: \"kubernetes.io/projected/b5a9aee5-872a-4cc6-b048-e023f59d5a9b-kube-api-access-qbtfw\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 03:27:47.115262 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:47.115226 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd2ts\" (UniqueName: \"kubernetes.io/projected/5670100b-9e6d-4813-8953-958ea9d305cd-kube-api-access-qd2ts\") pod \"test-curl-7146-140688457619328\" (UID: \"5670100b-9e6d-4813-8953-958ea9d305cd\") " pod="openshift-ingress/test-curl-7146-140688457619328" Jul 10 03:27:47.215873 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:47.215736 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd2ts\" (UniqueName: \"kubernetes.io/projected/5670100b-9e6d-4813-8953-958ea9d305cd-kube-api-access-qd2ts\") pod \"test-curl-7146-140688457619328\" (UID: \"5670100b-9e6d-4813-8953-958ea9d305cd\") " pod="openshift-ingress/test-curl-7146-140688457619328" Jul 10 03:27:47.224177 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:47.224154 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd2ts\" (UniqueName: \"kubernetes.io/projected/5670100b-9e6d-4813-8953-958ea9d305cd-kube-api-access-qd2ts\") pod \"test-curl-7146-140688457619328\" (UID: \"5670100b-9e6d-4813-8953-958ea9d305cd\") " pod="openshift-ingress/test-curl-7146-140688457619328" Jul 10 03:27:47.247167 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:47.247133 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7146-140688457619328" Jul 10 03:27:47.384997 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:47.384748 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7146-140688457619328"] Jul 10 03:27:47.387197 ip-10-0-130-139 kubenswrapper[2575]: W0710 03:27:47.387172 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5670100b_9e6d_4813_8953_958ea9d305cd.slice/crio-c7684df4096945fdb5da352929998f3ff27946bb4265f6cd3631b1328392c48a WatchSource:0}: Error finding container c7684df4096945fdb5da352929998f3ff27946bb4265f6cd3631b1328392c48a: Status 404 returned error can't find the container with id c7684df4096945fdb5da352929998f3ff27946bb4265f6cd3631b1328392c48a Jul 10 03:27:47.624984 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:47.624944 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7146-140688457619328" event={"ID":"5670100b-9e6d-4813-8953-958ea9d305cd","Type":"ContainerStarted","Data":"c7684df4096945fdb5da352929998f3ff27946bb4265f6cd3631b1328392c48a"} Jul 10 03:27:47.626088 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:47.626064 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7146-140688457619328" Jul 10 03:27:47.626088 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:47.626085 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50b805ab65acc01b3e31825f9621b0f31b2c50271bc1b2bc4a5fa00a36793678" Jul 10 03:27:47.637501 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:47.637462 2575 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="openshift-ingress/test-curl-7146-140688457619328" oldPodUID="b5a9aee5-872a-4cc6-b048-e023f59d5a9b" podUID="5670100b-9e6d-4813-8953-958ea9d305cd" Jul 10 03:27:48.290230 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:48.290197 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5a9aee5-872a-4cc6-b048-e023f59d5a9b" path="/var/lib/kubelet/pods/b5a9aee5-872a-4cc6-b048-e023f59d5a9b/volumes" Jul 10 03:27:48.633188 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:48.633095 2575 generic.go:356] "Generic (PLEG): container finished" podID="5670100b-9e6d-4813-8953-958ea9d305cd" containerID="dfdec478526444d30835bdb2b0b6e6573b5ce1928d1e1e7edb1b4dffe3703754" exitCode=0 Jul 10 03:27:48.633340 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:48.633181 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7146-140688457619328" event={"ID":"5670100b-9e6d-4813-8953-958ea9d305cd","Type":"ContainerDied","Data":"dfdec478526444d30835bdb2b0b6e6573b5ce1928d1e1e7edb1b4dffe3703754"} Jul 10 03:27:49.784985 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:49.784953 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7146-140688457619328" Jul 10 03:27:49.805587 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:49.805559 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7146-140688457619328_5670100b-9e6d-4813-8953-958ea9d305cd/test-curl-7146-140688457619328/0.log" Jul 10 03:27:49.831264 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:49.831228 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7146-140688457619328"] Jul 10 03:27:49.833295 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:49.833270 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7146-140688457619328"] Jul 10 03:27:49.943671 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:49.943581 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qd2ts\" (UniqueName: \"kubernetes.io/projected/5670100b-9e6d-4813-8953-958ea9d305cd-kube-api-access-qd2ts\") pod \"5670100b-9e6d-4813-8953-958ea9d305cd\" (UID: \"5670100b-9e6d-4813-8953-958ea9d305cd\") " Jul 10 03:27:49.945668 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:49.945639 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5670100b-9e6d-4813-8953-958ea9d305cd-kube-api-access-qd2ts" (OuterVolumeSpecName: "kube-api-access-qd2ts") pod "5670100b-9e6d-4813-8953-958ea9d305cd" (UID: "5670100b-9e6d-4813-8953-958ea9d305cd"). InnerVolumeSpecName "kube-api-access-qd2ts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 03:27:49.955335 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:49.955312 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7146-140688457620336"] Jul 10 03:27:49.955747 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:49.955735 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5670100b-9e6d-4813-8953-958ea9d305cd" containerName="test-curl-7146-140688457619328" Jul 10 03:27:49.955796 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:49.955758 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5670100b-9e6d-4813-8953-958ea9d305cd" containerName="test-curl-7146-140688457619328" Jul 10 03:27:49.955878 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:49.955868 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5670100b-9e6d-4813-8953-958ea9d305cd" containerName="test-curl-7146-140688457619328" Jul 10 03:27:49.958803 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:49.958789 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7146-140688457620336" Jul 10 03:27:49.961674 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:49.961650 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7146-140688457620336"] Jul 10 03:27:50.045038 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:50.045007 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhcsw\" (UniqueName: \"kubernetes.io/projected/5a3044ca-6e54-4250-9ef1-dfb297371bd3-kube-api-access-fhcsw\") pod \"test-curl-7146-140688457620336\" (UID: \"5a3044ca-6e54-4250-9ef1-dfb297371bd3\") " pod="openshift-ingress/test-curl-7146-140688457620336" Jul 10 03:27:50.045179 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:50.045063 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd2ts\" (UniqueName: \"kubernetes.io/projected/5670100b-9e6d-4813-8953-958ea9d305cd-kube-api-access-qd2ts\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 03:27:50.145693 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:50.145660 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhcsw\" (UniqueName: \"kubernetes.io/projected/5a3044ca-6e54-4250-9ef1-dfb297371bd3-kube-api-access-fhcsw\") pod \"test-curl-7146-140688457620336\" (UID: \"5a3044ca-6e54-4250-9ef1-dfb297371bd3\") " pod="openshift-ingress/test-curl-7146-140688457620336" Jul 10 03:27:50.152997 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:50.152979 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhcsw\" (UniqueName: \"kubernetes.io/projected/5a3044ca-6e54-4250-9ef1-dfb297371bd3-kube-api-access-fhcsw\") pod \"test-curl-7146-140688457620336\" (UID: \"5a3044ca-6e54-4250-9ef1-dfb297371bd3\") " pod="openshift-ingress/test-curl-7146-140688457620336" Jul 10 03:27:50.270296 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:50.270260 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7146-140688457620336" Jul 10 03:27:50.291207 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:50.291165 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5670100b-9e6d-4813-8953-958ea9d305cd" path="/var/lib/kubelet/pods/5670100b-9e6d-4813-8953-958ea9d305cd/volumes" Jul 10 03:27:50.411649 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:50.411625 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7146-140688457620336"] Jul 10 03:27:50.413585 ip-10-0-130-139 kubenswrapper[2575]: W0710 03:27:50.413555 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a3044ca_6e54_4250_9ef1_dfb297371bd3.slice/crio-da60e798421e1f83341572c510f6c1266ca7d65281eb1418a6b4349dac8ea2b3 WatchSource:0}: Error finding container da60e798421e1f83341572c510f6c1266ca7d65281eb1418a6b4349dac8ea2b3: Status 404 returned error can't find the container with id da60e798421e1f83341572c510f6c1266ca7d65281eb1418a6b4349dac8ea2b3 Jul 10 03:27:50.645680 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:50.645589 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7146-140688457620336" event={"ID":"5a3044ca-6e54-4250-9ef1-dfb297371bd3","Type":"ContainerStarted","Data":"da60e798421e1f83341572c510f6c1266ca7d65281eb1418a6b4349dac8ea2b3"} Jul 10 03:27:50.646797 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:50.646780 2575 scope.go:117] "RemoveContainer" containerID="dfdec478526444d30835bdb2b0b6e6573b5ce1928d1e1e7edb1b4dffe3703754" Jul 10 03:27:50.646797 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:50.646790 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7146-140688457619328" Jul 10 03:27:51.653265 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:51.653224 2575 generic.go:356] "Generic (PLEG): container finished" podID="5a3044ca-6e54-4250-9ef1-dfb297371bd3" containerID="97887c36745fb13734ba6ea806cc6580841193715d44a6e39e00833e1f151c09" exitCode=0 Jul 10 03:27:51.653734 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:51.653312 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7146-140688457620336" event={"ID":"5a3044ca-6e54-4250-9ef1-dfb297371bd3","Type":"ContainerDied","Data":"97887c36745fb13734ba6ea806cc6580841193715d44a6e39e00833e1f151c09"} Jul 10 03:27:52.809723 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:52.809701 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7146-140688457620336" Jul 10 03:27:52.829669 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:52.829636 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7146-140688457620336_5a3044ca-6e54-4250-9ef1-dfb297371bd3/test-curl-7146-140688457620336/0.log" Jul 10 03:27:52.853750 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:52.853700 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7146-140688457620336"] Jul 10 03:27:52.856047 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:52.856019 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7146-140688457620336"] Jul 10 03:27:52.868289 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:52.868263 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fhcsw\" (UniqueName: \"kubernetes.io/projected/5a3044ca-6e54-4250-9ef1-dfb297371bd3-kube-api-access-fhcsw\") pod \"5a3044ca-6e54-4250-9ef1-dfb297371bd3\" (UID: \"5a3044ca-6e54-4250-9ef1-dfb297371bd3\") " Jul 10 03:27:52.870471 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:52.870439 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a3044ca-6e54-4250-9ef1-dfb297371bd3-kube-api-access-fhcsw" (OuterVolumeSpecName: "kube-api-access-fhcsw") pod "5a3044ca-6e54-4250-9ef1-dfb297371bd3" (UID: "5a3044ca-6e54-4250-9ef1-dfb297371bd3"). InnerVolumeSpecName "kube-api-access-fhcsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 03:27:52.969268 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:52.969187 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhcsw\" (UniqueName: \"kubernetes.io/projected/5a3044ca-6e54-4250-9ef1-dfb297371bd3-kube-api-access-fhcsw\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 03:27:53.666489 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:53.666461 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7146-140688457620336" Jul 10 03:27:53.666489 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:53.666475 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da60e798421e1f83341572c510f6c1266ca7d65281eb1418a6b4349dac8ea2b3" Jul 10 03:27:53.676203 ip-10-0-130-139 kubenswrapper[2575]: E0710 03:27:53.676173 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7146-140688457620336\" is forbidden: User \"system:node:ip-10-0-130-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-130-139.ec2.internal' and this object" podUID="5a3044ca-6e54-4250-9ef1-dfb297371bd3" pod="openshift-ingress/test-curl-7146-140688457620336" Jul 10 03:27:54.290288 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:54.290255 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a3044ca-6e54-4250-9ef1-dfb297371bd3" path="/var/lib/kubelet/pods/5a3044ca-6e54-4250-9ef1-dfb297371bd3/volumes" Jul 10 03:27:57.051273 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:57.051240 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-f42020-66d5d4b868-hz9vq"] Jul 10 03:27:57.051685 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:57.051478 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f42020-66d5d4b868-hz9vq" podUID="008c1c15-3410-420a-894c-5c37b33c3273" containerName="maas-api" containerID="cri-o://7ff81a5ee7f905b9f221c556062aec9ca93120f1b77d957dcc9853e5b8709d44" gracePeriod=30 Jul 10 03:27:57.313570 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:57.313508 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f42020-66d5d4b868-hz9vq" Jul 10 03:27:57.410439 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:57.410406 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/008c1c15-3410-420a-894c-5c37b33c3273-openshift-service-ca\") pod \"008c1c15-3410-420a-894c-5c37b33c3273\" (UID: \"008c1c15-3410-420a-894c-5c37b33c3273\") " Jul 10 03:27:57.410626 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:57.410527 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86zpc\" (UniqueName: \"kubernetes.io/projected/008c1c15-3410-420a-894c-5c37b33c3273-kube-api-access-86zpc\") pod \"008c1c15-3410-420a-894c-5c37b33c3273\" (UID: \"008c1c15-3410-420a-894c-5c37b33c3273\") " Jul 10 03:27:57.410626 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:57.410565 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/008c1c15-3410-420a-894c-5c37b33c3273-maas-api-tls\") pod \"008c1c15-3410-420a-894c-5c37b33c3273\" (UID: \"008c1c15-3410-420a-894c-5c37b33c3273\") " Jul 10 03:27:57.410754 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:57.410732 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/008c1c15-3410-420a-894c-5c37b33c3273-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "008c1c15-3410-420a-894c-5c37b33c3273" (UID: "008c1c15-3410-420a-894c-5c37b33c3273"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 03:27:57.410886 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:57.410865 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/008c1c15-3410-420a-894c-5c37b33c3273-openshift-service-ca\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 03:27:57.412607 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:57.412576 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/008c1c15-3410-420a-894c-5c37b33c3273-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "008c1c15-3410-420a-894c-5c37b33c3273" (UID: "008c1c15-3410-420a-894c-5c37b33c3273"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 03:27:57.412708 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:57.412683 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/008c1c15-3410-420a-894c-5c37b33c3273-kube-api-access-86zpc" (OuterVolumeSpecName: "kube-api-access-86zpc") pod "008c1c15-3410-420a-894c-5c37b33c3273" (UID: "008c1c15-3410-420a-894c-5c37b33c3273"). InnerVolumeSpecName "kube-api-access-86zpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 03:27:57.512134 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:57.512087 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86zpc\" (UniqueName: \"kubernetes.io/projected/008c1c15-3410-420a-894c-5c37b33c3273-kube-api-access-86zpc\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 03:27:57.512134 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:57.512129 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/008c1c15-3410-420a-894c-5c37b33c3273-maas-api-tls\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 03:27:57.690332 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:57.690239 2575 generic.go:356] "Generic (PLEG): container finished" podID="008c1c15-3410-420a-894c-5c37b33c3273" containerID="7ff81a5ee7f905b9f221c556062aec9ca93120f1b77d957dcc9853e5b8709d44" exitCode=0 Jul 10 03:27:57.690332 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:57.690284 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f42020-66d5d4b868-hz9vq" Jul 10 03:27:57.690332 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:57.690321 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f42020-66d5d4b868-hz9vq" event={"ID":"008c1c15-3410-420a-894c-5c37b33c3273","Type":"ContainerDied","Data":"7ff81a5ee7f905b9f221c556062aec9ca93120f1b77d957dcc9853e5b8709d44"} Jul 10 03:27:57.690576 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:57.690381 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-f42020-66d5d4b868-hz9vq" event={"ID":"008c1c15-3410-420a-894c-5c37b33c3273","Type":"ContainerDied","Data":"80243a860c81c19e061504963dcc2fbb56e87dc79c2c464fcf17c97f5e28d065"} Jul 10 03:27:57.690576 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:57.690409 2575 scope.go:117] "RemoveContainer" containerID="7ff81a5ee7f905b9f221c556062aec9ca93120f1b77d957dcc9853e5b8709d44" Jul 10 03:27:57.703154 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:57.703130 2575 scope.go:117] "RemoveContainer" containerID="7ff81a5ee7f905b9f221c556062aec9ca93120f1b77d957dcc9853e5b8709d44" Jul 10 03:27:57.703828 ip-10-0-130-139 kubenswrapper[2575]: E0710 03:27:57.703784 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ff81a5ee7f905b9f221c556062aec9ca93120f1b77d957dcc9853e5b8709d44\": container with ID starting with 7ff81a5ee7f905b9f221c556062aec9ca93120f1b77d957dcc9853e5b8709d44 not found: ID does not exist" containerID="7ff81a5ee7f905b9f221c556062aec9ca93120f1b77d957dcc9853e5b8709d44" Jul 10 03:27:57.703891 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:57.703840 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ff81a5ee7f905b9f221c556062aec9ca93120f1b77d957dcc9853e5b8709d44"} err="failed to get container status \"7ff81a5ee7f905b9f221c556062aec9ca93120f1b77d957dcc9853e5b8709d44\": rpc error: code = NotFound desc = could not find container \"7ff81a5ee7f905b9f221c556062aec9ca93120f1b77d957dcc9853e5b8709d44\": container with ID starting with 7ff81a5ee7f905b9f221c556062aec9ca93120f1b77d957dcc9853e5b8709d44 not found: ID does not exist" Jul 10 03:27:57.712803 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:57.712782 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-f42020-66d5d4b868-hz9vq"] Jul 10 03:27:57.716642 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:57.716621 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-f42020-66d5d4b868-hz9vq"] Jul 10 03:27:58.289728 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:27:58.289694 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="008c1c15-3410-420a-894c-5c37b33c3273" path="/var/lib/kubelet/pods/008c1c15-3410-420a-894c-5c37b33c3273/volumes" Jul 10 03:28:02.426713 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:02.426685 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-xxlp6_372be3ce-a380-4c0e-bbee-49c2e3b80095/console-operator/1.log" Jul 10 03:28:02.431064 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:02.431041 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-xxlp6_372be3ce-a380-4c0e-bbee-49c2e3b80095/console-operator/1.log" Jul 10 03:28:03.968752 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:03.968717 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-45a5a5-5bd65885d-w672q"] Jul 10 03:28:03.969264 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:03.968967 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-45a5a5-5bd65885d-w672q" podUID="d659ebd2-bb90-4d2b-9676-0a14f9627e7b" containerName="maas-api" containerID="cri-o://267ef2cd97a5774d5adba82e6c0324e2f888cfcc2ce3ba5fdcb454c088313de1" gracePeriod=30 Jul 10 03:28:04.425751 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:04.425723 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-45a5a5-5bd65885d-w672q" Jul 10 03:28:04.478008 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:04.477974 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d659ebd2-bb90-4d2b-9676-0a14f9627e7b-maas-api-tls\") pod \"d659ebd2-bb90-4d2b-9676-0a14f9627e7b\" (UID: \"d659ebd2-bb90-4d2b-9676-0a14f9627e7b\") " Jul 10 03:28:04.478177 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:04.478021 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d659ebd2-bb90-4d2b-9676-0a14f9627e7b-openshift-service-ca\") pod \"d659ebd2-bb90-4d2b-9676-0a14f9627e7b\" (UID: \"d659ebd2-bb90-4d2b-9676-0a14f9627e7b\") " Jul 10 03:28:04.478177 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:04.478119 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s8bhl\" (UniqueName: \"kubernetes.io/projected/d659ebd2-bb90-4d2b-9676-0a14f9627e7b-kube-api-access-s8bhl\") pod \"d659ebd2-bb90-4d2b-9676-0a14f9627e7b\" (UID: \"d659ebd2-bb90-4d2b-9676-0a14f9627e7b\") " Jul 10 03:28:04.478455 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:04.478433 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d659ebd2-bb90-4d2b-9676-0a14f9627e7b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d659ebd2-bb90-4d2b-9676-0a14f9627e7b" (UID: "d659ebd2-bb90-4d2b-9676-0a14f9627e7b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 03:28:04.480151 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:04.480124 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d659ebd2-bb90-4d2b-9676-0a14f9627e7b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d659ebd2-bb90-4d2b-9676-0a14f9627e7b" (UID: "d659ebd2-bb90-4d2b-9676-0a14f9627e7b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 03:28:04.480247 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:04.480150 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d659ebd2-bb90-4d2b-9676-0a14f9627e7b-kube-api-access-s8bhl" (OuterVolumeSpecName: "kube-api-access-s8bhl") pod "d659ebd2-bb90-4d2b-9676-0a14f9627e7b" (UID: "d659ebd2-bb90-4d2b-9676-0a14f9627e7b"). InnerVolumeSpecName "kube-api-access-s8bhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 03:28:04.579530 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:04.579493 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8bhl\" (UniqueName: \"kubernetes.io/projected/d659ebd2-bb90-4d2b-9676-0a14f9627e7b-kube-api-access-s8bhl\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 03:28:04.579530 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:04.579523 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d659ebd2-bb90-4d2b-9676-0a14f9627e7b-maas-api-tls\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 03:28:04.579530 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:04.579536 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d659ebd2-bb90-4d2b-9676-0a14f9627e7b-openshift-service-ca\") on node \"ip-10-0-130-139.ec2.internal\" DevicePath \"\"" Jul 10 03:28:04.737501 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:04.737468 2575 generic.go:356] "Generic (PLEG): container finished" podID="d659ebd2-bb90-4d2b-9676-0a14f9627e7b" containerID="267ef2cd97a5774d5adba82e6c0324e2f888cfcc2ce3ba5fdcb454c088313de1" exitCode=0 Jul 10 03:28:04.737681 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:04.737513 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-45a5a5-5bd65885d-w672q" Jul 10 03:28:04.737681 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:04.737551 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-45a5a5-5bd65885d-w672q" event={"ID":"d659ebd2-bb90-4d2b-9676-0a14f9627e7b","Type":"ContainerDied","Data":"267ef2cd97a5774d5adba82e6c0324e2f888cfcc2ce3ba5fdcb454c088313de1"} Jul 10 03:28:04.737681 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:04.737598 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-45a5a5-5bd65885d-w672q" event={"ID":"d659ebd2-bb90-4d2b-9676-0a14f9627e7b","Type":"ContainerDied","Data":"f652037108e3388d42eb6941137b59fb9bcddbd70deae19fffe6647ff5275268"} Jul 10 03:28:04.737681 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:04.737621 2575 scope.go:117] "RemoveContainer" containerID="267ef2cd97a5774d5adba82e6c0324e2f888cfcc2ce3ba5fdcb454c088313de1" Jul 10 03:28:04.747428 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:04.747409 2575 scope.go:117] "RemoveContainer" containerID="267ef2cd97a5774d5adba82e6c0324e2f888cfcc2ce3ba5fdcb454c088313de1" Jul 10 03:28:04.747685 ip-10-0-130-139 kubenswrapper[2575]: E0710 03:28:04.747668 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"267ef2cd97a5774d5adba82e6c0324e2f888cfcc2ce3ba5fdcb454c088313de1\": container with ID starting with 267ef2cd97a5774d5adba82e6c0324e2f888cfcc2ce3ba5fdcb454c088313de1 not found: ID does not exist" containerID="267ef2cd97a5774d5adba82e6c0324e2f888cfcc2ce3ba5fdcb454c088313de1" Jul 10 03:28:04.747732 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:04.747695 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"267ef2cd97a5774d5adba82e6c0324e2f888cfcc2ce3ba5fdcb454c088313de1"} err="failed to get container status \"267ef2cd97a5774d5adba82e6c0324e2f888cfcc2ce3ba5fdcb454c088313de1\": rpc error: code = NotFound desc = could not find container \"267ef2cd97a5774d5adba82e6c0324e2f888cfcc2ce3ba5fdcb454c088313de1\": container with ID starting with 267ef2cd97a5774d5adba82e6c0324e2f888cfcc2ce3ba5fdcb454c088313de1 not found: ID does not exist" Jul 10 03:28:04.758436 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:04.758410 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-45a5a5-5bd65885d-w672q"] Jul 10 03:28:04.761621 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:04.761597 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-45a5a5-5bd65885d-w672q"] Jul 10 03:28:06.289119 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:06.289088 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d659ebd2-bb90-4d2b-9676-0a14f9627e7b" path="/var/lib/kubelet/pods/d659ebd2-bb90-4d2b-9676-0a14f9627e7b/volumes" Jul 10 03:28:15.414692 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:15.414659 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-r987c_0fb37c8b-e397-4f28-8dee-2b7ee86976d2/manager/0.log" Jul 10 03:28:15.667968 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:15.667886 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5dcc8799c9-nwbqh_8e7713f9-b289-4dd3-9545-5362a8bdbd4e/manager/0.log" Jul 10 03:28:15.781942 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:15.781913 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-6xkl7_dfc95468-c766-4463-9b94-ad59662147e6/server/0.log" Jul 10 03:28:15.901378 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:15.901348 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-hkdfx_4447dd2a-7547-4bbc-8738-7db73cc8c40f/manager/0.log" Jul 10 03:28:16.144960 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:16.144900 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84dd56b898-rmcl2_9fa796aa-f442-43b7-886f-575c72f33603/copy-manifests/0.log" Jul 10 03:28:16.152445 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:16.152421 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84dd56b898-rmcl2_9fa796aa-f442-43b7-886f-575c72f33603/manager/0.log" Jul 10 03:28:16.646895 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:16.646782 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5cbf495c54-x76lb_d9054c73-a611-4b55-8f91-bbc9b08201f4/maas-api/0.log" Jul 10 03:28:16.761521 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:16.761492 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-nrl67_b29e6d04-2b35-432e-9764-f047667337db/postgres/0.log" Jul 10 03:28:18.192678 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:18.192644 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-9zmcz_9ca0d756-cdbe-4517-b4d9-c3a6c96f44ed/manager/0.log" Jul 10 03:28:18.313300 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:18.313273 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-46t5x_fae1bef4-d254-4bed-8e45-4e65c35e78f8/manager/0.log" Jul 10 03:28:18.427866 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:18.427841 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-jtsd5_47f6a75f-d956-4969-8b7b-991b752d06af/kuadrant-console-plugin/0.log" Jul 10 03:28:18.546905 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:18.546872 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-zzg8l_4269dd60-fe03-48e4-8368-dd8e04a33335/registry-server/0.log" Jul 10 03:28:18.930565 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:18.930483 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-t2zbx_127c755f-b34b-452a-a83d-855155192db2/manager/0.log" Jul 10 03:28:19.636108 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:19.636075 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-864dd46fbc-vmnv7_74e98eb8-2d91-4e17-8be5-01a99f1d07b0/kube-auth-proxy/0.log" Jul 10 03:28:19.760824 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:19.760774 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-6zr5k_1e5039b0-ee00-4541-8216-c4a7015b93d1/istio-proxy/1.log" Jul 10 03:28:20.006220 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:20.006189 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-vhjvc_17135a96-76bf-434d-a6c1-0edd00d7901c/payload-processing/0.log" Jul 10 03:28:20.120824 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:20.120790 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-bfcf8947b-7fz4l_3adc0ae6-f77b-4909-8bef-8534f19e6601/router/0.log" Jul 10 03:28:20.574466 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:20.574433 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-vnxvn_79ab8a96-b91b-4a79-87e1-222a409a29e4/main/0.log" Jul 10 03:28:20.916358 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:20.916280 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-rmrxw_82b3cdab-2d8b-46a2-ba7c-c9d319b418a9/main/0.log" Jul 10 03:28:25.366183 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:25.366105 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lplzd/must-gather-ck448"] Jul 10 03:28:25.366531 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:25.366498 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a3044ca-6e54-4250-9ef1-dfb297371bd3" containerName="test-curl-7146-140688457620336" Jul 10 03:28:25.366531 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:25.366510 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a3044ca-6e54-4250-9ef1-dfb297371bd3" containerName="test-curl-7146-140688457620336" Jul 10 03:28:25.366603 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:25.366540 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d659ebd2-bb90-4d2b-9676-0a14f9627e7b" containerName="maas-api" Jul 10 03:28:25.366603 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:25.366546 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d659ebd2-bb90-4d2b-9676-0a14f9627e7b" containerName="maas-api" Jul 10 03:28:25.366603 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:25.366555 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="008c1c15-3410-420a-894c-5c37b33c3273" containerName="maas-api" Jul 10 03:28:25.366603 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:25.366560 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="008c1c15-3410-420a-894c-5c37b33c3273" containerName="maas-api" Jul 10 03:28:25.366767 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:25.366618 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="008c1c15-3410-420a-894c-5c37b33c3273" containerName="maas-api" Jul 10 03:28:25.366767 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:25.366631 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a3044ca-6e54-4250-9ef1-dfb297371bd3" containerName="test-curl-7146-140688457620336" Jul 10 03:28:25.366767 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:25.366639 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d659ebd2-bb90-4d2b-9676-0a14f9627e7b" containerName="maas-api" Jul 10 03:28:25.371664 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:25.371645 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lplzd/must-gather-ck448" Jul 10 03:28:25.375590 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:25.375566 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lplzd\"/\"kube-root-ca.crt\"" Jul 10 03:28:25.375590 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:25.375579 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lplzd\"/\"openshift-service-ca.crt\"" Jul 10 03:28:25.375879 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:25.375584 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lplzd\"/\"default-dockercfg-plrlg\"" Jul 10 03:28:25.378968 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:25.378704 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lplzd/must-gather-ck448"] Jul 10 03:28:25.469688 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:25.469660 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a4e710ce-0737-419e-9f4c-c6e7359105b2-must-gather-output\") pod \"must-gather-ck448\" (UID: \"a4e710ce-0737-419e-9f4c-c6e7359105b2\") " pod="openshift-must-gather-lplzd/must-gather-ck448" Jul 10 03:28:25.469858 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:25.469712 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77mxh\" (UniqueName: \"kubernetes.io/projected/a4e710ce-0737-419e-9f4c-c6e7359105b2-kube-api-access-77mxh\") pod \"must-gather-ck448\" (UID: \"a4e710ce-0737-419e-9f4c-c6e7359105b2\") " pod="openshift-must-gather-lplzd/must-gather-ck448" Jul 10 03:28:25.570695 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:25.570658 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a4e710ce-0737-419e-9f4c-c6e7359105b2-must-gather-output\") pod \"must-gather-ck448\" (UID: \"a4e710ce-0737-419e-9f4c-c6e7359105b2\") " pod="openshift-must-gather-lplzd/must-gather-ck448" Jul 10 03:28:25.570893 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:25.570718 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77mxh\" (UniqueName: \"kubernetes.io/projected/a4e710ce-0737-419e-9f4c-c6e7359105b2-kube-api-access-77mxh\") pod \"must-gather-ck448\" (UID: \"a4e710ce-0737-419e-9f4c-c6e7359105b2\") " pod="openshift-must-gather-lplzd/must-gather-ck448" Jul 10 03:28:25.571047 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:25.571026 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a4e710ce-0737-419e-9f4c-c6e7359105b2-must-gather-output\") pod \"must-gather-ck448\" (UID: \"a4e710ce-0737-419e-9f4c-c6e7359105b2\") " pod="openshift-must-gather-lplzd/must-gather-ck448" Jul 10 03:28:25.579953 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:25.579928 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77mxh\" (UniqueName: \"kubernetes.io/projected/a4e710ce-0737-419e-9f4c-c6e7359105b2-kube-api-access-77mxh\") pod \"must-gather-ck448\" (UID: \"a4e710ce-0737-419e-9f4c-c6e7359105b2\") " pod="openshift-must-gather-lplzd/must-gather-ck448" Jul 10 03:28:25.681709 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:25.681628 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lplzd/must-gather-ck448" Jul 10 03:28:25.825992 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:25.825962 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lplzd/must-gather-ck448"] Jul 10 03:28:25.826159 ip-10-0-130-139 kubenswrapper[2575]: W0710 03:28:25.826051 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda4e710ce_0737_419e_9f4c_c6e7359105b2.slice/crio-48557b1df03dd0720b93291c20b5bea823ff6ec304c6baf3823952a85d9221d7 WatchSource:0}: Error finding container 48557b1df03dd0720b93291c20b5bea823ff6ec304c6baf3823952a85d9221d7: Status 404 returned error can't find the container with id 48557b1df03dd0720b93291c20b5bea823ff6ec304c6baf3823952a85d9221d7 Jul 10 03:28:25.847344 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:25.847305 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lplzd/must-gather-ck448" event={"ID":"a4e710ce-0737-419e-9f4c-c6e7359105b2","Type":"ContainerStarted","Data":"48557b1df03dd0720b93291c20b5bea823ff6ec304c6baf3823952a85d9221d7"} Jul 10 03:28:26.858624 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:26.858589 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lplzd/must-gather-ck448" event={"ID":"a4e710ce-0737-419e-9f4c-c6e7359105b2","Type":"ContainerStarted","Data":"ac8ea5d63a283898a47e5139ef8becf23e8c422ab86a40b645c46da72babc0df"} Jul 10 03:28:26.859107 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:26.858642 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lplzd/must-gather-ck448" event={"ID":"a4e710ce-0737-419e-9f4c-c6e7359105b2","Type":"ContainerStarted","Data":"64a32ce4703e2142804680b183ea154b6ae45b18934fdf4ba3c13d2d9f95659b"} Jul 10 03:28:26.874685 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:26.874579 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-lplzd/must-gather-ck448" podStartSLOduration=1.086607954 podStartE2EDuration="1.874560343s" totalImagesPullingTime="787.952389ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 03:28:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 03:28:26.87285954 +0000 UTC m=+2425.153975828" watchObservedRunningTime="2026-07-10 03:28:26.874560343 +0000 UTC m=+2425.155676627" Jul 10 03:28:28.204011 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:28.203976 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-q28nr_dc1f4b26-669e-42a0-abd4-f858ac6b1543/global-pull-secret-syncer/0.log" Jul 10 03:28:28.330165 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:28.330124 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-kbfmm_cc6bf0c3-5e10-4619-9748-a2e9e46754ed/konnectivity-agent/0.log" Jul 10 03:28:28.387317 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:28.387288 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-139.ec2.internal_f238b60151ef7c6e38d6cb1b9cdf59c0/haproxy/0.log" Jul 10 03:28:33.451849 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:33.451797 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-9zmcz_9ca0d756-cdbe-4517-b4d9-c3a6c96f44ed/manager/0.log" Jul 10 03:28:33.484030 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:33.484001 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-46t5x_fae1bef4-d254-4bed-8e45-4e65c35e78f8/manager/0.log" Jul 10 03:28:33.511530 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:33.511492 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-jtsd5_47f6a75f-d956-4969-8b7b-991b752d06af/kuadrant-console-plugin/0.log" Jul 10 03:28:33.556260 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:33.556225 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-zzg8l_4269dd60-fe03-48e4-8368-dd8e04a33335/registry-server/0.log" Jul 10 03:28:33.804450 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:33.802675 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-t2zbx_127c755f-b34b-452a-a83d-855155192db2/manager/0.log" Jul 10 03:28:35.555312 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:35.555229 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-2rh2j_bcda0e99-fdd8-498f-8e73-ecf4c279be71/cluster-monitoring-operator/0.log" Jul 10 03:28:35.833153 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:35.833074 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jrfx9_d66bd742-5ded-43b8-a8df-b202ed2fce86/node-exporter/0.log" Jul 10 03:28:35.850497 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:35.850469 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jrfx9_d66bd742-5ded-43b8-a8df-b202ed2fce86/kube-rbac-proxy/0.log" Jul 10 03:28:35.872937 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:35.872905 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jrfx9_d66bd742-5ded-43b8-a8df-b202ed2fce86/init-textfile/0.log" Jul 10 03:28:36.265999 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:36.265961 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-6fjb8_9bd0b995-3a3a-47e8-b101-aaab0393550c/prometheus-operator-admission-webhook/0.log" Jul 10 03:28:37.699967 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:37.699927 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-7v4dt_c6492dae-a18a-45ed-95be-93bc92198ea9/networking-console-plugin/0.log" Jul 10 03:28:38.381913 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:38.381874 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-xxlp6_372be3ce-a380-4c0e-bbee-49c2e3b80095/console-operator/1.log" Jul 10 03:28:38.399843 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:38.398486 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-xxlp6_372be3ce-a380-4c0e-bbee-49c2e3b80095/console-operator/2.log" Jul 10 03:28:38.528386 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:38.528346 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lplzd/perf-node-gather-daemonset-r8gq8"] Jul 10 03:28:38.537158 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:38.537124 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lplzd/perf-node-gather-daemonset-r8gq8" Jul 10 03:28:38.544057 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:38.544027 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lplzd/perf-node-gather-daemonset-r8gq8"] Jul 10 03:28:38.602090 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:38.601631 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/09c7bb73-a055-4f36-ba34-48fffd11c0ae-lib-modules\") pod \"perf-node-gather-daemonset-r8gq8\" (UID: \"09c7bb73-a055-4f36-ba34-48fffd11c0ae\") " pod="openshift-must-gather-lplzd/perf-node-gather-daemonset-r8gq8" Jul 10 03:28:38.602090 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:38.601707 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/09c7bb73-a055-4f36-ba34-48fffd11c0ae-proc\") pod \"perf-node-gather-daemonset-r8gq8\" (UID: \"09c7bb73-a055-4f36-ba34-48fffd11c0ae\") " pod="openshift-must-gather-lplzd/perf-node-gather-daemonset-r8gq8" Jul 10 03:28:38.602090 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:38.601752 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/09c7bb73-a055-4f36-ba34-48fffd11c0ae-sys\") pod \"perf-node-gather-daemonset-r8gq8\" (UID: \"09c7bb73-a055-4f36-ba34-48fffd11c0ae\") " pod="openshift-must-gather-lplzd/perf-node-gather-daemonset-r8gq8" Jul 10 03:28:38.602090 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:38.601844 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/09c7bb73-a055-4f36-ba34-48fffd11c0ae-podres\") pod \"perf-node-gather-daemonset-r8gq8\" (UID: \"09c7bb73-a055-4f36-ba34-48fffd11c0ae\") " pod="openshift-must-gather-lplzd/perf-node-gather-daemonset-r8gq8" Jul 10 03:28:38.602090 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:38.601962 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhqqb\" (UniqueName: \"kubernetes.io/projected/09c7bb73-a055-4f36-ba34-48fffd11c0ae-kube-api-access-nhqqb\") pod \"perf-node-gather-daemonset-r8gq8\" (UID: \"09c7bb73-a055-4f36-ba34-48fffd11c0ae\") " pod="openshift-must-gather-lplzd/perf-node-gather-daemonset-r8gq8" Jul 10 03:28:38.703381 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:38.703292 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/09c7bb73-a055-4f36-ba34-48fffd11c0ae-lib-modules\") pod \"perf-node-gather-daemonset-r8gq8\" (UID: \"09c7bb73-a055-4f36-ba34-48fffd11c0ae\") " pod="openshift-must-gather-lplzd/perf-node-gather-daemonset-r8gq8" Jul 10 03:28:38.703381 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:38.703366 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/09c7bb73-a055-4f36-ba34-48fffd11c0ae-proc\") pod \"perf-node-gather-daemonset-r8gq8\" (UID: \"09c7bb73-a055-4f36-ba34-48fffd11c0ae\") " pod="openshift-must-gather-lplzd/perf-node-gather-daemonset-r8gq8" Jul 10 03:28:38.703892 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:38.703393 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/09c7bb73-a055-4f36-ba34-48fffd11c0ae-sys\") pod \"perf-node-gather-daemonset-r8gq8\" (UID: \"09c7bb73-a055-4f36-ba34-48fffd11c0ae\") " pod="openshift-must-gather-lplzd/perf-node-gather-daemonset-r8gq8" Jul 10 03:28:38.703892 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:38.703423 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/09c7bb73-a055-4f36-ba34-48fffd11c0ae-podres\") pod \"perf-node-gather-daemonset-r8gq8\" (UID: \"09c7bb73-a055-4f36-ba34-48fffd11c0ae\") " pod="openshift-must-gather-lplzd/perf-node-gather-daemonset-r8gq8" Jul 10 03:28:38.703892 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:38.703460 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhqqb\" (UniqueName: \"kubernetes.io/projected/09c7bb73-a055-4f36-ba34-48fffd11c0ae-kube-api-access-nhqqb\") pod \"perf-node-gather-daemonset-r8gq8\" (UID: \"09c7bb73-a055-4f36-ba34-48fffd11c0ae\") " pod="openshift-must-gather-lplzd/perf-node-gather-daemonset-r8gq8" Jul 10 03:28:38.703892 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:38.703497 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/09c7bb73-a055-4f36-ba34-48fffd11c0ae-proc\") pod \"perf-node-gather-daemonset-r8gq8\" (UID: \"09c7bb73-a055-4f36-ba34-48fffd11c0ae\") " pod="openshift-must-gather-lplzd/perf-node-gather-daemonset-r8gq8" Jul 10 03:28:38.703892 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:38.703505 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/09c7bb73-a055-4f36-ba34-48fffd11c0ae-lib-modules\") pod \"perf-node-gather-daemonset-r8gq8\" (UID: \"09c7bb73-a055-4f36-ba34-48fffd11c0ae\") " pod="openshift-must-gather-lplzd/perf-node-gather-daemonset-r8gq8" Jul 10 03:28:38.703892 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:38.703574 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/09c7bb73-a055-4f36-ba34-48fffd11c0ae-podres\") pod \"perf-node-gather-daemonset-r8gq8\" (UID: \"09c7bb73-a055-4f36-ba34-48fffd11c0ae\") " pod="openshift-must-gather-lplzd/perf-node-gather-daemonset-r8gq8" Jul 10 03:28:38.703892 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:38.703642 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/09c7bb73-a055-4f36-ba34-48fffd11c0ae-sys\") pod \"perf-node-gather-daemonset-r8gq8\" (UID: \"09c7bb73-a055-4f36-ba34-48fffd11c0ae\") " pod="openshift-must-gather-lplzd/perf-node-gather-daemonset-r8gq8" Jul 10 03:28:38.712791 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:38.712760 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhqqb\" (UniqueName: \"kubernetes.io/projected/09c7bb73-a055-4f36-ba34-48fffd11c0ae-kube-api-access-nhqqb\") pod \"perf-node-gather-daemonset-r8gq8\" (UID: \"09c7bb73-a055-4f36-ba34-48fffd11c0ae\") " pod="openshift-must-gather-lplzd/perf-node-gather-daemonset-r8gq8" Jul 10 03:28:38.857778 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:38.857527 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lplzd/perf-node-gather-daemonset-r8gq8" Jul 10 03:28:39.058468 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:39.058440 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lplzd/perf-node-gather-daemonset-r8gq8"] Jul 10 03:28:39.062303 ip-10-0-130-139 kubenswrapper[2575]: W0710 03:28:39.062273 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod09c7bb73_a055_4f36_ba34_48fffd11c0ae.slice/crio-a6224bce947b6c3cc0b1ab9849cc693b3309a77776b60ac69908935576f92738 WatchSource:0}: Error finding container a6224bce947b6c3cc0b1ab9849cc693b3309a77776b60ac69908935576f92738: Status 404 returned error can't find the container with id a6224bce947b6c3cc0b1ab9849cc693b3309a77776b60ac69908935576f92738 Jul 10 03:28:39.126188 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:39.126144 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-nnk5x_cbbc2bee-9865-401e-9bc3-23de969ff5eb/download-server/0.log" Jul 10 03:28:39.728208 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:39.728170 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78495597f6-n8kpv_00377a57-ff2d-4c74-a9de-1fb20db00f5e/volume-data-source-validator/0.log" Jul 10 03:28:39.973655 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:39.972711 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lplzd/perf-node-gather-daemonset-r8gq8" event={"ID":"09c7bb73-a055-4f36-ba34-48fffd11c0ae","Type":"ContainerStarted","Data":"d4ff0198d7853a09f5ee2e14d353bc5cfe69ec9ce5ef94834908cac3c9e606f5"} Jul 10 03:28:39.973655 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:39.972784 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lplzd/perf-node-gather-daemonset-r8gq8" event={"ID":"09c7bb73-a055-4f36-ba34-48fffd11c0ae","Type":"ContainerStarted","Data":"a6224bce947b6c3cc0b1ab9849cc693b3309a77776b60ac69908935576f92738"} Jul 10 03:28:39.973929 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:39.973663 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-lplzd/perf-node-gather-daemonset-r8gq8" Jul 10 03:28:40.804489 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:40.804460 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cc72k_5176df10-dfd5-4514-9a59-c854c4a0ec21/dns/0.log" Jul 10 03:28:40.825467 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:40.825434 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cc72k_5176df10-dfd5-4514-9a59-c854c4a0ec21/kube-rbac-proxy/0.log" Jul 10 03:28:40.942651 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:40.942619 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-shf9m_731666e1-c05f-4169-98f4-df0efa01817a/dns-node-resolver/0.log" Jul 10 03:28:41.511829 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:41.511775 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gp2gc_11f18b36-f920-47ba-8b04-10483f768ebf/node-ca/0.log" Jul 10 03:28:42.641509 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:42.641481 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-864dd46fbc-vmnv7_74e98eb8-2d91-4e17-8be5-01a99f1d07b0/kube-auth-proxy/0.log" Jul 10 03:28:42.676834 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:42.676782 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-6zr5k_1e5039b0-ee00-4541-8216-c4a7015b93d1/istio-proxy/0.log" Jul 10 03:28:42.742443 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:42.742413 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-6zr5k_1e5039b0-ee00-4541-8216-c4a7015b93d1/istio-proxy/1.log" Jul 10 03:28:42.829853 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:42.829821 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-vhjvc_17135a96-76bf-434d-a6c1-0edd00d7901c/payload-processing/0.log" Jul 10 03:28:42.866829 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:42.866771 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-bfcf8947b-7fz4l_3adc0ae6-f77b-4909-8bef-8534f19e6601/router/0.log" Jul 10 03:28:43.575458 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:43.575426 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9qrnw_6136f9e5-f2c2-4e54-94e5-f86957cc814e/serve-healthcheck-canary/0.log" Jul 10 03:28:44.099871 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:44.099833 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-x22bq_64e37f35-938c-457a-a377-354c4bc12d73/insights-operator/0.log" Jul 10 03:28:44.101764 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:44.101736 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-x22bq_64e37f35-938c-457a-a377-354c4bc12d73/insights-operator/1.log" Jul 10 03:28:44.290687 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:44.290660 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w7npn_b3661326-52c2-4c9b-b2db-420720fe4a86/kube-rbac-proxy/0.log" Jul 10 03:28:44.312777 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:44.312739 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w7npn_b3661326-52c2-4c9b-b2db-420720fe4a86/exporter/0.log" Jul 10 03:28:44.336956 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:44.336918 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w7npn_b3661326-52c2-4c9b-b2db-420720fe4a86/extractor/0.log" Jul 10 03:28:46.378741 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:46.378706 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-r987c_0fb37c8b-e397-4f28-8dee-2b7ee86976d2/manager/0.log" Jul 10 03:28:46.674204 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:46.674126 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5dcc8799c9-nwbqh_8e7713f9-b289-4dd3-9545-5362a8bdbd4e/manager/0.log" Jul 10 03:28:46.696008 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:46.695974 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-6xkl7_dfc95468-c766-4463-9b94-ad59662147e6/server/0.log" Jul 10 03:28:46.805791 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:46.805759 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-hkdfx_4447dd2a-7547-4bbc-8738-7db73cc8c40f/manager/0.log" Jul 10 03:28:46.920532 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:46.920500 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84dd56b898-rmcl2_9fa796aa-f442-43b7-886f-575c72f33603/manager/0.log" Jul 10 03:28:46.941388 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:46.941318 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84dd56b898-rmcl2_9fa796aa-f442-43b7-886f-575c72f33603/copy-manifests/0.log" Jul 10 03:28:47.000539 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:47.000506 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-lplzd/perf-node-gather-daemonset-r8gq8" Jul 10 03:28:47.015987 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:47.015927 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-lplzd/perf-node-gather-daemonset-r8gq8" podStartSLOduration=9.015906417 podStartE2EDuration="9.015906417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 03:28:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 03:28:39.990774605 +0000 UTC m=+2438.271890892" watchObservedRunningTime="2026-07-10 03:28:47.015906417 +0000 UTC m=+2445.297022703" Jul 10 03:28:48.565821 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:48.565725 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-864dd46fbc-vmnv7_74e98eb8-2d91-4e17-8be5-01a99f1d07b0/kube-auth-proxy/0.log" Jul 10 03:28:48.587174 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:48.587142 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-6zr5k_1e5039b0-ee00-4541-8216-c4a7015b93d1/istio-proxy/0.log" Jul 10 03:28:48.650644 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:48.650614 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-6zr5k_1e5039b0-ee00-4541-8216-c4a7015b93d1/istio-proxy/1.log" Jul 10 03:28:48.693880 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:48.693840 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-vhjvc_17135a96-76bf-434d-a6c1-0edd00d7901c/payload-processing/0.log" Jul 10 03:28:48.714338 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:48.714269 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-bfcf8947b-7fz4l_3adc0ae6-f77b-4909-8bef-8534f19e6601/router/0.log" Jul 10 03:28:50.189436 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:50.189401 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-864dd46fbc-vmnv7_74e98eb8-2d91-4e17-8be5-01a99f1d07b0/kube-auth-proxy/0.log" Jul 10 03:28:50.210536 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:50.210212 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-6zr5k_1e5039b0-ee00-4541-8216-c4a7015b93d1/istio-proxy/0.log" Jul 10 03:28:50.263198 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:50.263164 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-6zr5k_1e5039b0-ee00-4541-8216-c4a7015b93d1/istio-proxy/1.log" Jul 10 03:28:50.315769 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:50.315726 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-vhjvc_17135a96-76bf-434d-a6c1-0edd00d7901c/payload-processing/0.log" Jul 10 03:28:50.341588 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:50.341549 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-bfcf8947b-7fz4l_3adc0ae6-f77b-4909-8bef-8534f19e6601/router/0.log" Jul 10 03:28:53.468499 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:53.468471 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-7bqcj_c568f52e-6afa-453d-8b32-e5cb8b4f762e/migrator/0.log" Jul 10 03:28:53.487869 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:53.487839 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-7bqcj_c568f52e-6afa-453d-8b32-e5cb8b4f762e/graceful-termination/0.log" Jul 10 03:28:53.832964 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:53.832924 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-gtqxb_4094383d-2bb9-441b-8c30-94ff7551ce18/kube-storage-version-migrator-operator/1.log" Jul 10 03:28:53.835060 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:53.835029 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-gtqxb_4094383d-2bb9-441b-8c30-94ff7551ce18/kube-storage-version-migrator-operator/0.log" Jul 10 03:28:54.828350 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:54.828322 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-9hgr6_29d54ad7-c565-4c22-a182-0e9fe8c8b981/kube-multus/0.log" Jul 10 03:28:55.195316 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:55.195205 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tfrkp_ff323ccb-9efc-43fd-8ce7-9365fe9af996/kube-multus-additional-cni-plugins/0.log" Jul 10 03:28:55.221266 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:55.221239 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tfrkp_ff323ccb-9efc-43fd-8ce7-9365fe9af996/egress-router-binary-copy/0.log" Jul 10 03:28:55.248051 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:55.248028 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tfrkp_ff323ccb-9efc-43fd-8ce7-9365fe9af996/cni-plugins/0.log" Jul 10 03:28:55.269944 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:55.269915 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tfrkp_ff323ccb-9efc-43fd-8ce7-9365fe9af996/bond-cni-plugin/0.log" Jul 10 03:28:55.291854 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:55.291827 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tfrkp_ff323ccb-9efc-43fd-8ce7-9365fe9af996/routeoverride-cni/0.log" Jul 10 03:28:55.313323 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:55.313298 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tfrkp_ff323ccb-9efc-43fd-8ce7-9365fe9af996/whereabouts-cni-bincopy/0.log" Jul 10 03:28:55.333551 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:55.333518 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tfrkp_ff323ccb-9efc-43fd-8ce7-9365fe9af996/whereabouts-cni/0.log" Jul 10 03:28:55.471256 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:55.471182 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-bgbpr_7d037586-6a56-4354-ba36-a40c2cc2ab48/network-metrics-daemon/0.log" Jul 10 03:28:55.487819 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:55.487786 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-bgbpr_7d037586-6a56-4354-ba36-a40c2cc2ab48/kube-rbac-proxy/0.log" Jul 10 03:28:56.420935 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:56.420896 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-25tvc_b93349d9-10f0-430c-b00f-74f141f508de/ovn-controller/0.log" Jul 10 03:28:56.442256 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:56.442226 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-25tvc_b93349d9-10f0-430c-b00f-74f141f508de/ovn-acl-logging/0.log" Jul 10 03:28:56.471691 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:56.471659 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-25tvc_b93349d9-10f0-430c-b00f-74f141f508de/kube-rbac-proxy-node/0.log" Jul 10 03:28:56.499577 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:56.499546 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-25tvc_b93349d9-10f0-430c-b00f-74f141f508de/kube-rbac-proxy-ovn-metrics/0.log" Jul 10 03:28:56.528540 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:56.528514 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-25tvc_b93349d9-10f0-430c-b00f-74f141f508de/northd/0.log" Jul 10 03:28:56.562889 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:56.562860 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-25tvc_b93349d9-10f0-430c-b00f-74f141f508de/nbdb/0.log" Jul 10 03:28:56.591074 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:56.591047 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-25tvc_b93349d9-10f0-430c-b00f-74f141f508de/sbdb/0.log" Jul 10 03:28:56.696143 ip-10-0-130-139 kubenswrapper[2575]: I0710 03:28:56.696072 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-25tvc_b93349d9-10f0-430c-b00f-74f141f508de/ovnkube-controller/0.log"