Jul 14 09:24:50.886770 ip-10-0-132-2 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 14 09:24:50.886907 ip-10-0-132-2 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 14 09:24:50.887005 ip-10-0-132-2 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 14 09:24:50.887464 ip-10-0-132-2 systemd[1]: Failed to start Kubernetes Kubelet. Jul 14 09:25:00.670505 ip-10-0-132-2 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 14 09:25:00.670518 ip-10-0-132-2 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 49677f5c5ff846d2ade6fedce93568e6 -- Jul 14 09:26:50.392096 ip-10-0-132-2 systemd[1]: Starting Kubernetes Kubelet... Jul 14 09:26:50.736765 ip-10-0-132-2 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 09:26:50.736765 ip-10-0-132-2 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 14 09:26:50.736765 ip-10-0-132-2 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 09:26:50.736765 ip-10-0-132-2 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 14 09:26:50.736765 ip-10-0-132-2 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 09:26:50.737742 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.737593 2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 14 09:26:50.741592 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.741576 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 09:26:50.741592 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.741591 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 09:26:50.741663 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.741596 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 09:26:50.741663 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.741600 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 09:26:50.741663 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.741603 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 09:26:50.741663 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.741605 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 09:26:50.741663 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.741608 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 09:26:50.741663 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.741611 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 09:26:50.741663 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.741614 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 09:26:50.741663 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.741617 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 09:26:50.741663 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.741620 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 09:26:50.741663 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.741622 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 09:26:50.741663 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.741625 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 09:26:50.741663 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.741628 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 09:26:50.741663 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.741632 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 09:26:50.741663 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.741634 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 09:26:50.741663 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.741637 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 09:26:50.741663 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.741640 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 09:26:50.741663 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.741643 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 09:26:50.741663 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.741646 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 09:26:50.741663 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.741649 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 09:26:50.741663 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.741651 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 09:26:50.743083 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743073 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 09:26:50.743083 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743084 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 09:26:50.743153 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743089 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 09:26:50.743153 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743093 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 09:26:50.743153 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743096 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 09:26:50.743153 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743099 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 09:26:50.743153 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743103 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 09:26:50.743153 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743106 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 09:26:50.743153 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743109 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 09:26:50.743153 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743111 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 14 09:26:50.743153 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743114 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 09:26:50.743153 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743117 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 09:26:50.743153 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743120 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 09:26:50.743153 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743123 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 09:26:50.743153 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743126 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 09:26:50.743153 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743129 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 09:26:50.743153 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743132 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 09:26:50.743153 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743135 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 09:26:50.743153 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743138 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 09:26:50.743153 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743140 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 09:26:50.743153 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743143 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 09:26:50.743153 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743145 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 09:26:50.743668 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743149 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 09:26:50.743668 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743151 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 09:26:50.743668 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743154 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 09:26:50.743668 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743157 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 09:26:50.743668 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743160 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 09:26:50.743668 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743163 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 09:26:50.743668 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743169 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 09:26:50.743668 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743173 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 09:26:50.743668 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743177 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 09:26:50.743668 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743179 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 09:26:50.743668 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743182 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 09:26:50.743668 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743185 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 09:26:50.743668 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743187 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 09:26:50.743668 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743190 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 09:26:50.743668 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743193 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 09:26:50.743668 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743196 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 09:26:50.743668 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743199 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 09:26:50.743668 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743201 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 09:26:50.743668 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743204 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 09:26:50.743668 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743207 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 09:26:50.744195 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743209 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 09:26:50.744195 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743212 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 09:26:50.744195 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743215 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 09:26:50.744195 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743217 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 09:26:50.744195 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743220 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 09:26:50.744195 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743223 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 09:26:50.744195 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743227 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 09:26:50.744195 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743231 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 09:26:50.744195 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743234 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 09:26:50.744195 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743237 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 09:26:50.744195 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743240 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 09:26:50.744195 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743243 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 09:26:50.744195 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743247 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 09:26:50.744195 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743251 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 09:26:50.744195 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743254 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 09:26:50.744195 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743257 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 09:26:50.744195 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743260 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 09:26:50.744195 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743262 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 09:26:50.744195 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743265 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 09:26:50.744679 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743268 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 09:26:50.744679 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743271 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 09:26:50.744679 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743274 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 09:26:50.744679 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743278 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 09:26:50.744679 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743281 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 09:26:50.744679 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743284 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 09:26:50.744679 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743287 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 09:26:50.744679 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743290 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 09:26:50.744679 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743293 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 09:26:50.744679 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743296 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 09:26:50.744679 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743300 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 09:26:50.744679 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743303 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 09:26:50.744679 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743306 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 09:26:50.744679 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743310 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 09:26:50.744679 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743312 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 09:26:50.744679 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.743315 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 09:26:50.744679 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744089 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 09:26:50.744679 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744095 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 09:26:50.744679 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744100 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 09:26:50.745193 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744103 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 09:26:50.745193 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744107 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 09:26:50.745193 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744110 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 09:26:50.745193 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744113 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 09:26:50.745193 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744116 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 09:26:50.745193 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744119 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 09:26:50.745193 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744123 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 09:26:50.745193 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744126 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 09:26:50.745193 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744129 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 09:26:50.745193 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744132 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 09:26:50.745193 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744134 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 09:26:50.745193 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744137 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 09:26:50.745193 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744140 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 09:26:50.745193 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744143 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 09:26:50.745193 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744146 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 09:26:50.745193 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744149 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 09:26:50.745193 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744151 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 09:26:50.745193 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744154 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 09:26:50.745193 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744157 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 09:26:50.745193 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744160 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 09:26:50.745720 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744163 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 09:26:50.745720 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744166 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 09:26:50.745720 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744169 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 09:26:50.745720 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744172 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 09:26:50.745720 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744174 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 09:26:50.745720 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744177 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 09:26:50.745720 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744180 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 09:26:50.745720 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744183 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 09:26:50.745720 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744185 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 09:26:50.745720 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744188 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 09:26:50.745720 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744191 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 09:26:50.745720 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744194 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 09:26:50.745720 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744197 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 09:26:50.745720 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744200 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 09:26:50.745720 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744203 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 09:26:50.745720 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744207 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 09:26:50.745720 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744210 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 09:26:50.745720 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744213 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 09:26:50.745720 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744215 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 09:26:50.745720 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744218 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 09:26:50.746269 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744221 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 09:26:50.746269 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744224 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 09:26:50.746269 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744227 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 09:26:50.746269 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744230 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 09:26:50.746269 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744232 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 09:26:50.746269 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744235 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 09:26:50.746269 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744237 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 09:26:50.746269 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744240 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 09:26:50.746269 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744243 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 09:26:50.746269 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744246 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 09:26:50.746269 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744249 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 09:26:50.746269 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744252 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 09:26:50.746269 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744254 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 09:26:50.746269 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744257 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 09:26:50.746269 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744260 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 09:26:50.746269 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744264 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 09:26:50.746269 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744266 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 09:26:50.746269 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744269 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 09:26:50.746269 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744273 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 09:26:50.746767 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744277 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 09:26:50.746767 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744280 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 09:26:50.746767 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744284 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 09:26:50.746767 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744287 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 09:26:50.746767 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744290 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 09:26:50.746767 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744294 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 09:26:50.746767 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744296 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 09:26:50.746767 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744299 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 09:26:50.746767 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744303 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 09:26:50.746767 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744307 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 09:26:50.746767 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744310 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 09:26:50.746767 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744314 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 09:26:50.746767 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744317 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 09:26:50.746767 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744320 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 14 09:26:50.746767 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744324 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 09:26:50.746767 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744329 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 09:26:50.746767 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744332 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 09:26:50.746767 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744336 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 09:26:50.746767 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744339 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 09:26:50.747311 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744342 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 09:26:50.747311 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744345 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 09:26:50.747311 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744348 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 09:26:50.747311 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744351 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 09:26:50.747311 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744354 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 09:26:50.747311 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744357 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 09:26:50.747311 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744360 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 09:26:50.747311 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744363 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 09:26:50.747311 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744365 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 09:26:50.747311 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744369 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 09:26:50.747311 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744371 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 09:26:50.747311 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744374 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 09:26:50.747311 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744376 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 09:26:50.747311 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744379 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 09:26:50.747311 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744381 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 09:26:50.747311 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744384 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 09:26:50.747311 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744387 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 09:26:50.747311 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744389 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 09:26:50.747311 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744801 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 09:26:50.747311 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744806 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 09:26:50.747827 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744810 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 09:26:50.747827 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744813 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 09:26:50.747827 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744817 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 09:26:50.747827 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744820 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 09:26:50.747827 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744823 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 09:26:50.747827 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744826 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 09:26:50.747827 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744829 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 09:26:50.747827 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744832 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 09:26:50.747827 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744835 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 09:26:50.747827 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744838 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 09:26:50.747827 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744841 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 09:26:50.747827 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744844 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 09:26:50.747827 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744846 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 09:26:50.747827 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744849 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 09:26:50.747827 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744852 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 09:26:50.747827 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744855 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 09:26:50.747827 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744859 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 09:26:50.747827 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744873 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 09:26:50.747827 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744877 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 09:26:50.748452 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744894 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 09:26:50.748452 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744897 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 09:26:50.748452 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744901 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 09:26:50.748452 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744904 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 09:26:50.748452 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744907 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 09:26:50.748452 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744910 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 14 09:26:50.748452 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744913 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 09:26:50.748452 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744916 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 09:26:50.748452 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744919 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 09:26:50.748452 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744922 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 09:26:50.748452 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744925 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 09:26:50.748452 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744928 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 09:26:50.748452 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744931 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 09:26:50.748452 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744934 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 09:26:50.748452 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744937 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 09:26:50.748452 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744939 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 09:26:50.748452 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744943 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 09:26:50.748452 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744946 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 09:26:50.748452 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744949 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 09:26:50.749011 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744952 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 09:26:50.749011 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744956 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 09:26:50.749011 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744959 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 09:26:50.749011 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744962 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 09:26:50.749011 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744965 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 09:26:50.749011 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744968 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 09:26:50.749011 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744970 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 09:26:50.749011 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744973 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 09:26:50.749011 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744975 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 09:26:50.749011 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744978 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 09:26:50.749011 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744981 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 09:26:50.749011 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744984 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 09:26:50.749011 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744986 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 09:26:50.749011 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744989 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 09:26:50.749011 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744992 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 09:26:50.749011 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744995 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 09:26:50.749011 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.744997 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 09:26:50.749011 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745000 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 09:26:50.749011 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745003 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 09:26:50.749011 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745008 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 09:26:50.749543 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745011 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 09:26:50.749543 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745015 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 09:26:50.749543 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745017 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 09:26:50.749543 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745020 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 09:26:50.749543 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745023 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 09:26:50.749543 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745027 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 09:26:50.749543 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745030 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 09:26:50.749543 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745033 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 09:26:50.749543 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745036 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 09:26:50.749543 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745040 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 09:26:50.749543 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745044 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 09:26:50.749543 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745047 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 09:26:50.749543 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745050 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 09:26:50.749543 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745053 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 09:26:50.749543 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745056 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 09:26:50.749543 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745059 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 09:26:50.749543 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745061 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 09:26:50.749543 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745064 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 09:26:50.749543 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745067 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 09:26:50.749543 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745069 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 09:26:50.750081 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745072 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 09:26:50.750081 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745075 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 09:26:50.750081 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745077 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 09:26:50.750081 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745080 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 09:26:50.750081 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745083 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 09:26:50.750081 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745086 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 09:26:50.750081 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745089 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 09:26:50.750081 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745092 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 09:26:50.750081 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745094 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 09:26:50.750081 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745097 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 09:26:50.750081 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745100 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 09:26:50.750081 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745102 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 09:26:50.750081 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745105 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 09:26:50.750081 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745107 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 09:26:50.750081 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745111 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 09:26:50.750081 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745113 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 09:26:50.750081 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745116 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 09:26:50.750081 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745120 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 09:26:50.750081 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745122 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 09:26:50.750081 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745203 2565 flags.go:64] FLAG: --address="0.0.0.0" Jul 14 09:26:50.750081 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745213 2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 14 09:26:50.750662 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745223 2565 flags.go:64] FLAG: --anonymous-auth="true" Jul 14 09:26:50.750662 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745230 2565 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 14 09:26:50.750662 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745239 2565 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 14 09:26:50.750662 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745243 2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 14 09:26:50.750662 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745248 2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 14 09:26:50.750662 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745252 2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 14 09:26:50.750662 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745255 2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 14 09:26:50.750662 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745259 2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 14 09:26:50.750662 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745262 2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 14 09:26:50.750662 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745266 2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 14 09:26:50.750662 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745269 2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 14 09:26:50.750662 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745272 2565 flags.go:64] FLAG: --cgroup-root="" Jul 14 09:26:50.750662 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745276 2565 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 14 09:26:50.750662 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745279 2565 flags.go:64] FLAG: --client-ca-file="" Jul 14 09:26:50.750662 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745282 2565 flags.go:64] FLAG: --cloud-provider="external" Jul 14 09:26:50.750662 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745285 2565 flags.go:64] FLAG: --cluster-dns="[]" Jul 14 09:26:50.750662 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745291 2565 flags.go:64] FLAG: --cluster-domain="" Jul 14 09:26:50.750662 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745294 2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 14 09:26:50.750662 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745297 2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 14 09:26:50.750662 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745301 2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 14 09:26:50.750662 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745304 2565 flags.go:64] FLAG: --container-log-max-files="5" Jul 14 09:26:50.750662 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745308 2565 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 14 09:26:50.750662 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745311 2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 14 09:26:50.751264 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745315 2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 14 09:26:50.751264 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745318 2565 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 14 09:26:50.751264 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745321 2565 flags.go:64] FLAG: --contention-profiling="false" Jul 14 09:26:50.751264 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745324 2565 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 14 09:26:50.751264 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745328 2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 14 09:26:50.751264 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745334 2565 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 14 09:26:50.751264 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745337 2565 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 14 09:26:50.751264 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745342 2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 14 09:26:50.751264 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745345 2565 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 14 09:26:50.751264 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745349 2565 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 14 09:26:50.751264 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745352 2565 flags.go:64] FLAG: --enable-load-reader="false" Jul 14 09:26:50.751264 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745355 2565 flags.go:64] FLAG: --enable-server="true" Jul 14 09:26:50.751264 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745358 2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 14 09:26:50.751264 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745364 2565 flags.go:64] FLAG: --event-burst="100" Jul 14 09:26:50.751264 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745367 2565 flags.go:64] FLAG: --event-qps="50" Jul 14 09:26:50.751264 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745370 2565 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 14 09:26:50.751264 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745374 2565 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 14 09:26:50.751264 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745377 2565 flags.go:64] FLAG: --eviction-hard="" Jul 14 09:26:50.751264 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745381 2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 14 09:26:50.751264 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745384 2565 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 14 09:26:50.751264 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745387 2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 14 09:26:50.751264 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745390 2565 flags.go:64] FLAG: --eviction-soft="" Jul 14 09:26:50.751264 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745393 2565 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 14 09:26:50.751264 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745396 2565 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 14 09:26:50.751264 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745400 2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 14 09:26:50.751964 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745403 2565 flags.go:64] FLAG: --experimental-mounter-path="" Jul 14 09:26:50.751964 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745406 2565 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 14 09:26:50.751964 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745409 2565 flags.go:64] FLAG: --fail-swap-on="true" Jul 14 09:26:50.751964 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745414 2565 flags.go:64] FLAG: --feature-gates="" Jul 14 09:26:50.751964 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745418 2565 flags.go:64] FLAG: --file-check-frequency="20s" Jul 14 09:26:50.751964 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745422 2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 14 09:26:50.751964 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745425 2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 14 09:26:50.751964 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745428 2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 14 09:26:50.751964 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745431 2565 flags.go:64] FLAG: --healthz-port="10248" Jul 14 09:26:50.751964 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745435 2565 flags.go:64] FLAG: --help="false" Jul 14 09:26:50.751964 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745438 2565 flags.go:64] FLAG: --hostname-override="ip-10-0-132-2.ec2.internal" Jul 14 09:26:50.751964 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745441 2565 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 14 09:26:50.751964 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745446 2565 flags.go:64] FLAG: --http-check-frequency="20s" Jul 14 09:26:50.751964 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745449 2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 14 09:26:50.751964 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745453 2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 14 09:26:50.751964 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745457 2565 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 14 09:26:50.751964 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745460 2565 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 14 09:26:50.751964 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745463 2565 flags.go:64] FLAG: --image-service-endpoint="" Jul 14 09:26:50.751964 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745466 2565 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 14 09:26:50.751964 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745469 2565 flags.go:64] FLAG: --kube-api-burst="100" Jul 14 09:26:50.751964 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745473 2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 14 09:26:50.751964 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745476 2565 flags.go:64] FLAG: --kube-api-qps="50" Jul 14 09:26:50.751964 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745479 2565 flags.go:64] FLAG: --kube-reserved="" Jul 14 09:26:50.751964 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745482 2565 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 14 09:26:50.752594 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745485 2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 14 09:26:50.752594 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745489 2565 flags.go:64] FLAG: --kubelet-cgroups="" Jul 14 09:26:50.752594 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745492 2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 14 09:26:50.752594 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745495 2565 flags.go:64] FLAG: --lock-file="" Jul 14 09:26:50.752594 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745498 2565 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 14 09:26:50.752594 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745501 2565 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 14 09:26:50.752594 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745504 2565 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 14 09:26:50.752594 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745509 2565 flags.go:64] FLAG: --log-json-split-stream="false" Jul 14 09:26:50.752594 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745512 2565 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 14 09:26:50.752594 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745515 2565 flags.go:64] FLAG: --log-text-split-stream="false" Jul 14 09:26:50.752594 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745518 2565 flags.go:64] FLAG: --logging-format="text" Jul 14 09:26:50.752594 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745523 2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 14 09:26:50.752594 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745526 2565 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 14 09:26:50.752594 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745530 2565 flags.go:64] FLAG: --manifest-url="" Jul 14 09:26:50.752594 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745533 2565 flags.go:64] FLAG: --manifest-url-header="" Jul 14 09:26:50.752594 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745537 2565 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 14 09:26:50.752594 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745540 2565 flags.go:64] FLAG: --max-open-files="1000000" Jul 14 09:26:50.752594 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745545 2565 flags.go:64] FLAG: --max-pods="110" Jul 14 09:26:50.752594 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745548 2565 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 14 09:26:50.752594 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745551 2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 14 09:26:50.752594 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745556 2565 flags.go:64] FLAG: --memory-manager-policy="None" Jul 14 09:26:50.752594 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745560 2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 14 09:26:50.752594 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745563 2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 14 09:26:50.752594 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745566 2565 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 14 09:26:50.752594 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745569 2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 14 09:26:50.753263 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745576 2565 flags.go:64] FLAG: --node-status-max-images="50" Jul 14 09:26:50.753263 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745580 2565 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 14 09:26:50.753263 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745583 2565 flags.go:64] FLAG: --oom-score-adj="-999" Jul 14 09:26:50.753263 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745586 2565 flags.go:64] FLAG: --pod-cidr="" Jul 14 09:26:50.753263 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745590 2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 14 09:26:50.753263 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745596 2565 flags.go:64] FLAG: --pod-manifest-path="" Jul 14 09:26:50.753263 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745599 2565 flags.go:64] FLAG: --pod-max-pids="-1" Jul 14 09:26:50.753263 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745602 2565 flags.go:64] FLAG: --pods-per-core="0" Jul 14 09:26:50.753263 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745606 2565 flags.go:64] FLAG: --port="10250" Jul 14 09:26:50.753263 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745609 2565 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 14 09:26:50.753263 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745612 2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d3f73d75d667be28" Jul 14 09:26:50.753263 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745615 2565 flags.go:64] FLAG: --qos-reserved="" Jul 14 09:26:50.753263 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745618 2565 flags.go:64] FLAG: --read-only-port="10255" Jul 14 09:26:50.753263 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745622 2565 flags.go:64] FLAG: --register-node="true" Jul 14 09:26:50.753263 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745624 2565 flags.go:64] FLAG: --register-with-taints="" Jul 14 09:26:50.753263 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745629 2565 flags.go:64] FLAG: --registry-burst="10" Jul 14 09:26:50.753263 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745632 2565 flags.go:64] FLAG: --registry-qps="5" Jul 14 09:26:50.753263 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745635 2565 flags.go:64] FLAG: --reserved-cpus="" Jul 14 09:26:50.753263 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745640 2565 flags.go:64] FLAG: --reserved-memory="" Jul 14 09:26:50.753263 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745644 2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 14 09:26:50.753263 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745647 2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 14 09:26:50.753263 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745650 2565 flags.go:64] FLAG: --rotate-certificates="false" Jul 14 09:26:50.753263 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745653 2565 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 14 09:26:50.753263 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745656 2565 flags.go:64] FLAG: --runonce="false" Jul 14 09:26:50.753263 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745659 2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 14 09:26:50.753967 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745662 2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 14 09:26:50.753967 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745665 2565 flags.go:64] FLAG: --seccomp-default="false" Jul 14 09:26:50.753967 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745668 2565 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 14 09:26:50.753967 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745673 2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 14 09:26:50.753967 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745676 2565 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 14 09:26:50.753967 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745680 2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 14 09:26:50.753967 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745683 2565 flags.go:64] FLAG: --storage-driver-password="root" Jul 14 09:26:50.753967 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745686 2565 flags.go:64] FLAG: --storage-driver-secure="false" Jul 14 09:26:50.753967 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745689 2565 flags.go:64] FLAG: --storage-driver-table="stats" Jul 14 09:26:50.753967 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745692 2565 flags.go:64] FLAG: --storage-driver-user="root" Jul 14 09:26:50.753967 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745695 2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 14 09:26:50.753967 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745698 2565 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 14 09:26:50.753967 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745702 2565 flags.go:64] FLAG: --system-cgroups="" Jul 14 09:26:50.753967 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745705 2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 14 09:26:50.753967 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745710 2565 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 14 09:26:50.753967 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745713 2565 flags.go:64] FLAG: --tls-cert-file="" Jul 14 09:26:50.753967 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745716 2565 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 14 09:26:50.753967 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745720 2565 flags.go:64] FLAG: --tls-min-version="" Jul 14 09:26:50.753967 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745723 2565 flags.go:64] FLAG: --tls-private-key-file="" Jul 14 09:26:50.753967 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745726 2565 flags.go:64] FLAG: --topology-manager-policy="none" Jul 14 09:26:50.753967 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745729 2565 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 14 09:26:50.753967 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745733 2565 flags.go:64] FLAG: --topology-manager-scope="container" Jul 14 09:26:50.753967 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745736 2565 flags.go:64] FLAG: --v="2" Jul 14 09:26:50.753967 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745740 2565 flags.go:64] FLAG: --version="false" Jul 14 09:26:50.753967 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745744 2565 flags.go:64] FLAG: --vmodule="" Jul 14 09:26:50.754638 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745749 2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 14 09:26:50.754638 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.745752 2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 14 09:26:50.754638 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745918 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 09:26:50.754638 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745924 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 09:26:50.754638 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745928 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 09:26:50.754638 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745932 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 09:26:50.754638 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745934 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 09:26:50.754638 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745937 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 09:26:50.754638 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745940 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 09:26:50.754638 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745943 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 09:26:50.754638 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745947 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 09:26:50.754638 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745951 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 09:26:50.754638 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745955 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 09:26:50.754638 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745959 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 09:26:50.754638 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745962 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 09:26:50.754638 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745965 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 09:26:50.754638 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745967 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 09:26:50.754638 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745970 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 09:26:50.754638 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745973 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 09:26:50.755155 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745976 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 09:26:50.755155 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745979 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 09:26:50.755155 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745982 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 09:26:50.755155 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745985 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 09:26:50.755155 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745987 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 09:26:50.755155 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745990 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 09:26:50.755155 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745993 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 09:26:50.755155 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745996 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 09:26:50.755155 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.745999 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 09:26:50.755155 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746002 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 09:26:50.755155 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746005 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 09:26:50.755155 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746007 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 09:26:50.755155 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746010 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 09:26:50.755155 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746014 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 09:26:50.755155 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746017 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 09:26:50.755155 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746020 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 09:26:50.755155 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746023 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 09:26:50.755155 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746025 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 09:26:50.755155 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746028 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 09:26:50.755155 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746031 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 09:26:50.755675 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746034 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 09:26:50.755675 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746037 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 09:26:50.755675 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746039 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 09:26:50.755675 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746044 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 09:26:50.755675 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746046 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 09:26:50.755675 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746050 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 09:26:50.755675 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746052 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 09:26:50.755675 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746055 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 09:26:50.755675 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746058 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 09:26:50.755675 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746060 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 09:26:50.755675 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746063 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 09:26:50.755675 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746066 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 09:26:50.755675 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746069 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 09:26:50.755675 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746071 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 09:26:50.755675 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746075 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 09:26:50.755675 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746078 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 09:26:50.755675 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746081 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 09:26:50.755675 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746084 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 09:26:50.755675 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746087 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 09:26:50.755675 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746091 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 09:26:50.756282 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746094 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 09:26:50.756282 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746097 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 09:26:50.756282 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746101 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 09:26:50.756282 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746103 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 09:26:50.756282 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746106 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 09:26:50.756282 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746111 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 09:26:50.756282 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746113 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 14 09:26:50.756282 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746116 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 09:26:50.756282 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746119 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 09:26:50.756282 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746122 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 09:26:50.756282 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746124 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 09:26:50.756282 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746127 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 09:26:50.756282 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746130 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 09:26:50.756282 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746133 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 09:26:50.756282 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746136 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 09:26:50.756282 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746140 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 09:26:50.756282 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746142 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 09:26:50.756282 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746145 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 09:26:50.756282 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746148 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 09:26:50.756282 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746151 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 09:26:50.756802 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746153 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 09:26:50.756802 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746156 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 09:26:50.756802 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746159 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 09:26:50.756802 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746161 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 09:26:50.756802 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746164 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 09:26:50.756802 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746167 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 09:26:50.756802 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746170 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 09:26:50.756802 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746173 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 09:26:50.756802 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746176 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 09:26:50.756802 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746179 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 09:26:50.756802 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746181 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 09:26:50.756802 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746184 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 09:26:50.756802 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746187 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 09:26:50.756802 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746190 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 09:26:50.756802 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746193 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 09:26:50.756802 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746195 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 09:26:50.756802 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746198 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 09:26:50.756802 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746204 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 09:26:50.756802 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746207 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 09:26:50.757306 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746210 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 09:26:50.757306 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746214 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 09:26:50.757306 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.746216 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 09:26:50.757306 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.746769 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 09:26:50.757306 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.754874 2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 14 09:26:50.757306 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.754921 2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 14 09:26:50.757306 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.754946 2565 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 14 09:26:50.757306 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.754952 2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 14 09:26:50.757306 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755024 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 09:26:50.757306 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755030 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 09:26:50.757306 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755034 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 09:26:50.757306 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755038 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 09:26:50.757306 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755041 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 09:26:50.757306 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755044 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 09:26:50.757306 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755046 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 09:26:50.757713 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755049 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 09:26:50.757713 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755052 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 09:26:50.757713 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755054 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 09:26:50.757713 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755058 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 09:26:50.757713 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755060 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 09:26:50.757713 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755071 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 09:26:50.757713 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755075 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 09:26:50.757713 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755077 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 09:26:50.757713 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755080 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 09:26:50.757713 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755083 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 09:26:50.757713 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755086 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 09:26:50.757713 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755089 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 09:26:50.757713 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755091 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 09:26:50.757713 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755094 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 09:26:50.757713 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755097 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 09:26:50.757713 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755100 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 09:26:50.757713 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755104 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 09:26:50.757713 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755106 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 09:26:50.757713 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755113 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 09:26:50.757713 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755116 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 09:26:50.758243 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755119 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 09:26:50.758243 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755122 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 09:26:50.758243 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755125 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 09:26:50.758243 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755129 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 09:26:50.758243 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755133 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 09:26:50.758243 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755138 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 09:26:50.758243 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755142 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 09:26:50.758243 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755146 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 09:26:50.758243 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755149 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 09:26:50.758243 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755152 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 09:26:50.758243 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755155 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 09:26:50.758243 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755158 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 09:26:50.758243 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755161 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 09:26:50.758243 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755164 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 09:26:50.758243 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755167 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 09:26:50.758243 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755170 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 09:26:50.758243 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755173 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 09:26:50.758243 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755182 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 09:26:50.758243 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755191 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 09:26:50.758733 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755194 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 09:26:50.758733 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755197 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 09:26:50.758733 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755200 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 09:26:50.758733 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755202 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 09:26:50.758733 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755205 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 09:26:50.758733 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755208 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 09:26:50.758733 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755210 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 09:26:50.758733 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755213 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 09:26:50.758733 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755216 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 09:26:50.758733 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755219 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 09:26:50.758733 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755222 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 09:26:50.758733 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755224 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 09:26:50.758733 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755227 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 09:26:50.758733 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755230 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 09:26:50.758733 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755232 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 09:26:50.758733 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755235 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 09:26:50.758733 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755238 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 09:26:50.758733 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755240 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 09:26:50.758733 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755243 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 09:26:50.758733 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755246 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 09:26:50.759261 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755249 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 09:26:50.759261 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755251 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 09:26:50.759261 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755254 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 09:26:50.759261 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755257 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 09:26:50.759261 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755260 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 09:26:50.759261 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755262 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 09:26:50.759261 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755265 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 09:26:50.759261 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755268 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 09:26:50.759261 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755270 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 09:26:50.759261 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755273 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 09:26:50.759261 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755276 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 09:26:50.759261 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755279 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 09:26:50.759261 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755290 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 09:26:50.759261 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755292 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 09:26:50.759261 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755295 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 09:26:50.759261 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755298 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 09:26:50.759261 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755301 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 09:26:50.759261 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755303 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 09:26:50.759261 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755306 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 09:26:50.759261 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755308 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 09:26:50.759768 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755311 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 09:26:50.759768 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755314 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 09:26:50.759768 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755316 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 09:26:50.759768 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755321 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 09:26:50.759768 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755324 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 09:26:50.759768 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755327 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 09:26:50.759768 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755330 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 09:26:50.759768 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755333 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 09:26:50.759768 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755336 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 09:26:50.759768 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755339 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 14 09:26:50.759768 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755342 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 09:26:50.759768 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755345 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 09:26:50.759768 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755348 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 09:26:50.759768 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.755354 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 09:26:50.759768 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755529 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 09:26:50.760292 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755535 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 09:26:50.760292 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755538 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 09:26:50.760292 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755541 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 09:26:50.760292 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755544 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 09:26:50.760292 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755547 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 09:26:50.760292 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755551 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 09:26:50.760292 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755553 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 09:26:50.760292 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755557 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 09:26:50.760292 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755560 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 09:26:50.760292 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755563 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 09:26:50.760292 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755572 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 09:26:50.760292 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755575 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 09:26:50.760292 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755578 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 09:26:50.760292 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755580 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 09:26:50.760292 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755583 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 09:26:50.760292 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755586 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 09:26:50.760292 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755588 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 09:26:50.760292 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755591 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 09:26:50.760292 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755594 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 09:26:50.760292 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755597 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 09:26:50.760800 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755600 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 09:26:50.760800 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755603 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 09:26:50.760800 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755605 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 09:26:50.760800 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755608 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 09:26:50.760800 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755611 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 14 09:26:50.760800 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755613 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 09:26:50.760800 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755616 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 09:26:50.760800 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755618 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 09:26:50.760800 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755621 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 09:26:50.760800 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755624 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 09:26:50.760800 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755627 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 09:26:50.760800 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755630 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 09:26:50.760800 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755632 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 09:26:50.760800 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755635 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 09:26:50.760800 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755639 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 09:26:50.760800 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755641 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 09:26:50.760800 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755644 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 09:26:50.760800 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755647 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 09:26:50.760800 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755649 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 09:26:50.761308 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755652 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 09:26:50.761308 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755655 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 09:26:50.761308 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755657 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 09:26:50.761308 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755660 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 09:26:50.761308 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755669 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 09:26:50.761308 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755671 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 09:26:50.761308 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755674 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 09:26:50.761308 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755677 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 09:26:50.761308 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755680 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 09:26:50.761308 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755683 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 09:26:50.761308 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755687 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 09:26:50.761308 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755691 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 09:26:50.761308 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755694 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 09:26:50.761308 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755697 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 09:26:50.761308 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755700 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 09:26:50.761308 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755703 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 09:26:50.761308 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755708 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 09:26:50.761308 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755710 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 09:26:50.761308 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755714 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 09:26:50.761791 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755716 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 09:26:50.761791 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755719 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 09:26:50.761791 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755722 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 09:26:50.761791 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755724 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 09:26:50.761791 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755727 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 09:26:50.761791 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755730 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 09:26:50.761791 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755733 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 09:26:50.761791 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755735 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 09:26:50.761791 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755738 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 09:26:50.761791 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755741 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 09:26:50.761791 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755743 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 09:26:50.761791 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755746 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 09:26:50.761791 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755749 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 09:26:50.761791 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755751 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 09:26:50.761791 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755754 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 09:26:50.761791 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755757 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 09:26:50.761791 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755761 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 09:26:50.761791 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755764 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 09:26:50.761791 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755773 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 09:26:50.761791 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755777 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 09:26:50.762323 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755780 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 09:26:50.762323 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755783 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 09:26:50.762323 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755786 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 09:26:50.762323 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755789 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 09:26:50.762323 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755792 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 09:26:50.762323 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755795 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 09:26:50.762323 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755798 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 09:26:50.762323 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755802 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 09:26:50.762323 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755805 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 09:26:50.762323 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755810 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 09:26:50.762323 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755813 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 09:26:50.762323 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755816 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 09:26:50.762323 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755819 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 09:26:50.762323 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755822 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 09:26:50.762323 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755824 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 09:26:50.762323 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755827 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 09:26:50.762323 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755830 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 09:26:50.762323 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755833 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 09:26:50.762323 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755835 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 09:26:50.762805 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:50.755838 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 09:26:50.762805 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.755843 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 09:26:50.762805 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.755982 2565 server.go:962] "Client rotation is on, will bootstrap in background" Jul 14 09:26:50.762805 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.758985 2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 14 09:26:50.762805 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.761551 2565 server.go:1019] "Starting client certificate rotation" Jul 14 09:26:50.762805 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.761666 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 14 09:26:50.762805 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.761709 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 14 09:26:50.765968 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.765898 2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 14 09:26:50.768454 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.768436 2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 14 09:26:50.781414 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.781389 2565 log.go:25] "Validated CRI v1 runtime API" Jul 14 09:26:50.783680 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.783665 2565 log.go:25] "Validated CRI v1 image API" Jul 14 09:26:50.785152 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.785137 2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 14 09:26:50.787351 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.787324 2565 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b999fd11-abaa-4873-9a86-acf367e5a25d:/dev/nvme0n1p4 c431b919-06f4-472a-9893-9ed9fc5f14be:/dev/nvme0n1p3] Jul 14 09:26:50.787403 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.787351 2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 14 09:26:50.792216 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.792198 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 14 09:26:50.792565 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.792450 2565 manager.go:216] Machine: {Timestamp:2026-07-14 09:26:50.791276128 +0000 UTC m=+0.300507570 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100310 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2bfc2f04dd5ab49f7d55e17eafc35c SystemUUID:ec2bfc2f-04dd-5ab4-9f7d-55e17eafc35c BootID:49677f5c-5ff8-46d2-ade6-fedce93568e6 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:c1:8e:bf:1e:9f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:c1:8e:bf:1e:9f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9e:ed:73:a8:76:4c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 14 09:26:50.792565 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.792563 2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 14 09:26:50.792662 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.792650 2565 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 14 09:26:50.795660 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.795636 2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 14 09:26:50.795803 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.795663 2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-2.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 14 09:26:50.796468 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.796458 2565 topology_manager.go:138] "Creating topology manager with none policy" Jul 14 09:26:50.796546 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.796471 2565 container_manager_linux.go:309] "Creating device plugin manager" Jul 14 09:26:50.796546 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.796479 2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 14 09:26:50.796546 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.796492 2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 14 09:26:50.797930 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.797919 2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 14 09:26:50.802087 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.802075 2565 state_mem.go:36] "Initialized new in-memory state store" Jul 14 09:26:50.802216 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.802200 2565 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 14 09:26:50.802250 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.802240 2565 kubelet.go:486] "Attempting to sync node with API server" Jul 14 09:26:50.802288 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.802251 2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 14 09:26:50.802288 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.802273 2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 14 09:26:50.802288 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.802285 2565 kubelet.go:398] "Adding apiserver pod source" Jul 14 09:26:50.802372 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.802294 2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 14 09:26:50.802798 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.802786 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 14 09:26:50.802851 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.802808 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 14 09:26:50.804954 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.804938 2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 14 09:26:50.807218 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.807201 2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 14 09:26:50.807271 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.807251 2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 14 09:26:50.807520 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.807508 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 14 09:26:50.807559 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.807526 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 14 09:26:50.807559 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.807533 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 14 09:26:50.807559 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.807542 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 14 09:26:50.807559 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.807553 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 14 09:26:50.807559 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.807559 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 14 09:26:50.807840 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.807830 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 14 09:26:50.807871 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.807842 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 14 09:26:50.807871 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.807850 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 14 09:26:50.807871 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.807856 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 14 09:26:50.807977 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.807878 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 14 09:26:50.807977 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.807900 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 14 09:26:50.808569 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.808559 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 14 09:26:50.808604 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.808571 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 14 09:26:50.812681 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.812613 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8jqf6" Jul 14 09:26:50.814213 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.814198 2565 server.go:1268] "Started kubelet" Jul 14 09:26:50.814401 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.814348 2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 14 09:26:50.814486 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.814390 2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 14 09:26:50.814486 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.814462 2565 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 14 09:26:50.814486 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.814475 2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-2.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 14 09:26:50.814636 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:50.814559 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-2.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 14 09:26:50.814724 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:50.814680 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 14 09:26:50.815117 ip-10-0-132-2 systemd[1]: Started Kubernetes Kubelet. Jul 14 09:26:50.815963 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.815944 2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 14 09:26:50.816592 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.816576 2565 server.go:310] "Adding debug handlers to kubelet server" Jul 14 09:26:50.818569 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:50.818549 2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 14 09:26:50.819753 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:50.818889 2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-2.ec2.internal.18c21dd6ce565b97 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-2.ec2.internal,UID:ip-10-0-132-2.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-2.ec2.internal,},FirstTimestamp:2026-07-14 09:26:50.814110615 +0000 UTC m=+0.323342056,LastTimestamp:2026-07-14 09:26:50.814110615 +0000 UTC m=+0.323342056,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-2.ec2.internal,}" Jul 14 09:26:50.821108 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.821093 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8jqf6" Jul 14 09:26:50.822377 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.822359 2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 14 09:26:50.822377 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.822370 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 14 09:26:50.822951 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:50.822720 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-2.ec2.internal\" not found" Jul 14 09:26:50.823046 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.822959 2565 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 14 09:26:50.823046 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.822984 2565 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 14 09:26:50.823046 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.823042 2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 14 09:26:50.823205 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.823119 2565 reconstruct.go:97] "Volume reconstruction finished" Jul 14 09:26:50.823205 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.823147 2565 reconciler.go:29] "Reconciler: start to sync state" Jul 14 09:26:50.823395 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.823378 2565 factory.go:55] Registering systemd factory Jul 14 09:26:50.823458 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.823398 2565 factory.go:223] Registration of the systemd container factory successfully Jul 14 09:26:50.823639 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.823622 2565 factory.go:153] Registering CRI-O factory Jul 14 09:26:50.823639 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.823640 2565 factory.go:223] Registration of the crio container factory successfully Jul 14 09:26:50.823751 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.823701 2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 14 09:26:50.823751 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.823727 2565 factory.go:103] Registering Raw factory Jul 14 09:26:50.823751 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.823744 2565 manager.go:1196] Started watching for new ooms in manager Jul 14 09:26:50.824511 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.824496 2565 manager.go:349] Starting recovery of all containers Jul 14 09:26:50.833467 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.833440 2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 09:26:50.836106 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.835948 2565 manager.go:354] Recovery completed Jul 14 09:26:50.836633 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:50.836604 2565 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-132-2.ec2.internal\" not found" node="ip-10-0-132-2.ec2.internal" Jul 14 09:26:50.839027 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:50.838999 2565 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 14 09:26:50.842525 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.842512 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 09:26:50.844332 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.844281 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-2.ec2.internal" event="NodeHasSufficientMemory" Jul 14 09:26:50.844332 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.844314 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-2.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 09:26:50.844332 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.844328 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-2.ec2.internal" event="NodeHasSufficientPID" Jul 14 09:26:50.845005 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.844989 2565 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 14 09:26:50.845005 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.845004 2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 14 09:26:50.845086 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.845022 2565 state_mem.go:36] "Initialized new in-memory state store" Jul 14 09:26:50.845086 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.844992 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 09:26:50.845759 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.845745 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-2.ec2.internal" event="NodeHasSufficientMemory" Jul 14 09:26:50.845807 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.845771 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-2.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 09:26:50.845807 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.845785 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-2.ec2.internal" event="NodeHasSufficientPID" Jul 14 09:26:50.846865 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.846854 2565 policy_none.go:49] "None policy: Start" Jul 14 09:26:50.846938 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.846871 2565 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 14 09:26:50.846938 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.846900 2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 14 09:26:50.848646 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.848635 2565 policy_none.go:47] "Start" Jul 14 09:26:50.885225 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.885207 2565 manager.go:337] "Starting Device Plugin manager" Jul 14 09:26:50.885311 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:50.885248 2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 14 09:26:50.885311 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.885259 2565 server.go:87] "Starting device plugin registration server" Jul 14 09:26:50.885571 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.885557 2565 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 14 09:26:50.885610 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.885577 2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 14 09:26:50.885708 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.885682 2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 14 09:26:50.885857 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.885829 2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 14 09:26:50.885857 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.885842 2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 14 09:26:50.886289 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:50.886267 2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 14 09:26:50.886377 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:50.886328 2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-2.ec2.internal\" not found" Jul 14 09:26:50.960617 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.960575 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 14 09:26:50.962978 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.962068 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 14 09:26:50.962978 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.962114 2565 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 14 09:26:50.962978 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.962141 2565 kubelet.go:2448] "Starting kubelet main sync loop" Jul 14 09:26:50.962978 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:50.962184 2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 14 09:26:50.964620 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.964600 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 09:26:50.986800 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.986772 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 09:26:50.988598 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.988580 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-2.ec2.internal" event="NodeHasSufficientMemory" Jul 14 09:26:50.988700 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.988612 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-2.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 09:26:50.988700 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.988623 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-2.ec2.internal" event="NodeHasSufficientPID" Jul 14 09:26:50.988700 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.988657 2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-2.ec2.internal" Jul 14 09:26:50.998902 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:50.998886 2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-2.ec2.internal" Jul 14 09:26:50.998964 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:50.998909 2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-2.ec2.internal\": node \"ip-10-0-132-2.ec2.internal\" not found" Jul 14 09:26:51.050741 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:51.050709 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-2.ec2.internal\" not found" Jul 14 09:26:51.062642 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:51.062600 2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-132-2.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-2.ec2.internal"] Jul 14 09:26:51.062715 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:51.062685 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 09:26:51.063730 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:51.063715 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-2.ec2.internal" event="NodeHasSufficientMemory" Jul 14 09:26:51.063797 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:51.063746 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-2.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 09:26:51.063797 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:51.063763 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-2.ec2.internal" event="NodeHasSufficientPID" Jul 14 09:26:51.066246 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:51.066233 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 09:26:51.066372 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:51.066358 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-2.ec2.internal" Jul 14 09:26:51.066451 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:51.066391 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 09:26:51.067019 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:51.066998 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-2.ec2.internal" event="NodeHasSufficientMemory" Jul 14 09:26:51.067127 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:51.067028 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-2.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 09:26:51.067127 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:51.067003 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-2.ec2.internal" event="NodeHasSufficientMemory" Jul 14 09:26:51.067127 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:51.067039 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-2.ec2.internal" event="NodeHasSufficientPID" Jul 14 09:26:51.067127 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:51.067059 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-2.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 09:26:51.067127 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:51.067095 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-2.ec2.internal" event="NodeHasSufficientPID" Jul 14 09:26:51.069257 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:51.069240 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-2.ec2.internal" Jul 14 09:26:51.069341 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:51.069269 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 09:26:51.069957 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:51.069941 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-2.ec2.internal" event="NodeHasSufficientMemory" Jul 14 09:26:51.070050 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:51.069973 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-2.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 09:26:51.070050 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:51.069988 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-2.ec2.internal" event="NodeHasSufficientPID" Jul 14 09:26:51.086497 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:51.086475 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-2.ec2.internal\" not found" node="ip-10-0-132-2.ec2.internal" Jul 14 09:26:51.091197 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:51.091181 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-2.ec2.internal\" not found" node="ip-10-0-132-2.ec2.internal" Jul 14 09:26:51.125018 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:51.124942 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9a1079819acdf201f17fa12e60d64a01-config\") pod \"kube-apiserver-proxy-ip-10-0-132-2.ec2.internal\" (UID: \"9a1079819acdf201f17fa12e60d64a01\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-2.ec2.internal" Jul 14 09:26:51.125018 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:51.124986 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a3b116561a4900506d95c442fc3a920c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-2.ec2.internal\" (UID: \"a3b116561a4900506d95c442fc3a920c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-2.ec2.internal" Jul 14 09:26:51.125018 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:51.125013 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3b116561a4900506d95c442fc3a920c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-2.ec2.internal\" (UID: \"a3b116561a4900506d95c442fc3a920c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-2.ec2.internal" Jul 14 09:26:51.151847 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:51.151824 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-2.ec2.internal\" not found" Jul 14 09:26:51.225198 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:51.225162 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a3b116561a4900506d95c442fc3a920c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-2.ec2.internal\" (UID: \"a3b116561a4900506d95c442fc3a920c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-2.ec2.internal" Jul 14 09:26:51.225198 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:51.225197 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3b116561a4900506d95c442fc3a920c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-2.ec2.internal\" (UID: \"a3b116561a4900506d95c442fc3a920c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-2.ec2.internal" Jul 14 09:26:51.225417 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:51.225216 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9a1079819acdf201f17fa12e60d64a01-config\") pod \"kube-apiserver-proxy-ip-10-0-132-2.ec2.internal\" (UID: \"9a1079819acdf201f17fa12e60d64a01\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-2.ec2.internal" Jul 14 09:26:51.225417 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:51.225248 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9a1079819acdf201f17fa12e60d64a01-config\") pod \"kube-apiserver-proxy-ip-10-0-132-2.ec2.internal\" (UID: \"9a1079819acdf201f17fa12e60d64a01\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-2.ec2.internal" Jul 14 09:26:51.225417 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:51.225278 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3b116561a4900506d95c442fc3a920c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-2.ec2.internal\" (UID: \"a3b116561a4900506d95c442fc3a920c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-2.ec2.internal" Jul 14 09:26:51.225417 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:51.225276 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a3b116561a4900506d95c442fc3a920c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-2.ec2.internal\" (UID: \"a3b116561a4900506d95c442fc3a920c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-2.ec2.internal" Jul 14 09:26:51.253406 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:51.253372 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-2.ec2.internal\" not found" Jul 14 09:26:51.355694 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:51.355661 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-2.ec2.internal\" not found" Jul 14 09:26:51.390593 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:51.390509 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-2.ec2.internal" Jul 14 09:26:51.396040 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:51.395697 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-2.ec2.internal" Jul 14 09:26:51.455912 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:51.455872 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-2.ec2.internal\" not found" Jul 14 09:26:51.556134 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:51.556106 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-2.ec2.internal\" not found" Jul 14 09:26:51.656528 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:51.656440 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-2.ec2.internal\" not found" Jul 14 09:26:51.757159 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:51.757119 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-2.ec2.internal\" not found" Jul 14 09:26:51.761358 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:51.761325 2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 14 09:26:51.761547 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:51.761521 2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 14 09:26:51.761547 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:51.761526 2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 14 09:26:51.828782 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:51.828728 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-13 09:21:51 +0000 UTC" deadline="2028-04-22 08:26:55.229483502 +0000 UTC" Jul 14 09:26:51.828782 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:51.828771 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15551h0m3.400717073s" Jul 14 09:26:51.857718 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:51.857691 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-2.ec2.internal\" not found" Jul 14 09:26:51.903852 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:51.903808 2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 09:26:51.923701 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:51.923614 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-2.ec2.internal" Jul 14 09:26:51.934918 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:51.934872 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 14 09:26:51.937154 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:51.937132 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-2.ec2.internal" Jul 14 09:26:51.948475 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:51.948452 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 14 09:26:52.029039 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:52.029002 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a1079819acdf201f17fa12e60d64a01.slice/crio-49cf688c34f96204ebd0248d36b7c65c12d6ed5dc9f564db7374e89ec3089823 WatchSource:0}: Error finding container 49cf688c34f96204ebd0248d36b7c65c12d6ed5dc9f564db7374e89ec3089823: Status 404 returned error can't find the container with id 49cf688c34f96204ebd0248d36b7c65c12d6ed5dc9f564db7374e89ec3089823 Jul 14 09:26:52.032703 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.032682 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 09:26:52.039431 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:52.039404 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3b116561a4900506d95c442fc3a920c.slice/crio-629ee640c1025f8838f83855a0360a69ad56c924927b20cf0e682432f55785d8 WatchSource:0}: Error finding container 629ee640c1025f8838f83855a0360a69ad56c924927b20cf0e682432f55785d8: Status 404 returned error can't find the container with id 629ee640c1025f8838f83855a0360a69ad56c924927b20cf0e682432f55785d8 Jul 14 09:26:52.148316 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.148279 2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 09:26:52.715416 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.715383 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 09:26:52.775010 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.774974 2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 09:26:52.803592 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.803529 2565 apiserver.go:52] "Watching apiserver" Jul 14 09:26:52.812240 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.812028 2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 14 09:26:52.812568 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.812536 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-jjwzx","kube-system/kube-apiserver-proxy-ip-10-0-132-2.ec2.internal","openshift-dns/node-resolver-5r769","openshift-multus/network-metrics-daemon-p5547","openshift-network-diagnostics/network-check-target-n8c7q","openshift-network-operator/iptables-alerter-r5rg6","openshift-ovn-kubernetes/ovnkube-node-r79mk","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5rb","openshift-cluster-node-tuning-operator/tuned-ttvxk","openshift-image-registry/node-ca-4kb48","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-2.ec2.internal","openshift-multus/multus-9xsfz","openshift-multus/multus-additional-cni-plugins-9z4f8","kube-system/global-pull-secret-syncer-2tcgq"] Jul 14 09:26:52.816775 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.815835 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5rb" Jul 14 09:26:52.818738 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.818715 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9z4f8" Jul 14 09:26:52.820238 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.820218 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-sc9bh\"" Jul 14 09:26:52.820416 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.820400 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 14 09:26:52.821100 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.820648 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 14 09:26:52.821100 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.820874 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 14 09:26:52.825106 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.825086 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 14 09:26:52.825325 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.825309 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-68859\"" Jul 14 09:26:52.825945 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.825359 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ttvxk" Jul 14 09:26:52.825945 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.825586 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 14 09:26:52.825945 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.825679 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5r769" Jul 14 09:26:52.825945 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.825896 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 14 09:26:52.826170 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.826153 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 14 09:26:52.826227 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.826190 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 14 09:26:52.826648 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.826628 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 14 09:26:52.827015 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.827002 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8c7q" Jul 14 09:26:52.827189 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:52.827166 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8c7q" podUID="7225439c-6210-4bc5-b124-0d92487dcd0c" Jul 14 09:26:52.827573 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.827550 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 14 09:26:52.828379 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.827977 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-r5rg6" Jul 14 09:26:52.828379 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.828106 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 14 09:26:52.828722 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.828703 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gzgp7\"" Jul 14 09:26:52.829321 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.829058 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 14 09:26:52.829321 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.829168 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-45wtk\"" Jul 14 09:26:52.829813 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.829538 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 14 09:26:52.831338 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.831318 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:52.834485 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.833921 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jjwzx" Jul 14 09:26:52.834485 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.834005 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 14 09:26:52.834485 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.834106 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 14 09:26:52.834485 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.834371 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-86rnb\"" Jul 14 09:26:52.834768 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.834489 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 14 09:26:52.834768 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.834565 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 14 09:26:52.834768 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.834597 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wmlhg\"" Jul 14 09:26:52.834768 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.834727 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 14 09:26:52.835053 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.834939 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3da5cf33-ada6-411e-b56d-73c993bf77b7-etc-sysconfig\") pod \"tuned-ttvxk\" (UID: \"3da5cf33-ada6-411e-b56d-73c993bf77b7\") " pod="openshift-cluster-node-tuning-operator/tuned-ttvxk" Jul 14 09:26:52.835053 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.834972 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eebab604-45fc-4992-a289-96c5275ed378-iptables-alerter-script\") pod \"iptables-alerter-r5rg6\" (UID: \"eebab604-45fc-4992-a289-96c5275ed378\") " pod="openshift-network-operator/iptables-alerter-r5rg6" Jul 14 09:26:52.835053 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.834984 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 14 09:26:52.835053 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.835011 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/82f2aa67-8042-4bbb-930d-09ad08fd1696-system-cni-dir\") pod \"multus-additional-cni-plugins-9z4f8\" (UID: \"82f2aa67-8042-4bbb-930d-09ad08fd1696\") " pod="openshift-multus/multus-additional-cni-plugins-9z4f8" Jul 14 09:26:52.835053 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.835037 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3da5cf33-ada6-411e-b56d-73c993bf77b7-etc-kubernetes\") pod \"tuned-ttvxk\" (UID: \"3da5cf33-ada6-411e-b56d-73c993bf77b7\") " pod="openshift-cluster-node-tuning-operator/tuned-ttvxk" Jul 14 09:26:52.835346 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.835062 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3da5cf33-ada6-411e-b56d-73c993bf77b7-etc-sysctl-conf\") pod \"tuned-ttvxk\" (UID: \"3da5cf33-ada6-411e-b56d-73c993bf77b7\") " pod="openshift-cluster-node-tuning-operator/tuned-ttvxk" Jul 14 09:26:52.835346 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.835085 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3da5cf33-ada6-411e-b56d-73c993bf77b7-etc-tuned\") pod \"tuned-ttvxk\" (UID: \"3da5cf33-ada6-411e-b56d-73c993bf77b7\") " pod="openshift-cluster-node-tuning-operator/tuned-ttvxk" Jul 14 09:26:52.835346 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.835119 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs4nw\" (UniqueName: \"kubernetes.io/projected/7225439c-6210-4bc5-b124-0d92487dcd0c-kube-api-access-hs4nw\") pod \"network-check-target-n8c7q\" (UID: \"7225439c-6210-4bc5-b124-0d92487dcd0c\") " pod="openshift-network-diagnostics/network-check-target-n8c7q" Jul 14 09:26:52.835346 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.835143 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eebab604-45fc-4992-a289-96c5275ed378-host-slash\") pod \"iptables-alerter-r5rg6\" (UID: \"eebab604-45fc-4992-a289-96c5275ed378\") " pod="openshift-network-operator/iptables-alerter-r5rg6" Jul 14 09:26:52.835346 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.835168 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3da5cf33-ada6-411e-b56d-73c993bf77b7-run\") pod \"tuned-ttvxk\" (UID: \"3da5cf33-ada6-411e-b56d-73c993bf77b7\") " pod="openshift-cluster-node-tuning-operator/tuned-ttvxk" Jul 14 09:26:52.835346 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.835190 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3da5cf33-ada6-411e-b56d-73c993bf77b7-sys\") pod \"tuned-ttvxk\" (UID: \"3da5cf33-ada6-411e-b56d-73c993bf77b7\") " pod="openshift-cluster-node-tuning-operator/tuned-ttvxk" Jul 14 09:26:52.835346 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.835216 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 14 09:26:52.835786 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.835479 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 14 09:26:52.835786 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.835503 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 14 09:26:52.841906 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.835215 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3da5cf33-ada6-411e-b56d-73c993bf77b7-lib-modules\") pod \"tuned-ttvxk\" (UID: \"3da5cf33-ada6-411e-b56d-73c993bf77b7\") " pod="openshift-cluster-node-tuning-operator/tuned-ttvxk" Jul 14 09:26:52.841906 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.837235 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjvns\" (UniqueName: \"kubernetes.io/projected/3da5cf33-ada6-411e-b56d-73c993bf77b7-kube-api-access-rjvns\") pod \"tuned-ttvxk\" (UID: \"3da5cf33-ada6-411e-b56d-73c993bf77b7\") " pod="openshift-cluster-node-tuning-operator/tuned-ttvxk" Jul 14 09:26:52.841906 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.837278 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dc1cb2ed-2875-4a0a-9089-c1f9240dd7be-socket-dir\") pod \"aws-ebs-csi-driver-node-mf5rb\" (UID: \"dc1cb2ed-2875-4a0a-9089-c1f9240dd7be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5rb" Jul 14 09:26:52.841906 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.837315 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/82f2aa67-8042-4bbb-930d-09ad08fd1696-cnibin\") pod \"multus-additional-cni-plugins-9z4f8\" (UID: \"82f2aa67-8042-4bbb-930d-09ad08fd1696\") " pod="openshift-multus/multus-additional-cni-plugins-9z4f8" Jul 14 09:26:52.841906 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.837346 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/82f2aa67-8042-4bbb-930d-09ad08fd1696-cni-binary-copy\") pod \"multus-additional-cni-plugins-9z4f8\" (UID: \"82f2aa67-8042-4bbb-930d-09ad08fd1696\") " pod="openshift-multus/multus-additional-cni-plugins-9z4f8" Jul 14 09:26:52.841906 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.837383 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/82f2aa67-8042-4bbb-930d-09ad08fd1696-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9z4f8\" (UID: \"82f2aa67-8042-4bbb-930d-09ad08fd1696\") " pod="openshift-multus/multus-additional-cni-plugins-9z4f8" Jul 14 09:26:52.841906 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.837414 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3da5cf33-ada6-411e-b56d-73c993bf77b7-tmp\") pod \"tuned-ttvxk\" (UID: \"3da5cf33-ada6-411e-b56d-73c993bf77b7\") " pod="openshift-cluster-node-tuning-operator/tuned-ttvxk" Jul 14 09:26:52.841906 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.837547 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl5tn\" (UniqueName: \"kubernetes.io/projected/dc1cb2ed-2875-4a0a-9089-c1f9240dd7be-kube-api-access-tl5tn\") pod \"aws-ebs-csi-driver-node-mf5rb\" (UID: \"dc1cb2ed-2875-4a0a-9089-c1f9240dd7be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5rb" Jul 14 09:26:52.841906 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.837623 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/82f2aa67-8042-4bbb-930d-09ad08fd1696-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9z4f8\" (UID: \"82f2aa67-8042-4bbb-930d-09ad08fd1696\") " pod="openshift-multus/multus-additional-cni-plugins-9z4f8" Jul 14 09:26:52.841906 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.837653 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/82f2aa67-8042-4bbb-930d-09ad08fd1696-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9z4f8\" (UID: \"82f2aa67-8042-4bbb-930d-09ad08fd1696\") " pod="openshift-multus/multus-additional-cni-plugins-9z4f8" Jul 14 09:26:52.841906 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.837687 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3da5cf33-ada6-411e-b56d-73c993bf77b7-etc-systemd\") pod \"tuned-ttvxk\" (UID: \"3da5cf33-ada6-411e-b56d-73c993bf77b7\") " pod="openshift-cluster-node-tuning-operator/tuned-ttvxk" Jul 14 09:26:52.841906 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.837720 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmpgj\" (UniqueName: \"kubernetes.io/projected/eebab604-45fc-4992-a289-96c5275ed378-kube-api-access-pmpgj\") pod \"iptables-alerter-r5rg6\" (UID: \"eebab604-45fc-4992-a289-96c5275ed378\") " pod="openshift-network-operator/iptables-alerter-r5rg6" Jul 14 09:26:52.841906 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.837832 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dc1cb2ed-2875-4a0a-9089-c1f9240dd7be-etc-selinux\") pod \"aws-ebs-csi-driver-node-mf5rb\" (UID: \"dc1cb2ed-2875-4a0a-9089-c1f9240dd7be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5rb" Jul 14 09:26:52.841906 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.837871 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/82f2aa67-8042-4bbb-930d-09ad08fd1696-os-release\") pod \"multus-additional-cni-plugins-9z4f8\" (UID: \"82f2aa67-8042-4bbb-930d-09ad08fd1696\") " pod="openshift-multus/multus-additional-cni-plugins-9z4f8" Jul 14 09:26:52.841906 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.837926 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzzvn\" (UniqueName: \"kubernetes.io/projected/82f2aa67-8042-4bbb-930d-09ad08fd1696-kube-api-access-dzzvn\") pod \"multus-additional-cni-plugins-9z4f8\" (UID: \"82f2aa67-8042-4bbb-930d-09ad08fd1696\") " pod="openshift-multus/multus-additional-cni-plugins-9z4f8" Jul 14 09:26:52.841906 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.837960 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z8j6\" (UniqueName: \"kubernetes.io/projected/4a7e6b7e-3c6a-4def-bc24-b9a00ccad67b-kube-api-access-2z8j6\") pod \"node-resolver-5r769\" (UID: \"4a7e6b7e-3c6a-4def-bc24-b9a00ccad67b\") " pod="openshift-dns/node-resolver-5r769" Jul 14 09:26:52.842806 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.837990 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3da5cf33-ada6-411e-b56d-73c993bf77b7-etc-modprobe-d\") pod \"tuned-ttvxk\" (UID: \"3da5cf33-ada6-411e-b56d-73c993bf77b7\") " pod="openshift-cluster-node-tuning-operator/tuned-ttvxk" Jul 14 09:26:52.842806 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.838020 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dc1cb2ed-2875-4a0a-9089-c1f9240dd7be-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mf5rb\" (UID: \"dc1cb2ed-2875-4a0a-9089-c1f9240dd7be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5rb" Jul 14 09:26:52.842806 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.838155 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dc1cb2ed-2875-4a0a-9089-c1f9240dd7be-registration-dir\") pod \"aws-ebs-csi-driver-node-mf5rb\" (UID: \"dc1cb2ed-2875-4a0a-9089-c1f9240dd7be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5rb" Jul 14 09:26:52.842806 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.838190 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3da5cf33-ada6-411e-b56d-73c993bf77b7-etc-sysctl-d\") pod \"tuned-ttvxk\" (UID: \"3da5cf33-ada6-411e-b56d-73c993bf77b7\") " pod="openshift-cluster-node-tuning-operator/tuned-ttvxk" Jul 14 09:26:52.842806 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.838219 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3da5cf33-ada6-411e-b56d-73c993bf77b7-host\") pod \"tuned-ttvxk\" (UID: \"3da5cf33-ada6-411e-b56d-73c993bf77b7\") " pod="openshift-cluster-node-tuning-operator/tuned-ttvxk" Jul 14 09:26:52.842806 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.838253 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dc1cb2ed-2875-4a0a-9089-c1f9240dd7be-device-dir\") pod \"aws-ebs-csi-driver-node-mf5rb\" (UID: \"dc1cb2ed-2875-4a0a-9089-c1f9240dd7be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5rb" Jul 14 09:26:52.842806 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.838327 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3da5cf33-ada6-411e-b56d-73c993bf77b7-var-lib-kubelet\") pod \"tuned-ttvxk\" (UID: \"3da5cf33-ada6-411e-b56d-73c993bf77b7\") " pod="openshift-cluster-node-tuning-operator/tuned-ttvxk" Jul 14 09:26:52.842806 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.838360 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dc1cb2ed-2875-4a0a-9089-c1f9240dd7be-sys-fs\") pod \"aws-ebs-csi-driver-node-mf5rb\" (UID: \"dc1cb2ed-2875-4a0a-9089-c1f9240dd7be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5rb" Jul 14 09:26:52.842806 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.838385 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4a7e6b7e-3c6a-4def-bc24-b9a00ccad67b-hosts-file\") pod \"node-resolver-5r769\" (UID: \"4a7e6b7e-3c6a-4def-bc24-b9a00ccad67b\") " pod="openshift-dns/node-resolver-5r769" Jul 14 09:26:52.842806 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.838398 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 14 09:26:52.842806 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.838415 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a7e6b7e-3c6a-4def-bc24-b9a00ccad67b-tmp-dir\") pod \"node-resolver-5r769\" (UID: \"4a7e6b7e-3c6a-4def-bc24-b9a00ccad67b\") " pod="openshift-dns/node-resolver-5r769" Jul 14 09:26:52.842806 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.839004 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 14 09:26:52.842806 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.839176 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p5547" Jul 14 09:26:52.842806 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:52.839436 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p5547" podUID="c2190520-5806-41dd-b25f-b7d1076cfcf3" Jul 14 09:26:52.842806 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.839563 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7h9hb\"" Jul 14 09:26:52.842806 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.841629 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4kb48" Jul 14 09:26:52.843666 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.843187 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 14 09:26:52.844440 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.844195 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 14 09:26:52.845712 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.845398 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9xsfz" Jul 14 09:26:52.846444 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.846212 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 14 09:26:52.846444 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.846302 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9wzq8\"" Jul 14 09:26:52.846444 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.846328 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 14 09:26:52.847904 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.847853 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tcgq" Jul 14 09:26:52.848004 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:52.847948 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tcgq" podUID="a5c62b6a-f781-4066-9773-a5f4e8921276" Jul 14 09:26:52.848198 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.848175 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 14 09:26:52.848400 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.848376 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-98x9m\"" Jul 14 09:26:52.867327 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.867298 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-pd2nd" Jul 14 09:26:52.873468 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.873442 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-pd2nd" Jul 14 09:26:52.924016 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.923956 2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 14 09:26:52.939256 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.939222 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0274da0d-7c52-4932-9ac3-732af03b0c85-multus-socket-dir-parent\") pod \"multus-9xsfz\" (UID: \"0274da0d-7c52-4932-9ac3-732af03b0c85\") " pod="openshift-multus/multus-9xsfz" Jul 14 09:26:52.939449 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.939269 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2190520-5806-41dd-b25f-b7d1076cfcf3-metrics-certs\") pod \"network-metrics-daemon-p5547\" (UID: \"c2190520-5806-41dd-b25f-b7d1076cfcf3\") " pod="openshift-multus/network-metrics-daemon-p5547" Jul 14 09:26:52.939449 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.939307 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/82f2aa67-8042-4bbb-930d-09ad08fd1696-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9z4f8\" (UID: \"82f2aa67-8042-4bbb-930d-09ad08fd1696\") " pod="openshift-multus/multus-additional-cni-plugins-9z4f8" Jul 14 09:26:52.939449 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.939338 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dc1cb2ed-2875-4a0a-9089-c1f9240dd7be-etc-selinux\") pod \"aws-ebs-csi-driver-node-mf5rb\" (UID: \"dc1cb2ed-2875-4a0a-9089-c1f9240dd7be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5rb" Jul 14 09:26:52.939449 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.939365 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0274da0d-7c52-4932-9ac3-732af03b0c85-host-run-multus-certs\") pod \"multus-9xsfz\" (UID: \"0274da0d-7c52-4932-9ac3-732af03b0c85\") " pod="openshift-multus/multus-9xsfz" Jul 14 09:26:52.939449 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.939393 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9a0e148-5f32-4d80-8963-321cbc4e5de0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:52.939449 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.939422 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z8j6\" (UniqueName: \"kubernetes.io/projected/4a7e6b7e-3c6a-4def-bc24-b9a00ccad67b-kube-api-access-2z8j6\") pod \"node-resolver-5r769\" (UID: \"4a7e6b7e-3c6a-4def-bc24-b9a00ccad67b\") " pod="openshift-dns/node-resolver-5r769" Jul 14 09:26:52.939449 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.939445 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3da5cf33-ada6-411e-b56d-73c993bf77b7-etc-modprobe-d\") pod \"tuned-ttvxk\" (UID: \"3da5cf33-ada6-411e-b56d-73c993bf77b7\") " pod="openshift-cluster-node-tuning-operator/tuned-ttvxk" Jul 14 09:26:52.939828 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.939470 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dc1cb2ed-2875-4a0a-9089-c1f9240dd7be-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mf5rb\" (UID: \"dc1cb2ed-2875-4a0a-9089-c1f9240dd7be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5rb" Jul 14 09:26:52.939828 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.939512 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dc1cb2ed-2875-4a0a-9089-c1f9240dd7be-etc-selinux\") pod \"aws-ebs-csi-driver-node-mf5rb\" (UID: \"dc1cb2ed-2875-4a0a-9089-c1f9240dd7be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5rb" Jul 14 09:26:52.939828 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.939513 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dc1cb2ed-2875-4a0a-9089-c1f9240dd7be-registration-dir\") pod \"aws-ebs-csi-driver-node-mf5rb\" (UID: \"dc1cb2ed-2875-4a0a-9089-c1f9240dd7be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5rb" Jul 14 09:26:52.939828 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.939568 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0274da0d-7c52-4932-9ac3-732af03b0c85-host-var-lib-cni-multus\") pod \"multus-9xsfz\" (UID: \"0274da0d-7c52-4932-9ac3-732af03b0c85\") " pod="openshift-multus/multus-9xsfz" Jul 14 09:26:52.939828 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.939587 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dc1cb2ed-2875-4a0a-9089-c1f9240dd7be-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mf5rb\" (UID: \"dc1cb2ed-2875-4a0a-9089-c1f9240dd7be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5rb" Jul 14 09:26:52.939828 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.939601 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dc1cb2ed-2875-4a0a-9089-c1f9240dd7be-registration-dir\") pod \"aws-ebs-csi-driver-node-mf5rb\" (UID: \"dc1cb2ed-2875-4a0a-9089-c1f9240dd7be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5rb" Jul 14 09:26:52.939828 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.939650 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a5c62b6a-f781-4066-9773-a5f4e8921276-kubelet-config\") pod \"global-pull-secret-syncer-2tcgq\" (UID: \"a5c62b6a-f781-4066-9773-a5f4e8921276\") " pod="kube-system/global-pull-secret-syncer-2tcgq" Jul 14 09:26:52.939828 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.939686 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3da5cf33-ada6-411e-b56d-73c993bf77b7-etc-sysctl-d\") pod \"tuned-ttvxk\" (UID: \"3da5cf33-ada6-411e-b56d-73c993bf77b7\") " pod="openshift-cluster-node-tuning-operator/tuned-ttvxk" Jul 14 09:26:52.939828 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.939709 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3da5cf33-ada6-411e-b56d-73c993bf77b7-etc-modprobe-d\") pod \"tuned-ttvxk\" (UID: \"3da5cf33-ada6-411e-b56d-73c993bf77b7\") " pod="openshift-cluster-node-tuning-operator/tuned-ttvxk" Jul 14 09:26:52.939828 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.939716 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3da5cf33-ada6-411e-b56d-73c993bf77b7-host\") pod \"tuned-ttvxk\" (UID: \"3da5cf33-ada6-411e-b56d-73c993bf77b7\") " pod="openshift-cluster-node-tuning-operator/tuned-ttvxk" Jul 14 09:26:52.939828 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.939749 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dc1cb2ed-2875-4a0a-9089-c1f9240dd7be-device-dir\") pod \"aws-ebs-csi-driver-node-mf5rb\" (UID: \"dc1cb2ed-2875-4a0a-9089-c1f9240dd7be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5rb" Jul 14 09:26:52.939828 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.939767 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3da5cf33-ada6-411e-b56d-73c993bf77b7-host\") pod \"tuned-ttvxk\" (UID: \"3da5cf33-ada6-411e-b56d-73c993bf77b7\") " pod="openshift-cluster-node-tuning-operator/tuned-ttvxk" Jul 14 09:26:52.939828 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.939776 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f9a0e148-5f32-4d80-8963-321cbc4e5de0-systemd-units\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:52.939828 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.939802 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f9a0e148-5f32-4d80-8963-321cbc4e5de0-host-slash\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:52.940552 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.939841 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f9a0e148-5f32-4d80-8963-321cbc4e5de0-log-socket\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:52.940552 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.939884 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dc4d1b63-99f8-4738-9dee-76fe0e4e4e4d-konnectivity-ca\") pod \"konnectivity-agent-jjwzx\" (UID: \"dc4d1b63-99f8-4738-9dee-76fe0e4e4e4d\") " pod="kube-system/konnectivity-agent-jjwzx" Jul 14 09:26:52.940552 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.939887 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3da5cf33-ada6-411e-b56d-73c993bf77b7-etc-sysctl-d\") pod \"tuned-ttvxk\" (UID: \"3da5cf33-ada6-411e-b56d-73c993bf77b7\") " pod="openshift-cluster-node-tuning-operator/tuned-ttvxk" Jul 14 09:26:52.940552 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.939912 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3da5cf33-ada6-411e-b56d-73c993bf77b7-var-lib-kubelet\") pod \"tuned-ttvxk\" (UID: \"3da5cf33-ada6-411e-b56d-73c993bf77b7\") " pod="openshift-cluster-node-tuning-operator/tuned-ttvxk" Jul 14 09:26:52.940552 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.939934 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dc1cb2ed-2875-4a0a-9089-c1f9240dd7be-device-dir\") pod \"aws-ebs-csi-driver-node-mf5rb\" (UID: \"dc1cb2ed-2875-4a0a-9089-c1f9240dd7be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5rb" Jul 14 09:26:52.940552 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.939940 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3cfb37de-4170-43d8-aa35-d2f1c3c6cc90-host\") pod \"node-ca-4kb48\" (UID: \"3cfb37de-4170-43d8-aa35-d2f1c3c6cc90\") " pod="openshift-image-registry/node-ca-4kb48" Jul 14 09:26:52.940552 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.939998 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3da5cf33-ada6-411e-b56d-73c993bf77b7-var-lib-kubelet\") pod \"tuned-ttvxk\" (UID: \"3da5cf33-ada6-411e-b56d-73c993bf77b7\") " pod="openshift-cluster-node-tuning-operator/tuned-ttvxk" Jul 14 09:26:52.940552 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.939999 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a7e6b7e-3c6a-4def-bc24-b9a00ccad67b-tmp-dir\") pod \"node-resolver-5r769\" (UID: \"4a7e6b7e-3c6a-4def-bc24-b9a00ccad67b\") " pod="openshift-dns/node-resolver-5r769" Jul 14 09:26:52.940552 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.940025 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/82f2aa67-8042-4bbb-930d-09ad08fd1696-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9z4f8\" (UID: \"82f2aa67-8042-4bbb-930d-09ad08fd1696\") " pod="openshift-multus/multus-additional-cni-plugins-9z4f8" Jul 14 09:26:52.940552 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.940065 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0274da0d-7c52-4932-9ac3-732af03b0c85-system-cni-dir\") pod \"multus-9xsfz\" (UID: \"0274da0d-7c52-4932-9ac3-732af03b0c85\") " pod="openshift-multus/multus-9xsfz" Jul 14 09:26:52.940552 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.940100 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3da5cf33-ada6-411e-b56d-73c993bf77b7-etc-kubernetes\") pod \"tuned-ttvxk\" (UID: \"3da5cf33-ada6-411e-b56d-73c993bf77b7\") " pod="openshift-cluster-node-tuning-operator/tuned-ttvxk" Jul 14 09:26:52.940552 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.940138 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3da5cf33-ada6-411e-b56d-73c993bf77b7-etc-sysctl-conf\") pod \"tuned-ttvxk\" (UID: \"3da5cf33-ada6-411e-b56d-73c993bf77b7\") " pod="openshift-cluster-node-tuning-operator/tuned-ttvxk" Jul 14 09:26:52.940552 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.940163 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3da5cf33-ada6-411e-b56d-73c993bf77b7-etc-tuned\") pod \"tuned-ttvxk\" (UID: \"3da5cf33-ada6-411e-b56d-73c993bf77b7\") " pod="openshift-cluster-node-tuning-operator/tuned-ttvxk" Jul 14 09:26:52.940552 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.940184 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3da5cf33-ada6-411e-b56d-73c993bf77b7-etc-kubernetes\") pod \"tuned-ttvxk\" (UID: \"3da5cf33-ada6-411e-b56d-73c993bf77b7\") " pod="openshift-cluster-node-tuning-operator/tuned-ttvxk" Jul 14 09:26:52.940552 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.940192 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs4nw\" (UniqueName: \"kubernetes.io/projected/7225439c-6210-4bc5-b124-0d92487dcd0c-kube-api-access-hs4nw\") pod \"network-check-target-n8c7q\" (UID: \"7225439c-6210-4bc5-b124-0d92487dcd0c\") " pod="openshift-network-diagnostics/network-check-target-n8c7q" Jul 14 09:26:52.940552 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.940225 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eebab604-45fc-4992-a289-96c5275ed378-host-slash\") pod \"iptables-alerter-r5rg6\" (UID: \"eebab604-45fc-4992-a289-96c5275ed378\") " pod="openshift-network-operator/iptables-alerter-r5rg6" Jul 14 09:26:52.940552 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.940251 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0274da0d-7c52-4932-9ac3-732af03b0c85-os-release\") pod \"multus-9xsfz\" (UID: \"0274da0d-7c52-4932-9ac3-732af03b0c85\") " pod="openshift-multus/multus-9xsfz" Jul 14 09:26:52.941449 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.940278 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlzqc\" (UniqueName: \"kubernetes.io/projected/0274da0d-7c52-4932-9ac3-732af03b0c85-kube-api-access-wlzqc\") pod \"multus-9xsfz\" (UID: \"0274da0d-7c52-4932-9ac3-732af03b0c85\") " pod="openshift-multus/multus-9xsfz" Jul 14 09:26:52.941449 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.940280 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3da5cf33-ada6-411e-b56d-73c993bf77b7-etc-sysctl-conf\") pod \"tuned-ttvxk\" (UID: \"3da5cf33-ada6-411e-b56d-73c993bf77b7\") " pod="openshift-cluster-node-tuning-operator/tuned-ttvxk" Jul 14 09:26:52.941449 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.940304 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f9a0e148-5f32-4d80-8963-321cbc4e5de0-host-run-netns\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:52.941449 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.940296 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a7e6b7e-3c6a-4def-bc24-b9a00ccad67b-tmp-dir\") pod \"node-resolver-5r769\" (UID: \"4a7e6b7e-3c6a-4def-bc24-b9a00ccad67b\") " pod="openshift-dns/node-resolver-5r769" Jul 14 09:26:52.941449 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.940319 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eebab604-45fc-4992-a289-96c5275ed378-host-slash\") pod \"iptables-alerter-r5rg6\" (UID: \"eebab604-45fc-4992-a289-96c5275ed378\") " pod="openshift-network-operator/iptables-alerter-r5rg6" Jul 14 09:26:52.941449 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.940330 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a5c62b6a-f781-4066-9773-a5f4e8921276-dbus\") pod \"global-pull-secret-syncer-2tcgq\" (UID: \"a5c62b6a-f781-4066-9773-a5f4e8921276\") " pod="kube-system/global-pull-secret-syncer-2tcgq" Jul 14 09:26:52.941449 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.940452 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a5c62b6a-f781-4066-9773-a5f4e8921276-original-pull-secret\") pod \"global-pull-secret-syncer-2tcgq\" (UID: \"a5c62b6a-f781-4066-9773-a5f4e8921276\") " pod="kube-system/global-pull-secret-syncer-2tcgq" Jul 14 09:26:52.941449 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.940489 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f9a0e148-5f32-4d80-8963-321cbc4e5de0-run-systemd\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:52.941449 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.940482 2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 14 09:26:52.941449 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.940519 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9a0e148-5f32-4d80-8963-321cbc4e5de0-etc-openvswitch\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:52.941449 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.940564 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f9a0e148-5f32-4d80-8963-321cbc4e5de0-host-cni-bin\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:52.941449 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.940602 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f9a0e148-5f32-4d80-8963-321cbc4e5de0-ovn-node-metrics-cert\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:52.941449 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.940635 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dc4d1b63-99f8-4738-9dee-76fe0e4e4e4d-agent-certs\") pod \"konnectivity-agent-jjwzx\" (UID: \"dc4d1b63-99f8-4738-9dee-76fe0e4e4e4d\") " pod="kube-system/konnectivity-agent-jjwzx" Jul 14 09:26:52.941449 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.940668 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/82f2aa67-8042-4bbb-930d-09ad08fd1696-cni-binary-copy\") pod \"multus-additional-cni-plugins-9z4f8\" (UID: \"82f2aa67-8042-4bbb-930d-09ad08fd1696\") " pod="openshift-multus/multus-additional-cni-plugins-9z4f8" Jul 14 09:26:52.941449 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.940694 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3da5cf33-ada6-411e-b56d-73c993bf77b7-tmp\") pod \"tuned-ttvxk\" (UID: \"3da5cf33-ada6-411e-b56d-73c993bf77b7\") " pod="openshift-cluster-node-tuning-operator/tuned-ttvxk" Jul 14 09:26:52.941449 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.940719 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl5tn\" (UniqueName: \"kubernetes.io/projected/dc1cb2ed-2875-4a0a-9089-c1f9240dd7be-kube-api-access-tl5tn\") pod \"aws-ebs-csi-driver-node-mf5rb\" (UID: \"dc1cb2ed-2875-4a0a-9089-c1f9240dd7be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5rb" Jul 14 09:26:52.941449 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.940745 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9a0e148-5f32-4d80-8963-321cbc4e5de0-run-openvswitch\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:52.942315 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.940771 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/82f2aa67-8042-4bbb-930d-09ad08fd1696-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9z4f8\" (UID: \"82f2aa67-8042-4bbb-930d-09ad08fd1696\") " pod="openshift-multus/multus-additional-cni-plugins-9z4f8" Jul 14 09:26:52.942315 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.940807 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3da5cf33-ada6-411e-b56d-73c993bf77b7-etc-systemd\") pod \"tuned-ttvxk\" (UID: \"3da5cf33-ada6-411e-b56d-73c993bf77b7\") " pod="openshift-cluster-node-tuning-operator/tuned-ttvxk" Jul 14 09:26:52.942315 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.940832 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmpgj\" (UniqueName: \"kubernetes.io/projected/eebab604-45fc-4992-a289-96c5275ed378-kube-api-access-pmpgj\") pod \"iptables-alerter-r5rg6\" (UID: \"eebab604-45fc-4992-a289-96c5275ed378\") " pod="openshift-network-operator/iptables-alerter-r5rg6" Jul 14 09:26:52.942315 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.940873 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0274da0d-7c52-4932-9ac3-732af03b0c85-host-run-netns\") pod \"multus-9xsfz\" (UID: \"0274da0d-7c52-4932-9ac3-732af03b0c85\") " pod="openshift-multus/multus-9xsfz" Jul 14 09:26:52.942315 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.940900 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2ktj\" (UniqueName: \"kubernetes.io/projected/f9a0e148-5f32-4d80-8963-321cbc4e5de0-kube-api-access-g2ktj\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:52.942315 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.940932 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3da5cf33-ada6-411e-b56d-73c993bf77b7-etc-systemd\") pod \"tuned-ttvxk\" (UID: \"3da5cf33-ada6-411e-b56d-73c993bf77b7\") " pod="openshift-cluster-node-tuning-operator/tuned-ttvxk" Jul 14 09:26:52.942315 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.940938 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rc2d\" (UniqueName: \"kubernetes.io/projected/3cfb37de-4170-43d8-aa35-d2f1c3c6cc90-kube-api-access-5rc2d\") pod \"node-ca-4kb48\" (UID: \"3cfb37de-4170-43d8-aa35-d2f1c3c6cc90\") " pod="openshift-image-registry/node-ca-4kb48" Jul 14 09:26:52.942315 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.940991 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/82f2aa67-8042-4bbb-930d-09ad08fd1696-os-release\") pod \"multus-additional-cni-plugins-9z4f8\" (UID: \"82f2aa67-8042-4bbb-930d-09ad08fd1696\") " pod="openshift-multus/multus-additional-cni-plugins-9z4f8" Jul 14 09:26:52.942315 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.941007 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/82f2aa67-8042-4bbb-930d-09ad08fd1696-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9z4f8\" (UID: \"82f2aa67-8042-4bbb-930d-09ad08fd1696\") " pod="openshift-multus/multus-additional-cni-plugins-9z4f8" Jul 14 09:26:52.942315 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.941021 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzzvn\" (UniqueName: \"kubernetes.io/projected/82f2aa67-8042-4bbb-930d-09ad08fd1696-kube-api-access-dzzvn\") pod \"multus-additional-cni-plugins-9z4f8\" (UID: \"82f2aa67-8042-4bbb-930d-09ad08fd1696\") " pod="openshift-multus/multus-additional-cni-plugins-9z4f8" Jul 14 09:26:52.942315 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.941061 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0274da0d-7c52-4932-9ac3-732af03b0c85-multus-daemon-config\") pod \"multus-9xsfz\" (UID: \"0274da0d-7c52-4932-9ac3-732af03b0c85\") " pod="openshift-multus/multus-9xsfz" Jul 14 09:26:52.942315 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.941090 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f9a0e148-5f32-4d80-8963-321cbc4e5de0-ovnkube-config\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:52.942315 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.941117 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f9a0e148-5f32-4d80-8963-321cbc4e5de0-env-overrides\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:52.942315 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.941143 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0274da0d-7c52-4932-9ac3-732af03b0c85-cni-binary-copy\") pod \"multus-9xsfz\" (UID: \"0274da0d-7c52-4932-9ac3-732af03b0c85\") " pod="openshift-multus/multus-9xsfz" Jul 14 09:26:52.942315 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.941168 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0274da0d-7c52-4932-9ac3-732af03b0c85-host-run-k8s-cni-cncf-io\") pod \"multus-9xsfz\" (UID: \"0274da0d-7c52-4932-9ac3-732af03b0c85\") " pod="openshift-multus/multus-9xsfz" Jul 14 09:26:52.942315 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.941209 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/82f2aa67-8042-4bbb-930d-09ad08fd1696-cni-binary-copy\") pod \"multus-additional-cni-plugins-9z4f8\" (UID: \"82f2aa67-8042-4bbb-930d-09ad08fd1696\") " pod="openshift-multus/multus-additional-cni-plugins-9z4f8" Jul 14 09:26:52.942315 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.941232 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/82f2aa67-8042-4bbb-930d-09ad08fd1696-os-release\") pod \"multus-additional-cni-plugins-9z4f8\" (UID: \"82f2aa67-8042-4bbb-930d-09ad08fd1696\") " pod="openshift-multus/multus-additional-cni-plugins-9z4f8" Jul 14 09:26:52.943082 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.941260 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0274da0d-7c52-4932-9ac3-732af03b0c85-host-var-lib-kubelet\") pod \"multus-9xsfz\" (UID: \"0274da0d-7c52-4932-9ac3-732af03b0c85\") " pod="openshift-multus/multus-9xsfz" Jul 14 09:26:52.943082 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.941299 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0274da0d-7c52-4932-9ac3-732af03b0c85-hostroot\") pod \"multus-9xsfz\" (UID: \"0274da0d-7c52-4932-9ac3-732af03b0c85\") " pod="openshift-multus/multus-9xsfz" Jul 14 09:26:52.943082 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.941337 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0274da0d-7c52-4932-9ac3-732af03b0c85-multus-conf-dir\") pod \"multus-9xsfz\" (UID: \"0274da0d-7c52-4932-9ac3-732af03b0c85\") " pod="openshift-multus/multus-9xsfz" Jul 14 09:26:52.943082 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.941370 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f9a0e148-5f32-4d80-8963-321cbc4e5de0-node-log\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:52.943082 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.941398 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9a0e148-5f32-4d80-8963-321cbc4e5de0-host-run-ovn-kubernetes\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:52.943082 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.941450 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f9a0e148-5f32-4d80-8963-321cbc4e5de0-host-cni-netd\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:52.943082 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.941478 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dc1cb2ed-2875-4a0a-9089-c1f9240dd7be-sys-fs\") pod \"aws-ebs-csi-driver-node-mf5rb\" (UID: \"dc1cb2ed-2875-4a0a-9089-c1f9240dd7be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5rb" Jul 14 09:26:52.943082 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.941501 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3cfb37de-4170-43d8-aa35-d2f1c3c6cc90-serviceca\") pod \"node-ca-4kb48\" (UID: \"3cfb37de-4170-43d8-aa35-d2f1c3c6cc90\") " pod="openshift-image-registry/node-ca-4kb48" Jul 14 09:26:52.943082 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.941524 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4a7e6b7e-3c6a-4def-bc24-b9a00ccad67b-hosts-file\") pod \"node-resolver-5r769\" (UID: \"4a7e6b7e-3c6a-4def-bc24-b9a00ccad67b\") " pod="openshift-dns/node-resolver-5r769" Jul 14 09:26:52.943082 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.941544 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3da5cf33-ada6-411e-b56d-73c993bf77b7-etc-sysconfig\") pod \"tuned-ttvxk\" (UID: \"3da5cf33-ada6-411e-b56d-73c993bf77b7\") " pod="openshift-cluster-node-tuning-operator/tuned-ttvxk" Jul 14 09:26:52.943082 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.941559 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dc1cb2ed-2875-4a0a-9089-c1f9240dd7be-sys-fs\") pod \"aws-ebs-csi-driver-node-mf5rb\" (UID: \"dc1cb2ed-2875-4a0a-9089-c1f9240dd7be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5rb" Jul 14 09:26:52.943082 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.941567 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eebab604-45fc-4992-a289-96c5275ed378-iptables-alerter-script\") pod \"iptables-alerter-r5rg6\" (UID: \"eebab604-45fc-4992-a289-96c5275ed378\") " pod="openshift-network-operator/iptables-alerter-r5rg6" Jul 14 09:26:52.943082 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.941603 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0274da0d-7c52-4932-9ac3-732af03b0c85-host-var-lib-cni-bin\") pod \"multus-9xsfz\" (UID: \"0274da0d-7c52-4932-9ac3-732af03b0c85\") " pod="openshift-multus/multus-9xsfz" Jul 14 09:26:52.943082 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.941634 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0274da0d-7c52-4932-9ac3-732af03b0c85-etc-kubernetes\") pod \"multus-9xsfz\" (UID: \"0274da0d-7c52-4932-9ac3-732af03b0c85\") " pod="openshift-multus/multus-9xsfz" Jul 14 09:26:52.943082 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.941659 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f9a0e148-5f32-4d80-8963-321cbc4e5de0-run-ovn\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:52.943082 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.941685 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr2rs\" (UniqueName: \"kubernetes.io/projected/c2190520-5806-41dd-b25f-b7d1076cfcf3-kube-api-access-kr2rs\") pod \"network-metrics-daemon-p5547\" (UID: \"c2190520-5806-41dd-b25f-b7d1076cfcf3\") " pod="openshift-multus/network-metrics-daemon-p5547" Jul 14 09:26:52.943082 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.941716 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/82f2aa67-8042-4bbb-930d-09ad08fd1696-system-cni-dir\") pod \"multus-additional-cni-plugins-9z4f8\" (UID: \"82f2aa67-8042-4bbb-930d-09ad08fd1696\") " pod="openshift-multus/multus-additional-cni-plugins-9z4f8" Jul 14 09:26:52.943784 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.941741 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0274da0d-7c52-4932-9ac3-732af03b0c85-multus-cni-dir\") pod \"multus-9xsfz\" (UID: \"0274da0d-7c52-4932-9ac3-732af03b0c85\") " pod="openshift-multus/multus-9xsfz" Jul 14 09:26:52.943784 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.941747 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4a7e6b7e-3c6a-4def-bc24-b9a00ccad67b-hosts-file\") pod \"node-resolver-5r769\" (UID: \"4a7e6b7e-3c6a-4def-bc24-b9a00ccad67b\") " pod="openshift-dns/node-resolver-5r769" Jul 14 09:26:52.943784 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.941764 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0274da0d-7c52-4932-9ac3-732af03b0c85-cnibin\") pod \"multus-9xsfz\" (UID: \"0274da0d-7c52-4932-9ac3-732af03b0c85\") " pod="openshift-multus/multus-9xsfz" Jul 14 09:26:52.943784 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.941790 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9a0e148-5f32-4d80-8963-321cbc4e5de0-host-kubelet\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:52.943784 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.941803 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3da5cf33-ada6-411e-b56d-73c993bf77b7-etc-sysconfig\") pod \"tuned-ttvxk\" (UID: \"3da5cf33-ada6-411e-b56d-73c993bf77b7\") " pod="openshift-cluster-node-tuning-operator/tuned-ttvxk" Jul 14 09:26:52.943784 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.941818 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3da5cf33-ada6-411e-b56d-73c993bf77b7-run\") pod \"tuned-ttvxk\" (UID: \"3da5cf33-ada6-411e-b56d-73c993bf77b7\") " pod="openshift-cluster-node-tuning-operator/tuned-ttvxk" Jul 14 09:26:52.943784 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.941864 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3da5cf33-ada6-411e-b56d-73c993bf77b7-sys\") pod \"tuned-ttvxk\" (UID: \"3da5cf33-ada6-411e-b56d-73c993bf77b7\") " pod="openshift-cluster-node-tuning-operator/tuned-ttvxk" Jul 14 09:26:52.943784 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.941892 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3da5cf33-ada6-411e-b56d-73c993bf77b7-lib-modules\") pod \"tuned-ttvxk\" (UID: \"3da5cf33-ada6-411e-b56d-73c993bf77b7\") " pod="openshift-cluster-node-tuning-operator/tuned-ttvxk" Jul 14 09:26:52.943784 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.941902 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/82f2aa67-8042-4bbb-930d-09ad08fd1696-system-cni-dir\") pod \"multus-additional-cni-plugins-9z4f8\" (UID: \"82f2aa67-8042-4bbb-930d-09ad08fd1696\") " pod="openshift-multus/multus-additional-cni-plugins-9z4f8" Jul 14 09:26:52.943784 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.941919 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjvns\" (UniqueName: \"kubernetes.io/projected/3da5cf33-ada6-411e-b56d-73c993bf77b7-kube-api-access-rjvns\") pod \"tuned-ttvxk\" (UID: \"3da5cf33-ada6-411e-b56d-73c993bf77b7\") " pod="openshift-cluster-node-tuning-operator/tuned-ttvxk" Jul 14 09:26:52.943784 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.941949 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dc1cb2ed-2875-4a0a-9089-c1f9240dd7be-socket-dir\") pod \"aws-ebs-csi-driver-node-mf5rb\" (UID: \"dc1cb2ed-2875-4a0a-9089-c1f9240dd7be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5rb" Jul 14 09:26:52.943784 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.941957 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3da5cf33-ada6-411e-b56d-73c993bf77b7-sys\") pod \"tuned-ttvxk\" (UID: \"3da5cf33-ada6-411e-b56d-73c993bf77b7\") " pod="openshift-cluster-node-tuning-operator/tuned-ttvxk" Jul 14 09:26:52.943784 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.941961 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3da5cf33-ada6-411e-b56d-73c993bf77b7-run\") pod \"tuned-ttvxk\" (UID: \"3da5cf33-ada6-411e-b56d-73c993bf77b7\") " pod="openshift-cluster-node-tuning-operator/tuned-ttvxk" Jul 14 09:26:52.943784 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.941974 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9a0e148-5f32-4d80-8963-321cbc4e5de0-var-lib-openvswitch\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:52.943784 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.942009 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f9a0e148-5f32-4d80-8963-321cbc4e5de0-ovnkube-script-lib\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:52.943784 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.942040 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/82f2aa67-8042-4bbb-930d-09ad08fd1696-cnibin\") pod \"multus-additional-cni-plugins-9z4f8\" (UID: \"82f2aa67-8042-4bbb-930d-09ad08fd1696\") " pod="openshift-multus/multus-additional-cni-plugins-9z4f8" Jul 14 09:26:52.943784 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.942067 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/82f2aa67-8042-4bbb-930d-09ad08fd1696-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9z4f8\" (UID: \"82f2aa67-8042-4bbb-930d-09ad08fd1696\") " pod="openshift-multus/multus-additional-cni-plugins-9z4f8" Jul 14 09:26:52.944648 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.942088 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eebab604-45fc-4992-a289-96c5275ed378-iptables-alerter-script\") pod \"iptables-alerter-r5rg6\" (UID: \"eebab604-45fc-4992-a289-96c5275ed378\") " pod="openshift-network-operator/iptables-alerter-r5rg6" Jul 14 09:26:52.944648 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.942095 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dc1cb2ed-2875-4a0a-9089-c1f9240dd7be-socket-dir\") pod \"aws-ebs-csi-driver-node-mf5rb\" (UID: \"dc1cb2ed-2875-4a0a-9089-c1f9240dd7be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5rb" Jul 14 09:26:52.944648 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.942150 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/82f2aa67-8042-4bbb-930d-09ad08fd1696-cnibin\") pod \"multus-additional-cni-plugins-9z4f8\" (UID: \"82f2aa67-8042-4bbb-930d-09ad08fd1696\") " pod="openshift-multus/multus-additional-cni-plugins-9z4f8" Jul 14 09:26:52.944648 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.942215 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3da5cf33-ada6-411e-b56d-73c993bf77b7-lib-modules\") pod \"tuned-ttvxk\" (UID: \"3da5cf33-ada6-411e-b56d-73c993bf77b7\") " pod="openshift-cluster-node-tuning-operator/tuned-ttvxk" Jul 14 09:26:52.944648 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.942600 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/82f2aa67-8042-4bbb-930d-09ad08fd1696-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9z4f8\" (UID: \"82f2aa67-8042-4bbb-930d-09ad08fd1696\") " pod="openshift-multus/multus-additional-cni-plugins-9z4f8" Jul 14 09:26:52.944922 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.944898 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3da5cf33-ada6-411e-b56d-73c993bf77b7-etc-tuned\") pod \"tuned-ttvxk\" (UID: \"3da5cf33-ada6-411e-b56d-73c993bf77b7\") " pod="openshift-cluster-node-tuning-operator/tuned-ttvxk" Jul 14 09:26:52.945057 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.944941 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3da5cf33-ada6-411e-b56d-73c993bf77b7-tmp\") pod \"tuned-ttvxk\" (UID: \"3da5cf33-ada6-411e-b56d-73c993bf77b7\") " pod="openshift-cluster-node-tuning-operator/tuned-ttvxk" Jul 14 09:26:52.946876 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:52.946847 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 09:26:52.946876 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:52.946872 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 09:26:52.947067 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:52.946901 2565 projected.go:196] Error preparing data for projected volume kube-api-access-hs4nw for pod openshift-network-diagnostics/network-check-target-n8c7q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 09:26:52.947067 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:52.946995 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7225439c-6210-4bc5-b124-0d92487dcd0c-kube-api-access-hs4nw podName:7225439c-6210-4bc5-b124-0d92487dcd0c nodeName:}" failed. No retries permitted until 2026-07-14 09:26:53.446971627 +0000 UTC m=+2.956203069 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hs4nw" (UniqueName: "kubernetes.io/projected/7225439c-6210-4bc5-b124-0d92487dcd0c-kube-api-access-hs4nw") pod "network-check-target-n8c7q" (UID: "7225439c-6210-4bc5-b124-0d92487dcd0c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 09:26:52.950282 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.950228 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z8j6\" (UniqueName: \"kubernetes.io/projected/4a7e6b7e-3c6a-4def-bc24-b9a00ccad67b-kube-api-access-2z8j6\") pod \"node-resolver-5r769\" (UID: \"4a7e6b7e-3c6a-4def-bc24-b9a00ccad67b\") " pod="openshift-dns/node-resolver-5r769" Jul 14 09:26:52.951412 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.951382 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmpgj\" (UniqueName: \"kubernetes.io/projected/eebab604-45fc-4992-a289-96c5275ed378-kube-api-access-pmpgj\") pod \"iptables-alerter-r5rg6\" (UID: \"eebab604-45fc-4992-a289-96c5275ed378\") " pod="openshift-network-operator/iptables-alerter-r5rg6" Jul 14 09:26:52.953542 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.953497 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl5tn\" (UniqueName: \"kubernetes.io/projected/dc1cb2ed-2875-4a0a-9089-c1f9240dd7be-kube-api-access-tl5tn\") pod \"aws-ebs-csi-driver-node-mf5rb\" (UID: \"dc1cb2ed-2875-4a0a-9089-c1f9240dd7be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5rb" Jul 14 09:26:52.953992 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.953969 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjvns\" (UniqueName: \"kubernetes.io/projected/3da5cf33-ada6-411e-b56d-73c993bf77b7-kube-api-access-rjvns\") pod \"tuned-ttvxk\" (UID: \"3da5cf33-ada6-411e-b56d-73c993bf77b7\") " pod="openshift-cluster-node-tuning-operator/tuned-ttvxk" Jul 14 09:26:52.954501 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.954479 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzzvn\" (UniqueName: \"kubernetes.io/projected/82f2aa67-8042-4bbb-930d-09ad08fd1696-kube-api-access-dzzvn\") pod \"multus-additional-cni-plugins-9z4f8\" (UID: \"82f2aa67-8042-4bbb-930d-09ad08fd1696\") " pod="openshift-multus/multus-additional-cni-plugins-9z4f8" Jul 14 09:26:52.967645 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.967542 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-2.ec2.internal" event={"ID":"a3b116561a4900506d95c442fc3a920c","Type":"ContainerStarted","Data":"629ee640c1025f8838f83855a0360a69ad56c924927b20cf0e682432f55785d8"} Jul 14 09:26:52.969925 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:52.969892 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-2.ec2.internal" event={"ID":"9a1079819acdf201f17fa12e60d64a01","Type":"ContainerStarted","Data":"49cf688c34f96204ebd0248d36b7c65c12d6ed5dc9f564db7374e89ec3089823"} Jul 14 09:26:53.042388 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.042344 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0274da0d-7c52-4932-9ac3-732af03b0c85-multus-daemon-config\") pod \"multus-9xsfz\" (UID: \"0274da0d-7c52-4932-9ac3-732af03b0c85\") " pod="openshift-multus/multus-9xsfz" Jul 14 09:26:53.042388 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.042395 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f9a0e148-5f32-4d80-8963-321cbc4e5de0-ovnkube-config\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:53.042652 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.042423 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f9a0e148-5f32-4d80-8963-321cbc4e5de0-env-overrides\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:53.042652 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.042448 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0274da0d-7c52-4932-9ac3-732af03b0c85-cni-binary-copy\") pod \"multus-9xsfz\" (UID: \"0274da0d-7c52-4932-9ac3-732af03b0c85\") " pod="openshift-multus/multus-9xsfz" Jul 14 09:26:53.042652 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.042471 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0274da0d-7c52-4932-9ac3-732af03b0c85-host-run-k8s-cni-cncf-io\") pod \"multus-9xsfz\" (UID: \"0274da0d-7c52-4932-9ac3-732af03b0c85\") " pod="openshift-multus/multus-9xsfz" Jul 14 09:26:53.042652 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.042496 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0274da0d-7c52-4932-9ac3-732af03b0c85-host-var-lib-kubelet\") pod \"multus-9xsfz\" (UID: \"0274da0d-7c52-4932-9ac3-732af03b0c85\") " pod="openshift-multus/multus-9xsfz" Jul 14 09:26:53.042652 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.042522 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0274da0d-7c52-4932-9ac3-732af03b0c85-hostroot\") pod \"multus-9xsfz\" (UID: \"0274da0d-7c52-4932-9ac3-732af03b0c85\") " pod="openshift-multus/multus-9xsfz" Jul 14 09:26:53.042652 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.042554 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0274da0d-7c52-4932-9ac3-732af03b0c85-multus-conf-dir\") pod \"multus-9xsfz\" (UID: \"0274da0d-7c52-4932-9ac3-732af03b0c85\") " pod="openshift-multus/multus-9xsfz" Jul 14 09:26:53.042652 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.042578 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f9a0e148-5f32-4d80-8963-321cbc4e5de0-node-log\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:53.042652 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.042603 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9a0e148-5f32-4d80-8963-321cbc4e5de0-host-run-ovn-kubernetes\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:53.042652 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.042626 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f9a0e148-5f32-4d80-8963-321cbc4e5de0-host-cni-netd\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:53.042652 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.042652 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3cfb37de-4170-43d8-aa35-d2f1c3c6cc90-serviceca\") pod \"node-ca-4kb48\" (UID: \"3cfb37de-4170-43d8-aa35-d2f1c3c6cc90\") " pod="openshift-image-registry/node-ca-4kb48" Jul 14 09:26:53.043231 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.042678 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0274da0d-7c52-4932-9ac3-732af03b0c85-host-var-lib-cni-bin\") pod \"multus-9xsfz\" (UID: \"0274da0d-7c52-4932-9ac3-732af03b0c85\") " pod="openshift-multus/multus-9xsfz" Jul 14 09:26:53.043231 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.042703 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0274da0d-7c52-4932-9ac3-732af03b0c85-etc-kubernetes\") pod \"multus-9xsfz\" (UID: \"0274da0d-7c52-4932-9ac3-732af03b0c85\") " pod="openshift-multus/multus-9xsfz" Jul 14 09:26:53.043231 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.042729 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f9a0e148-5f32-4d80-8963-321cbc4e5de0-run-ovn\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:53.043231 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.042754 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr2rs\" (UniqueName: \"kubernetes.io/projected/c2190520-5806-41dd-b25f-b7d1076cfcf3-kube-api-access-kr2rs\") pod \"network-metrics-daemon-p5547\" (UID: \"c2190520-5806-41dd-b25f-b7d1076cfcf3\") " pod="openshift-multus/network-metrics-daemon-p5547" Jul 14 09:26:53.043231 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.042779 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0274da0d-7c52-4932-9ac3-732af03b0c85-multus-cni-dir\") pod \"multus-9xsfz\" (UID: \"0274da0d-7c52-4932-9ac3-732af03b0c85\") " pod="openshift-multus/multus-9xsfz" Jul 14 09:26:53.043231 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.042805 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0274da0d-7c52-4932-9ac3-732af03b0c85-cnibin\") pod \"multus-9xsfz\" (UID: \"0274da0d-7c52-4932-9ac3-732af03b0c85\") " pod="openshift-multus/multus-9xsfz" Jul 14 09:26:53.043231 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.042832 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9a0e148-5f32-4d80-8963-321cbc4e5de0-host-kubelet\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:53.043231 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.042880 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9a0e148-5f32-4d80-8963-321cbc4e5de0-var-lib-openvswitch\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:53.043231 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.042908 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f9a0e148-5f32-4d80-8963-321cbc4e5de0-ovnkube-script-lib\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:53.043231 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.042936 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0274da0d-7c52-4932-9ac3-732af03b0c85-multus-socket-dir-parent\") pod \"multus-9xsfz\" (UID: \"0274da0d-7c52-4932-9ac3-732af03b0c85\") " pod="openshift-multus/multus-9xsfz" Jul 14 09:26:53.043231 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.042964 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2190520-5806-41dd-b25f-b7d1076cfcf3-metrics-certs\") pod \"network-metrics-daemon-p5547\" (UID: \"c2190520-5806-41dd-b25f-b7d1076cfcf3\") " pod="openshift-multus/network-metrics-daemon-p5547" Jul 14 09:26:53.043231 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.042993 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0274da0d-7c52-4932-9ac3-732af03b0c85-host-run-multus-certs\") pod \"multus-9xsfz\" (UID: \"0274da0d-7c52-4932-9ac3-732af03b0c85\") " pod="openshift-multus/multus-9xsfz" Jul 14 09:26:53.043231 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.043019 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9a0e148-5f32-4d80-8963-321cbc4e5de0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:53.043231 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.043055 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0274da0d-7c52-4932-9ac3-732af03b0c85-host-var-lib-cni-multus\") pod \"multus-9xsfz\" (UID: \"0274da0d-7c52-4932-9ac3-732af03b0c85\") " pod="openshift-multus/multus-9xsfz" Jul 14 09:26:53.043231 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.043054 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0274da0d-7c52-4932-9ac3-732af03b0c85-multus-daemon-config\") pod \"multus-9xsfz\" (UID: \"0274da0d-7c52-4932-9ac3-732af03b0c85\") " pod="openshift-multus/multus-9xsfz" Jul 14 09:26:53.043231 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.043083 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a5c62b6a-f781-4066-9773-a5f4e8921276-kubelet-config\") pod \"global-pull-secret-syncer-2tcgq\" (UID: \"a5c62b6a-f781-4066-9773-a5f4e8921276\") " pod="kube-system/global-pull-secret-syncer-2tcgq" Jul 14 09:26:53.043231 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.043138 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f9a0e148-5f32-4d80-8963-321cbc4e5de0-env-overrides\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:53.044090 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.043168 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a5c62b6a-f781-4066-9773-a5f4e8921276-kubelet-config\") pod \"global-pull-secret-syncer-2tcgq\" (UID: \"a5c62b6a-f781-4066-9773-a5f4e8921276\") " pod="kube-system/global-pull-secret-syncer-2tcgq" Jul 14 09:26:53.044090 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.043210 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f9a0e148-5f32-4d80-8963-321cbc4e5de0-run-ovn\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:53.044090 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.043216 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0274da0d-7c52-4932-9ac3-732af03b0c85-host-var-lib-kubelet\") pod \"multus-9xsfz\" (UID: \"0274da0d-7c52-4932-9ac3-732af03b0c85\") " pod="openshift-multus/multus-9xsfz" Jul 14 09:26:53.044090 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.043248 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0274da0d-7c52-4932-9ac3-732af03b0c85-host-run-k8s-cni-cncf-io\") pod \"multus-9xsfz\" (UID: \"0274da0d-7c52-4932-9ac3-732af03b0c85\") " pod="openshift-multus/multus-9xsfz" Jul 14 09:26:53.044090 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.043268 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0274da0d-7c52-4932-9ac3-732af03b0c85-hostroot\") pod \"multus-9xsfz\" (UID: \"0274da0d-7c52-4932-9ac3-732af03b0c85\") " pod="openshift-multus/multus-9xsfz" Jul 14 09:26:53.044090 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.043312 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0274da0d-7c52-4932-9ac3-732af03b0c85-multus-conf-dir\") pod \"multus-9xsfz\" (UID: \"0274da0d-7c52-4932-9ac3-732af03b0c85\") " pod="openshift-multus/multus-9xsfz" Jul 14 09:26:53.044090 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.043412 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0274da0d-7c52-4932-9ac3-732af03b0c85-multus-cni-dir\") pod \"multus-9xsfz\" (UID: \"0274da0d-7c52-4932-9ac3-732af03b0c85\") " pod="openshift-multus/multus-9xsfz" Jul 14 09:26:53.044090 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.043494 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0274da0d-7c52-4932-9ac3-732af03b0c85-cnibin\") pod \"multus-9xsfz\" (UID: \"0274da0d-7c52-4932-9ac3-732af03b0c85\") " pod="openshift-multus/multus-9xsfz" Jul 14 09:26:53.044090 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.043532 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f9a0e148-5f32-4d80-8963-321cbc4e5de0-ovnkube-config\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:53.044090 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.043562 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f9a0e148-5f32-4d80-8963-321cbc4e5de0-node-log\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:53.044090 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.043572 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f9a0e148-5f32-4d80-8963-321cbc4e5de0-systemd-units\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:53.044090 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.043591 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f9a0e148-5f32-4d80-8963-321cbc4e5de0-host-slash\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:53.044090 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.043600 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0274da0d-7c52-4932-9ac3-732af03b0c85-multus-socket-dir-parent\") pod \"multus-9xsfz\" (UID: \"0274da0d-7c52-4932-9ac3-732af03b0c85\") " pod="openshift-multus/multus-9xsfz" Jul 14 09:26:53.044090 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.043535 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9a0e148-5f32-4d80-8963-321cbc4e5de0-host-kubelet\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:53.044090 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.043619 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9a0e148-5f32-4d80-8963-321cbc4e5de0-host-run-ovn-kubernetes\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:53.044090 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.043642 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9a0e148-5f32-4d80-8963-321cbc4e5de0-var-lib-openvswitch\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:53.044090 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.043642 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f9a0e148-5f32-4d80-8963-321cbc4e5de0-log-socket\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:53.044090 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.043666 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f9a0e148-5f32-4d80-8963-321cbc4e5de0-host-cni-netd\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:53.045074 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.043671 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f9a0e148-5f32-4d80-8963-321cbc4e5de0-systemd-units\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:53.045074 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.043691 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f9a0e148-5f32-4d80-8963-321cbc4e5de0-host-slash\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:53.045074 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.043720 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0274da0d-7c52-4932-9ac3-732af03b0c85-host-var-lib-cni-bin\") pod \"multus-9xsfz\" (UID: \"0274da0d-7c52-4932-9ac3-732af03b0c85\") " pod="openshift-multus/multus-9xsfz" Jul 14 09:26:53.045074 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.043732 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0274da0d-7c52-4932-9ac3-732af03b0c85-etc-kubernetes\") pod \"multus-9xsfz\" (UID: \"0274da0d-7c52-4932-9ac3-732af03b0c85\") " pod="openshift-multus/multus-9xsfz" Jul 14 09:26:53.045074 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.043777 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9a0e148-5f32-4d80-8963-321cbc4e5de0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:53.045074 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:53.043867 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 09:26:53.045074 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:53.043947 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2190520-5806-41dd-b25f-b7d1076cfcf3-metrics-certs podName:c2190520-5806-41dd-b25f-b7d1076cfcf3 nodeName:}" failed. No retries permitted until 2026-07-14 09:26:53.543929027 +0000 UTC m=+3.053160489 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c2190520-5806-41dd-b25f-b7d1076cfcf3-metrics-certs") pod "network-metrics-daemon-p5547" (UID: "c2190520-5806-41dd-b25f-b7d1076cfcf3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 09:26:53.045074 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.044021 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3cfb37de-4170-43d8-aa35-d2f1c3c6cc90-serviceca\") pod \"node-ca-4kb48\" (UID: \"3cfb37de-4170-43d8-aa35-d2f1c3c6cc90\") " pod="openshift-image-registry/node-ca-4kb48" Jul 14 09:26:53.045074 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.044164 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f9a0e148-5f32-4d80-8963-321cbc4e5de0-ovnkube-script-lib\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:53.045074 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.044184 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0274da0d-7c52-4932-9ac3-732af03b0c85-host-run-multus-certs\") pod \"multus-9xsfz\" (UID: \"0274da0d-7c52-4932-9ac3-732af03b0c85\") " pod="openshift-multus/multus-9xsfz" Jul 14 09:26:53.045074 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.043607 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f9a0e148-5f32-4d80-8963-321cbc4e5de0-log-socket\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:53.045074 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.044224 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dc4d1b63-99f8-4738-9dee-76fe0e4e4e4d-konnectivity-ca\") pod \"konnectivity-agent-jjwzx\" (UID: \"dc4d1b63-99f8-4738-9dee-76fe0e4e4e4d\") " pod="kube-system/konnectivity-agent-jjwzx" Jul 14 09:26:53.045074 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.044248 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0274da0d-7c52-4932-9ac3-732af03b0c85-host-var-lib-cni-multus\") pod \"multus-9xsfz\" (UID: \"0274da0d-7c52-4932-9ac3-732af03b0c85\") " pod="openshift-multus/multus-9xsfz" Jul 14 09:26:53.045074 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.044251 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3cfb37de-4170-43d8-aa35-d2f1c3c6cc90-host\") pod \"node-ca-4kb48\" (UID: \"3cfb37de-4170-43d8-aa35-d2f1c3c6cc90\") " pod="openshift-image-registry/node-ca-4kb48" Jul 14 09:26:53.045074 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.044290 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0274da0d-7c52-4932-9ac3-732af03b0c85-system-cni-dir\") pod \"multus-9xsfz\" (UID: \"0274da0d-7c52-4932-9ac3-732af03b0c85\") " pod="openshift-multus/multus-9xsfz" Jul 14 09:26:53.045074 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.044310 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3cfb37de-4170-43d8-aa35-d2f1c3c6cc90-host\") pod \"node-ca-4kb48\" (UID: \"3cfb37de-4170-43d8-aa35-d2f1c3c6cc90\") " pod="openshift-image-registry/node-ca-4kb48" Jul 14 09:26:53.045074 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.044322 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0274da0d-7c52-4932-9ac3-732af03b0c85-cni-binary-copy\") pod \"multus-9xsfz\" (UID: \"0274da0d-7c52-4932-9ac3-732af03b0c85\") " pod="openshift-multus/multus-9xsfz" Jul 14 09:26:53.045074 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.044333 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0274da0d-7c52-4932-9ac3-732af03b0c85-os-release\") pod \"multus-9xsfz\" (UID: \"0274da0d-7c52-4932-9ac3-732af03b0c85\") " pod="openshift-multus/multus-9xsfz" Jul 14 09:26:53.046002 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.044359 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlzqc\" (UniqueName: \"kubernetes.io/projected/0274da0d-7c52-4932-9ac3-732af03b0c85-kube-api-access-wlzqc\") pod \"multus-9xsfz\" (UID: \"0274da0d-7c52-4932-9ac3-732af03b0c85\") " pod="openshift-multus/multus-9xsfz" Jul 14 09:26:53.046002 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.044367 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0274da0d-7c52-4932-9ac3-732af03b0c85-system-cni-dir\") pod \"multus-9xsfz\" (UID: \"0274da0d-7c52-4932-9ac3-732af03b0c85\") " pod="openshift-multus/multus-9xsfz" Jul 14 09:26:53.046002 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.044389 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f9a0e148-5f32-4d80-8963-321cbc4e5de0-host-run-netns\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:53.046002 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.044425 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a5c62b6a-f781-4066-9773-a5f4e8921276-dbus\") pod \"global-pull-secret-syncer-2tcgq\" (UID: \"a5c62b6a-f781-4066-9773-a5f4e8921276\") " pod="kube-system/global-pull-secret-syncer-2tcgq" Jul 14 09:26:53.046002 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.044442 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a5c62b6a-f781-4066-9773-a5f4e8921276-original-pull-secret\") pod \"global-pull-secret-syncer-2tcgq\" (UID: \"a5c62b6a-f781-4066-9773-a5f4e8921276\") " pod="kube-system/global-pull-secret-syncer-2tcgq" Jul 14 09:26:53.046002 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.044460 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f9a0e148-5f32-4d80-8963-321cbc4e5de0-run-systemd\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:53.046002 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.044477 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9a0e148-5f32-4d80-8963-321cbc4e5de0-etc-openvswitch\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:53.046002 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.044492 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f9a0e148-5f32-4d80-8963-321cbc4e5de0-host-cni-bin\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:53.046002 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.044507 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f9a0e148-5f32-4d80-8963-321cbc4e5de0-ovn-node-metrics-cert\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:53.046002 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.044522 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dc4d1b63-99f8-4738-9dee-76fe0e4e4e4d-agent-certs\") pod \"konnectivity-agent-jjwzx\" (UID: \"dc4d1b63-99f8-4738-9dee-76fe0e4e4e4d\") " pod="kube-system/konnectivity-agent-jjwzx" Jul 14 09:26:53.046002 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.044544 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9a0e148-5f32-4d80-8963-321cbc4e5de0-run-openvswitch\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:53.046002 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.044561 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0274da0d-7c52-4932-9ac3-732af03b0c85-host-run-netns\") pod \"multus-9xsfz\" (UID: \"0274da0d-7c52-4932-9ac3-732af03b0c85\") " pod="openshift-multus/multus-9xsfz" Jul 14 09:26:53.046002 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.044591 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f9a0e148-5f32-4d80-8963-321cbc4e5de0-host-cni-bin\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:53.046002 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.044598 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2ktj\" (UniqueName: \"kubernetes.io/projected/f9a0e148-5f32-4d80-8963-321cbc4e5de0-kube-api-access-g2ktj\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:53.046002 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.044666 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rc2d\" (UniqueName: \"kubernetes.io/projected/3cfb37de-4170-43d8-aa35-d2f1c3c6cc90-kube-api-access-5rc2d\") pod \"node-ca-4kb48\" (UID: \"3cfb37de-4170-43d8-aa35-d2f1c3c6cc90\") " pod="openshift-image-registry/node-ca-4kb48" Jul 14 09:26:53.046002 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.044689 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0274da0d-7c52-4932-9ac3-732af03b0c85-os-release\") pod \"multus-9xsfz\" (UID: \"0274da0d-7c52-4932-9ac3-732af03b0c85\") " pod="openshift-multus/multus-9xsfz" Jul 14 09:26:53.046002 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.044797 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dc4d1b63-99f8-4738-9dee-76fe0e4e4e4d-konnectivity-ca\") pod \"konnectivity-agent-jjwzx\" (UID: \"dc4d1b63-99f8-4738-9dee-76fe0e4e4e4d\") " pod="kube-system/konnectivity-agent-jjwzx" Jul 14 09:26:53.046002 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.044992 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f9a0e148-5f32-4d80-8963-321cbc4e5de0-host-run-netns\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:53.046819 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.045113 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a5c62b6a-f781-4066-9773-a5f4e8921276-dbus\") pod \"global-pull-secret-syncer-2tcgq\" (UID: \"a5c62b6a-f781-4066-9773-a5f4e8921276\") " pod="kube-system/global-pull-secret-syncer-2tcgq" Jul 14 09:26:53.046819 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:53.045196 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 09:26:53.046819 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:53.045241 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5c62b6a-f781-4066-9773-a5f4e8921276-original-pull-secret podName:a5c62b6a-f781-4066-9773-a5f4e8921276 nodeName:}" failed. No retries permitted until 2026-07-14 09:26:53.545225687 +0000 UTC m=+3.054457119 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a5c62b6a-f781-4066-9773-a5f4e8921276-original-pull-secret") pod "global-pull-secret-syncer-2tcgq" (UID: "a5c62b6a-f781-4066-9773-a5f4e8921276") : object "kube-system"/"original-pull-secret" not registered Jul 14 09:26:53.046819 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.045281 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f9a0e148-5f32-4d80-8963-321cbc4e5de0-run-systemd\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:53.046819 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.045319 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9a0e148-5f32-4d80-8963-321cbc4e5de0-run-openvswitch\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:53.046819 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.045344 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9a0e148-5f32-4d80-8963-321cbc4e5de0-etc-openvswitch\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:53.046819 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.045380 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0274da0d-7c52-4932-9ac3-732af03b0c85-host-run-netns\") pod \"multus-9xsfz\" (UID: \"0274da0d-7c52-4932-9ac3-732af03b0c85\") " pod="openshift-multus/multus-9xsfz" Jul 14 09:26:53.047593 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.047555 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f9a0e148-5f32-4d80-8963-321cbc4e5de0-ovn-node-metrics-cert\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:53.047692 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.047667 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dc4d1b63-99f8-4738-9dee-76fe0e4e4e4d-agent-certs\") pod \"konnectivity-agent-jjwzx\" (UID: \"dc4d1b63-99f8-4738-9dee-76fe0e4e4e4d\") " pod="kube-system/konnectivity-agent-jjwzx" Jul 14 09:26:53.054381 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.054282 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr2rs\" (UniqueName: \"kubernetes.io/projected/c2190520-5806-41dd-b25f-b7d1076cfcf3-kube-api-access-kr2rs\") pod \"network-metrics-daemon-p5547\" (UID: \"c2190520-5806-41dd-b25f-b7d1076cfcf3\") " pod="openshift-multus/network-metrics-daemon-p5547" Jul 14 09:26:53.054506 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.054388 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlzqc\" (UniqueName: \"kubernetes.io/projected/0274da0d-7c52-4932-9ac3-732af03b0c85-kube-api-access-wlzqc\") pod \"multus-9xsfz\" (UID: \"0274da0d-7c52-4932-9ac3-732af03b0c85\") " pod="openshift-multus/multus-9xsfz" Jul 14 09:26:53.054786 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.054763 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rc2d\" (UniqueName: \"kubernetes.io/projected/3cfb37de-4170-43d8-aa35-d2f1c3c6cc90-kube-api-access-5rc2d\") pod \"node-ca-4kb48\" (UID: \"3cfb37de-4170-43d8-aa35-d2f1c3c6cc90\") " pod="openshift-image-registry/node-ca-4kb48" Jul 14 09:26:53.054786 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.054773 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2ktj\" (UniqueName: \"kubernetes.io/projected/f9a0e148-5f32-4d80-8963-321cbc4e5de0-kube-api-access-g2ktj\") pod \"ovnkube-node-r79mk\" (UID: \"f9a0e148-5f32-4d80-8963-321cbc4e5de0\") " pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:53.145585 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.145542 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5rb" Jul 14 09:26:53.157152 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.157120 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9z4f8" Jul 14 09:26:53.168276 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.168253 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ttvxk" Jul 14 09:26:53.183798 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.183770 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5r769" Jul 14 09:26:53.192762 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.192731 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-r5rg6" Jul 14 09:26:53.201896 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.201871 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:26:53.209867 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.209848 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jjwzx" Jul 14 09:26:53.217804 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.217748 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4kb48" Jul 14 09:26:53.223654 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.223634 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9xsfz" Jul 14 09:26:53.448620 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.448584 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs4nw\" (UniqueName: \"kubernetes.io/projected/7225439c-6210-4bc5-b124-0d92487dcd0c-kube-api-access-hs4nw\") pod \"network-check-target-n8c7q\" (UID: \"7225439c-6210-4bc5-b124-0d92487dcd0c\") " pod="openshift-network-diagnostics/network-check-target-n8c7q" Jul 14 09:26:53.448794 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:53.448773 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 09:26:53.448834 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:53.448800 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 09:26:53.448834 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:53.448810 2565 projected.go:196] Error preparing data for projected volume kube-api-access-hs4nw for pod openshift-network-diagnostics/network-check-target-n8c7q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 09:26:53.448915 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:53.448881 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7225439c-6210-4bc5-b124-0d92487dcd0c-kube-api-access-hs4nw podName:7225439c-6210-4bc5-b124-0d92487dcd0c nodeName:}" failed. No retries permitted until 2026-07-14 09:26:54.448862364 +0000 UTC m=+3.958093793 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hs4nw" (UniqueName: "kubernetes.io/projected/7225439c-6210-4bc5-b124-0d92487dcd0c-kube-api-access-hs4nw") pod "network-check-target-n8c7q" (UID: "7225439c-6210-4bc5-b124-0d92487dcd0c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 09:26:53.549265 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.549161 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a5c62b6a-f781-4066-9773-a5f4e8921276-original-pull-secret\") pod \"global-pull-secret-syncer-2tcgq\" (UID: \"a5c62b6a-f781-4066-9773-a5f4e8921276\") " pod="kube-system/global-pull-secret-syncer-2tcgq" Jul 14 09:26:53.549426 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.549273 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2190520-5806-41dd-b25f-b7d1076cfcf3-metrics-certs\") pod \"network-metrics-daemon-p5547\" (UID: \"c2190520-5806-41dd-b25f-b7d1076cfcf3\") " pod="openshift-multus/network-metrics-daemon-p5547" Jul 14 09:26:53.549426 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:53.549341 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 09:26:53.549426 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:53.549382 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 09:26:53.549593 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:53.549432 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5c62b6a-f781-4066-9773-a5f4e8921276-original-pull-secret podName:a5c62b6a-f781-4066-9773-a5f4e8921276 nodeName:}" failed. No retries permitted until 2026-07-14 09:26:54.549403155 +0000 UTC m=+4.058634584 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a5c62b6a-f781-4066-9773-a5f4e8921276-original-pull-secret") pod "global-pull-secret-syncer-2tcgq" (UID: "a5c62b6a-f781-4066-9773-a5f4e8921276") : object "kube-system"/"original-pull-secret" not registered Jul 14 09:26:53.549593 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:53.549463 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2190520-5806-41dd-b25f-b7d1076cfcf3-metrics-certs podName:c2190520-5806-41dd-b25f-b7d1076cfcf3 nodeName:}" failed. No retries permitted until 2026-07-14 09:26:54.54944282 +0000 UTC m=+4.058674247 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c2190520-5806-41dd-b25f-b7d1076cfcf3-metrics-certs") pod "network-metrics-daemon-p5547" (UID: "c2190520-5806-41dd-b25f-b7d1076cfcf3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 09:26:53.876741 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.876652 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 09:21:53 +0000 UTC" deadline="2028-01-12 16:29:19.080842902 +0000 UTC" Jul 14 09:26:53.876741 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:53.876690 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13135h2m25.204156893s" Jul 14 09:26:54.182212 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:54.182121 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cfb37de_4170_43d8_aa35_d2f1c3c6cc90.slice/crio-2197bdd5db68f0c11c2257c1d87fefaa38746c8ac4ccd9755f392a517c2e1078 WatchSource:0}: Error finding container 2197bdd5db68f0c11c2257c1d87fefaa38746c8ac4ccd9755f392a517c2e1078: Status 404 returned error can't find the container with id 2197bdd5db68f0c11c2257c1d87fefaa38746c8ac4ccd9755f392a517c2e1078 Jul 14 09:26:54.183802 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:54.183667 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc1cb2ed_2875_4a0a_9089_c1f9240dd7be.slice/crio-ef013f5606466c4e8e69368e66b5c0ec500bbb6cfdcc4c3b1d6ca83da837b018 WatchSource:0}: Error finding container ef013f5606466c4e8e69368e66b5c0ec500bbb6cfdcc4c3b1d6ca83da837b018: Status 404 returned error can't find the container with id ef013f5606466c4e8e69368e66b5c0ec500bbb6cfdcc4c3b1d6ca83da837b018 Jul 14 09:26:54.187027 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:54.187003 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82f2aa67_8042_4bbb_930d_09ad08fd1696.slice/crio-869806b596b272b34ac3582f3a67ab03eddabc0dc20608eb466ed09b73431f77 WatchSource:0}: Error finding container 869806b596b272b34ac3582f3a67ab03eddabc0dc20608eb466ed09b73431f77: Status 404 returned error can't find the container with id 869806b596b272b34ac3582f3a67ab03eddabc0dc20608eb466ed09b73431f77 Jul 14 09:26:54.189167 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:26:54.188930 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9a0e148_5f32_4d80_8963_321cbc4e5de0.slice/crio-196d6107202631a6e940f3cdcc7f288e5e228e26b33be7420c33cfff76e35743 WatchSource:0}: Error finding container 196d6107202631a6e940f3cdcc7f288e5e228e26b33be7420c33cfff76e35743: Status 404 returned error can't find the container with id 196d6107202631a6e940f3cdcc7f288e5e228e26b33be7420c33cfff76e35743 Jul 14 09:26:54.455901 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:54.455707 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs4nw\" (UniqueName: \"kubernetes.io/projected/7225439c-6210-4bc5-b124-0d92487dcd0c-kube-api-access-hs4nw\") pod \"network-check-target-n8c7q\" (UID: \"7225439c-6210-4bc5-b124-0d92487dcd0c\") " pod="openshift-network-diagnostics/network-check-target-n8c7q" Jul 14 09:26:54.456011 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:54.455941 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 09:26:54.456011 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:54.455972 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 09:26:54.456011 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:54.455986 2565 projected.go:196] Error preparing data for projected volume kube-api-access-hs4nw for pod openshift-network-diagnostics/network-check-target-n8c7q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 09:26:54.456145 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:54.456060 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7225439c-6210-4bc5-b124-0d92487dcd0c-kube-api-access-hs4nw podName:7225439c-6210-4bc5-b124-0d92487dcd0c nodeName:}" failed. No retries permitted until 2026-07-14 09:26:56.456037557 +0000 UTC m=+5.965269004 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hs4nw" (UniqueName: "kubernetes.io/projected/7225439c-6210-4bc5-b124-0d92487dcd0c-kube-api-access-hs4nw") pod "network-check-target-n8c7q" (UID: "7225439c-6210-4bc5-b124-0d92487dcd0c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 09:26:54.556829 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:54.556792 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2190520-5806-41dd-b25f-b7d1076cfcf3-metrics-certs\") pod \"network-metrics-daemon-p5547\" (UID: \"c2190520-5806-41dd-b25f-b7d1076cfcf3\") " pod="openshift-multus/network-metrics-daemon-p5547" Jul 14 09:26:54.556977 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:54.556885 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a5c62b6a-f781-4066-9773-a5f4e8921276-original-pull-secret\") pod \"global-pull-secret-syncer-2tcgq\" (UID: \"a5c62b6a-f781-4066-9773-a5f4e8921276\") " pod="kube-system/global-pull-secret-syncer-2tcgq" Jul 14 09:26:54.557042 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:54.556972 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 09:26:54.557042 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:54.556994 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 09:26:54.557122 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:54.557047 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2190520-5806-41dd-b25f-b7d1076cfcf3-metrics-certs podName:c2190520-5806-41dd-b25f-b7d1076cfcf3 nodeName:}" failed. No retries permitted until 2026-07-14 09:26:56.557028089 +0000 UTC m=+6.066259535 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c2190520-5806-41dd-b25f-b7d1076cfcf3-metrics-certs") pod "network-metrics-daemon-p5547" (UID: "c2190520-5806-41dd-b25f-b7d1076cfcf3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 09:26:54.557122 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:54.557065 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5c62b6a-f781-4066-9773-a5f4e8921276-original-pull-secret podName:a5c62b6a-f781-4066-9773-a5f4e8921276 nodeName:}" failed. No retries permitted until 2026-07-14 09:26:56.557057431 +0000 UTC m=+6.066288859 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a5c62b6a-f781-4066-9773-a5f4e8921276-original-pull-secret") pod "global-pull-secret-syncer-2tcgq" (UID: "a5c62b6a-f781-4066-9773-a5f4e8921276") : object "kube-system"/"original-pull-secret" not registered Jul 14 09:26:54.886836 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:54.886761 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 09:21:53 +0000 UTC" deadline="2028-01-11 12:34:52.299981707 +0000 UTC" Jul 14 09:26:54.886836 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:54.886834 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13107h7m57.413151665s" Jul 14 09:26:54.965269 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:54.965238 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p5547" Jul 14 09:26:54.965487 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:54.965384 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p5547" podUID="c2190520-5806-41dd-b25f-b7d1076cfcf3" Jul 14 09:26:54.965808 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:54.965790 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tcgq" Jul 14 09:26:54.966245 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:54.965971 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8c7q" Jul 14 09:26:54.966245 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:54.966051 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8c7q" podUID="7225439c-6210-4bc5-b124-0d92487dcd0c" Jul 14 09:26:54.966549 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:54.966486 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tcgq" podUID="a5c62b6a-f781-4066-9773-a5f4e8921276" Jul 14 09:26:54.989449 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:54.989411 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-2.ec2.internal" event={"ID":"9a1079819acdf201f17fa12e60d64a01","Type":"ContainerStarted","Data":"c1483b93a26bad01d44c0b2e76d897851989d2e85b7c31fa521babb0e3b4c138"} Jul 14 09:26:54.993670 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:54.993634 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jjwzx" event={"ID":"dc4d1b63-99f8-4738-9dee-76fe0e4e4e4d","Type":"ContainerStarted","Data":"c6249458bc269392a3b5b7577eb0a5743b17c431a9987006c2d3888de01e2001"} Jul 14 09:26:54.998139 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:54.998108 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-r5rg6" event={"ID":"eebab604-45fc-4992-a289-96c5275ed378","Type":"ContainerStarted","Data":"4ab12bc1cd15d0acaeec562a950d4ea8bc5fef17ca6017d7af0f6c7037bdf0c6"} Jul 14 09:26:55.002790 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:55.002755 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ttvxk" event={"ID":"3da5cf33-ada6-411e-b56d-73c993bf77b7","Type":"ContainerStarted","Data":"83595ed8e4163a598fb3ebd21717055f83846c9760a313d57c06c9bb76b95712"} Jul 14 09:26:55.009811 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:55.009778 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9xsfz" event={"ID":"0274da0d-7c52-4932-9ac3-732af03b0c85","Type":"ContainerStarted","Data":"51ad1425743e5a5c415168dac3d87463b1359f39309d15ad8ec75b111c07fb79"} Jul 14 09:26:55.019166 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:55.019136 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" event={"ID":"f9a0e148-5f32-4d80-8963-321cbc4e5de0","Type":"ContainerStarted","Data":"196d6107202631a6e940f3cdcc7f288e5e228e26b33be7420c33cfff76e35743"} Jul 14 09:26:55.023615 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:55.023549 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9z4f8" event={"ID":"82f2aa67-8042-4bbb-930d-09ad08fd1696","Type":"ContainerStarted","Data":"869806b596b272b34ac3582f3a67ab03eddabc0dc20608eb466ed09b73431f77"} Jul 14 09:26:55.035395 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:55.035358 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5rb" event={"ID":"dc1cb2ed-2875-4a0a-9089-c1f9240dd7be","Type":"ContainerStarted","Data":"ef013f5606466c4e8e69368e66b5c0ec500bbb6cfdcc4c3b1d6ca83da837b018"} Jul 14 09:26:55.040739 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:55.040705 2565 generic.go:356] "Generic (PLEG): container finished" podID="a3b116561a4900506d95c442fc3a920c" containerID="0530fa2135a923dcd3ca33b9eee0f36cab6223949dd78ad6d6976bb1570dd625" exitCode=0 Jul 14 09:26:55.041348 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:55.041322 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-2.ec2.internal" event={"ID":"a3b116561a4900506d95c442fc3a920c","Type":"ContainerDied","Data":"0530fa2135a923dcd3ca33b9eee0f36cab6223949dd78ad6d6976bb1570dd625"} Jul 14 09:26:55.063111 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:55.062801 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-2.ec2.internal" podStartSLOduration=3.062776134 podStartE2EDuration="3.062776134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 09:26:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:26:55.006224707 +0000 UTC m=+4.515456159" watchObservedRunningTime="2026-07-14 09:26:55.062776134 +0000 UTC m=+4.572007591" Jul 14 09:26:55.067110 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:55.067021 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5r769" event={"ID":"4a7e6b7e-3c6a-4def-bc24-b9a00ccad67b","Type":"ContainerStarted","Data":"402d126192018aba56253da93e9ba79b935bddabbf3bdbf6f5cd448f7ef5d781"} Jul 14 09:26:55.073816 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:55.073780 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4kb48" event={"ID":"3cfb37de-4170-43d8-aa35-d2f1c3c6cc90","Type":"ContainerStarted","Data":"2197bdd5db68f0c11c2257c1d87fefaa38746c8ac4ccd9755f392a517c2e1078"} Jul 14 09:26:56.095605 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:56.095565 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-2.ec2.internal" event={"ID":"a3b116561a4900506d95c442fc3a920c","Type":"ContainerStarted","Data":"28d411f7f6362e4cfd96523f0324c9dfa766f72bce7069108aab3d6e063bebf8"} Jul 14 09:26:56.111746 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:56.111679 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-2.ec2.internal" podStartSLOduration=4.111657746 podStartE2EDuration="4.111657746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 09:26:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:26:56.11148074 +0000 UTC m=+5.620712193" watchObservedRunningTime="2026-07-14 09:26:56.111657746 +0000 UTC m=+5.620889197" Jul 14 09:26:56.475292 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:56.474482 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs4nw\" (UniqueName: \"kubernetes.io/projected/7225439c-6210-4bc5-b124-0d92487dcd0c-kube-api-access-hs4nw\") pod \"network-check-target-n8c7q\" (UID: \"7225439c-6210-4bc5-b124-0d92487dcd0c\") " pod="openshift-network-diagnostics/network-check-target-n8c7q" Jul 14 09:26:56.475292 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:56.474683 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 09:26:56.475292 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:56.474706 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 09:26:56.475292 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:56.474719 2565 projected.go:196] Error preparing data for projected volume kube-api-access-hs4nw for pod openshift-network-diagnostics/network-check-target-n8c7q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 09:26:56.475292 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:56.474783 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7225439c-6210-4bc5-b124-0d92487dcd0c-kube-api-access-hs4nw podName:7225439c-6210-4bc5-b124-0d92487dcd0c nodeName:}" failed. No retries permitted until 2026-07-14 09:27:00.47476283 +0000 UTC m=+9.983994262 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hs4nw" (UniqueName: "kubernetes.io/projected/7225439c-6210-4bc5-b124-0d92487dcd0c-kube-api-access-hs4nw") pod "network-check-target-n8c7q" (UID: "7225439c-6210-4bc5-b124-0d92487dcd0c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 09:26:56.576589 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:56.575755 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2190520-5806-41dd-b25f-b7d1076cfcf3-metrics-certs\") pod \"network-metrics-daemon-p5547\" (UID: \"c2190520-5806-41dd-b25f-b7d1076cfcf3\") " pod="openshift-multus/network-metrics-daemon-p5547" Jul 14 09:26:56.576589 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:56.575853 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a5c62b6a-f781-4066-9773-a5f4e8921276-original-pull-secret\") pod \"global-pull-secret-syncer-2tcgq\" (UID: \"a5c62b6a-f781-4066-9773-a5f4e8921276\") " pod="kube-system/global-pull-secret-syncer-2tcgq" Jul 14 09:26:56.576589 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:56.576038 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 09:26:56.576589 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:56.576105 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5c62b6a-f781-4066-9773-a5f4e8921276-original-pull-secret podName:a5c62b6a-f781-4066-9773-a5f4e8921276 nodeName:}" failed. No retries permitted until 2026-07-14 09:27:00.576085652 +0000 UTC m=+10.085317081 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a5c62b6a-f781-4066-9773-a5f4e8921276-original-pull-secret") pod "global-pull-secret-syncer-2tcgq" (UID: "a5c62b6a-f781-4066-9773-a5f4e8921276") : object "kube-system"/"original-pull-secret" not registered Jul 14 09:26:56.576589 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:56.576504 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 09:26:56.576589 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:56.576552 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2190520-5806-41dd-b25f-b7d1076cfcf3-metrics-certs podName:c2190520-5806-41dd-b25f-b7d1076cfcf3 nodeName:}" failed. No retries permitted until 2026-07-14 09:27:00.576537067 +0000 UTC m=+10.085768500 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c2190520-5806-41dd-b25f-b7d1076cfcf3-metrics-certs") pod "network-metrics-daemon-p5547" (UID: "c2190520-5806-41dd-b25f-b7d1076cfcf3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 09:26:56.964627 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:56.964041 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8c7q" Jul 14 09:26:56.964627 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:56.964063 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p5547" Jul 14 09:26:56.964627 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:56.964186 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8c7q" podUID="7225439c-6210-4bc5-b124-0d92487dcd0c" Jul 14 09:26:56.964627 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:56.964254 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tcgq" Jul 14 09:26:56.964627 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:56.964370 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p5547" podUID="c2190520-5806-41dd-b25f-b7d1076cfcf3" Jul 14 09:26:56.964627 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:56.964443 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tcgq" podUID="a5c62b6a-f781-4066-9773-a5f4e8921276" Jul 14 09:26:58.962854 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:58.962795 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p5547" Jul 14 09:26:58.963397 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:58.962972 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p5547" podUID="c2190520-5806-41dd-b25f-b7d1076cfcf3" Jul 14 09:26:58.963397 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:58.962987 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8c7q" Jul 14 09:26:58.963397 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:58.963113 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8c7q" podUID="7225439c-6210-4bc5-b124-0d92487dcd0c" Jul 14 09:26:58.963397 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:26:58.963187 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tcgq" Jul 14 09:26:58.963397 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:26:58.963265 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tcgq" podUID="a5c62b6a-f781-4066-9773-a5f4e8921276" Jul 14 09:27:00.509964 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:00.509197 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs4nw\" (UniqueName: \"kubernetes.io/projected/7225439c-6210-4bc5-b124-0d92487dcd0c-kube-api-access-hs4nw\") pod \"network-check-target-n8c7q\" (UID: \"7225439c-6210-4bc5-b124-0d92487dcd0c\") " pod="openshift-network-diagnostics/network-check-target-n8c7q" Jul 14 09:27:00.509964 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:00.509434 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 09:27:00.509964 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:00.509461 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 09:27:00.509964 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:00.509475 2565 projected.go:196] Error preparing data for projected volume kube-api-access-hs4nw for pod openshift-network-diagnostics/network-check-target-n8c7q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 09:27:00.509964 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:00.509540 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7225439c-6210-4bc5-b124-0d92487dcd0c-kube-api-access-hs4nw podName:7225439c-6210-4bc5-b124-0d92487dcd0c nodeName:}" failed. No retries permitted until 2026-07-14 09:27:08.509519773 +0000 UTC m=+18.018751208 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hs4nw" (UniqueName: "kubernetes.io/projected/7225439c-6210-4bc5-b124-0d92487dcd0c-kube-api-access-hs4nw") pod "network-check-target-n8c7q" (UID: "7225439c-6210-4bc5-b124-0d92487dcd0c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 09:27:00.609883 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:00.609638 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2190520-5806-41dd-b25f-b7d1076cfcf3-metrics-certs\") pod \"network-metrics-daemon-p5547\" (UID: \"c2190520-5806-41dd-b25f-b7d1076cfcf3\") " pod="openshift-multus/network-metrics-daemon-p5547" Jul 14 09:27:00.609883 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:00.609761 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a5c62b6a-f781-4066-9773-a5f4e8921276-original-pull-secret\") pod \"global-pull-secret-syncer-2tcgq\" (UID: \"a5c62b6a-f781-4066-9773-a5f4e8921276\") " pod="kube-system/global-pull-secret-syncer-2tcgq" Jul 14 09:27:00.610151 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:00.609967 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 09:27:00.610151 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:00.610064 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2190520-5806-41dd-b25f-b7d1076cfcf3-metrics-certs podName:c2190520-5806-41dd-b25f-b7d1076cfcf3 nodeName:}" failed. No retries permitted until 2026-07-14 09:27:08.610041816 +0000 UTC m=+18.119273247 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c2190520-5806-41dd-b25f-b7d1076cfcf3-metrics-certs") pod "network-metrics-daemon-p5547" (UID: "c2190520-5806-41dd-b25f-b7d1076cfcf3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 09:27:00.610151 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:00.610076 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 09:27:00.610151 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:00.610147 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5c62b6a-f781-4066-9773-a5f4e8921276-original-pull-secret podName:a5c62b6a-f781-4066-9773-a5f4e8921276 nodeName:}" failed. No retries permitted until 2026-07-14 09:27:08.610126566 +0000 UTC m=+18.119358013 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a5c62b6a-f781-4066-9773-a5f4e8921276-original-pull-secret") pod "global-pull-secret-syncer-2tcgq" (UID: "a5c62b6a-f781-4066-9773-a5f4e8921276") : object "kube-system"/"original-pull-secret" not registered Jul 14 09:27:00.963890 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:00.963794 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tcgq" Jul 14 09:27:00.964047 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:00.963945 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tcgq" podUID="a5c62b6a-f781-4066-9773-a5f4e8921276" Jul 14 09:27:00.964384 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:00.964358 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8c7q" Jul 14 09:27:00.964488 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:00.964467 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8c7q" podUID="7225439c-6210-4bc5-b124-0d92487dcd0c" Jul 14 09:27:00.964566 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:00.964532 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p5547" Jul 14 09:27:00.964639 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:00.964621 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p5547" podUID="c2190520-5806-41dd-b25f-b7d1076cfcf3" Jul 14 09:27:02.963010 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:02.962972 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tcgq" Jul 14 09:27:02.963518 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:02.963114 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tcgq" podUID="a5c62b6a-f781-4066-9773-a5f4e8921276" Jul 14 09:27:02.964475 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:02.964264 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8c7q" Jul 14 09:27:02.964475 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:02.964312 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p5547" Jul 14 09:27:02.964475 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:02.964420 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8c7q" podUID="7225439c-6210-4bc5-b124-0d92487dcd0c" Jul 14 09:27:02.964710 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:02.964592 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p5547" podUID="c2190520-5806-41dd-b25f-b7d1076cfcf3" Jul 14 09:27:04.963361 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:04.963323 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p5547" Jul 14 09:27:04.963866 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:04.963323 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8c7q" Jul 14 09:27:04.963866 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:04.963464 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p5547" podUID="c2190520-5806-41dd-b25f-b7d1076cfcf3" Jul 14 09:27:04.963866 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:04.963336 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tcgq" Jul 14 09:27:04.963866 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:04.963530 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8c7q" podUID="7225439c-6210-4bc5-b124-0d92487dcd0c" Jul 14 09:27:04.963866 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:04.963606 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tcgq" podUID="a5c62b6a-f781-4066-9773-a5f4e8921276" Jul 14 09:27:06.962780 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:06.962748 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p5547" Jul 14 09:27:06.962780 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:06.962784 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8c7q" Jul 14 09:27:06.963247 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:06.962748 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tcgq" Jul 14 09:27:06.963247 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:06.962923 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p5547" podUID="c2190520-5806-41dd-b25f-b7d1076cfcf3" Jul 14 09:27:06.963247 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:06.963104 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8c7q" podUID="7225439c-6210-4bc5-b124-0d92487dcd0c" Jul 14 09:27:06.963353 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:06.963257 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tcgq" podUID="a5c62b6a-f781-4066-9773-a5f4e8921276" Jul 14 09:27:08.579068 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:08.579029 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs4nw\" (UniqueName: \"kubernetes.io/projected/7225439c-6210-4bc5-b124-0d92487dcd0c-kube-api-access-hs4nw\") pod \"network-check-target-n8c7q\" (UID: \"7225439c-6210-4bc5-b124-0d92487dcd0c\") " pod="openshift-network-diagnostics/network-check-target-n8c7q" Jul 14 09:27:08.579544 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:08.579198 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 09:27:08.579544 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:08.579219 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 09:27:08.579544 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:08.579230 2565 projected.go:196] Error preparing data for projected volume kube-api-access-hs4nw for pod openshift-network-diagnostics/network-check-target-n8c7q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 09:27:08.579544 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:08.579298 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7225439c-6210-4bc5-b124-0d92487dcd0c-kube-api-access-hs4nw podName:7225439c-6210-4bc5-b124-0d92487dcd0c nodeName:}" failed. No retries permitted until 2026-07-14 09:27:24.579281068 +0000 UTC m=+34.088512506 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hs4nw" (UniqueName: "kubernetes.io/projected/7225439c-6210-4bc5-b124-0d92487dcd0c-kube-api-access-hs4nw") pod "network-check-target-n8c7q" (UID: "7225439c-6210-4bc5-b124-0d92487dcd0c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 09:27:08.679558 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:08.679529 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a5c62b6a-f781-4066-9773-a5f4e8921276-original-pull-secret\") pod \"global-pull-secret-syncer-2tcgq\" (UID: \"a5c62b6a-f781-4066-9773-a5f4e8921276\") " pod="kube-system/global-pull-secret-syncer-2tcgq" Jul 14 09:27:08.679739 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:08.679585 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2190520-5806-41dd-b25f-b7d1076cfcf3-metrics-certs\") pod \"network-metrics-daemon-p5547\" (UID: \"c2190520-5806-41dd-b25f-b7d1076cfcf3\") " pod="openshift-multus/network-metrics-daemon-p5547" Jul 14 09:27:08.679739 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:08.679669 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 09:27:08.679739 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:08.679693 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 09:27:08.679739 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:08.679726 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2190520-5806-41dd-b25f-b7d1076cfcf3-metrics-certs podName:c2190520-5806-41dd-b25f-b7d1076cfcf3 nodeName:}" failed. No retries permitted until 2026-07-14 09:27:24.679711757 +0000 UTC m=+34.188943186 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c2190520-5806-41dd-b25f-b7d1076cfcf3-metrics-certs") pod "network-metrics-daemon-p5547" (UID: "c2190520-5806-41dd-b25f-b7d1076cfcf3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 09:27:08.679942 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:08.679759 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5c62b6a-f781-4066-9773-a5f4e8921276-original-pull-secret podName:a5c62b6a-f781-4066-9773-a5f4e8921276 nodeName:}" failed. No retries permitted until 2026-07-14 09:27:24.679739637 +0000 UTC m=+34.188971065 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a5c62b6a-f781-4066-9773-a5f4e8921276-original-pull-secret") pod "global-pull-secret-syncer-2tcgq" (UID: "a5c62b6a-f781-4066-9773-a5f4e8921276") : object "kube-system"/"original-pull-secret" not registered Jul 14 09:27:08.962966 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:08.962877 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p5547" Jul 14 09:27:08.963139 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:08.962877 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8c7q" Jul 14 09:27:08.963139 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:08.963019 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p5547" podUID="c2190520-5806-41dd-b25f-b7d1076cfcf3" Jul 14 09:27:08.963139 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:08.962889 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tcgq" Jul 14 09:27:08.963139 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:08.963085 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8c7q" podUID="7225439c-6210-4bc5-b124-0d92487dcd0c" Jul 14 09:27:08.963348 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:08.963149 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tcgq" podUID="a5c62b6a-f781-4066-9773-a5f4e8921276" Jul 14 09:27:10.963847 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:10.963682 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8c7q" Jul 14 09:27:10.963847 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:10.963819 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8c7q" podUID="7225439c-6210-4bc5-b124-0d92487dcd0c" Jul 14 09:27:10.963847 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:10.963829 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p5547" Jul 14 09:27:10.964442 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:10.963942 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p5547" podUID="c2190520-5806-41dd-b25f-b7d1076cfcf3" Jul 14 09:27:10.964442 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:10.963967 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tcgq" Jul 14 09:27:10.964442 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:10.964050 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tcgq" podUID="a5c62b6a-f781-4066-9773-a5f4e8921276" Jul 14 09:27:12.962704 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:12.962672 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8c7q" Jul 14 09:27:12.963205 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:12.962676 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tcgq" Jul 14 09:27:12.963205 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:12.962801 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8c7q" podUID="7225439c-6210-4bc5-b124-0d92487dcd0c" Jul 14 09:27:12.963205 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:12.962908 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tcgq" podUID="a5c62b6a-f781-4066-9773-a5f4e8921276" Jul 14 09:27:12.963205 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:12.962675 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p5547" Jul 14 09:27:12.963205 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:12.963011 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p5547" podUID="c2190520-5806-41dd-b25f-b7d1076cfcf3" Jul 14 09:27:14.138518 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:14.138174 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jjwzx" event={"ID":"dc4d1b63-99f8-4738-9dee-76fe0e4e4e4d","Type":"ContainerStarted","Data":"304eb572590c528e1916a349bcc58269cab578f0a9b648de3522bf94454a74a7"} Jul 14 09:27:14.139689 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:14.139665 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ttvxk" event={"ID":"3da5cf33-ada6-411e-b56d-73c993bf77b7","Type":"ContainerStarted","Data":"cbbec9247a97cab151e51fd669a6d8fdbeff574e5cef8ef2852f80e836b48bcb"} Jul 14 09:27:14.141077 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:14.141053 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9xsfz" event={"ID":"0274da0d-7c52-4932-9ac3-732af03b0c85","Type":"ContainerStarted","Data":"2ea86de5bec20d9c7e8ef4a4336016963f6ba0cd1cffb30f7786312c43233d08"} Jul 14 09:27:14.142959 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:14.142931 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" event={"ID":"f9a0e148-5f32-4d80-8963-321cbc4e5de0","Type":"ContainerStarted","Data":"443584a9bd99c16c81a3e17517f75ad4adf0bdaa6fe91cf2248e77cb8bfd5328"} Jul 14 09:27:14.143056 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:14.143011 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" event={"ID":"f9a0e148-5f32-4d80-8963-321cbc4e5de0","Type":"ContainerStarted","Data":"14cb3db6d78ab3e7bdebcb88543aa35c95bac2d96c7b1c6a2722e9dbdeaca3d0"} Jul 14 09:27:14.144485 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:14.144459 2565 generic.go:356] "Generic (PLEG): container finished" podID="82f2aa67-8042-4bbb-930d-09ad08fd1696" containerID="5b56ad78abca3662bfd88ab5e9e6287ed22e944f253560b0e20536dfd135c378" exitCode=0 Jul 14 09:27:14.144572 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:14.144549 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9z4f8" event={"ID":"82f2aa67-8042-4bbb-930d-09ad08fd1696","Type":"ContainerDied","Data":"5b56ad78abca3662bfd88ab5e9e6287ed22e944f253560b0e20536dfd135c378"} Jul 14 09:27:14.146285 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:14.146258 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5rb" event={"ID":"dc1cb2ed-2875-4a0a-9089-c1f9240dd7be","Type":"ContainerStarted","Data":"3238676d3cbd197e88b79ba4791e2156dd134d1b64bf1432695a9ccd18f4d749"} Jul 14 09:27:14.147790 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:14.147763 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5r769" event={"ID":"4a7e6b7e-3c6a-4def-bc24-b9a00ccad67b","Type":"ContainerStarted","Data":"227ea60e15797d6c8f39d89c6bbe780b22c8b32fc8029c2886d4826590c421b4"} Jul 14 09:27:14.149112 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:14.149092 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4kb48" event={"ID":"3cfb37de-4170-43d8-aa35-d2f1c3c6cc90","Type":"ContainerStarted","Data":"36753dd217f3fa4bdf5275d2957c84a2d707530727c2e05765a0b93092c59036"} Jul 14 09:27:14.153501 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:14.153466 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jjwzx" podStartSLOduration=6.9342917889999995 podStartE2EDuration="22.153455384s" totalImagesPullingTime="15.219163595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 09:26:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:27:14.153196355 +0000 UTC m=+23.662427815" watchObservedRunningTime="2026-07-14 09:27:14.153455384 +0000 UTC m=+23.662686832" Jul 14 09:27:14.201292 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:14.201247 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ttvxk" podStartSLOduration=3.671478279 podStartE2EDuration="23.20123124s" totalImagesPullingTime="19.529752961s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 09:26:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:27:14.200989271 +0000 UTC m=+23.710220762" watchObservedRunningTime="2026-07-14 09:27:14.20123124 +0000 UTC m=+23.710462688" Jul 14 09:27:14.201417 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:14.201337 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4kb48" podStartSLOduration=2.666033811 podStartE2EDuration="22.201332711s" totalImagesPullingTime="19.5352989s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 09:26:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:27:14.1841381 +0000 UTC m=+23.693369548" watchObservedRunningTime="2026-07-14 09:27:14.201332711 +0000 UTC m=+23.710564160" Jul 14 09:27:14.215378 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:14.215337 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-5r769" podStartSLOduration=7.996267979 podStartE2EDuration="23.215322576s" totalImagesPullingTime="15.219054597s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 09:26:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:27:14.215167529 +0000 UTC m=+23.724398977" watchObservedRunningTime="2026-07-14 09:27:14.215322576 +0000 UTC m=+23.724554023" Jul 14 09:27:14.232245 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:14.232203 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-9xsfz" podStartSLOduration=2.655570778 podStartE2EDuration="22.232186461s" totalImagesPullingTime="19.576615683s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 09:26:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:27:14.231454171 +0000 UTC m=+23.740685618" watchObservedRunningTime="2026-07-14 09:27:14.232186461 +0000 UTC m=+23.741417909" Jul 14 09:27:14.963387 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:14.963355 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p5547" Jul 14 09:27:14.963648 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:14.963354 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tcgq" Jul 14 09:27:14.963648 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:14.963485 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p5547" podUID="c2190520-5806-41dd-b25f-b7d1076cfcf3" Jul 14 09:27:14.963648 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:14.963573 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tcgq" podUID="a5c62b6a-f781-4066-9773-a5f4e8921276" Jul 14 09:27:14.963648 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:14.963368 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8c7q" Jul 14 09:27:14.963874 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:14.963704 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8c7q" podUID="7225439c-6210-4bc5-b124-0d92487dcd0c" Jul 14 09:27:15.154405 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:15.154374 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-r5rg6" event={"ID":"eebab604-45fc-4992-a289-96c5275ed378","Type":"ContainerStarted","Data":"084e157acaf7f9e7b9830aaed10eafd5b195ba84791476520315707ba624a32b"} Jul 14 09:27:15.158507 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:15.158020 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" event={"ID":"f9a0e148-5f32-4d80-8963-321cbc4e5de0","Type":"ContainerStarted","Data":"e5488676cd60e9b20a84dc95b0a61819ff933dc7bd7f3d1d3b452acbe7658ae8"} Jul 14 09:27:15.158507 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:15.158099 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" event={"ID":"f9a0e148-5f32-4d80-8963-321cbc4e5de0","Type":"ContainerStarted","Data":"b6e987496f5e8d749b9479a6199e3352f2b71f5a22f4a6fc3e22f0326c4b8983"} Jul 14 09:27:15.158507 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:15.158133 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" event={"ID":"f9a0e148-5f32-4d80-8963-321cbc4e5de0","Type":"ContainerStarted","Data":"66840b476917a4e2ec655c810631a4ed90e29542fcd330a6ed9c2b437667781d"} Jul 14 09:27:15.158507 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:15.158161 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" event={"ID":"f9a0e148-5f32-4d80-8963-321cbc4e5de0","Type":"ContainerStarted","Data":"060e2ab0526d2f37e1309b6c961706ac25b2f542a155ed790e35dc8bb7ca257a"} Jul 14 09:27:15.267972 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:15.267942 2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 14 09:27:15.902983 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:15.902861 2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-14T09:27:15.267965316Z","UUID":"af3bad5a-15aa-4921-a797-251ad637d068","Handler":null,"Name":"","Endpoint":""} Jul 14 09:27:15.904778 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:15.904757 2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 14 09:27:15.904929 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:15.904789 2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 14 09:27:16.162606 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:16.162520 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5rb" event={"ID":"dc1cb2ed-2875-4a0a-9089-c1f9240dd7be","Type":"ContainerStarted","Data":"91dbc1a11eafba500090e38d74b586f60671d558e9ea93921457d8cbe30a4665"} Jul 14 09:27:16.963091 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:16.962836 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p5547" Jul 14 09:27:16.963252 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:16.963155 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p5547" podUID="c2190520-5806-41dd-b25f-b7d1076cfcf3" Jul 14 09:27:16.963252 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:16.962888 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8c7q" Jul 14 09:27:16.963252 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:16.963245 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8c7q" podUID="7225439c-6210-4bc5-b124-0d92487dcd0c" Jul 14 09:27:16.963422 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:16.962840 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tcgq" Jul 14 09:27:16.963422 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:16.963322 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tcgq" podUID="a5c62b6a-f781-4066-9773-a5f4e8921276" Jul 14 09:27:17.168468 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:17.168430 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" event={"ID":"f9a0e148-5f32-4d80-8963-321cbc4e5de0","Type":"ContainerStarted","Data":"5a288a25dc7f570d9ff0396278715d875ca39e17c19a02156f39388352bba831"} Jul 14 09:27:17.170670 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:17.170640 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5rb" event={"ID":"dc1cb2ed-2875-4a0a-9089-c1f9240dd7be","Type":"ContainerStarted","Data":"551516f6c945cd4b5e6e29cceff0dac32ca5da1d670793a9c94acb9cf3eb596e"} Jul 14 09:27:17.190075 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:17.190014 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-r5rg6" podStartSLOduration=6.668782466 podStartE2EDuration="26.189997897s" totalImagesPullingTime="19.521215431s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 09:26:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:27:15.169280117 +0000 UTC m=+24.678511568" watchObservedRunningTime="2026-07-14 09:27:17.189997897 +0000 UTC m=+26.699229346" Jul 14 09:27:17.190296 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:17.190269 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mf5rb" podStartSLOduration=4.054785764 podStartE2EDuration="26.190259879s" totalImagesPullingTime="22.135474115s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 09:26:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:27:17.18963508 +0000 UTC m=+26.698866547" watchObservedRunningTime="2026-07-14 09:27:17.190259879 +0000 UTC m=+26.699491330" Jul 14 09:27:18.504330 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:18.504296 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jjwzx" Jul 14 09:27:18.504928 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:18.504911 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jjwzx" Jul 14 09:27:18.962713 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:18.962689 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p5547" Jul 14 09:27:18.962854 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:18.962688 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8c7q" Jul 14 09:27:18.962854 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:18.962823 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p5547" podUID="c2190520-5806-41dd-b25f-b7d1076cfcf3" Jul 14 09:27:18.962958 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:18.962695 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tcgq" Jul 14 09:27:18.962958 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:18.962884 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8c7q" podUID="7225439c-6210-4bc5-b124-0d92487dcd0c" Jul 14 09:27:18.962958 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:18.962948 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tcgq" podUID="a5c62b6a-f781-4066-9773-a5f4e8921276" Jul 14 09:27:19.178647 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:19.178396 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" event={"ID":"f9a0e148-5f32-4d80-8963-321cbc4e5de0","Type":"ContainerStarted","Data":"0ccc090ff6c0b4cc5a416be3d6a7f0ea9f3b185f5bfbeb3551198e54dcd45cd7"} Jul 14 09:27:19.178830 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:19.178657 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:27:19.178830 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:19.178732 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:27:19.182306 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:19.182280 2565 generic.go:356] "Generic (PLEG): container finished" podID="82f2aa67-8042-4bbb-930d-09ad08fd1696" containerID="bb431fa568e249daf1621bb9d2defe3650c29b766024bcae0211ca31af92ef90" exitCode=0 Jul 14 09:27:19.182452 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:19.182329 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9z4f8" event={"ID":"82f2aa67-8042-4bbb-930d-09ad08fd1696","Type":"ContainerDied","Data":"bb431fa568e249daf1621bb9d2defe3650c29b766024bcae0211ca31af92ef90"} Jul 14 09:27:19.195218 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:19.195200 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:27:19.210660 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:19.210619 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" podStartSLOduration=7.613671051 podStartE2EDuration="27.210605874s" totalImagesPullingTime="19.596934823s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 09:26:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:27:19.210426772 +0000 UTC m=+28.719658221" watchObservedRunningTime="2026-07-14 09:27:19.210605874 +0000 UTC m=+28.719837322" Jul 14 09:27:20.005173 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:20.005129 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jjwzx" Jul 14 09:27:20.005945 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:20.005734 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jjwzx" Jul 14 09:27:20.186768 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:20.186665 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:27:20.205033 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:20.205008 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:27:20.955103 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:20.955067 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-n8c7q"] Jul 14 09:27:20.955343 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:20.955245 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8c7q" Jul 14 09:27:20.955416 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:20.955371 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8c7q" podUID="7225439c-6210-4bc5-b124-0d92487dcd0c" Jul 14 09:27:20.957072 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:20.957034 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-p5547"] Jul 14 09:27:20.957212 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:20.957169 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p5547" Jul 14 09:27:20.957306 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:20.957284 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p5547" podUID="c2190520-5806-41dd-b25f-b7d1076cfcf3" Jul 14 09:27:20.957664 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:20.957639 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2tcgq"] Jul 14 09:27:20.957769 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:20.957750 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tcgq" Jul 14 09:27:20.957891 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:20.957870 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tcgq" podUID="a5c62b6a-f781-4066-9773-a5f4e8921276" Jul 14 09:27:21.189032 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:21.188997 2565 generic.go:356] "Generic (PLEG): container finished" podID="82f2aa67-8042-4bbb-930d-09ad08fd1696" containerID="a89dabddd6bad63870e21b2c318136c77b5443c883a420a394d88c91a9a0e0ac" exitCode=0 Jul 14 09:27:21.189522 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:21.189091 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9z4f8" event={"ID":"82f2aa67-8042-4bbb-930d-09ad08fd1696","Type":"ContainerDied","Data":"a89dabddd6bad63870e21b2c318136c77b5443c883a420a394d88c91a9a0e0ac"} Jul 14 09:27:22.507103 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:22.506881 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5r769_4a7e6b7e-3c6a-4def-bc24-b9a00ccad67b/dns-node-resolver/0.log" Jul 14 09:27:22.962525 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:22.962494 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tcgq" Jul 14 09:27:22.962525 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:22.962521 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p5547" Jul 14 09:27:22.962766 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:22.962493 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8c7q" Jul 14 09:27:22.962766 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:22.962639 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p5547" podUID="c2190520-5806-41dd-b25f-b7d1076cfcf3" Jul 14 09:27:22.962766 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:22.962669 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8c7q" podUID="7225439c-6210-4bc5-b124-0d92487dcd0c" Jul 14 09:27:22.962937 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:22.962774 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tcgq" podUID="a5c62b6a-f781-4066-9773-a5f4e8921276" Jul 14 09:27:23.194989 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:23.194909 2565 generic.go:356] "Generic (PLEG): container finished" podID="82f2aa67-8042-4bbb-930d-09ad08fd1696" containerID="c61d3401c0cc66b77de1cbaf0007da492b8d0f95d1dac211cc31b438e95edf50" exitCode=0 Jul 14 09:27:23.194989 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:23.194958 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9z4f8" event={"ID":"82f2aa67-8042-4bbb-930d-09ad08fd1696","Type":"ContainerDied","Data":"c61d3401c0cc66b77de1cbaf0007da492b8d0f95d1dac211cc31b438e95edf50"} Jul 14 09:27:23.482367 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:23.482297 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4kb48_3cfb37de-4170-43d8-aa35-d2f1c3c6cc90/node-ca/0.log" Jul 14 09:27:24.603697 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:24.603659 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs4nw\" (UniqueName: \"kubernetes.io/projected/7225439c-6210-4bc5-b124-0d92487dcd0c-kube-api-access-hs4nw\") pod \"network-check-target-n8c7q\" (UID: \"7225439c-6210-4bc5-b124-0d92487dcd0c\") " pod="openshift-network-diagnostics/network-check-target-n8c7q" Jul 14 09:27:24.604426 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:24.603859 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 09:27:24.604426 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:24.603890 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 09:27:24.604426 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:24.603904 2565 projected.go:196] Error preparing data for projected volume kube-api-access-hs4nw for pod openshift-network-diagnostics/network-check-target-n8c7q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 09:27:24.604426 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:24.603980 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7225439c-6210-4bc5-b124-0d92487dcd0c-kube-api-access-hs4nw podName:7225439c-6210-4bc5-b124-0d92487dcd0c nodeName:}" failed. No retries permitted until 2026-07-14 09:27:56.603953903 +0000 UTC m=+66.113185353 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hs4nw" (UniqueName: "kubernetes.io/projected/7225439c-6210-4bc5-b124-0d92487dcd0c-kube-api-access-hs4nw") pod "network-check-target-n8c7q" (UID: "7225439c-6210-4bc5-b124-0d92487dcd0c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 09:27:24.705089 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:24.705052 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2190520-5806-41dd-b25f-b7d1076cfcf3-metrics-certs\") pod \"network-metrics-daemon-p5547\" (UID: \"c2190520-5806-41dd-b25f-b7d1076cfcf3\") " pod="openshift-multus/network-metrics-daemon-p5547" Jul 14 09:27:24.705249 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:24.705120 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a5c62b6a-f781-4066-9773-a5f4e8921276-original-pull-secret\") pod \"global-pull-secret-syncer-2tcgq\" (UID: \"a5c62b6a-f781-4066-9773-a5f4e8921276\") " pod="kube-system/global-pull-secret-syncer-2tcgq" Jul 14 09:27:24.705249 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:24.705207 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 09:27:24.705366 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:24.705266 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5c62b6a-f781-4066-9773-a5f4e8921276-original-pull-secret podName:a5c62b6a-f781-4066-9773-a5f4e8921276 nodeName:}" failed. No retries permitted until 2026-07-14 09:27:56.70525172 +0000 UTC m=+66.214483171 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a5c62b6a-f781-4066-9773-a5f4e8921276-original-pull-secret") pod "global-pull-secret-syncer-2tcgq" (UID: "a5c62b6a-f781-4066-9773-a5f4e8921276") : object "kube-system"/"original-pull-secret" not registered Jul 14 09:27:24.705366 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:24.705207 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 09:27:24.705366 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:24.705325 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2190520-5806-41dd-b25f-b7d1076cfcf3-metrics-certs podName:c2190520-5806-41dd-b25f-b7d1076cfcf3 nodeName:}" failed. No retries permitted until 2026-07-14 09:27:56.705311297 +0000 UTC m=+66.214542737 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c2190520-5806-41dd-b25f-b7d1076cfcf3-metrics-certs") pod "network-metrics-daemon-p5547" (UID: "c2190520-5806-41dd-b25f-b7d1076cfcf3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 09:27:24.962696 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:24.962659 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tcgq" Jul 14 09:27:24.962911 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:24.962659 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8c7q" Jul 14 09:27:24.962911 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:24.962788 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tcgq" podUID="a5c62b6a-f781-4066-9773-a5f4e8921276" Jul 14 09:27:24.962911 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:24.962674 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p5547" Jul 14 09:27:24.962911 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:24.962874 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8c7q" podUID="7225439c-6210-4bc5-b124-0d92487dcd0c" Jul 14 09:27:24.963132 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:24.962957 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p5547" podUID="c2190520-5806-41dd-b25f-b7d1076cfcf3" Jul 14 09:27:26.962500 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:26.962456 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tcgq" Jul 14 09:27:26.963100 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:26.962459 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p5547" Jul 14 09:27:26.963100 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:26.962583 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tcgq" podUID="a5c62b6a-f781-4066-9773-a5f4e8921276" Jul 14 09:27:26.963100 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:26.962665 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p5547" podUID="c2190520-5806-41dd-b25f-b7d1076cfcf3" Jul 14 09:27:26.963100 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:26.962706 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8c7q" Jul 14 09:27:26.963100 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:26.962783 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8c7q" podUID="7225439c-6210-4bc5-b124-0d92487dcd0c" Jul 14 09:27:28.209774 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:28.209739 2565 generic.go:356] "Generic (PLEG): container finished" podID="82f2aa67-8042-4bbb-930d-09ad08fd1696" containerID="2a900d437045c2a1c0545120e3a9b8bbebad79f5d0e1da39efa23cdce8363d32" exitCode=0 Jul 14 09:27:28.209774 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:28.209780 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9z4f8" event={"ID":"82f2aa67-8042-4bbb-930d-09ad08fd1696","Type":"ContainerDied","Data":"2a900d437045c2a1c0545120e3a9b8bbebad79f5d0e1da39efa23cdce8363d32"} Jul 14 09:27:28.963060 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:28.963025 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p5547" Jul 14 09:27:28.963245 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:28.963074 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8c7q" Jul 14 09:27:28.963245 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:28.963025 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tcgq" Jul 14 09:27:28.963245 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:28.963167 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p5547" podUID="c2190520-5806-41dd-b25f-b7d1076cfcf3" Jul 14 09:27:28.963245 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:28.963214 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8c7q" podUID="7225439c-6210-4bc5-b124-0d92487dcd0c" Jul 14 09:27:28.963424 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:28.963315 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tcgq" podUID="a5c62b6a-f781-4066-9773-a5f4e8921276" Jul 14 09:27:29.214029 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:29.213953 2565 generic.go:356] "Generic (PLEG): container finished" podID="82f2aa67-8042-4bbb-930d-09ad08fd1696" containerID="e2a71afa6e4ba5272c72503d19a3f16dbb736ad00982c00583711ac54c147bd7" exitCode=0 Jul 14 09:27:29.214029 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:29.214019 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9z4f8" event={"ID":"82f2aa67-8042-4bbb-930d-09ad08fd1696","Type":"ContainerDied","Data":"e2a71afa6e4ba5272c72503d19a3f16dbb736ad00982c00583711ac54c147bd7"} Jul 14 09:27:30.218919 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:30.218732 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9z4f8" event={"ID":"82f2aa67-8042-4bbb-930d-09ad08fd1696","Type":"ContainerStarted","Data":"30b9867d2d21bb664627861962709489b572a228adf2c8f4e40ee95ab351b005"} Jul 14 09:27:30.243100 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:30.243050 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9z4f8" podStartSLOduration=9.284889199 podStartE2EDuration="39.243037045s" totalImagesPullingTime="29.958147846s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 09:26:51 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:27:30.242616081 +0000 UTC m=+39.751847528" watchObservedRunningTime="2026-07-14 09:27:30.243037045 +0000 UTC m=+39.752268493" Jul 14 09:27:30.964206 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:30.964168 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8c7q" Jul 14 09:27:30.964393 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:30.964269 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tcgq" Jul 14 09:27:30.964393 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:30.964285 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8c7q" podUID="7225439c-6210-4bc5-b124-0d92487dcd0c" Jul 14 09:27:30.964393 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:30.964336 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p5547" Jul 14 09:27:30.964560 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:30.964436 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p5547" podUID="c2190520-5806-41dd-b25f-b7d1076cfcf3" Jul 14 09:27:30.964560 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:30.964525 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tcgq" podUID="a5c62b6a-f781-4066-9773-a5f4e8921276" Jul 14 09:27:32.962954 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:32.962922 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8c7q" Jul 14 09:27:32.963430 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:32.962922 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tcgq" Jul 14 09:27:32.963430 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:32.963048 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p5547" Jul 14 09:27:32.963430 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:32.963037 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8c7q" podUID="7225439c-6210-4bc5-b124-0d92487dcd0c" Jul 14 09:27:32.963430 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:32.963147 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tcgq" podUID="a5c62b6a-f781-4066-9773-a5f4e8921276" Jul 14 09:27:32.963430 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:32.963222 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p5547" podUID="c2190520-5806-41dd-b25f-b7d1076cfcf3" Jul 14 09:27:34.963217 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:34.963178 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p5547" Jul 14 09:27:34.963600 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:34.963178 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8c7q" Jul 14 09:27:34.963600 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:34.963304 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p5547" podUID="c2190520-5806-41dd-b25f-b7d1076cfcf3" Jul 14 09:27:34.963600 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:34.963198 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tcgq" Jul 14 09:27:34.963600 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:34.963358 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8c7q" podUID="7225439c-6210-4bc5-b124-0d92487dcd0c" Jul 14 09:27:34.963600 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:34.963443 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tcgq" podUID="a5c62b6a-f781-4066-9773-a5f4e8921276" Jul 14 09:27:36.963254 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:36.963213 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8c7q" Jul 14 09:27:36.963770 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:36.963227 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p5547" Jul 14 09:27:36.963770 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:36.963323 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8c7q" podUID="7225439c-6210-4bc5-b124-0d92487dcd0c" Jul 14 09:27:36.963770 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:36.963349 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tcgq" Jul 14 09:27:36.963770 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:36.963470 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p5547" podUID="c2190520-5806-41dd-b25f-b7d1076cfcf3" Jul 14 09:27:36.963770 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:36.963573 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tcgq" podUID="a5c62b6a-f781-4066-9773-a5f4e8921276" Jul 14 09:27:38.962324 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:38.962286 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p5547" Jul 14 09:27:38.962776 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:38.962393 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p5547" podUID="c2190520-5806-41dd-b25f-b7d1076cfcf3" Jul 14 09:27:38.962776 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:38.962286 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tcgq" Jul 14 09:27:38.962776 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:38.962291 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8c7q" Jul 14 09:27:38.962776 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:38.962478 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tcgq" podUID="a5c62b6a-f781-4066-9773-a5f4e8921276" Jul 14 09:27:38.962776 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:38.962527 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8c7q" podUID="7225439c-6210-4bc5-b124-0d92487dcd0c" Jul 14 09:27:40.965494 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:40.965461 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8c7q" Jul 14 09:27:40.965494 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:40.965487 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tcgq" Jul 14 09:27:40.965915 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:40.965550 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8c7q" podUID="7225439c-6210-4bc5-b124-0d92487dcd0c" Jul 14 09:27:40.965915 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:40.965468 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p5547" Jul 14 09:27:40.965915 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:40.965647 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tcgq" podUID="a5c62b6a-f781-4066-9773-a5f4e8921276" Jul 14 09:27:40.965915 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:40.965729 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p5547" podUID="c2190520-5806-41dd-b25f-b7d1076cfcf3" Jul 14 09:27:42.962537 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:42.962493 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8c7q" Jul 14 09:27:42.962956 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:42.962493 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p5547" Jul 14 09:27:42.962956 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:42.962606 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8c7q" podUID="7225439c-6210-4bc5-b124-0d92487dcd0c" Jul 14 09:27:42.962956 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:42.962711 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p5547" podUID="c2190520-5806-41dd-b25f-b7d1076cfcf3" Jul 14 09:27:42.962956 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:42.962493 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tcgq" Jul 14 09:27:42.962956 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:42.962828 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tcgq" podUID="a5c62b6a-f781-4066-9773-a5f4e8921276" Jul 14 09:27:44.963179 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:44.963142 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8c7q" Jul 14 09:27:44.963706 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:44.963156 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p5547" Jul 14 09:27:44.963706 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:44.963298 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8c7q" podUID="7225439c-6210-4bc5-b124-0d92487dcd0c" Jul 14 09:27:44.963706 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:44.963382 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p5547" podUID="c2190520-5806-41dd-b25f-b7d1076cfcf3" Jul 14 09:27:44.963706 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:44.963440 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tcgq" Jul 14 09:27:44.963706 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:44.963521 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2tcgq" podUID="a5c62b6a-f781-4066-9773-a5f4e8921276" Jul 14 09:27:45.338840 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.338795 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-2.ec2.internal" event="NodeReady" Jul 14 09:27:45.338987 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.338951 2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 14 09:27:45.386500 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.386463 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-m7mrj"] Jul 14 09:27:45.410864 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.410836 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-zlzfk"] Jul 14 09:27:45.411016 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.410926 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-m7mrj" Jul 14 09:27:45.413638 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.413616 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jncc8\"" Jul 14 09:27:45.413918 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.413899 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 14 09:27:45.414010 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.413903 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 14 09:27:45.428742 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.428722 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-m7mrj"] Jul 14 09:27:45.428850 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.428757 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zlzfk"] Jul 14 09:27:45.428850 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.428833 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zlzfk" Jul 14 09:27:45.431423 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.431403 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 14 09:27:45.431632 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.431611 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 14 09:27:45.431731 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.431708 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 14 09:27:45.431799 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.431753 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-fxlg4\"" Jul 14 09:27:45.438796 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.438776 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8f2133f7-7ee2-48c5-bc8f-1f17049b6c46-cert\") pod \"ingress-canary-zlzfk\" (UID: \"8f2133f7-7ee2-48c5-bc8f-1f17049b6c46\") " pod="openshift-ingress-canary/ingress-canary-zlzfk" Jul 14 09:27:45.438910 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.438826 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg2xl\" (UniqueName: \"kubernetes.io/projected/537a71a2-5db0-421b-8f77-76c91b07ac77-kube-api-access-dg2xl\") pod \"dns-default-m7mrj\" (UID: \"537a71a2-5db0-421b-8f77-76c91b07ac77\") " pod="openshift-dns/dns-default-m7mrj" Jul 14 09:27:45.438910 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.438855 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/537a71a2-5db0-421b-8f77-76c91b07ac77-config-volume\") pod \"dns-default-m7mrj\" (UID: \"537a71a2-5db0-421b-8f77-76c91b07ac77\") " pod="openshift-dns/dns-default-m7mrj" Jul 14 09:27:45.438910 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.438905 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/537a71a2-5db0-421b-8f77-76c91b07ac77-tmp-dir\") pod \"dns-default-m7mrj\" (UID: \"537a71a2-5db0-421b-8f77-76c91b07ac77\") " pod="openshift-dns/dns-default-m7mrj" Jul 14 09:27:45.439012 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.438982 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rwh6\" (UniqueName: \"kubernetes.io/projected/8f2133f7-7ee2-48c5-bc8f-1f17049b6c46-kube-api-access-2rwh6\") pod \"ingress-canary-zlzfk\" (UID: \"8f2133f7-7ee2-48c5-bc8f-1f17049b6c46\") " pod="openshift-ingress-canary/ingress-canary-zlzfk" Jul 14 09:27:45.439012 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.439005 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/537a71a2-5db0-421b-8f77-76c91b07ac77-metrics-tls\") pod \"dns-default-m7mrj\" (UID: \"537a71a2-5db0-421b-8f77-76c91b07ac77\") " pod="openshift-dns/dns-default-m7mrj" Jul 14 09:27:45.492818 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.492732 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hr4j6"] Jul 14 09:27:45.518775 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.518748 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hr4j6"] Jul 14 09:27:45.518916 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.518886 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hr4j6" Jul 14 09:27:45.521483 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.521460 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 14 09:27:45.521624 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.521558 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 14 09:27:45.521624 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.521573 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 14 09:27:45.521755 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.521657 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qsz5b\"" Jul 14 09:27:45.521867 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.521851 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 14 09:27:45.539431 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.539410 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg2xl\" (UniqueName: \"kubernetes.io/projected/537a71a2-5db0-421b-8f77-76c91b07ac77-kube-api-access-dg2xl\") pod \"dns-default-m7mrj\" (UID: \"537a71a2-5db0-421b-8f77-76c91b07ac77\") " pod="openshift-dns/dns-default-m7mrj" Jul 14 09:27:45.539541 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.539440 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5cffbd14-98ff-4790-aaaa-3b0ca91450fa-crio-socket\") pod \"insights-runtime-extractor-hr4j6\" (UID: \"5cffbd14-98ff-4790-aaaa-3b0ca91450fa\") " pod="openshift-insights/insights-runtime-extractor-hr4j6" Jul 14 09:27:45.539541 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.539461 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5cffbd14-98ff-4790-aaaa-3b0ca91450fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hr4j6\" (UID: \"5cffbd14-98ff-4790-aaaa-3b0ca91450fa\") " pod="openshift-insights/insights-runtime-extractor-hr4j6" Jul 14 09:27:45.539541 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.539481 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/537a71a2-5db0-421b-8f77-76c91b07ac77-config-volume\") pod \"dns-default-m7mrj\" (UID: \"537a71a2-5db0-421b-8f77-76c91b07ac77\") " pod="openshift-dns/dns-default-m7mrj" Jul 14 09:27:45.539647 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.539552 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5cffbd14-98ff-4790-aaaa-3b0ca91450fa-data-volume\") pod \"insights-runtime-extractor-hr4j6\" (UID: \"5cffbd14-98ff-4790-aaaa-3b0ca91450fa\") " pod="openshift-insights/insights-runtime-extractor-hr4j6" Jul 14 09:27:45.539647 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.539571 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/537a71a2-5db0-421b-8f77-76c91b07ac77-tmp-dir\") pod \"dns-default-m7mrj\" (UID: \"537a71a2-5db0-421b-8f77-76c91b07ac77\") " pod="openshift-dns/dns-default-m7mrj" Jul 14 09:27:45.539755 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.539726 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5cffbd14-98ff-4790-aaaa-3b0ca91450fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hr4j6\" (UID: \"5cffbd14-98ff-4790-aaaa-3b0ca91450fa\") " pod="openshift-insights/insights-runtime-extractor-hr4j6" Jul 14 09:27:45.539831 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.539765 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8ktm\" (UniqueName: \"kubernetes.io/projected/5cffbd14-98ff-4790-aaaa-3b0ca91450fa-kube-api-access-t8ktm\") pod \"insights-runtime-extractor-hr4j6\" (UID: \"5cffbd14-98ff-4790-aaaa-3b0ca91450fa\") " pod="openshift-insights/insights-runtime-extractor-hr4j6" Jul 14 09:27:45.539831 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.539823 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rwh6\" (UniqueName: \"kubernetes.io/projected/8f2133f7-7ee2-48c5-bc8f-1f17049b6c46-kube-api-access-2rwh6\") pod \"ingress-canary-zlzfk\" (UID: \"8f2133f7-7ee2-48c5-bc8f-1f17049b6c46\") " pod="openshift-ingress-canary/ingress-canary-zlzfk" Jul 14 09:27:45.539940 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.539834 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/537a71a2-5db0-421b-8f77-76c91b07ac77-tmp-dir\") pod \"dns-default-m7mrj\" (UID: \"537a71a2-5db0-421b-8f77-76c91b07ac77\") " pod="openshift-dns/dns-default-m7mrj" Jul 14 09:27:45.539940 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.539856 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/537a71a2-5db0-421b-8f77-76c91b07ac77-metrics-tls\") pod \"dns-default-m7mrj\" (UID: \"537a71a2-5db0-421b-8f77-76c91b07ac77\") " pod="openshift-dns/dns-default-m7mrj" Jul 14 09:27:45.539940 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.539894 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8f2133f7-7ee2-48c5-bc8f-1f17049b6c46-cert\") pod \"ingress-canary-zlzfk\" (UID: \"8f2133f7-7ee2-48c5-bc8f-1f17049b6c46\") " pod="openshift-ingress-canary/ingress-canary-zlzfk" Jul 14 09:27:45.540118 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.540101 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/537a71a2-5db0-421b-8f77-76c91b07ac77-config-volume\") pod \"dns-default-m7mrj\" (UID: \"537a71a2-5db0-421b-8f77-76c91b07ac77\") " pod="openshift-dns/dns-default-m7mrj" Jul 14 09:27:45.543616 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.543597 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/537a71a2-5db0-421b-8f77-76c91b07ac77-metrics-tls\") pod \"dns-default-m7mrj\" (UID: \"537a71a2-5db0-421b-8f77-76c91b07ac77\") " pod="openshift-dns/dns-default-m7mrj" Jul 14 09:27:45.543726 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.543708 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8f2133f7-7ee2-48c5-bc8f-1f17049b6c46-cert\") pod \"ingress-canary-zlzfk\" (UID: \"8f2133f7-7ee2-48c5-bc8f-1f17049b6c46\") " pod="openshift-ingress-canary/ingress-canary-zlzfk" Jul 14 09:27:45.547543 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.547518 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg2xl\" (UniqueName: \"kubernetes.io/projected/537a71a2-5db0-421b-8f77-76c91b07ac77-kube-api-access-dg2xl\") pod \"dns-default-m7mrj\" (UID: \"537a71a2-5db0-421b-8f77-76c91b07ac77\") " pod="openshift-dns/dns-default-m7mrj" Jul 14 09:27:45.548435 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.548413 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rwh6\" (UniqueName: \"kubernetes.io/projected/8f2133f7-7ee2-48c5-bc8f-1f17049b6c46-kube-api-access-2rwh6\") pod \"ingress-canary-zlzfk\" (UID: \"8f2133f7-7ee2-48c5-bc8f-1f17049b6c46\") " pod="openshift-ingress-canary/ingress-canary-zlzfk" Jul 14 09:27:45.641150 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.641114 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5cffbd14-98ff-4790-aaaa-3b0ca91450fa-data-volume\") pod \"insights-runtime-extractor-hr4j6\" (UID: \"5cffbd14-98ff-4790-aaaa-3b0ca91450fa\") " pod="openshift-insights/insights-runtime-extractor-hr4j6" Jul 14 09:27:45.641341 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.641166 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5cffbd14-98ff-4790-aaaa-3b0ca91450fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hr4j6\" (UID: \"5cffbd14-98ff-4790-aaaa-3b0ca91450fa\") " pod="openshift-insights/insights-runtime-extractor-hr4j6" Jul 14 09:27:45.641341 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.641183 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8ktm\" (UniqueName: \"kubernetes.io/projected/5cffbd14-98ff-4790-aaaa-3b0ca91450fa-kube-api-access-t8ktm\") pod \"insights-runtime-extractor-hr4j6\" (UID: \"5cffbd14-98ff-4790-aaaa-3b0ca91450fa\") " pod="openshift-insights/insights-runtime-extractor-hr4j6" Jul 14 09:27:45.641341 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.641216 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5cffbd14-98ff-4790-aaaa-3b0ca91450fa-crio-socket\") pod \"insights-runtime-extractor-hr4j6\" (UID: \"5cffbd14-98ff-4790-aaaa-3b0ca91450fa\") " pod="openshift-insights/insights-runtime-extractor-hr4j6" Jul 14 09:27:45.641341 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.641233 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5cffbd14-98ff-4790-aaaa-3b0ca91450fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hr4j6\" (UID: \"5cffbd14-98ff-4790-aaaa-3b0ca91450fa\") " pod="openshift-insights/insights-runtime-extractor-hr4j6" Jul 14 09:27:45.641341 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.641324 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5cffbd14-98ff-4790-aaaa-3b0ca91450fa-crio-socket\") pod \"insights-runtime-extractor-hr4j6\" (UID: \"5cffbd14-98ff-4790-aaaa-3b0ca91450fa\") " pod="openshift-insights/insights-runtime-extractor-hr4j6" Jul 14 09:27:45.641587 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.641527 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5cffbd14-98ff-4790-aaaa-3b0ca91450fa-data-volume\") pod \"insights-runtime-extractor-hr4j6\" (UID: \"5cffbd14-98ff-4790-aaaa-3b0ca91450fa\") " pod="openshift-insights/insights-runtime-extractor-hr4j6" Jul 14 09:27:45.641712 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.641696 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5cffbd14-98ff-4790-aaaa-3b0ca91450fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hr4j6\" (UID: \"5cffbd14-98ff-4790-aaaa-3b0ca91450fa\") " pod="openshift-insights/insights-runtime-extractor-hr4j6" Jul 14 09:27:45.643549 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.643528 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5cffbd14-98ff-4790-aaaa-3b0ca91450fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hr4j6\" (UID: \"5cffbd14-98ff-4790-aaaa-3b0ca91450fa\") " pod="openshift-insights/insights-runtime-extractor-hr4j6" Jul 14 09:27:45.649188 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.649168 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8ktm\" (UniqueName: \"kubernetes.io/projected/5cffbd14-98ff-4790-aaaa-3b0ca91450fa-kube-api-access-t8ktm\") pod \"insights-runtime-extractor-hr4j6\" (UID: \"5cffbd14-98ff-4790-aaaa-3b0ca91450fa\") " pod="openshift-insights/insights-runtime-extractor-hr4j6" Jul 14 09:27:45.719693 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.719652 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-m7mrj" Jul 14 09:27:45.736879 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.736844 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zlzfk" Jul 14 09:27:45.827513 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.827478 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hr4j6" Jul 14 09:27:45.927675 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.927646 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-m7mrj"] Jul 14 09:27:45.931353 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:27:45.931327 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod537a71a2_5db0_421b_8f77_76c91b07ac77.slice/crio-ff0ff0b7a7b7764ca2801dbdc9f36fd9a2d9c1fe13021731ba88594982103a11 WatchSource:0}: Error finding container ff0ff0b7a7b7764ca2801dbdc9f36fd9a2d9c1fe13021731ba88594982103a11: Status 404 returned error can't find the container with id ff0ff0b7a7b7764ca2801dbdc9f36fd9a2d9c1fe13021731ba88594982103a11 Jul 14 09:27:45.940699 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.940673 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zlzfk"] Jul 14 09:27:45.944075 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:27:45.944051 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f2133f7_7ee2_48c5_bc8f_1f17049b6c46.slice/crio-d51d5e252703065f69b3d89a42f09a7ca4a20b6f1950f55c9b6adfb7b4f2de78 WatchSource:0}: Error finding container d51d5e252703065f69b3d89a42f09a7ca4a20b6f1950f55c9b6adfb7b4f2de78: Status 404 returned error can't find the container with id d51d5e252703065f69b3d89a42f09a7ca4a20b6f1950f55c9b6adfb7b4f2de78 Jul 14 09:27:45.983546 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:45.983519 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hr4j6"] Jul 14 09:27:45.986896 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:27:45.986865 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cffbd14_98ff_4790_aaaa_3b0ca91450fa.slice/crio-d7c08dee7575e6a94267143a8bc8968c9da1f1679851b09254396865453f9bec WatchSource:0}: Error finding container d7c08dee7575e6a94267143a8bc8968c9da1f1679851b09254396865453f9bec: Status 404 returned error can't find the container with id d7c08dee7575e6a94267143a8bc8968c9da1f1679851b09254396865453f9bec Jul 14 09:27:46.255016 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:46.254975 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zlzfk" event={"ID":"8f2133f7-7ee2-48c5-bc8f-1f17049b6c46","Type":"ContainerStarted","Data":"d51d5e252703065f69b3d89a42f09a7ca4a20b6f1950f55c9b6adfb7b4f2de78"} Jul 14 09:27:46.256099 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:46.256071 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-m7mrj" event={"ID":"537a71a2-5db0-421b-8f77-76c91b07ac77","Type":"ContainerStarted","Data":"ff0ff0b7a7b7764ca2801dbdc9f36fd9a2d9c1fe13021731ba88594982103a11"} Jul 14 09:27:46.256997 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:46.256976 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hr4j6" event={"ID":"5cffbd14-98ff-4790-aaaa-3b0ca91450fa","Type":"ContainerStarted","Data":"d7c08dee7575e6a94267143a8bc8968c9da1f1679851b09254396865453f9bec"} Jul 14 09:27:46.962575 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:46.962501 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p5547" Jul 14 09:27:46.962575 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:46.962524 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tcgq" Jul 14 09:27:46.964057 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:46.963575 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8c7q" Jul 14 09:27:46.969887 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:46.967633 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-frh48\"" Jul 14 09:27:46.969887 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:46.968494 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 14 09:27:46.969887 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:46.968917 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ks9qb\"" Jul 14 09:27:46.969887 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:46.969576 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 14 09:27:46.971116 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:46.970577 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 14 09:27:46.971116 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:46.970704 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 14 09:27:47.538664 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:47.538140 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7555476c54-7f4b2"] Jul 14 09:27:47.549503 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:47.549472 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7555476c54-7f4b2" Jul 14 09:27:47.551789 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:47.551759 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7555476c54-7f4b2"] Jul 14 09:27:47.552047 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:47.551974 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 14 09:27:47.555584 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:47.555558 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 14 09:27:47.555924 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:47.555905 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 14 09:27:47.556153 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:47.556135 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 14 09:27:47.556371 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:47.556345 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 14 09:27:47.556540 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:47.556523 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 14 09:27:47.558319 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:47.556705 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6v5zj\"" Jul 14 09:27:47.558858 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:47.558674 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 14 09:27:47.658752 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:47.658723 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d088b26-558e-44e9-a8c3-5e836a0936e2-oauth-serving-cert\") pod \"console-7555476c54-7f4b2\" (UID: \"1d088b26-558e-44e9-a8c3-5e836a0936e2\") " pod="openshift-console/console-7555476c54-7f4b2" Jul 14 09:27:47.659024 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:47.658797 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d088b26-558e-44e9-a8c3-5e836a0936e2-console-config\") pod \"console-7555476c54-7f4b2\" (UID: \"1d088b26-558e-44e9-a8c3-5e836a0936e2\") " pod="openshift-console/console-7555476c54-7f4b2" Jul 14 09:27:47.659024 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:47.658861 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d088b26-558e-44e9-a8c3-5e836a0936e2-console-serving-cert\") pod \"console-7555476c54-7f4b2\" (UID: \"1d088b26-558e-44e9-a8c3-5e836a0936e2\") " pod="openshift-console/console-7555476c54-7f4b2" Jul 14 09:27:47.659024 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:47.658896 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hntz\" (UniqueName: \"kubernetes.io/projected/1d088b26-558e-44e9-a8c3-5e836a0936e2-kube-api-access-8hntz\") pod \"console-7555476c54-7f4b2\" (UID: \"1d088b26-558e-44e9-a8c3-5e836a0936e2\") " pod="openshift-console/console-7555476c54-7f4b2" Jul 14 09:27:47.659024 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:47.658970 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d088b26-558e-44e9-a8c3-5e836a0936e2-console-oauth-config\") pod \"console-7555476c54-7f4b2\" (UID: \"1d088b26-558e-44e9-a8c3-5e836a0936e2\") " pod="openshift-console/console-7555476c54-7f4b2" Jul 14 09:27:47.659024 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:47.659009 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d088b26-558e-44e9-a8c3-5e836a0936e2-service-ca\") pod \"console-7555476c54-7f4b2\" (UID: \"1d088b26-558e-44e9-a8c3-5e836a0936e2\") " pod="openshift-console/console-7555476c54-7f4b2" Jul 14 09:27:47.759951 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:47.759914 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d088b26-558e-44e9-a8c3-5e836a0936e2-console-config\") pod \"console-7555476c54-7f4b2\" (UID: \"1d088b26-558e-44e9-a8c3-5e836a0936e2\") " pod="openshift-console/console-7555476c54-7f4b2" Jul 14 09:27:47.760177 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:47.759968 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d088b26-558e-44e9-a8c3-5e836a0936e2-console-serving-cert\") pod \"console-7555476c54-7f4b2\" (UID: \"1d088b26-558e-44e9-a8c3-5e836a0936e2\") " pod="openshift-console/console-7555476c54-7f4b2" Jul 14 09:27:47.760177 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:47.760014 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hntz\" (UniqueName: \"kubernetes.io/projected/1d088b26-558e-44e9-a8c3-5e836a0936e2-kube-api-access-8hntz\") pod \"console-7555476c54-7f4b2\" (UID: \"1d088b26-558e-44e9-a8c3-5e836a0936e2\") " pod="openshift-console/console-7555476c54-7f4b2" Jul 14 09:27:47.760177 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:47.760089 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d088b26-558e-44e9-a8c3-5e836a0936e2-console-oauth-config\") pod \"console-7555476c54-7f4b2\" (UID: \"1d088b26-558e-44e9-a8c3-5e836a0936e2\") " pod="openshift-console/console-7555476c54-7f4b2" Jul 14 09:27:47.760177 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:47.760118 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d088b26-558e-44e9-a8c3-5e836a0936e2-service-ca\") pod \"console-7555476c54-7f4b2\" (UID: \"1d088b26-558e-44e9-a8c3-5e836a0936e2\") " pod="openshift-console/console-7555476c54-7f4b2" Jul 14 09:27:47.760177 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:47.760157 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d088b26-558e-44e9-a8c3-5e836a0936e2-oauth-serving-cert\") pod \"console-7555476c54-7f4b2\" (UID: \"1d088b26-558e-44e9-a8c3-5e836a0936e2\") " pod="openshift-console/console-7555476c54-7f4b2" Jul 14 09:27:47.760841 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:47.760718 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d088b26-558e-44e9-a8c3-5e836a0936e2-console-config\") pod \"console-7555476c54-7f4b2\" (UID: \"1d088b26-558e-44e9-a8c3-5e836a0936e2\") " pod="openshift-console/console-7555476c54-7f4b2" Jul 14 09:27:47.760841 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:47.760742 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d088b26-558e-44e9-a8c3-5e836a0936e2-oauth-serving-cert\") pod \"console-7555476c54-7f4b2\" (UID: \"1d088b26-558e-44e9-a8c3-5e836a0936e2\") " pod="openshift-console/console-7555476c54-7f4b2" Jul 14 09:27:47.762875 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:47.762826 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d088b26-558e-44e9-a8c3-5e836a0936e2-console-serving-cert\") pod \"console-7555476c54-7f4b2\" (UID: \"1d088b26-558e-44e9-a8c3-5e836a0936e2\") " pod="openshift-console/console-7555476c54-7f4b2" Jul 14 09:27:47.762992 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:47.762973 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d088b26-558e-44e9-a8c3-5e836a0936e2-console-oauth-config\") pod \"console-7555476c54-7f4b2\" (UID: \"1d088b26-558e-44e9-a8c3-5e836a0936e2\") " pod="openshift-console/console-7555476c54-7f4b2" Jul 14 09:27:47.765306 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:47.765282 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d088b26-558e-44e9-a8c3-5e836a0936e2-service-ca\") pod \"console-7555476c54-7f4b2\" (UID: \"1d088b26-558e-44e9-a8c3-5e836a0936e2\") " pod="openshift-console/console-7555476c54-7f4b2" Jul 14 09:27:47.777647 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:47.777616 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hntz\" (UniqueName: \"kubernetes.io/projected/1d088b26-558e-44e9-a8c3-5e836a0936e2-kube-api-access-8hntz\") pod \"console-7555476c54-7f4b2\" (UID: \"1d088b26-558e-44e9-a8c3-5e836a0936e2\") " pod="openshift-console/console-7555476c54-7f4b2" Jul 14 09:27:47.867971 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:47.867883 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7555476c54-7f4b2" Jul 14 09:27:47.994968 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:47.994927 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vtqjr"] Jul 14 09:27:48.003824 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:48.003773 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vtqjr" Jul 14 09:27:48.007107 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:48.006905 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 14 09:27:48.007107 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:48.006929 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-7r5nq\"" Jul 14 09:27:48.007107 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:48.007046 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vtqjr"] Jul 14 09:27:48.062205 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:48.062127 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a2766a64-d5c3-4685-b76a-b61e0ffb0104-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-vtqjr\" (UID: \"a2766a64-d5c3-4685-b76a-b61e0ffb0104\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vtqjr" Jul 14 09:27:48.163035 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:48.162994 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a2766a64-d5c3-4685-b76a-b61e0ffb0104-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-vtqjr\" (UID: \"a2766a64-d5c3-4685-b76a-b61e0ffb0104\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vtqjr" Jul 14 09:27:48.163229 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:48.163171 2565 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 14 09:27:48.163299 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:48.163258 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2766a64-d5c3-4685-b76a-b61e0ffb0104-tls-certificates podName:a2766a64-d5c3-4685-b76a-b61e0ffb0104 nodeName:}" failed. No retries permitted until 2026-07-14 09:27:48.663241297 +0000 UTC m=+58.172472724 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/a2766a64-d5c3-4685-b76a-b61e0ffb0104-tls-certificates") pod "prometheus-operator-admission-webhook-77d98bc786-vtqjr" (UID: "a2766a64-d5c3-4685-b76a-b61e0ffb0104") : secret "prometheus-operator-admission-webhook-tls" not found Jul 14 09:27:48.667235 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:48.667199 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a2766a64-d5c3-4685-b76a-b61e0ffb0104-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-vtqjr\" (UID: \"a2766a64-d5c3-4685-b76a-b61e0ffb0104\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vtqjr" Jul 14 09:27:48.670020 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:48.669992 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a2766a64-d5c3-4685-b76a-b61e0ffb0104-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-vtqjr\" (UID: \"a2766a64-d5c3-4685-b76a-b61e0ffb0104\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vtqjr" Jul 14 09:27:48.917037 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:48.916996 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vtqjr" Jul 14 09:27:51.539794 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:51.539693 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vtqjr"] Jul 14 09:27:51.570010 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:51.568715 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7555476c54-7f4b2"] Jul 14 09:27:51.574681 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:27:51.574629 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d088b26_558e_44e9_a8c3_5e836a0936e2.slice/crio-d2f3ae8567aae739c1a8f4f01a2ff474ec762e03a0886ed1e7ac06e6459e5de6 WatchSource:0}: Error finding container d2f3ae8567aae739c1a8f4f01a2ff474ec762e03a0886ed1e7ac06e6459e5de6: Status 404 returned error can't find the container with id d2f3ae8567aae739c1a8f4f01a2ff474ec762e03a0886ed1e7ac06e6459e5de6 Jul 14 09:27:52.205268 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:52.205238 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r79mk" Jul 14 09:27:52.275497 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:52.275464 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7555476c54-7f4b2" event={"ID":"1d088b26-558e-44e9-a8c3-5e836a0936e2","Type":"ContainerStarted","Data":"d2f3ae8567aae739c1a8f4f01a2ff474ec762e03a0886ed1e7ac06e6459e5de6"} Jul 14 09:27:52.277789 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:52.277682 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hr4j6" event={"ID":"5cffbd14-98ff-4790-aaaa-3b0ca91450fa","Type":"ContainerStarted","Data":"981145f661909feef76710886789768061968cbcc3355df24d722fccca163b55"} Jul 14 09:27:52.277789 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:52.277750 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hr4j6" event={"ID":"5cffbd14-98ff-4790-aaaa-3b0ca91450fa","Type":"ContainerStarted","Data":"bad933fcbf20162985cf419ddcf94f1aec5842a7e9828bfc82b8a917bbb90021"} Jul 14 09:27:52.279677 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:52.279642 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zlzfk" event={"ID":"8f2133f7-7ee2-48c5-bc8f-1f17049b6c46","Type":"ContainerStarted","Data":"dfcf9a21bd04f49d4df3fb8f95817e6bcc49dea1126457fc316e831b201ea5c2"} Jul 14 09:27:52.282587 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:52.282563 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-m7mrj" event={"ID":"537a71a2-5db0-421b-8f77-76c91b07ac77","Type":"ContainerStarted","Data":"2400519accd08eb00b5a5f7ae10a0369f1caac1de767ac960a4f1e81df734d72"} Jul 14 09:27:52.282695 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:52.282610 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-m7mrj" event={"ID":"537a71a2-5db0-421b-8f77-76c91b07ac77","Type":"ContainerStarted","Data":"9456933a8ecc3dd63d3af5bb0bc1d6d3e5828017db4728c58f6bc7b91708a38d"} Jul 14 09:27:52.283155 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:52.283114 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-m7mrj" Jul 14 09:27:52.284508 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:52.284486 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vtqjr" event={"ID":"a2766a64-d5c3-4685-b76a-b61e0ffb0104","Type":"ContainerStarted","Data":"98ae5ce2b5a568d7b6ccbfa7ecb4f36c51c29b12f3faf5fc886f07b102027a06"} Jul 14 09:27:52.298533 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:52.298483 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-zlzfk" podStartSLOduration=1.832892723 podStartE2EDuration="7.298466348s" totalImagesPullingTime="5.465573625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 09:27:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:27:52.297318474 +0000 UTC m=+61.806549937" watchObservedRunningTime="2026-07-14 09:27:52.298466348 +0000 UTC m=+61.807697794" Jul 14 09:27:52.313059 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:52.312998 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-m7mrj" podStartSLOduration=1.859609049 podStartE2EDuration="7.312978159s" totalImagesPullingTime="5.45336911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 09:27:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:27:52.312788639 +0000 UTC m=+61.822020087" watchObservedRunningTime="2026-07-14 09:27:52.312978159 +0000 UTC m=+61.822209609" Jul 14 09:27:53.290630 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:53.290567 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vtqjr" event={"ID":"a2766a64-d5c3-4685-b76a-b61e0ffb0104","Type":"ContainerStarted","Data":"95957959c9b2e0d9a951df37a18b5dafa5f505efe576bf88918ead74cd6fb632"} Jul 14 09:27:53.291129 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:53.290922 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vtqjr" Jul 14 09:27:53.297276 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:53.297228 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vtqjr" Jul 14 09:27:53.305410 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:53.305363 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-vtqjr" podStartSLOduration=5.298873734 podStartE2EDuration="6.305345964s" totalImagesPullingTime="1.00647223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 09:27:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:27:53.305106308 +0000 UTC m=+62.814337787" watchObservedRunningTime="2026-07-14 09:27:53.305345964 +0000 UTC m=+62.814577412" Jul 14 09:27:54.102333 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:54.102298 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-qszj9"] Jul 14 09:27:54.106427 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:54.106401 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-qszj9" Jul 14 09:27:54.110020 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:54.109888 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 14 09:27:54.110020 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:54.109941 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 14 09:27:54.110586 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:54.110558 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 14 09:27:54.110718 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:54.110622 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 14 09:27:54.111032 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:54.111014 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 14 09:27:54.111476 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:54.111459 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-mblxz\"" Jul 14 09:27:54.116630 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:54.116610 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-qszj9"] Jul 14 09:27:54.213052 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:54.213006 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48183b82-41a3-4075-b553-36bfd3742176-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-qszj9\" (UID: \"48183b82-41a3-4075-b553-36bfd3742176\") " pod="openshift-monitoring/prometheus-operator-64785b6675-qszj9" Jul 14 09:27:54.213235 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:54.213080 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/48183b82-41a3-4075-b553-36bfd3742176-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-qszj9\" (UID: \"48183b82-41a3-4075-b553-36bfd3742176\") " pod="openshift-monitoring/prometheus-operator-64785b6675-qszj9" Jul 14 09:27:54.213235 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:54.213127 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mfgl\" (UniqueName: \"kubernetes.io/projected/48183b82-41a3-4075-b553-36bfd3742176-kube-api-access-9mfgl\") pod \"prometheus-operator-64785b6675-qszj9\" (UID: \"48183b82-41a3-4075-b553-36bfd3742176\") " pod="openshift-monitoring/prometheus-operator-64785b6675-qszj9" Jul 14 09:27:54.213235 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:54.213185 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48183b82-41a3-4075-b553-36bfd3742176-metrics-client-ca\") pod \"prometheus-operator-64785b6675-qszj9\" (UID: \"48183b82-41a3-4075-b553-36bfd3742176\") " pod="openshift-monitoring/prometheus-operator-64785b6675-qszj9" Jul 14 09:27:54.313559 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:54.313523 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48183b82-41a3-4075-b553-36bfd3742176-metrics-client-ca\") pod \"prometheus-operator-64785b6675-qszj9\" (UID: \"48183b82-41a3-4075-b553-36bfd3742176\") " pod="openshift-monitoring/prometheus-operator-64785b6675-qszj9" Jul 14 09:27:54.314012 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:54.313573 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48183b82-41a3-4075-b553-36bfd3742176-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-qszj9\" (UID: \"48183b82-41a3-4075-b553-36bfd3742176\") " pod="openshift-monitoring/prometheus-operator-64785b6675-qszj9" Jul 14 09:27:54.314012 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:54.313635 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/48183b82-41a3-4075-b553-36bfd3742176-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-qszj9\" (UID: \"48183b82-41a3-4075-b553-36bfd3742176\") " pod="openshift-monitoring/prometheus-operator-64785b6675-qszj9" Jul 14 09:27:54.314012 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:54.313669 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mfgl\" (UniqueName: \"kubernetes.io/projected/48183b82-41a3-4075-b553-36bfd3742176-kube-api-access-9mfgl\") pod \"prometheus-operator-64785b6675-qszj9\" (UID: \"48183b82-41a3-4075-b553-36bfd3742176\") " pod="openshift-monitoring/prometheus-operator-64785b6675-qszj9" Jul 14 09:27:54.314377 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:54.314327 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48183b82-41a3-4075-b553-36bfd3742176-metrics-client-ca\") pod \"prometheus-operator-64785b6675-qszj9\" (UID: \"48183b82-41a3-4075-b553-36bfd3742176\") " pod="openshift-monitoring/prometheus-operator-64785b6675-qszj9" Jul 14 09:27:54.316436 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:54.316410 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48183b82-41a3-4075-b553-36bfd3742176-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-qszj9\" (UID: \"48183b82-41a3-4075-b553-36bfd3742176\") " pod="openshift-monitoring/prometheus-operator-64785b6675-qszj9" Jul 14 09:27:54.316542 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:54.316411 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/48183b82-41a3-4075-b553-36bfd3742176-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-qszj9\" (UID: \"48183b82-41a3-4075-b553-36bfd3742176\") " pod="openshift-monitoring/prometheus-operator-64785b6675-qszj9" Jul 14 09:27:54.333526 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:54.333494 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mfgl\" (UniqueName: \"kubernetes.io/projected/48183b82-41a3-4075-b553-36bfd3742176-kube-api-access-9mfgl\") pod \"prometheus-operator-64785b6675-qszj9\" (UID: \"48183b82-41a3-4075-b553-36bfd3742176\") " pod="openshift-monitoring/prometheus-operator-64785b6675-qszj9" Jul 14 09:27:54.336045 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:54.336020 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-948c58bdb-g9g2g"] Jul 14 09:27:54.340470 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:54.340451 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-948c58bdb-g9g2g" Jul 14 09:27:54.348633 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:54.348606 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 14 09:27:54.349834 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:54.349552 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-948c58bdb-g9g2g"] Jul 14 09:27:54.414580 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:54.414544 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/394653f7-9641-443f-b1e2-8ea222d9ae02-service-ca\") pod \"console-948c58bdb-g9g2g\" (UID: \"394653f7-9641-443f-b1e2-8ea222d9ae02\") " pod="openshift-console/console-948c58bdb-g9g2g" Jul 14 09:27:54.414752 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:54.414593 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/394653f7-9641-443f-b1e2-8ea222d9ae02-console-config\") pod \"console-948c58bdb-g9g2g\" (UID: \"394653f7-9641-443f-b1e2-8ea222d9ae02\") " pod="openshift-console/console-948c58bdb-g9g2g" Jul 14 09:27:54.414752 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:54.414628 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/394653f7-9641-443f-b1e2-8ea222d9ae02-console-oauth-config\") pod \"console-948c58bdb-g9g2g\" (UID: \"394653f7-9641-443f-b1e2-8ea222d9ae02\") " pod="openshift-console/console-948c58bdb-g9g2g" Jul 14 09:27:54.414752 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:54.414656 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/394653f7-9641-443f-b1e2-8ea222d9ae02-trusted-ca-bundle\") pod \"console-948c58bdb-g9g2g\" (UID: \"394653f7-9641-443f-b1e2-8ea222d9ae02\") " pod="openshift-console/console-948c58bdb-g9g2g" Jul 14 09:27:54.414752 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:54.414705 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bj78\" (UniqueName: \"kubernetes.io/projected/394653f7-9641-443f-b1e2-8ea222d9ae02-kube-api-access-5bj78\") pod \"console-948c58bdb-g9g2g\" (UID: \"394653f7-9641-443f-b1e2-8ea222d9ae02\") " pod="openshift-console/console-948c58bdb-g9g2g" Jul 14 09:27:54.414752 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:54.414750 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/394653f7-9641-443f-b1e2-8ea222d9ae02-console-serving-cert\") pod \"console-948c58bdb-g9g2g\" (UID: \"394653f7-9641-443f-b1e2-8ea222d9ae02\") " pod="openshift-console/console-948c58bdb-g9g2g" Jul 14 09:27:54.415036 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:54.414838 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/394653f7-9641-443f-b1e2-8ea222d9ae02-oauth-serving-cert\") pod \"console-948c58bdb-g9g2g\" (UID: \"394653f7-9641-443f-b1e2-8ea222d9ae02\") " pod="openshift-console/console-948c58bdb-g9g2g" Jul 14 09:27:54.418488 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:54.418469 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-qszj9" Jul 14 09:27:54.515440 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:54.515403 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/394653f7-9641-443f-b1e2-8ea222d9ae02-service-ca\") pod \"console-948c58bdb-g9g2g\" (UID: \"394653f7-9641-443f-b1e2-8ea222d9ae02\") " pod="openshift-console/console-948c58bdb-g9g2g" Jul 14 09:27:54.515740 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:54.515457 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/394653f7-9641-443f-b1e2-8ea222d9ae02-console-config\") pod \"console-948c58bdb-g9g2g\" (UID: \"394653f7-9641-443f-b1e2-8ea222d9ae02\") " pod="openshift-console/console-948c58bdb-g9g2g" Jul 14 09:27:54.515740 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:54.515493 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/394653f7-9641-443f-b1e2-8ea222d9ae02-console-oauth-config\") pod \"console-948c58bdb-g9g2g\" (UID: \"394653f7-9641-443f-b1e2-8ea222d9ae02\") " pod="openshift-console/console-948c58bdb-g9g2g" Jul 14 09:27:54.515740 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:54.515519 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/394653f7-9641-443f-b1e2-8ea222d9ae02-trusted-ca-bundle\") pod \"console-948c58bdb-g9g2g\" (UID: \"394653f7-9641-443f-b1e2-8ea222d9ae02\") " pod="openshift-console/console-948c58bdb-g9g2g" Jul 14 09:27:54.515740 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:54.515554 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bj78\" (UniqueName: \"kubernetes.io/projected/394653f7-9641-443f-b1e2-8ea222d9ae02-kube-api-access-5bj78\") pod \"console-948c58bdb-g9g2g\" (UID: \"394653f7-9641-443f-b1e2-8ea222d9ae02\") " pod="openshift-console/console-948c58bdb-g9g2g" Jul 14 09:27:54.515740 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:54.515587 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/394653f7-9641-443f-b1e2-8ea222d9ae02-console-serving-cert\") pod \"console-948c58bdb-g9g2g\" (UID: \"394653f7-9641-443f-b1e2-8ea222d9ae02\") " pod="openshift-console/console-948c58bdb-g9g2g" Jul 14 09:27:54.515740 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:54.515624 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/394653f7-9641-443f-b1e2-8ea222d9ae02-oauth-serving-cert\") pod \"console-948c58bdb-g9g2g\" (UID: \"394653f7-9641-443f-b1e2-8ea222d9ae02\") " pod="openshift-console/console-948c58bdb-g9g2g" Jul 14 09:27:54.516631 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:54.516400 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/394653f7-9641-443f-b1e2-8ea222d9ae02-service-ca\") pod \"console-948c58bdb-g9g2g\" (UID: \"394653f7-9641-443f-b1e2-8ea222d9ae02\") " pod="openshift-console/console-948c58bdb-g9g2g" Jul 14 09:27:54.516631 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:54.516590 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/394653f7-9641-443f-b1e2-8ea222d9ae02-trusted-ca-bundle\") pod \"console-948c58bdb-g9g2g\" (UID: \"394653f7-9641-443f-b1e2-8ea222d9ae02\") " pod="openshift-console/console-948c58bdb-g9g2g" Jul 14 09:27:54.516789 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:54.516637 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/394653f7-9641-443f-b1e2-8ea222d9ae02-console-config\") pod \"console-948c58bdb-g9g2g\" (UID: \"394653f7-9641-443f-b1e2-8ea222d9ae02\") " pod="openshift-console/console-948c58bdb-g9g2g" Jul 14 09:27:54.516789 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:54.516720 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/394653f7-9641-443f-b1e2-8ea222d9ae02-oauth-serving-cert\") pod \"console-948c58bdb-g9g2g\" (UID: \"394653f7-9641-443f-b1e2-8ea222d9ae02\") " pod="openshift-console/console-948c58bdb-g9g2g" Jul 14 09:27:54.518385 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:54.518338 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/394653f7-9641-443f-b1e2-8ea222d9ae02-console-oauth-config\") pod \"console-948c58bdb-g9g2g\" (UID: \"394653f7-9641-443f-b1e2-8ea222d9ae02\") " pod="openshift-console/console-948c58bdb-g9g2g" Jul 14 09:27:54.518774 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:54.518750 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/394653f7-9641-443f-b1e2-8ea222d9ae02-console-serving-cert\") pod \"console-948c58bdb-g9g2g\" (UID: \"394653f7-9641-443f-b1e2-8ea222d9ae02\") " pod="openshift-console/console-948c58bdb-g9g2g" Jul 14 09:27:54.530247 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:54.530224 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bj78\" (UniqueName: \"kubernetes.io/projected/394653f7-9641-443f-b1e2-8ea222d9ae02-kube-api-access-5bj78\") pod \"console-948c58bdb-g9g2g\" (UID: \"394653f7-9641-443f-b1e2-8ea222d9ae02\") " pod="openshift-console/console-948c58bdb-g9g2g" Jul 14 09:27:54.652407 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:54.652362 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-948c58bdb-g9g2g" Jul 14 09:27:55.736334 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:55.736272 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-948c58bdb-g9g2g"] Jul 14 09:27:55.738907 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:27:55.738878 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod394653f7_9641_443f_b1e2_8ea222d9ae02.slice/crio-df565999602e1101590270cbe027035ca9cb1ac7b31463b4730e94a77d9a13e0 WatchSource:0}: Error finding container df565999602e1101590270cbe027035ca9cb1ac7b31463b4730e94a77d9a13e0: Status 404 returned error can't find the container with id df565999602e1101590270cbe027035ca9cb1ac7b31463b4730e94a77d9a13e0 Jul 14 09:27:55.747952 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:55.747930 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-qszj9"] Jul 14 09:27:55.751477 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:27:55.751453 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48183b82_41a3_4075_b553_36bfd3742176.slice/crio-2eeace6377adf773491c806e7e35e9f0e0b8742ff27604fdafc644c13cf3ae46 WatchSource:0}: Error finding container 2eeace6377adf773491c806e7e35e9f0e0b8742ff27604fdafc644c13cf3ae46: Status 404 returned error can't find the container with id 2eeace6377adf773491c806e7e35e9f0e0b8742ff27604fdafc644c13cf3ae46 Jul 14 09:27:56.306255 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:56.306208 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-948c58bdb-g9g2g" event={"ID":"394653f7-9641-443f-b1e2-8ea222d9ae02","Type":"ContainerStarted","Data":"7dfc67eca4620e1462f07c58aa943d8fbaa8b393c85927db9ce6edbb0f36d546"} Jul 14 09:27:56.306454 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:56.306304 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-948c58bdb-g9g2g" event={"ID":"394653f7-9641-443f-b1e2-8ea222d9ae02","Type":"ContainerStarted","Data":"df565999602e1101590270cbe027035ca9cb1ac7b31463b4730e94a77d9a13e0"} Jul 14 09:27:56.308940 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:56.308901 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hr4j6" event={"ID":"5cffbd14-98ff-4790-aaaa-3b0ca91450fa","Type":"ContainerStarted","Data":"b21d17fcc12e8bbeabef21a6c00d1e98e74a61175dd07a2ce20125b9092eeefb"} Jul 14 09:27:56.310710 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:56.310683 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7555476c54-7f4b2" event={"ID":"1d088b26-558e-44e9-a8c3-5e836a0936e2","Type":"ContainerStarted","Data":"f5b5e9584e5529cbd49e78300ba4444448aa702b7daeba57183261a4d3fe8dd5"} Jul 14 09:27:56.311934 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:56.311910 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-qszj9" event={"ID":"48183b82-41a3-4075-b553-36bfd3742176","Type":"ContainerStarted","Data":"2eeace6377adf773491c806e7e35e9f0e0b8742ff27604fdafc644c13cf3ae46"} Jul 14 09:27:56.331062 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:56.331000 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-948c58bdb-g9g2g" podStartSLOduration=2.330983768 podStartE2EDuration="2.330983768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 09:27:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:27:56.3301692 +0000 UTC m=+65.839400652" watchObservedRunningTime="2026-07-14 09:27:56.330983768 +0000 UTC m=+65.840215204" Jul 14 09:27:56.351000 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:56.350940 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7555476c54-7f4b2" podStartSLOduration=5.339229606 podStartE2EDuration="9.350923915s" totalImagesPullingTime="4.011694309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 09:27:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:27:56.350440608 +0000 UTC m=+65.859672076" watchObservedRunningTime="2026-07-14 09:27:56.350923915 +0000 UTC m=+65.860155364" Jul 14 09:27:56.370286 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:56.370231 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hr4j6" podStartSLOduration=1.955911987 podStartE2EDuration="11.370213244s" totalImagesPullingTime="9.414301257s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 09:27:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:27:56.369506751 +0000 UTC m=+65.878738199" watchObservedRunningTime="2026-07-14 09:27:56.370213244 +0000 UTC m=+65.879444692" Jul 14 09:27:56.636153 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:56.636119 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs4nw\" (UniqueName: \"kubernetes.io/projected/7225439c-6210-4bc5-b124-0d92487dcd0c-kube-api-access-hs4nw\") pod \"network-check-target-n8c7q\" (UID: \"7225439c-6210-4bc5-b124-0d92487dcd0c\") " pod="openshift-network-diagnostics/network-check-target-n8c7q" Jul 14 09:27:56.638939 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:56.638917 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 14 09:27:56.649410 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:56.649377 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 14 09:27:56.660144 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:56.660116 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs4nw\" (UniqueName: \"kubernetes.io/projected/7225439c-6210-4bc5-b124-0d92487dcd0c-kube-api-access-hs4nw\") pod \"network-check-target-n8c7q\" (UID: \"7225439c-6210-4bc5-b124-0d92487dcd0c\") " pod="openshift-network-diagnostics/network-check-target-n8c7q" Jul 14 09:27:56.736864 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:56.736824 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a5c62b6a-f781-4066-9773-a5f4e8921276-original-pull-secret\") pod \"global-pull-secret-syncer-2tcgq\" (UID: \"a5c62b6a-f781-4066-9773-a5f4e8921276\") " pod="kube-system/global-pull-secret-syncer-2tcgq" Jul 14 09:27:56.737316 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:56.736925 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2190520-5806-41dd-b25f-b7d1076cfcf3-metrics-certs\") pod \"network-metrics-daemon-p5547\" (UID: \"c2190520-5806-41dd-b25f-b7d1076cfcf3\") " pod="openshift-multus/network-metrics-daemon-p5547" Jul 14 09:27:56.739730 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:56.739702 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 14 09:27:56.739730 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:56.739717 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 14 09:27:56.749626 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:56.749600 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a5c62b6a-f781-4066-9773-a5f4e8921276-original-pull-secret\") pod \"global-pull-secret-syncer-2tcgq\" (UID: \"a5c62b6a-f781-4066-9773-a5f4e8921276\") " pod="kube-system/global-pull-secret-syncer-2tcgq" Jul 14 09:27:56.749626 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:56.749619 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2190520-5806-41dd-b25f-b7d1076cfcf3-metrics-certs\") pod \"network-metrics-daemon-p5547\" (UID: \"c2190520-5806-41dd-b25f-b7d1076cfcf3\") " pod="openshift-multus/network-metrics-daemon-p5547" Jul 14 09:27:56.887200 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:56.887126 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-frh48\"" Jul 14 09:27:56.891540 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:56.891517 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2tcgq" Jul 14 09:27:56.894718 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:56.894698 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p5547" Jul 14 09:27:56.901962 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:56.901945 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ks9qb\"" Jul 14 09:27:56.910110 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:56.910084 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8c7q" Jul 14 09:27:57.124034 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:57.123973 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-n8c7q"] Jul 14 09:27:57.127305 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:27:57.127280 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7225439c_6210_4bc5_b124_0d92487dcd0c.slice/crio-65a1f6f30d18af2111193a499fb20eeafd76b43ac989d814a6d330e7a062ca8e WatchSource:0}: Error finding container 65a1f6f30d18af2111193a499fb20eeafd76b43ac989d814a6d330e7a062ca8e: Status 404 returned error can't find the container with id 65a1f6f30d18af2111193a499fb20eeafd76b43ac989d814a6d330e7a062ca8e Jul 14 09:27:57.317310 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:57.317230 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-n8c7q" event={"ID":"7225439c-6210-4bc5-b124-0d92487dcd0c","Type":"ContainerStarted","Data":"65a1f6f30d18af2111193a499fb20eeafd76b43ac989d814a6d330e7a062ca8e"} Jul 14 09:27:57.319083 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:57.319036 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-qszj9" event={"ID":"48183b82-41a3-4075-b553-36bfd3742176","Type":"ContainerStarted","Data":"1c2184987ef09ea731f85a529e480f81d152d55ca5c88ac980b96d637224b223"} Jul 14 09:27:57.319212 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:57.319092 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-qszj9" event={"ID":"48183b82-41a3-4075-b553-36bfd3742176","Type":"ContainerStarted","Data":"10fd16521e7aa66eae279f99eb926eab62a18d0e45d804b8f62610e865bcefe5"} Jul 14 09:27:57.338201 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:57.338150 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-qszj9" podStartSLOduration=2.129304264 podStartE2EDuration="3.338133637s" totalImagesPullingTime="1.208829373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 09:27:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:27:57.337466593 +0000 UTC m=+66.846698042" watchObservedRunningTime="2026-07-14 09:27:57.338133637 +0000 UTC m=+66.847365114" Jul 14 09:27:57.348142 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:57.348112 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2tcgq"] Jul 14 09:27:57.351287 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:57.351266 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-p5547"] Jul 14 09:27:57.353187 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:27:57.353128 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5c62b6a_f781_4066_9773_a5f4e8921276.slice/crio-4e3e265c6ab163e85f1b04cd1ea89ef3996d3f4ee006f3fac3325298a80487fd WatchSource:0}: Error finding container 4e3e265c6ab163e85f1b04cd1ea89ef3996d3f4ee006f3fac3325298a80487fd: Status 404 returned error can't find the container with id 4e3e265c6ab163e85f1b04cd1ea89ef3996d3f4ee006f3fac3325298a80487fd Jul 14 09:27:57.355157 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:27:57.355133 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2190520_5806_41dd_b25f_b7d1076cfcf3.slice/crio-6d638c1990688584a5a4ac60718fabbcf9674d7c3d0607371747a4c0178679ec WatchSource:0}: Error finding container 6d638c1990688584a5a4ac60718fabbcf9674d7c3d0607371747a4c0178679ec: Status 404 returned error can't find the container with id 6d638c1990688584a5a4ac60718fabbcf9674d7c3d0607371747a4c0178679ec Jul 14 09:27:57.869151 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:57.869109 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7555476c54-7f4b2" Jul 14 09:27:57.869622 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:57.869181 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7555476c54-7f4b2" Jul 14 09:27:57.877145 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:57.876959 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7555476c54-7f4b2" Jul 14 09:27:58.325538 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:58.325497 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p5547" event={"ID":"c2190520-5806-41dd-b25f-b7d1076cfcf3","Type":"ContainerStarted","Data":"6d638c1990688584a5a4ac60718fabbcf9674d7c3d0607371747a4c0178679ec"} Jul 14 09:27:58.327223 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:58.327191 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2tcgq" event={"ID":"a5c62b6a-f781-4066-9773-a5f4e8921276","Type":"ContainerStarted","Data":"4e3e265c6ab163e85f1b04cd1ea89ef3996d3f4ee006f3fac3325298a80487fd"} Jul 14 09:27:58.333018 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:58.332991 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7555476c54-7f4b2" Jul 14 09:27:59.335889 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.335582 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p5547" event={"ID":"c2190520-5806-41dd-b25f-b7d1076cfcf3","Type":"ContainerStarted","Data":"147ec1df4fb09a93ee60bb9a4e43a2b28a9541d434a2ef17385f9fdc937073d2"} Jul 14 09:27:59.335889 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.335650 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p5547" event={"ID":"c2190520-5806-41dd-b25f-b7d1076cfcf3","Type":"ContainerStarted","Data":"465988f5fd3d2a1072b0757a873d8012ea419b936d3c81cdb8825f6439bfa345"} Jul 14 09:27:59.355053 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.354974 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-p5547" podStartSLOduration=66.282055319 podStartE2EDuration="1m7.354950594s" totalImagesPullingTime="1.072895275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 09:26:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:27:59.353703439 +0000 UTC m=+68.862934879" watchObservedRunningTime="2026-07-14 09:27:59.354950594 +0000 UTC m=+68.864182043" Jul 14 09:27:59.586085 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.585990 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-5x5kh"] Jul 14 09:27:59.589319 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.589287 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-sw226"] Jul 14 09:27:59.589620 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.589598 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5x5kh" Jul 14 09:27:59.591821 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.591784 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-sw226" Jul 14 09:27:59.592577 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.592447 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-nrqdw\"" Jul 14 09:27:59.592577 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.592454 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 14 09:27:59.592577 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.592498 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 14 09:27:59.597144 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.596788 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 14 09:27:59.597144 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.596858 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 14 09:27:59.597144 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.596882 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 14 09:27:59.597144 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.596904 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-qxmvj\"" Jul 14 09:27:59.608521 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.608495 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-5x5kh"] Jul 14 09:27:59.615604 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.614226 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mh6sh"] Jul 14 09:27:59.616700 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.616680 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-sw226"] Jul 14 09:27:59.616879 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.616860 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mh6sh" Jul 14 09:27:59.619722 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.619471 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 14 09:27:59.619722 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.619519 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 14 09:27:59.619722 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.619552 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pkgzk\"" Jul 14 09:27:59.621247 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.621084 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 14 09:27:59.666569 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.666529 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/be19da8e-9e05-4c57-87b8-3da6b97e9514-node-exporter-textfile\") pod \"node-exporter-mh6sh\" (UID: \"be19da8e-9e05-4c57-87b8-3da6b97e9514\") " pod="openshift-monitoring/node-exporter-mh6sh" Jul 14 09:27:59.666569 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.666575 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/493cb380-3b2f-4f83-b251-c40f63b0b655-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-sw226\" (UID: \"493cb380-3b2f-4f83-b251-c40f63b0b655\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-sw226" Jul 14 09:27:59.666775 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.666650 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/be19da8e-9e05-4c57-87b8-3da6b97e9514-node-exporter-wtmp\") pod \"node-exporter-mh6sh\" (UID: \"be19da8e-9e05-4c57-87b8-3da6b97e9514\") " pod="openshift-monitoring/node-exporter-mh6sh" Jul 14 09:27:59.666775 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.666677 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q68rx\" (UniqueName: \"kubernetes.io/projected/be19da8e-9e05-4c57-87b8-3da6b97e9514-kube-api-access-q68rx\") pod \"node-exporter-mh6sh\" (UID: \"be19da8e-9e05-4c57-87b8-3da6b97e9514\") " pod="openshift-monitoring/node-exporter-mh6sh" Jul 14 09:27:59.666775 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.666736 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/be19da8e-9e05-4c57-87b8-3da6b97e9514-node-exporter-tls\") pod \"node-exporter-mh6sh\" (UID: \"be19da8e-9e05-4c57-87b8-3da6b97e9514\") " pod="openshift-monitoring/node-exporter-mh6sh" Jul 14 09:27:59.666949 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.666779 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/493cb380-3b2f-4f83-b251-c40f63b0b655-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-sw226\" (UID: \"493cb380-3b2f-4f83-b251-c40f63b0b655\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-sw226" Jul 14 09:27:59.666949 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.666829 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v69rs\" (UniqueName: \"kubernetes.io/projected/493cb380-3b2f-4f83-b251-c40f63b0b655-kube-api-access-v69rs\") pod \"kube-state-metrics-785778ff74-sw226\" (UID: \"493cb380-3b2f-4f83-b251-c40f63b0b655\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-sw226" Jul 14 09:27:59.666949 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.666874 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/be19da8e-9e05-4c57-87b8-3da6b97e9514-root\") pod \"node-exporter-mh6sh\" (UID: \"be19da8e-9e05-4c57-87b8-3da6b97e9514\") " pod="openshift-monitoring/node-exporter-mh6sh" Jul 14 09:27:59.666949 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.666908 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be19da8e-9e05-4c57-87b8-3da6b97e9514-metrics-client-ca\") pod \"node-exporter-mh6sh\" (UID: \"be19da8e-9e05-4c57-87b8-3da6b97e9514\") " pod="openshift-monitoring/node-exporter-mh6sh" Jul 14 09:27:59.666949 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.666937 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/493cb380-3b2f-4f83-b251-c40f63b0b655-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-sw226\" (UID: \"493cb380-3b2f-4f83-b251-c40f63b0b655\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-sw226" Jul 14 09:27:59.667153 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.666968 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6eab5a46-4afa-4e95-afa5-2a235e5e463c-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-5x5kh\" (UID: \"6eab5a46-4afa-4e95-afa5-2a235e5e463c\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5x5kh" Jul 14 09:27:59.667153 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.666996 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/be19da8e-9e05-4c57-87b8-3da6b97e9514-node-exporter-accelerators-collector-config\") pod \"node-exporter-mh6sh\" (UID: \"be19da8e-9e05-4c57-87b8-3da6b97e9514\") " pod="openshift-monitoring/node-exporter-mh6sh" Jul 14 09:27:59.667153 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.667021 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndkhs\" (UniqueName: \"kubernetes.io/projected/6eab5a46-4afa-4e95-afa5-2a235e5e463c-kube-api-access-ndkhs\") pod \"openshift-state-metrics-84bd68b896-5x5kh\" (UID: \"6eab5a46-4afa-4e95-afa5-2a235e5e463c\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5x5kh" Jul 14 09:27:59.667153 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.667062 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/493cb380-3b2f-4f83-b251-c40f63b0b655-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-sw226\" (UID: \"493cb380-3b2f-4f83-b251-c40f63b0b655\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-sw226" Jul 14 09:27:59.667153 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.667103 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be19da8e-9e05-4c57-87b8-3da6b97e9514-sys\") pod \"node-exporter-mh6sh\" (UID: \"be19da8e-9e05-4c57-87b8-3da6b97e9514\") " pod="openshift-monitoring/node-exporter-mh6sh" Jul 14 09:27:59.667153 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.667132 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6eab5a46-4afa-4e95-afa5-2a235e5e463c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-5x5kh\" (UID: \"6eab5a46-4afa-4e95-afa5-2a235e5e463c\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5x5kh" Jul 14 09:27:59.667343 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.667168 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6eab5a46-4afa-4e95-afa5-2a235e5e463c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-5x5kh\" (UID: \"6eab5a46-4afa-4e95-afa5-2a235e5e463c\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5x5kh" Jul 14 09:27:59.667343 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.667197 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/be19da8e-9e05-4c57-87b8-3da6b97e9514-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mh6sh\" (UID: \"be19da8e-9e05-4c57-87b8-3da6b97e9514\") " pod="openshift-monitoring/node-exporter-mh6sh" Jul 14 09:27:59.667343 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.667224 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/493cb380-3b2f-4f83-b251-c40f63b0b655-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-sw226\" (UID: \"493cb380-3b2f-4f83-b251-c40f63b0b655\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-sw226" Jul 14 09:27:59.767635 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.767589 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/be19da8e-9e05-4c57-87b8-3da6b97e9514-node-exporter-wtmp\") pod \"node-exporter-mh6sh\" (UID: \"be19da8e-9e05-4c57-87b8-3da6b97e9514\") " pod="openshift-monitoring/node-exporter-mh6sh" Jul 14 09:27:59.767635 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.767634 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q68rx\" (UniqueName: \"kubernetes.io/projected/be19da8e-9e05-4c57-87b8-3da6b97e9514-kube-api-access-q68rx\") pod \"node-exporter-mh6sh\" (UID: \"be19da8e-9e05-4c57-87b8-3da6b97e9514\") " pod="openshift-monitoring/node-exporter-mh6sh" Jul 14 09:27:59.767896 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.767669 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/be19da8e-9e05-4c57-87b8-3da6b97e9514-node-exporter-tls\") pod \"node-exporter-mh6sh\" (UID: \"be19da8e-9e05-4c57-87b8-3da6b97e9514\") " pod="openshift-monitoring/node-exporter-mh6sh" Jul 14 09:27:59.767896 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.767700 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/493cb380-3b2f-4f83-b251-c40f63b0b655-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-sw226\" (UID: \"493cb380-3b2f-4f83-b251-c40f63b0b655\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-sw226" Jul 14 09:27:59.767896 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.767738 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v69rs\" (UniqueName: \"kubernetes.io/projected/493cb380-3b2f-4f83-b251-c40f63b0b655-kube-api-access-v69rs\") pod \"kube-state-metrics-785778ff74-sw226\" (UID: \"493cb380-3b2f-4f83-b251-c40f63b0b655\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-sw226" Jul 14 09:27:59.767896 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.767763 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/be19da8e-9e05-4c57-87b8-3da6b97e9514-root\") pod \"node-exporter-mh6sh\" (UID: \"be19da8e-9e05-4c57-87b8-3da6b97e9514\") " pod="openshift-monitoring/node-exporter-mh6sh" Jul 14 09:27:59.767896 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.767784 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/be19da8e-9e05-4c57-87b8-3da6b97e9514-node-exporter-wtmp\") pod \"node-exporter-mh6sh\" (UID: \"be19da8e-9e05-4c57-87b8-3da6b97e9514\") " pod="openshift-monitoring/node-exporter-mh6sh" Jul 14 09:27:59.767896 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.767793 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be19da8e-9e05-4c57-87b8-3da6b97e9514-metrics-client-ca\") pod \"node-exporter-mh6sh\" (UID: \"be19da8e-9e05-4c57-87b8-3da6b97e9514\") " pod="openshift-monitoring/node-exporter-mh6sh" Jul 14 09:27:59.767896 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.767871 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/493cb380-3b2f-4f83-b251-c40f63b0b655-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-sw226\" (UID: \"493cb380-3b2f-4f83-b251-c40f63b0b655\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-sw226" Jul 14 09:27:59.768245 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.767905 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6eab5a46-4afa-4e95-afa5-2a235e5e463c-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-5x5kh\" (UID: \"6eab5a46-4afa-4e95-afa5-2a235e5e463c\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5x5kh" Jul 14 09:27:59.768245 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.767942 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/be19da8e-9e05-4c57-87b8-3da6b97e9514-node-exporter-accelerators-collector-config\") pod \"node-exporter-mh6sh\" (UID: \"be19da8e-9e05-4c57-87b8-3da6b97e9514\") " pod="openshift-monitoring/node-exporter-mh6sh" Jul 14 09:27:59.768245 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.767969 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndkhs\" (UniqueName: \"kubernetes.io/projected/6eab5a46-4afa-4e95-afa5-2a235e5e463c-kube-api-access-ndkhs\") pod \"openshift-state-metrics-84bd68b896-5x5kh\" (UID: \"6eab5a46-4afa-4e95-afa5-2a235e5e463c\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5x5kh" Jul 14 09:27:59.768245 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:59.767985 2565 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 14 09:27:59.768245 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.767994 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/493cb380-3b2f-4f83-b251-c40f63b0b655-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-sw226\" (UID: \"493cb380-3b2f-4f83-b251-c40f63b0b655\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-sw226" Jul 14 09:27:59.768245 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.768017 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be19da8e-9e05-4c57-87b8-3da6b97e9514-sys\") pod \"node-exporter-mh6sh\" (UID: \"be19da8e-9e05-4c57-87b8-3da6b97e9514\") " pod="openshift-monitoring/node-exporter-mh6sh" Jul 14 09:27:59.768245 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:59.768053 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be19da8e-9e05-4c57-87b8-3da6b97e9514-node-exporter-tls podName:be19da8e-9e05-4c57-87b8-3da6b97e9514 nodeName:}" failed. No retries permitted until 2026-07-14 09:28:00.268023793 +0000 UTC m=+69.777255226 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/be19da8e-9e05-4c57-87b8-3da6b97e9514-node-exporter-tls") pod "node-exporter-mh6sh" (UID: "be19da8e-9e05-4c57-87b8-3da6b97e9514") : secret "node-exporter-tls" not found Jul 14 09:27:59.768245 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.768068 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/be19da8e-9e05-4c57-87b8-3da6b97e9514-sys\") pod \"node-exporter-mh6sh\" (UID: \"be19da8e-9e05-4c57-87b8-3da6b97e9514\") " pod="openshift-monitoring/node-exporter-mh6sh" Jul 14 09:27:59.768245 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.768080 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6eab5a46-4afa-4e95-afa5-2a235e5e463c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-5x5kh\" (UID: \"6eab5a46-4afa-4e95-afa5-2a235e5e463c\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5x5kh" Jul 14 09:27:59.768245 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.768117 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6eab5a46-4afa-4e95-afa5-2a235e5e463c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-5x5kh\" (UID: \"6eab5a46-4afa-4e95-afa5-2a235e5e463c\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5x5kh" Jul 14 09:27:59.768245 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.768147 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/be19da8e-9e05-4c57-87b8-3da6b97e9514-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mh6sh\" (UID: \"be19da8e-9e05-4c57-87b8-3da6b97e9514\") " pod="openshift-monitoring/node-exporter-mh6sh" Jul 14 09:27:59.768245 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.768173 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/493cb380-3b2f-4f83-b251-c40f63b0b655-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-sw226\" (UID: \"493cb380-3b2f-4f83-b251-c40f63b0b655\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-sw226" Jul 14 09:27:59.768245 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.768208 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/be19da8e-9e05-4c57-87b8-3da6b97e9514-node-exporter-textfile\") pod \"node-exporter-mh6sh\" (UID: \"be19da8e-9e05-4c57-87b8-3da6b97e9514\") " pod="openshift-monitoring/node-exporter-mh6sh" Jul 14 09:27:59.768245 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.768236 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/493cb380-3b2f-4f83-b251-c40f63b0b655-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-sw226\" (UID: \"493cb380-3b2f-4f83-b251-c40f63b0b655\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-sw226" Jul 14 09:27:59.768915 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.768468 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be19da8e-9e05-4c57-87b8-3da6b97e9514-metrics-client-ca\") pod \"node-exporter-mh6sh\" (UID: \"be19da8e-9e05-4c57-87b8-3da6b97e9514\") " pod="openshift-monitoring/node-exporter-mh6sh" Jul 14 09:27:59.768915 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.768538 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/be19da8e-9e05-4c57-87b8-3da6b97e9514-root\") pod \"node-exporter-mh6sh\" (UID: \"be19da8e-9e05-4c57-87b8-3da6b97e9514\") " pod="openshift-monitoring/node-exporter-mh6sh" Jul 14 09:27:59.768915 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:59.768627 2565 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 14 09:27:59.768915 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:27:59.768689 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6eab5a46-4afa-4e95-afa5-2a235e5e463c-openshift-state-metrics-tls podName:6eab5a46-4afa-4e95-afa5-2a235e5e463c nodeName:}" failed. No retries permitted until 2026-07-14 09:28:00.268671337 +0000 UTC m=+69.777902780 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/6eab5a46-4afa-4e95-afa5-2a235e5e463c-openshift-state-metrics-tls") pod "openshift-state-metrics-84bd68b896-5x5kh" (UID: "6eab5a46-4afa-4e95-afa5-2a235e5e463c") : secret "openshift-state-metrics-tls" not found Jul 14 09:27:59.769115 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.768969 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/493cb380-3b2f-4f83-b251-c40f63b0b655-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-sw226\" (UID: \"493cb380-3b2f-4f83-b251-c40f63b0b655\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-sw226" Jul 14 09:27:59.769115 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.768993 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/be19da8e-9e05-4c57-87b8-3da6b97e9514-node-exporter-textfile\") pod \"node-exporter-mh6sh\" (UID: \"be19da8e-9e05-4c57-87b8-3da6b97e9514\") " pod="openshift-monitoring/node-exporter-mh6sh" Jul 14 09:27:59.769115 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.769071 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/be19da8e-9e05-4c57-87b8-3da6b97e9514-node-exporter-accelerators-collector-config\") pod \"node-exporter-mh6sh\" (UID: \"be19da8e-9e05-4c57-87b8-3da6b97e9514\") " pod="openshift-monitoring/node-exporter-mh6sh" Jul 14 09:27:59.770209 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.769620 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6eab5a46-4afa-4e95-afa5-2a235e5e463c-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-5x5kh\" (UID: \"6eab5a46-4afa-4e95-afa5-2a235e5e463c\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5x5kh" Jul 14 09:27:59.770209 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.770054 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/493cb380-3b2f-4f83-b251-c40f63b0b655-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-sw226\" (UID: \"493cb380-3b2f-4f83-b251-c40f63b0b655\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-sw226" Jul 14 09:27:59.770209 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.770170 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/493cb380-3b2f-4f83-b251-c40f63b0b655-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-sw226\" (UID: \"493cb380-3b2f-4f83-b251-c40f63b0b655\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-sw226" Jul 14 09:27:59.771624 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.771446 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/493cb380-3b2f-4f83-b251-c40f63b0b655-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-sw226\" (UID: \"493cb380-3b2f-4f83-b251-c40f63b0b655\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-sw226" Jul 14 09:27:59.771624 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.771518 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6eab5a46-4afa-4e95-afa5-2a235e5e463c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-5x5kh\" (UID: \"6eab5a46-4afa-4e95-afa5-2a235e5e463c\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5x5kh" Jul 14 09:27:59.771844 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.771786 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/be19da8e-9e05-4c57-87b8-3da6b97e9514-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mh6sh\" (UID: \"be19da8e-9e05-4c57-87b8-3da6b97e9514\") " pod="openshift-monitoring/node-exporter-mh6sh" Jul 14 09:27:59.773493 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.773465 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/493cb380-3b2f-4f83-b251-c40f63b0b655-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-sw226\" (UID: \"493cb380-3b2f-4f83-b251-c40f63b0b655\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-sw226" Jul 14 09:27:59.778722 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.778674 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q68rx\" (UniqueName: \"kubernetes.io/projected/be19da8e-9e05-4c57-87b8-3da6b97e9514-kube-api-access-q68rx\") pod \"node-exporter-mh6sh\" (UID: \"be19da8e-9e05-4c57-87b8-3da6b97e9514\") " pod="openshift-monitoring/node-exporter-mh6sh" Jul 14 09:27:59.779090 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.779066 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndkhs\" (UniqueName: \"kubernetes.io/projected/6eab5a46-4afa-4e95-afa5-2a235e5e463c-kube-api-access-ndkhs\") pod \"openshift-state-metrics-84bd68b896-5x5kh\" (UID: \"6eab5a46-4afa-4e95-afa5-2a235e5e463c\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5x5kh" Jul 14 09:27:59.779435 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.779415 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v69rs\" (UniqueName: \"kubernetes.io/projected/493cb380-3b2f-4f83-b251-c40f63b0b655-kube-api-access-v69rs\") pod \"kube-state-metrics-785778ff74-sw226\" (UID: \"493cb380-3b2f-4f83-b251-c40f63b0b655\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-sw226" Jul 14 09:27:59.909706 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:27:59.909661 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-sw226" Jul 14 09:28:00.273579 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:00.273433 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6eab5a46-4afa-4e95-afa5-2a235e5e463c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-5x5kh\" (UID: \"6eab5a46-4afa-4e95-afa5-2a235e5e463c\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5x5kh" Jul 14 09:28:00.273579 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:00.273525 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/be19da8e-9e05-4c57-87b8-3da6b97e9514-node-exporter-tls\") pod \"node-exporter-mh6sh\" (UID: \"be19da8e-9e05-4c57-87b8-3da6b97e9514\") " pod="openshift-monitoring/node-exporter-mh6sh" Jul 14 09:28:00.276531 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:00.276501 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/be19da8e-9e05-4c57-87b8-3da6b97e9514-node-exporter-tls\") pod \"node-exporter-mh6sh\" (UID: \"be19da8e-9e05-4c57-87b8-3da6b97e9514\") " pod="openshift-monitoring/node-exporter-mh6sh" Jul 14 09:28:00.277206 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:00.277163 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6eab5a46-4afa-4e95-afa5-2a235e5e463c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-5x5kh\" (UID: \"6eab5a46-4afa-4e95-afa5-2a235e5e463c\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5x5kh" Jul 14 09:28:00.502034 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:00.501999 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5x5kh" Jul 14 09:28:00.528153 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:00.528051 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mh6sh" Jul 14 09:28:00.575398 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:00.575359 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 14 09:28:00.581585 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:00.581549 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 14 09:28:00.584743 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:00.584657 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 14 09:28:00.584932 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:00.584758 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 14 09:28:00.584932 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:00.584910 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-8bjl7\"" Jul 14 09:28:00.586940 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:00.586917 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 14 09:28:00.587107 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:00.586969 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 14 09:28:00.587107 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:00.586982 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 14 09:28:00.587240 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:00.587131 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 14 09:28:00.587292 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:00.587249 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 14 09:28:00.587338 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:00.587314 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 14 09:28:00.587517 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:00.587503 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 14 09:28:00.599595 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:00.599569 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 14 09:28:00.677283 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:00.677252 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n4qh\" (UniqueName: \"kubernetes.io/projected/5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc-kube-api-access-2n4qh\") pod \"alertmanager-main-0\" (UID: \"5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 09:28:00.677283 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:00.677287 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 09:28:00.677522 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:00.677322 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 09:28:00.677522 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:00.677385 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 09:28:00.677522 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:00.677442 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 09:28:00.677522 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:00.677464 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 09:28:00.677522 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:00.677480 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 09:28:00.677710 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:00.677526 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc-config-volume\") pod \"alertmanager-main-0\" (UID: \"5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 09:28:00.677710 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:00.677605 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 09:28:00.677710 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:00.677631 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 09:28:00.677710 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:00.677662 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc-config-out\") pod \"alertmanager-main-0\" (UID: \"5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 09:28:00.677911 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:00.677724 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 09:28:00.677911 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:00.677760 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc-web-config\") pod \"alertmanager-main-0\" (UID: \"5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 09:28:00.779050 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:00.778960 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n4qh\" (UniqueName: \"kubernetes.io/projected/5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc-kube-api-access-2n4qh\") pod \"alertmanager-main-0\" (UID: \"5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 09:28:00.779201 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:00.779107 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 09:28:00.779201 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:00.779142 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 09:28:00.779201 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:00.779181 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 09:28:00.779365 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:00.779212 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 09:28:00.779365 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:00.779252 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 09:28:00.779365 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:00.779275 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 09:28:00.779365 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:00.779304 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc-config-volume\") pod \"alertmanager-main-0\" (UID: \"5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 09:28:00.779600 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:28:00.779367 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc-alertmanager-trusted-ca-bundle podName:5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc nodeName:}" failed. No retries permitted until 2026-07-14 09:28:01.279341046 +0000 UTC m=+70.788572491 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc") : configmap references non-existent config key: ca-bundle.crt Jul 14 09:28:00.779600 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:00.779400 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 09:28:00.779600 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:00.779453 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 09:28:00.779600 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:00.779498 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc-config-out\") pod \"alertmanager-main-0\" (UID: \"5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 09:28:00.779600 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:00.779549 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 09:28:00.779600 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:00.779585 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc-web-config\") pod \"alertmanager-main-0\" (UID: \"5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 09:28:00.779941 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:00.779603 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 09:28:00.780571 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:28:00.780519 2565 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 14 09:28:00.780663 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:28:00.780584 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc-secret-alertmanager-main-tls podName:5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc nodeName:}" failed. No retries permitted until 2026-07-14 09:28:01.280563208 +0000 UTC m=+70.789794633 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc") : secret "alertmanager-main-tls" not found Jul 14 09:28:00.782517 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:00.782489 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc-web-config\") pod \"alertmanager-main-0\" (UID: \"5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 09:28:00.782640 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:00.782607 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 09:28:00.782701 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:00.782657 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc-config-out\") pod \"alertmanager-main-0\" (UID: \"5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 09:28:00.782701 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:00.782666 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 09:28:00.783242 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:00.783217 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 09:28:00.783336 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:00.783227 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 09:28:00.784173 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:00.784132 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 09:28:00.784252 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:00.784214 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 09:28:00.784945 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:00.784912 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc-config-volume\") pod \"alertmanager-main-0\" (UID: \"5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 09:28:00.795076 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:00.795033 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n4qh\" (UniqueName: \"kubernetes.io/projected/5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc-kube-api-access-2n4qh\") pod \"alertmanager-main-0\" (UID: \"5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 09:28:01.283686 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:01.283640 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 09:28:01.283899 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:01.283749 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 09:28:01.284753 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:01.284726 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 09:28:01.286602 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:01.286575 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 09:28:01.495467 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:01.495428 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 14 09:28:02.387978 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:28:02.387943 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe19da8e_9e05_4c57_87b8_3da6b97e9514.slice/crio-8edd9b90edfa11ad90afa4da48b761469624bc645739c5e9d51cf02444c0293b WatchSource:0}: Error finding container 8edd9b90edfa11ad90afa4da48b761469624bc645739c5e9d51cf02444c0293b: Status 404 returned error can't find the container with id 8edd9b90edfa11ad90afa4da48b761469624bc645739c5e9d51cf02444c0293b Jul 14 09:28:02.554043 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:02.553985 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-5x5kh"] Jul 14 09:28:02.572359 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:02.571410 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-9c5fd5b4b-gbbk7"] Jul 14 09:28:02.575594 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:02.575570 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-9c5fd5b4b-gbbk7" Jul 14 09:28:02.579326 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:02.579193 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-acjhn96tn1uvv\"" Jul 14 09:28:02.579611 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:02.579510 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 14 09:28:02.579966 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:02.579798 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-2dgst\"" Jul 14 09:28:02.579966 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:02.579862 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 14 09:28:02.580117 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:02.580020 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 14 09:28:02.580117 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:02.580094 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 14 09:28:02.580217 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:02.580198 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 14 09:28:02.581374 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:02.581336 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-sw226"] Jul 14 09:28:02.583528 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:28:02.583503 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod493cb380_3b2f_4f83_b251_c40f63b0b655.slice/crio-8c3399c35b95bca1bba0404bd648fc08c9e4fccd2f1cebdad7e5f5cc1653a077 WatchSource:0}: Error finding container 8c3399c35b95bca1bba0404bd648fc08c9e4fccd2f1cebdad7e5f5cc1653a077: Status 404 returned error can't find the container with id 8c3399c35b95bca1bba0404bd648fc08c9e4fccd2f1cebdad7e5f5cc1653a077 Jul 14 09:28:02.591769 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:02.591735 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-9c5fd5b4b-gbbk7"] Jul 14 09:28:02.702396 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:02.702365 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/85e26c8a-faa2-43bb-90e9-1d8175b8f8a3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-9c5fd5b4b-gbbk7\" (UID: \"85e26c8a-faa2-43bb-90e9-1d8175b8f8a3\") " pod="openshift-monitoring/thanos-querier-9c5fd5b4b-gbbk7" Jul 14 09:28:02.702537 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:02.702421 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/85e26c8a-faa2-43bb-90e9-1d8175b8f8a3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-9c5fd5b4b-gbbk7\" (UID: \"85e26c8a-faa2-43bb-90e9-1d8175b8f8a3\") " pod="openshift-monitoring/thanos-querier-9c5fd5b4b-gbbk7" Jul 14 09:28:02.702537 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:02.702460 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/85e26c8a-faa2-43bb-90e9-1d8175b8f8a3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-9c5fd5b4b-gbbk7\" (UID: \"85e26c8a-faa2-43bb-90e9-1d8175b8f8a3\") " pod="openshift-monitoring/thanos-querier-9c5fd5b4b-gbbk7" Jul 14 09:28:02.702537 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:02.702498 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/85e26c8a-faa2-43bb-90e9-1d8175b8f8a3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-9c5fd5b4b-gbbk7\" (UID: \"85e26c8a-faa2-43bb-90e9-1d8175b8f8a3\") " pod="openshift-monitoring/thanos-querier-9c5fd5b4b-gbbk7" Jul 14 09:28:02.702742 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:02.702595 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/85e26c8a-faa2-43bb-90e9-1d8175b8f8a3-metrics-client-ca\") pod \"thanos-querier-9c5fd5b4b-gbbk7\" (UID: \"85e26c8a-faa2-43bb-90e9-1d8175b8f8a3\") " pod="openshift-monitoring/thanos-querier-9c5fd5b4b-gbbk7" Jul 14 09:28:02.702742 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:02.702643 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/85e26c8a-faa2-43bb-90e9-1d8175b8f8a3-secret-thanos-querier-tls\") pod \"thanos-querier-9c5fd5b4b-gbbk7\" (UID: \"85e26c8a-faa2-43bb-90e9-1d8175b8f8a3\") " pod="openshift-monitoring/thanos-querier-9c5fd5b4b-gbbk7" Jul 14 09:28:02.702742 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:02.702674 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgtdg\" (UniqueName: \"kubernetes.io/projected/85e26c8a-faa2-43bb-90e9-1d8175b8f8a3-kube-api-access-wgtdg\") pod \"thanos-querier-9c5fd5b4b-gbbk7\" (UID: \"85e26c8a-faa2-43bb-90e9-1d8175b8f8a3\") " pod="openshift-monitoring/thanos-querier-9c5fd5b4b-gbbk7" Jul 14 09:28:02.702742 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:02.702732 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/85e26c8a-faa2-43bb-90e9-1d8175b8f8a3-secret-grpc-tls\") pod \"thanos-querier-9c5fd5b4b-gbbk7\" (UID: \"85e26c8a-faa2-43bb-90e9-1d8175b8f8a3\") " pod="openshift-monitoring/thanos-querier-9c5fd5b4b-gbbk7" Jul 14 09:28:02.794986 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:02.794949 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 14 09:28:02.795245 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:28:02.795212 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c3172dc_8d4c_41cb_b7dd_4b9b29ba7adc.slice/crio-f00aa52213f173b9ac6d7c728efa74788984caef11a6917f4922e103c9916caf WatchSource:0}: Error finding container f00aa52213f173b9ac6d7c728efa74788984caef11a6917f4922e103c9916caf: Status 404 returned error can't find the container with id f00aa52213f173b9ac6d7c728efa74788984caef11a6917f4922e103c9916caf Jul 14 09:28:02.803261 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:02.803228 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/85e26c8a-faa2-43bb-90e9-1d8175b8f8a3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-9c5fd5b4b-gbbk7\" (UID: \"85e26c8a-faa2-43bb-90e9-1d8175b8f8a3\") " pod="openshift-monitoring/thanos-querier-9c5fd5b4b-gbbk7" Jul 14 09:28:02.803363 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:02.803283 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/85e26c8a-faa2-43bb-90e9-1d8175b8f8a3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-9c5fd5b4b-gbbk7\" (UID: \"85e26c8a-faa2-43bb-90e9-1d8175b8f8a3\") " pod="openshift-monitoring/thanos-querier-9c5fd5b4b-gbbk7" Jul 14 09:28:02.803363 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:02.803318 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/85e26c8a-faa2-43bb-90e9-1d8175b8f8a3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-9c5fd5b4b-gbbk7\" (UID: \"85e26c8a-faa2-43bb-90e9-1d8175b8f8a3\") " pod="openshift-monitoring/thanos-querier-9c5fd5b4b-gbbk7" Jul 14 09:28:02.803363 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:02.803352 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/85e26c8a-faa2-43bb-90e9-1d8175b8f8a3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-9c5fd5b4b-gbbk7\" (UID: \"85e26c8a-faa2-43bb-90e9-1d8175b8f8a3\") " pod="openshift-monitoring/thanos-querier-9c5fd5b4b-gbbk7" Jul 14 09:28:02.803514 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:02.803397 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/85e26c8a-faa2-43bb-90e9-1d8175b8f8a3-metrics-client-ca\") pod \"thanos-querier-9c5fd5b4b-gbbk7\" (UID: \"85e26c8a-faa2-43bb-90e9-1d8175b8f8a3\") " pod="openshift-monitoring/thanos-querier-9c5fd5b4b-gbbk7" Jul 14 09:28:02.803514 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:02.803434 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/85e26c8a-faa2-43bb-90e9-1d8175b8f8a3-secret-thanos-querier-tls\") pod \"thanos-querier-9c5fd5b4b-gbbk7\" (UID: \"85e26c8a-faa2-43bb-90e9-1d8175b8f8a3\") " pod="openshift-monitoring/thanos-querier-9c5fd5b4b-gbbk7" Jul 14 09:28:02.803514 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:02.803460 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgtdg\" (UniqueName: \"kubernetes.io/projected/85e26c8a-faa2-43bb-90e9-1d8175b8f8a3-kube-api-access-wgtdg\") pod \"thanos-querier-9c5fd5b4b-gbbk7\" (UID: \"85e26c8a-faa2-43bb-90e9-1d8175b8f8a3\") " pod="openshift-monitoring/thanos-querier-9c5fd5b4b-gbbk7" Jul 14 09:28:02.803514 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:02.803495 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/85e26c8a-faa2-43bb-90e9-1d8175b8f8a3-secret-grpc-tls\") pod \"thanos-querier-9c5fd5b4b-gbbk7\" (UID: \"85e26c8a-faa2-43bb-90e9-1d8175b8f8a3\") " pod="openshift-monitoring/thanos-querier-9c5fd5b4b-gbbk7" Jul 14 09:28:02.804468 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:02.804442 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/85e26c8a-faa2-43bb-90e9-1d8175b8f8a3-metrics-client-ca\") pod \"thanos-querier-9c5fd5b4b-gbbk7\" (UID: \"85e26c8a-faa2-43bb-90e9-1d8175b8f8a3\") " pod="openshift-monitoring/thanos-querier-9c5fd5b4b-gbbk7" Jul 14 09:28:02.806251 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:02.806222 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/85e26c8a-faa2-43bb-90e9-1d8175b8f8a3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-9c5fd5b4b-gbbk7\" (UID: \"85e26c8a-faa2-43bb-90e9-1d8175b8f8a3\") " pod="openshift-monitoring/thanos-querier-9c5fd5b4b-gbbk7" Jul 14 09:28:02.806462 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:02.806411 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/85e26c8a-faa2-43bb-90e9-1d8175b8f8a3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-9c5fd5b4b-gbbk7\" (UID: \"85e26c8a-faa2-43bb-90e9-1d8175b8f8a3\") " pod="openshift-monitoring/thanos-querier-9c5fd5b4b-gbbk7" Jul 14 09:28:02.806574 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:02.806522 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/85e26c8a-faa2-43bb-90e9-1d8175b8f8a3-secret-thanos-querier-tls\") pod \"thanos-querier-9c5fd5b4b-gbbk7\" (UID: \"85e26c8a-faa2-43bb-90e9-1d8175b8f8a3\") " pod="openshift-monitoring/thanos-querier-9c5fd5b4b-gbbk7" Jul 14 09:28:02.807051 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:02.807021 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/85e26c8a-faa2-43bb-90e9-1d8175b8f8a3-secret-grpc-tls\") pod \"thanos-querier-9c5fd5b4b-gbbk7\" (UID: \"85e26c8a-faa2-43bb-90e9-1d8175b8f8a3\") " pod="openshift-monitoring/thanos-querier-9c5fd5b4b-gbbk7" Jul 14 09:28:02.807273 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:02.807256 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/85e26c8a-faa2-43bb-90e9-1d8175b8f8a3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-9c5fd5b4b-gbbk7\" (UID: \"85e26c8a-faa2-43bb-90e9-1d8175b8f8a3\") " pod="openshift-monitoring/thanos-querier-9c5fd5b4b-gbbk7" Jul 14 09:28:02.807345 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:02.807287 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/85e26c8a-faa2-43bb-90e9-1d8175b8f8a3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-9c5fd5b4b-gbbk7\" (UID: \"85e26c8a-faa2-43bb-90e9-1d8175b8f8a3\") " pod="openshift-monitoring/thanos-querier-9c5fd5b4b-gbbk7" Jul 14 09:28:02.813730 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:02.813707 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgtdg\" (UniqueName: \"kubernetes.io/projected/85e26c8a-faa2-43bb-90e9-1d8175b8f8a3-kube-api-access-wgtdg\") pod \"thanos-querier-9c5fd5b4b-gbbk7\" (UID: \"85e26c8a-faa2-43bb-90e9-1d8175b8f8a3\") " pod="openshift-monitoring/thanos-querier-9c5fd5b4b-gbbk7" Jul 14 09:28:02.888686 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:02.888651 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-9c5fd5b4b-gbbk7" Jul 14 09:28:03.037108 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:03.037077 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-9c5fd5b4b-gbbk7"] Jul 14 09:28:03.155261 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:28:03.155193 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85e26c8a_faa2_43bb_90e9_1d8175b8f8a3.slice/crio-86d14e69e233873ce40b900d7ccc432453e94a9fc168874c109b19c6cdc764c3 WatchSource:0}: Error finding container 86d14e69e233873ce40b900d7ccc432453e94a9fc168874c109b19c6cdc764c3: Status 404 returned error can't find the container with id 86d14e69e233873ce40b900d7ccc432453e94a9fc168874c109b19c6cdc764c3 Jul 14 09:28:03.296720 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:03.296692 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-m7mrj" Jul 14 09:28:03.355713 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:03.355669 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc","Type":"ContainerStarted","Data":"f00aa52213f173b9ac6d7c728efa74788984caef11a6917f4922e103c9916caf"} Jul 14 09:28:03.357464 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:03.357433 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-n8c7q" event={"ID":"7225439c-6210-4bc5-b124-0d92487dcd0c","Type":"ContainerStarted","Data":"b7b9fae884669865926559c3eebed85381c817e217a0311114aff9623ccb397e"} Jul 14 09:28:03.357606 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:03.357556 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-n8c7q" Jul 14 09:28:03.358897 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:03.358869 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9c5fd5b4b-gbbk7" event={"ID":"85e26c8a-faa2-43bb-90e9-1d8175b8f8a3","Type":"ContainerStarted","Data":"86d14e69e233873ce40b900d7ccc432453e94a9fc168874c109b19c6cdc764c3"} Jul 14 09:28:03.360191 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:03.360167 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-sw226" event={"ID":"493cb380-3b2f-4f83-b251-c40f63b0b655","Type":"ContainerStarted","Data":"8c3399c35b95bca1bba0404bd648fc08c9e4fccd2f1cebdad7e5f5cc1653a077"} Jul 14 09:28:03.362611 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:03.362543 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5x5kh" event={"ID":"6eab5a46-4afa-4e95-afa5-2a235e5e463c","Type":"ContainerStarted","Data":"f9a33ad24069bdaf3904474546bb1668aeb53b393533296626bf543d0793c55e"} Jul 14 09:28:03.362611 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:03.362598 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5x5kh" event={"ID":"6eab5a46-4afa-4e95-afa5-2a235e5e463c","Type":"ContainerStarted","Data":"795edc053d0b600006e0162dd9d7b2232e46603b3dc74497364cbb5bd552890e"} Jul 14 09:28:03.362772 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:03.362629 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5x5kh" event={"ID":"6eab5a46-4afa-4e95-afa5-2a235e5e463c","Type":"ContainerStarted","Data":"9e6ba6bbb8e4096dda7c414cbf8f569df43461225e700540f3ac938605379726"} Jul 14 09:28:03.364197 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:03.364175 2565 generic.go:356] "Generic (PLEG): container finished" podID="be19da8e-9e05-4c57-87b8-3da6b97e9514" containerID="952a7fb26ad4b5341a108cbf1fac250ea40336cf88d66d0f85ddb57976a38543" exitCode=0 Jul 14 09:28:03.364282 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:03.364247 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mh6sh" event={"ID":"be19da8e-9e05-4c57-87b8-3da6b97e9514","Type":"ContainerDied","Data":"952a7fb26ad4b5341a108cbf1fac250ea40336cf88d66d0f85ddb57976a38543"} Jul 14 09:28:03.364355 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:03.364298 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mh6sh" event={"ID":"be19da8e-9e05-4c57-87b8-3da6b97e9514","Type":"ContainerStarted","Data":"8edd9b90edfa11ad90afa4da48b761469624bc645739c5e9d51cf02444c0293b"} Jul 14 09:28:03.366082 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:03.366037 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2tcgq" event={"ID":"a5c62b6a-f781-4066-9773-a5f4e8921276","Type":"ContainerStarted","Data":"af5aad90f7bab1e3b164f06f6d1583f829ff8b9df029be5f1da8c3f10362e26d"} Jul 14 09:28:03.377291 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:03.377248 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-n8c7q" podStartSLOduration=67.106854651 podStartE2EDuration="1m12.377233376s" totalImagesPullingTime="5.270378725s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 09:26:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:28:03.376138975 +0000 UTC m=+72.885370424" watchObservedRunningTime="2026-07-14 09:28:03.377233376 +0000 UTC m=+72.886464823" Jul 14 09:28:03.416384 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:03.416277 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-2tcgq" podStartSLOduration=66.367907055 podStartE2EDuration="1m11.416256863s" totalImagesPullingTime="5.048349808s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 09:26:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:28:03.414690502 +0000 UTC m=+72.923921951" watchObservedRunningTime="2026-07-14 09:28:03.416256863 +0000 UTC m=+72.925488292" Jul 14 09:28:03.796478 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:03.795419 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6cdc69fb5d-nqq6h"] Jul 14 09:28:03.802321 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:03.802288 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6cdc69fb5d-nqq6h" Jul 14 09:28:03.805050 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:03.805019 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-6mf9qv7a1ii32\"" Jul 14 09:28:03.805050 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:03.805029 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 14 09:28:03.805496 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:03.805477 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-h8h4t\"" Jul 14 09:28:03.806572 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:03.806331 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 14 09:28:03.806572 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:03.806422 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 14 09:28:03.807329 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:03.806725 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 14 09:28:03.809502 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:03.809444 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6cdc69fb5d-nqq6h"] Jul 14 09:28:03.914429 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:03.914324 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0f13c0a0-4bcc-4dde-b3ea-a738d8d129e0-metrics-server-audit-profiles\") pod \"metrics-server-6cdc69fb5d-nqq6h\" (UID: \"0f13c0a0-4bcc-4dde-b3ea-a738d8d129e0\") " pod="openshift-monitoring/metrics-server-6cdc69fb5d-nqq6h" Jul 14 09:28:03.914429 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:03.914378 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f13c0a0-4bcc-4dde-b3ea-a738d8d129e0-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6cdc69fb5d-nqq6h\" (UID: \"0f13c0a0-4bcc-4dde-b3ea-a738d8d129e0\") " pod="openshift-monitoring/metrics-server-6cdc69fb5d-nqq6h" Jul 14 09:28:03.914693 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:03.914466 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0f13c0a0-4bcc-4dde-b3ea-a738d8d129e0-client-ca-bundle\") pod \"metrics-server-6cdc69fb5d-nqq6h\" (UID: \"0f13c0a0-4bcc-4dde-b3ea-a738d8d129e0\") " pod="openshift-monitoring/metrics-server-6cdc69fb5d-nqq6h" Jul 14 09:28:03.914693 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:03.914517 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0f13c0a0-4bcc-4dde-b3ea-a738d8d129e0-secret-metrics-server-tls\") pod \"metrics-server-6cdc69fb5d-nqq6h\" (UID: \"0f13c0a0-4bcc-4dde-b3ea-a738d8d129e0\") " pod="openshift-monitoring/metrics-server-6cdc69fb5d-nqq6h" Jul 14 09:28:03.914693 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:03.914555 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4b6m\" (UniqueName: \"kubernetes.io/projected/0f13c0a0-4bcc-4dde-b3ea-a738d8d129e0-kube-api-access-g4b6m\") pod \"metrics-server-6cdc69fb5d-nqq6h\" (UID: \"0f13c0a0-4bcc-4dde-b3ea-a738d8d129e0\") " pod="openshift-monitoring/metrics-server-6cdc69fb5d-nqq6h" Jul 14 09:28:03.914693 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:03.914605 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0f13c0a0-4bcc-4dde-b3ea-a738d8d129e0-secret-metrics-server-client-certs\") pod \"metrics-server-6cdc69fb5d-nqq6h\" (UID: \"0f13c0a0-4bcc-4dde-b3ea-a738d8d129e0\") " pod="openshift-monitoring/metrics-server-6cdc69fb5d-nqq6h" Jul 14 09:28:03.914693 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:03.914664 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0f13c0a0-4bcc-4dde-b3ea-a738d8d129e0-audit-log\") pod \"metrics-server-6cdc69fb5d-nqq6h\" (UID: \"0f13c0a0-4bcc-4dde-b3ea-a738d8d129e0\") " pod="openshift-monitoring/metrics-server-6cdc69fb5d-nqq6h" Jul 14 09:28:04.016051 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.016007 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0f13c0a0-4bcc-4dde-b3ea-a738d8d129e0-metrics-server-audit-profiles\") pod \"metrics-server-6cdc69fb5d-nqq6h\" (UID: \"0f13c0a0-4bcc-4dde-b3ea-a738d8d129e0\") " pod="openshift-monitoring/metrics-server-6cdc69fb5d-nqq6h" Jul 14 09:28:04.016220 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.016083 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f13c0a0-4bcc-4dde-b3ea-a738d8d129e0-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6cdc69fb5d-nqq6h\" (UID: \"0f13c0a0-4bcc-4dde-b3ea-a738d8d129e0\") " pod="openshift-monitoring/metrics-server-6cdc69fb5d-nqq6h" Jul 14 09:28:04.016220 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.016130 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0f13c0a0-4bcc-4dde-b3ea-a738d8d129e0-client-ca-bundle\") pod \"metrics-server-6cdc69fb5d-nqq6h\" (UID: \"0f13c0a0-4bcc-4dde-b3ea-a738d8d129e0\") " pod="openshift-monitoring/metrics-server-6cdc69fb5d-nqq6h" Jul 14 09:28:04.016220 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.016168 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0f13c0a0-4bcc-4dde-b3ea-a738d8d129e0-secret-metrics-server-tls\") pod \"metrics-server-6cdc69fb5d-nqq6h\" (UID: \"0f13c0a0-4bcc-4dde-b3ea-a738d8d129e0\") " pod="openshift-monitoring/metrics-server-6cdc69fb5d-nqq6h" Jul 14 09:28:04.016220 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.016200 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4b6m\" (UniqueName: \"kubernetes.io/projected/0f13c0a0-4bcc-4dde-b3ea-a738d8d129e0-kube-api-access-g4b6m\") pod \"metrics-server-6cdc69fb5d-nqq6h\" (UID: \"0f13c0a0-4bcc-4dde-b3ea-a738d8d129e0\") " pod="openshift-monitoring/metrics-server-6cdc69fb5d-nqq6h" Jul 14 09:28:04.016450 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.016243 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0f13c0a0-4bcc-4dde-b3ea-a738d8d129e0-secret-metrics-server-client-certs\") pod \"metrics-server-6cdc69fb5d-nqq6h\" (UID: \"0f13c0a0-4bcc-4dde-b3ea-a738d8d129e0\") " pod="openshift-monitoring/metrics-server-6cdc69fb5d-nqq6h" Jul 14 09:28:04.016450 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.016275 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0f13c0a0-4bcc-4dde-b3ea-a738d8d129e0-audit-log\") pod \"metrics-server-6cdc69fb5d-nqq6h\" (UID: \"0f13c0a0-4bcc-4dde-b3ea-a738d8d129e0\") " pod="openshift-monitoring/metrics-server-6cdc69fb5d-nqq6h" Jul 14 09:28:04.017109 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.017032 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f13c0a0-4bcc-4dde-b3ea-a738d8d129e0-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6cdc69fb5d-nqq6h\" (UID: \"0f13c0a0-4bcc-4dde-b3ea-a738d8d129e0\") " pod="openshift-monitoring/metrics-server-6cdc69fb5d-nqq6h" Jul 14 09:28:04.017538 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.017351 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0f13c0a0-4bcc-4dde-b3ea-a738d8d129e0-metrics-server-audit-profiles\") pod \"metrics-server-6cdc69fb5d-nqq6h\" (UID: \"0f13c0a0-4bcc-4dde-b3ea-a738d8d129e0\") " pod="openshift-monitoring/metrics-server-6cdc69fb5d-nqq6h" Jul 14 09:28:04.019368 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.017404 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0f13c0a0-4bcc-4dde-b3ea-a738d8d129e0-audit-log\") pod \"metrics-server-6cdc69fb5d-nqq6h\" (UID: \"0f13c0a0-4bcc-4dde-b3ea-a738d8d129e0\") " pod="openshift-monitoring/metrics-server-6cdc69fb5d-nqq6h" Jul 14 09:28:04.025242 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.025163 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0f13c0a0-4bcc-4dde-b3ea-a738d8d129e0-secret-metrics-server-client-certs\") pod \"metrics-server-6cdc69fb5d-nqq6h\" (UID: \"0f13c0a0-4bcc-4dde-b3ea-a738d8d129e0\") " pod="openshift-monitoring/metrics-server-6cdc69fb5d-nqq6h" Jul 14 09:28:04.026340 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.026307 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0f13c0a0-4bcc-4dde-b3ea-a738d8d129e0-client-ca-bundle\") pod \"metrics-server-6cdc69fb5d-nqq6h\" (UID: \"0f13c0a0-4bcc-4dde-b3ea-a738d8d129e0\") " pod="openshift-monitoring/metrics-server-6cdc69fb5d-nqq6h" Jul 14 09:28:04.026524 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.026501 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4b6m\" (UniqueName: \"kubernetes.io/projected/0f13c0a0-4bcc-4dde-b3ea-a738d8d129e0-kube-api-access-g4b6m\") pod \"metrics-server-6cdc69fb5d-nqq6h\" (UID: \"0f13c0a0-4bcc-4dde-b3ea-a738d8d129e0\") " pod="openshift-monitoring/metrics-server-6cdc69fb5d-nqq6h" Jul 14 09:28:04.026608 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.026564 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0f13c0a0-4bcc-4dde-b3ea-a738d8d129e0-secret-metrics-server-tls\") pod \"metrics-server-6cdc69fb5d-nqq6h\" (UID: \"0f13c0a0-4bcc-4dde-b3ea-a738d8d129e0\") " pod="openshift-monitoring/metrics-server-6cdc69fb5d-nqq6h" Jul 14 09:28:04.116184 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.116092 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6cdc69fb5d-nqq6h" Jul 14 09:28:04.312683 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.312647 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-48hk5"] Jul 14 09:28:04.316391 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.316360 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-948c58bdb-g9g2g"] Jul 14 09:28:04.316531 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.316512 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-48hk5" Jul 14 09:28:04.321014 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.320747 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 14 09:28:04.321014 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.320787 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-cd852\"" Jul 14 09:28:04.337344 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.337314 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-48hk5"] Jul 14 09:28:04.375758 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.375291 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mh6sh" event={"ID":"be19da8e-9e05-4c57-87b8-3da6b97e9514","Type":"ContainerStarted","Data":"4f15d93b2fb55db77de3c79f51bbc7a61f4fe534d41e7d647d1fe87a242bf18a"} Jul 14 09:28:04.375758 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.375353 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mh6sh" event={"ID":"be19da8e-9e05-4c57-87b8-3da6b97e9514","Type":"ContainerStarted","Data":"bf47beecd731f5a63f40a269f38f1dd55959ddfcd1e10dc65d2d8e1a3e3fe8de"} Jul 14 09:28:04.377291 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.377256 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-cdcbcdb48-r2pqc"] Jul 14 09:28:04.380964 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.380943 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cdcbcdb48-r2pqc" Jul 14 09:28:04.392982 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.392928 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cdcbcdb48-r2pqc"] Jul 14 09:28:04.420341 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.420306 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhcvq\" (UniqueName: \"kubernetes.io/projected/3834c168-54af-47d8-9787-8a74fe157c53-kube-api-access-lhcvq\") pod \"monitoring-plugin-5f68df7f58-48hk5\" (UID: \"3834c168-54af-47d8-9787-8a74fe157c53\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-48hk5" Jul 14 09:28:04.420774 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.420356 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3834c168-54af-47d8-9787-8a74fe157c53-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-48hk5\" (UID: \"3834c168-54af-47d8-9787-8a74fe157c53\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-48hk5" Jul 14 09:28:04.426171 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.424366 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mh6sh" podStartSLOduration=4.579623767 podStartE2EDuration="5.424346057s" totalImagesPullingTime="777.914761ms" totalInitContainerRuntime="66.807529ms" isStatefulPod=true podCreationTimestamp="2026-07-14 09:27:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:28:04.422520299 +0000 UTC m=+73.931751776" watchObservedRunningTime="2026-07-14 09:28:04.424346057 +0000 UTC m=+73.933577507" Jul 14 09:28:04.521390 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.521350 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/74493a16-c3ff-4158-bd5a-7a6a4b691817-console-oauth-config\") pod \"console-cdcbcdb48-r2pqc\" (UID: \"74493a16-c3ff-4158-bd5a-7a6a4b691817\") " pod="openshift-console/console-cdcbcdb48-r2pqc" Jul 14 09:28:04.521571 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.521420 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrn7k\" (UniqueName: \"kubernetes.io/projected/74493a16-c3ff-4158-bd5a-7a6a4b691817-kube-api-access-mrn7k\") pod \"console-cdcbcdb48-r2pqc\" (UID: \"74493a16-c3ff-4158-bd5a-7a6a4b691817\") " pod="openshift-console/console-cdcbcdb48-r2pqc" Jul 14 09:28:04.521722 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.521700 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/74493a16-c3ff-4158-bd5a-7a6a4b691817-oauth-serving-cert\") pod \"console-cdcbcdb48-r2pqc\" (UID: \"74493a16-c3ff-4158-bd5a-7a6a4b691817\") " pod="openshift-console/console-cdcbcdb48-r2pqc" Jul 14 09:28:04.521837 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.521740 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/74493a16-c3ff-4158-bd5a-7a6a4b691817-service-ca\") pod \"console-cdcbcdb48-r2pqc\" (UID: \"74493a16-c3ff-4158-bd5a-7a6a4b691817\") " pod="openshift-console/console-cdcbcdb48-r2pqc" Jul 14 09:28:04.521908 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.521864 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhcvq\" (UniqueName: \"kubernetes.io/projected/3834c168-54af-47d8-9787-8a74fe157c53-kube-api-access-lhcvq\") pod \"monitoring-plugin-5f68df7f58-48hk5\" (UID: \"3834c168-54af-47d8-9787-8a74fe157c53\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-48hk5" Jul 14 09:28:04.521908 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.521902 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/74493a16-c3ff-4158-bd5a-7a6a4b691817-console-config\") pod \"console-cdcbcdb48-r2pqc\" (UID: \"74493a16-c3ff-4158-bd5a-7a6a4b691817\") " pod="openshift-console/console-cdcbcdb48-r2pqc" Jul 14 09:28:04.522016 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.521938 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/74493a16-c3ff-4158-bd5a-7a6a4b691817-console-serving-cert\") pod \"console-cdcbcdb48-r2pqc\" (UID: \"74493a16-c3ff-4158-bd5a-7a6a4b691817\") " pod="openshift-console/console-cdcbcdb48-r2pqc" Jul 14 09:28:04.522016 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.521960 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74493a16-c3ff-4158-bd5a-7a6a4b691817-trusted-ca-bundle\") pod \"console-cdcbcdb48-r2pqc\" (UID: \"74493a16-c3ff-4158-bd5a-7a6a4b691817\") " pod="openshift-console/console-cdcbcdb48-r2pqc" Jul 14 09:28:04.522016 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.521985 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3834c168-54af-47d8-9787-8a74fe157c53-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-48hk5\" (UID: \"3834c168-54af-47d8-9787-8a74fe157c53\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-48hk5" Jul 14 09:28:04.524614 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.524593 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3834c168-54af-47d8-9787-8a74fe157c53-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-48hk5\" (UID: \"3834c168-54af-47d8-9787-8a74fe157c53\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-48hk5" Jul 14 09:28:04.531109 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.531090 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhcvq\" (UniqueName: \"kubernetes.io/projected/3834c168-54af-47d8-9787-8a74fe157c53-kube-api-access-lhcvq\") pod \"monitoring-plugin-5f68df7f58-48hk5\" (UID: \"3834c168-54af-47d8-9787-8a74fe157c53\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-48hk5" Jul 14 09:28:04.623271 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.622997 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/74493a16-c3ff-4158-bd5a-7a6a4b691817-console-oauth-config\") pod \"console-cdcbcdb48-r2pqc\" (UID: \"74493a16-c3ff-4158-bd5a-7a6a4b691817\") " pod="openshift-console/console-cdcbcdb48-r2pqc" Jul 14 09:28:04.623271 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.623062 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrn7k\" (UniqueName: \"kubernetes.io/projected/74493a16-c3ff-4158-bd5a-7a6a4b691817-kube-api-access-mrn7k\") pod \"console-cdcbcdb48-r2pqc\" (UID: \"74493a16-c3ff-4158-bd5a-7a6a4b691817\") " pod="openshift-console/console-cdcbcdb48-r2pqc" Jul 14 09:28:04.623271 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.623143 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/74493a16-c3ff-4158-bd5a-7a6a4b691817-oauth-serving-cert\") pod \"console-cdcbcdb48-r2pqc\" (UID: \"74493a16-c3ff-4158-bd5a-7a6a4b691817\") " pod="openshift-console/console-cdcbcdb48-r2pqc" Jul 14 09:28:04.623271 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.623173 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/74493a16-c3ff-4158-bd5a-7a6a4b691817-service-ca\") pod \"console-cdcbcdb48-r2pqc\" (UID: \"74493a16-c3ff-4158-bd5a-7a6a4b691817\") " pod="openshift-console/console-cdcbcdb48-r2pqc" Jul 14 09:28:04.623271 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.623218 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/74493a16-c3ff-4158-bd5a-7a6a4b691817-console-config\") pod \"console-cdcbcdb48-r2pqc\" (UID: \"74493a16-c3ff-4158-bd5a-7a6a4b691817\") " pod="openshift-console/console-cdcbcdb48-r2pqc" Jul 14 09:28:04.623701 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.623365 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/74493a16-c3ff-4158-bd5a-7a6a4b691817-console-serving-cert\") pod \"console-cdcbcdb48-r2pqc\" (UID: \"74493a16-c3ff-4158-bd5a-7a6a4b691817\") " pod="openshift-console/console-cdcbcdb48-r2pqc" Jul 14 09:28:04.623701 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.623404 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74493a16-c3ff-4158-bd5a-7a6a4b691817-trusted-ca-bundle\") pod \"console-cdcbcdb48-r2pqc\" (UID: \"74493a16-c3ff-4158-bd5a-7a6a4b691817\") " pod="openshift-console/console-cdcbcdb48-r2pqc" Jul 14 09:28:04.624008 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.623960 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/74493a16-c3ff-4158-bd5a-7a6a4b691817-console-config\") pod \"console-cdcbcdb48-r2pqc\" (UID: \"74493a16-c3ff-4158-bd5a-7a6a4b691817\") " pod="openshift-console/console-cdcbcdb48-r2pqc" Jul 14 09:28:04.624465 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.624439 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/74493a16-c3ff-4158-bd5a-7a6a4b691817-service-ca\") pod \"console-cdcbcdb48-r2pqc\" (UID: \"74493a16-c3ff-4158-bd5a-7a6a4b691817\") " pod="openshift-console/console-cdcbcdb48-r2pqc" Jul 14 09:28:04.624728 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.624647 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/74493a16-c3ff-4158-bd5a-7a6a4b691817-oauth-serving-cert\") pod \"console-cdcbcdb48-r2pqc\" (UID: \"74493a16-c3ff-4158-bd5a-7a6a4b691817\") " pod="openshift-console/console-cdcbcdb48-r2pqc" Jul 14 09:28:04.624853 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.624761 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74493a16-c3ff-4158-bd5a-7a6a4b691817-trusted-ca-bundle\") pod \"console-cdcbcdb48-r2pqc\" (UID: \"74493a16-c3ff-4158-bd5a-7a6a4b691817\") " pod="openshift-console/console-cdcbcdb48-r2pqc" Jul 14 09:28:04.626270 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.626216 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/74493a16-c3ff-4158-bd5a-7a6a4b691817-console-oauth-config\") pod \"console-cdcbcdb48-r2pqc\" (UID: \"74493a16-c3ff-4158-bd5a-7a6a4b691817\") " pod="openshift-console/console-cdcbcdb48-r2pqc" Jul 14 09:28:04.626376 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.626354 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/74493a16-c3ff-4158-bd5a-7a6a4b691817-console-serving-cert\") pod \"console-cdcbcdb48-r2pqc\" (UID: \"74493a16-c3ff-4158-bd5a-7a6a4b691817\") " pod="openshift-console/console-cdcbcdb48-r2pqc" Jul 14 09:28:04.630004 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.629983 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-48hk5" Jul 14 09:28:04.632230 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.632208 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrn7k\" (UniqueName: \"kubernetes.io/projected/74493a16-c3ff-4158-bd5a-7a6a4b691817-kube-api-access-mrn7k\") pod \"console-cdcbcdb48-r2pqc\" (UID: \"74493a16-c3ff-4158-bd5a-7a6a4b691817\") " pod="openshift-console/console-cdcbcdb48-r2pqc" Jul 14 09:28:04.652471 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.652448 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-948c58bdb-g9g2g" Jul 14 09:28:04.691954 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.691921 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cdcbcdb48-r2pqc" Jul 14 09:28:04.734599 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.734570 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-69cf5d869b-qwtcz"] Jul 14 09:28:04.738273 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.738252 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-69cf5d869b-qwtcz" Jul 14 09:28:04.740958 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.740920 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 14 09:28:04.740958 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.740947 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 14 09:28:04.741179 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.741017 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 14 09:28:04.741179 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.741082 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 14 09:28:04.741179 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.741126 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-ftz6m\"" Jul 14 09:28:04.741411 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.741395 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 14 09:28:04.746662 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.746644 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 14 09:28:04.754050 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.754031 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-69cf5d869b-qwtcz"] Jul 14 09:28:04.825555 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.825512 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f3c3a84-58b7-4794-b743-4ea53077b8ce-serving-certs-ca-bundle\") pod \"telemeter-client-69cf5d869b-qwtcz\" (UID: \"7f3c3a84-58b7-4794-b743-4ea53077b8ce\") " pod="openshift-monitoring/telemeter-client-69cf5d869b-qwtcz" Jul 14 09:28:04.825555 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.825554 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f3c3a84-58b7-4794-b743-4ea53077b8ce-telemeter-trusted-ca-bundle\") pod \"telemeter-client-69cf5d869b-qwtcz\" (UID: \"7f3c3a84-58b7-4794-b743-4ea53077b8ce\") " pod="openshift-monitoring/telemeter-client-69cf5d869b-qwtcz" Jul 14 09:28:04.825795 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.825631 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7f3c3a84-58b7-4794-b743-4ea53077b8ce-secret-telemeter-client\") pod \"telemeter-client-69cf5d869b-qwtcz\" (UID: \"7f3c3a84-58b7-4794-b743-4ea53077b8ce\") " pod="openshift-monitoring/telemeter-client-69cf5d869b-qwtcz" Jul 14 09:28:04.825795 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.825670 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7f3c3a84-58b7-4794-b743-4ea53077b8ce-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-69cf5d869b-qwtcz\" (UID: \"7f3c3a84-58b7-4794-b743-4ea53077b8ce\") " pod="openshift-monitoring/telemeter-client-69cf5d869b-qwtcz" Jul 14 09:28:04.825795 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.825698 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7f3c3a84-58b7-4794-b743-4ea53077b8ce-federate-client-tls\") pod \"telemeter-client-69cf5d869b-qwtcz\" (UID: \"7f3c3a84-58b7-4794-b743-4ea53077b8ce\") " pod="openshift-monitoring/telemeter-client-69cf5d869b-qwtcz" Jul 14 09:28:04.825795 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.825729 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw6dn\" (UniqueName: \"kubernetes.io/projected/7f3c3a84-58b7-4794-b743-4ea53077b8ce-kube-api-access-sw6dn\") pod \"telemeter-client-69cf5d869b-qwtcz\" (UID: \"7f3c3a84-58b7-4794-b743-4ea53077b8ce\") " pod="openshift-monitoring/telemeter-client-69cf5d869b-qwtcz" Jul 14 09:28:04.825795 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.825761 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7f3c3a84-58b7-4794-b743-4ea53077b8ce-metrics-client-ca\") pod \"telemeter-client-69cf5d869b-qwtcz\" (UID: \"7f3c3a84-58b7-4794-b743-4ea53077b8ce\") " pod="openshift-monitoring/telemeter-client-69cf5d869b-qwtcz" Jul 14 09:28:04.825795 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.825786 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7f3c3a84-58b7-4794-b743-4ea53077b8ce-telemeter-client-tls\") pod \"telemeter-client-69cf5d869b-qwtcz\" (UID: \"7f3c3a84-58b7-4794-b743-4ea53077b8ce\") " pod="openshift-monitoring/telemeter-client-69cf5d869b-qwtcz" Jul 14 09:28:04.926451 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.926413 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7f3c3a84-58b7-4794-b743-4ea53077b8ce-metrics-client-ca\") pod \"telemeter-client-69cf5d869b-qwtcz\" (UID: \"7f3c3a84-58b7-4794-b743-4ea53077b8ce\") " pod="openshift-monitoring/telemeter-client-69cf5d869b-qwtcz" Jul 14 09:28:04.926644 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.926463 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7f3c3a84-58b7-4794-b743-4ea53077b8ce-telemeter-client-tls\") pod \"telemeter-client-69cf5d869b-qwtcz\" (UID: \"7f3c3a84-58b7-4794-b743-4ea53077b8ce\") " pod="openshift-monitoring/telemeter-client-69cf5d869b-qwtcz" Jul 14 09:28:04.926644 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.926529 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f3c3a84-58b7-4794-b743-4ea53077b8ce-serving-certs-ca-bundle\") pod \"telemeter-client-69cf5d869b-qwtcz\" (UID: \"7f3c3a84-58b7-4794-b743-4ea53077b8ce\") " pod="openshift-monitoring/telemeter-client-69cf5d869b-qwtcz" Jul 14 09:28:04.926644 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.926561 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f3c3a84-58b7-4794-b743-4ea53077b8ce-telemeter-trusted-ca-bundle\") pod \"telemeter-client-69cf5d869b-qwtcz\" (UID: \"7f3c3a84-58b7-4794-b743-4ea53077b8ce\") " pod="openshift-monitoring/telemeter-client-69cf5d869b-qwtcz" Jul 14 09:28:04.926644 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.926627 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7f3c3a84-58b7-4794-b743-4ea53077b8ce-secret-telemeter-client\") pod \"telemeter-client-69cf5d869b-qwtcz\" (UID: \"7f3c3a84-58b7-4794-b743-4ea53077b8ce\") " pod="openshift-monitoring/telemeter-client-69cf5d869b-qwtcz" Jul 14 09:28:04.926883 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.926662 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7f3c3a84-58b7-4794-b743-4ea53077b8ce-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-69cf5d869b-qwtcz\" (UID: \"7f3c3a84-58b7-4794-b743-4ea53077b8ce\") " pod="openshift-monitoring/telemeter-client-69cf5d869b-qwtcz" Jul 14 09:28:04.926883 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.926698 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7f3c3a84-58b7-4794-b743-4ea53077b8ce-federate-client-tls\") pod \"telemeter-client-69cf5d869b-qwtcz\" (UID: \"7f3c3a84-58b7-4794-b743-4ea53077b8ce\") " pod="openshift-monitoring/telemeter-client-69cf5d869b-qwtcz" Jul 14 09:28:04.926883 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.926726 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw6dn\" (UniqueName: \"kubernetes.io/projected/7f3c3a84-58b7-4794-b743-4ea53077b8ce-kube-api-access-sw6dn\") pod \"telemeter-client-69cf5d869b-qwtcz\" (UID: \"7f3c3a84-58b7-4794-b743-4ea53077b8ce\") " pod="openshift-monitoring/telemeter-client-69cf5d869b-qwtcz" Jul 14 09:28:04.927354 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.927303 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7f3c3a84-58b7-4794-b743-4ea53077b8ce-metrics-client-ca\") pod \"telemeter-client-69cf5d869b-qwtcz\" (UID: \"7f3c3a84-58b7-4794-b743-4ea53077b8ce\") " pod="openshift-monitoring/telemeter-client-69cf5d869b-qwtcz" Jul 14 09:28:04.927832 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.927780 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f3c3a84-58b7-4794-b743-4ea53077b8ce-telemeter-trusted-ca-bundle\") pod \"telemeter-client-69cf5d869b-qwtcz\" (UID: \"7f3c3a84-58b7-4794-b743-4ea53077b8ce\") " pod="openshift-monitoring/telemeter-client-69cf5d869b-qwtcz" Jul 14 09:28:04.927946 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.927787 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f3c3a84-58b7-4794-b743-4ea53077b8ce-serving-certs-ca-bundle\") pod \"telemeter-client-69cf5d869b-qwtcz\" (UID: \"7f3c3a84-58b7-4794-b743-4ea53077b8ce\") " pod="openshift-monitoring/telemeter-client-69cf5d869b-qwtcz" Jul 14 09:28:04.929515 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.929446 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7f3c3a84-58b7-4794-b743-4ea53077b8ce-secret-telemeter-client\") pod \"telemeter-client-69cf5d869b-qwtcz\" (UID: \"7f3c3a84-58b7-4794-b743-4ea53077b8ce\") " pod="openshift-monitoring/telemeter-client-69cf5d869b-qwtcz" Jul 14 09:28:04.929655 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.929595 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7f3c3a84-58b7-4794-b743-4ea53077b8ce-telemeter-client-tls\") pod \"telemeter-client-69cf5d869b-qwtcz\" (UID: \"7f3c3a84-58b7-4794-b743-4ea53077b8ce\") " pod="openshift-monitoring/telemeter-client-69cf5d869b-qwtcz" Jul 14 09:28:04.929655 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.929641 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7f3c3a84-58b7-4794-b743-4ea53077b8ce-federate-client-tls\") pod \"telemeter-client-69cf5d869b-qwtcz\" (UID: \"7f3c3a84-58b7-4794-b743-4ea53077b8ce\") " pod="openshift-monitoring/telemeter-client-69cf5d869b-qwtcz" Jul 14 09:28:04.929995 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.929974 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7f3c3a84-58b7-4794-b743-4ea53077b8ce-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-69cf5d869b-qwtcz\" (UID: \"7f3c3a84-58b7-4794-b743-4ea53077b8ce\") " pod="openshift-monitoring/telemeter-client-69cf5d869b-qwtcz" Jul 14 09:28:04.939969 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:04.939936 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw6dn\" (UniqueName: \"kubernetes.io/projected/7f3c3a84-58b7-4794-b743-4ea53077b8ce-kube-api-access-sw6dn\") pod \"telemeter-client-69cf5d869b-qwtcz\" (UID: \"7f3c3a84-58b7-4794-b743-4ea53077b8ce\") " pod="openshift-monitoring/telemeter-client-69cf5d869b-qwtcz" Jul 14 09:28:05.050062 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:05.050021 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-69cf5d869b-qwtcz" Jul 14 09:28:05.405908 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:05.405778 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6cdc69fb5d-nqq6h"] Jul 14 09:28:05.411782 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:28:05.411454 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f13c0a0_4bcc_4dde_b3ea_a738d8d129e0.slice/crio-55a775c57c46b188225bf236fb3c1a9b69c90536abc9e5f04777d06aff9d9f48 WatchSource:0}: Error finding container 55a775c57c46b188225bf236fb3c1a9b69c90536abc9e5f04777d06aff9d9f48: Status 404 returned error can't find the container with id 55a775c57c46b188225bf236fb3c1a9b69c90536abc9e5f04777d06aff9d9f48 Jul 14 09:28:05.626623 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:05.626588 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-48hk5"] Jul 14 09:28:05.635879 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:05.634978 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cdcbcdb48-r2pqc"] Jul 14 09:28:05.637099 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:05.636940 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-69cf5d869b-qwtcz"] Jul 14 09:28:05.638657 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:28:05.638600 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74493a16_c3ff_4158_bd5a_7a6a4b691817.slice/crio-fb5474e9a7f613f23c11f8251e7764aa35756cc0b94a56a961fd265b113bee0f WatchSource:0}: Error finding container fb5474e9a7f613f23c11f8251e7764aa35756cc0b94a56a961fd265b113bee0f: Status 404 returned error can't find the container with id fb5474e9a7f613f23c11f8251e7764aa35756cc0b94a56a961fd265b113bee0f Jul 14 09:28:06.405539 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:06.405486 2565 generic.go:356] "Generic (PLEG): container finished" podID="5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc" containerID="a844addd637f1e20ef0ad29592e9423d010c4e40fa342446682d38d17e2b952a" exitCode=0 Jul 14 09:28:06.405731 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:06.405586 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc","Type":"ContainerDied","Data":"a844addd637f1e20ef0ad29592e9423d010c4e40fa342446682d38d17e2b952a"} Jul 14 09:28:06.409069 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:06.408942 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-sw226" event={"ID":"493cb380-3b2f-4f83-b251-c40f63b0b655","Type":"ContainerStarted","Data":"d51e7325315166d18772b233ea3b1f2692ab8114d71d62db691c01b95cb86ba3"} Jul 14 09:28:06.409069 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:06.409006 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-sw226" event={"ID":"493cb380-3b2f-4f83-b251-c40f63b0b655","Type":"ContainerStarted","Data":"d340ffe37851ed167b8d24c43ef334e5abe6c2679282d047505962aab9d0890a"} Jul 14 09:28:06.409069 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:06.409034 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-sw226" event={"ID":"493cb380-3b2f-4f83-b251-c40f63b0b655","Type":"ContainerStarted","Data":"c0c2e6e98146d24d15d73c9d5c241e0a77101fa6a49695c4889391b0e6b697a0"} Jul 14 09:28:06.411005 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:06.410764 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-48hk5" event={"ID":"3834c168-54af-47d8-9787-8a74fe157c53","Type":"ContainerStarted","Data":"64f39f6eab0fc5397947ee9acbb9b375c50917564c752b31e491ee6c53c90b37"} Jul 14 09:28:06.412525 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:06.412487 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-69cf5d869b-qwtcz" event={"ID":"7f3c3a84-58b7-4794-b743-4ea53077b8ce","Type":"ContainerStarted","Data":"c6c6f58f3d3e4b6b688f66b121108b4c8caba1ea47c0636b4d9aaa2b612dd2f0"} Jul 14 09:28:06.415009 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:06.414982 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9c5fd5b4b-gbbk7" event={"ID":"85e26c8a-faa2-43bb-90e9-1d8175b8f8a3","Type":"ContainerStarted","Data":"b11fb65d2ad92f59d91db71c231ecd675548953bfab7625b2eef4ee4d9d6589e"} Jul 14 09:28:06.415096 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:06.415051 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9c5fd5b4b-gbbk7" event={"ID":"85e26c8a-faa2-43bb-90e9-1d8175b8f8a3","Type":"ContainerStarted","Data":"9fc24ebc44d0a7551ace2b8e0d2e0d573a5ee764f05ea93f07ff536598d2a600"} Jul 14 09:28:06.415096 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:06.415088 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9c5fd5b4b-gbbk7" event={"ID":"85e26c8a-faa2-43bb-90e9-1d8175b8f8a3","Type":"ContainerStarted","Data":"304126d15d369fadd3a66e3c81f317782aff0110d7f05331901774ad0accaf12"} Jul 14 09:28:06.419957 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:06.418824 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5x5kh" event={"ID":"6eab5a46-4afa-4e95-afa5-2a235e5e463c","Type":"ContainerStarted","Data":"d7e7d873f50946515e5536b908bb226813e149ea274b90d12ee2501486d98eb0"} Jul 14 09:28:06.421627 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:06.421552 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6cdc69fb5d-nqq6h" event={"ID":"0f13c0a0-4bcc-4dde-b3ea-a738d8d129e0","Type":"ContainerStarted","Data":"55a775c57c46b188225bf236fb3c1a9b69c90536abc9e5f04777d06aff9d9f48"} Jul 14 09:28:06.423514 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:06.423493 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cdcbcdb48-r2pqc" event={"ID":"74493a16-c3ff-4158-bd5a-7a6a4b691817","Type":"ContainerStarted","Data":"d51902194ae879e6855e7cd420b2b2582642501a37d954564937ff2ab4e41c55"} Jul 14 09:28:06.423615 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:06.423532 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cdcbcdb48-r2pqc" event={"ID":"74493a16-c3ff-4158-bd5a-7a6a4b691817","Type":"ContainerStarted","Data":"fb5474e9a7f613f23c11f8251e7764aa35756cc0b94a56a961fd265b113bee0f"} Jul 14 09:28:06.455993 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:06.455937 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-cdcbcdb48-r2pqc" podStartSLOduration=2.45592016 podStartE2EDuration="2.45592016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 09:28:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:28:06.454566314 +0000 UTC m=+75.963797776" watchObservedRunningTime="2026-07-14 09:28:06.45592016 +0000 UTC m=+75.965151608" Jul 14 09:28:06.476469 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:06.476400 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5x5kh" podStartSLOduration=4.954906116 podStartE2EDuration="7.476377254s" totalImagesPullingTime="2.521471138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 09:27:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:28:06.475427386 +0000 UTC m=+75.984658855" watchObservedRunningTime="2026-07-14 09:28:06.476377254 +0000 UTC m=+75.985608702" Jul 14 09:28:06.497275 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:06.497213 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-sw226" podStartSLOduration=4.870948381 podStartE2EDuration="7.49719171s" totalImagesPullingTime="2.626243329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 09:27:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:28:06.496116656 +0000 UTC m=+76.005348125" watchObservedRunningTime="2026-07-14 09:28:06.49719171 +0000 UTC m=+76.006423159" Jul 14 09:28:07.433390 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:07.433350 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9c5fd5b4b-gbbk7" event={"ID":"85e26c8a-faa2-43bb-90e9-1d8175b8f8a3","Type":"ContainerStarted","Data":"01221b73f021486025449455f39cc2e0b78eb3dfdd5fc85c84d87dc19ed25211"} Jul 14 09:28:07.433858 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:07.433433 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9c5fd5b4b-gbbk7" event={"ID":"85e26c8a-faa2-43bb-90e9-1d8175b8f8a3","Type":"ContainerStarted","Data":"970f4c9af4be15c79ff551bcdc61178e95271e754e4ee07b88581ffc7ae68906"} Jul 14 09:28:09.445238 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:09.445200 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc","Type":"ContainerStarted","Data":"7181800cf4c12973bb2c02ed9ebc61100ec0336ce0a65b11b1395917bcbca5c2"} Jul 14 09:28:09.445672 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:09.445286 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc","Type":"ContainerStarted","Data":"2fd5c70ad36e517725a4674c44b5a2736641cbaae78e77a579caf49386ab023c"} Jul 14 09:28:09.445672 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:09.445336 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc","Type":"ContainerStarted","Data":"d7483a04689547a2c5b7120ddb46de09ff7bb40131860a78a7bfe7334071eed4"} Jul 14 09:28:09.445672 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:09.445377 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc","Type":"ContainerStarted","Data":"f768ed47ac67dd9891b5d0dca92d8b66d0b722addfc8d7e23ca71a8723ba3a3a"} Jul 14 09:28:09.445672 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:09.445417 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc","Type":"ContainerStarted","Data":"443956ed57dad8056b21390a26c96d3b919e1700cb522c3b748db17f28cd362a"} Jul 14 09:28:09.445672 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:09.445454 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc","Type":"ContainerStarted","Data":"ec1ee724f56888b6384f414f91281ff01ce0840c7ffe317192e0b2e1baa1ae11"} Jul 14 09:28:09.446905 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:09.446876 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-48hk5" event={"ID":"3834c168-54af-47d8-9787-8a74fe157c53","Type":"ContainerStarted","Data":"f7005d81d0dd678d19d5bc3ad6fd12ee6ef8a75e898804530c124d759011836e"} Jul 14 09:28:09.447050 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:09.447027 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-48hk5" Jul 14 09:28:09.449319 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:09.449293 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-69cf5d869b-qwtcz" event={"ID":"7f3c3a84-58b7-4794-b743-4ea53077b8ce","Type":"ContainerStarted","Data":"6a13827dd302509dae55027066a2f9bc42450826543bd13d7fc10b5bdbe3f2cb"} Jul 14 09:28:09.449420 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:09.449354 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-69cf5d869b-qwtcz" event={"ID":"7f3c3a84-58b7-4794-b743-4ea53077b8ce","Type":"ContainerStarted","Data":"f60facadb4c6bfbb4a2b22dc6afe99059abaf6d501958e7068a5e301c5c7cf35"} Jul 14 09:28:09.449420 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:09.449386 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-69cf5d869b-qwtcz" event={"ID":"7f3c3a84-58b7-4794-b743-4ea53077b8ce","Type":"ContainerStarted","Data":"05ad99c35a1ae31c58d1986ccd00e5315b620ab7f854ea86abfd05c9c20b2e46"} Jul 14 09:28:09.452746 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:09.452726 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-48hk5" Jul 14 09:28:09.453145 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:09.453125 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9c5fd5b4b-gbbk7" event={"ID":"85e26c8a-faa2-43bb-90e9-1d8175b8f8a3","Type":"ContainerStarted","Data":"4173e5e4d022b164bfb91c6092d4da3fe29c58c696e87cd9c8fff8fb43b4ab9d"} Jul 14 09:28:09.453276 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:09.453252 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-9c5fd5b4b-gbbk7" Jul 14 09:28:09.454645 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:09.454627 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6cdc69fb5d-nqq6h" event={"ID":"0f13c0a0-4bcc-4dde-b3ea-a738d8d129e0","Type":"ContainerStarted","Data":"29e18206a8c27540418b5b526f22169e2f19027507de6abb3bb9067911350e99"} Jul 14 09:28:09.473279 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:09.473236 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.914823341 podStartE2EDuration="9.473219306s" totalImagesPullingTime="4.52788838s" totalInitContainerRuntime="30.507585ms" isStatefulPod=false podCreationTimestamp="2026-07-14 09:28:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:28:09.472605707 +0000 UTC m=+78.981837154" watchObservedRunningTime="2026-07-14 09:28:09.473219306 +0000 UTC m=+78.982450754" Jul 14 09:28:09.497983 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:09.497912 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-9c5fd5b4b-gbbk7" podStartSLOduration=4.540408848 podStartE2EDuration="7.497892343s" totalImagesPullingTime="2.957483495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 09:28:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:28:09.49618025 +0000 UTC m=+79.005411698" watchObservedRunningTime="2026-07-14 09:28:09.497892343 +0000 UTC m=+79.007123792" Jul 14 09:28:09.520680 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:09.520630 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-69cf5d869b-qwtcz" podStartSLOduration=2.714899639 podStartE2EDuration="5.520612462s" totalImagesPullingTime="2.805712823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 09:28:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:28:09.519006462 +0000 UTC m=+79.028237940" watchObservedRunningTime="2026-07-14 09:28:09.520612462 +0000 UTC m=+79.029843941" Jul 14 09:28:09.536767 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:09.536715 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6cdc69fb5d-nqq6h" podStartSLOduration=3.503976173 podStartE2EDuration="6.536698225s" totalImagesPullingTime="3.032722052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 09:28:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:28:09.535799019 +0000 UTC m=+79.045030466" watchObservedRunningTime="2026-07-14 09:28:09.536698225 +0000 UTC m=+79.045929672" Jul 14 09:28:09.551849 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:09.551773 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-48hk5" podStartSLOduration=2.7418454260000003 podStartE2EDuration="5.551756326s" totalImagesPullingTime="2.8099109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 09:28:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:28:09.550388399 +0000 UTC m=+79.059619864" watchObservedRunningTime="2026-07-14 09:28:09.551756326 +0000 UTC m=+79.060987773" Jul 14 09:28:10.348029 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:10.347992 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cdcbcdb48-r2pqc"] Jul 14 09:28:10.371035 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:10.371002 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-78cbb6bb89-nw5sx"] Jul 14 09:28:10.374698 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:10.374673 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78cbb6bb89-nw5sx" Jul 14 09:28:10.383737 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:10.383713 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78cbb6bb89-nw5sx"] Jul 14 09:28:10.471628 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:10.471595 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-9c5fd5b4b-gbbk7" Jul 14 09:28:10.485912 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:10.485876 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59sgt\" (UniqueName: \"kubernetes.io/projected/527dcd95-98da-4412-930f-06a3feaf4cc2-kube-api-access-59sgt\") pod \"console-78cbb6bb89-nw5sx\" (UID: \"527dcd95-98da-4412-930f-06a3feaf4cc2\") " pod="openshift-console/console-78cbb6bb89-nw5sx" Jul 14 09:28:10.486047 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:10.485931 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/527dcd95-98da-4412-930f-06a3feaf4cc2-console-config\") pod \"console-78cbb6bb89-nw5sx\" (UID: \"527dcd95-98da-4412-930f-06a3feaf4cc2\") " pod="openshift-console/console-78cbb6bb89-nw5sx" Jul 14 09:28:10.486047 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:10.485960 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/527dcd95-98da-4412-930f-06a3feaf4cc2-trusted-ca-bundle\") pod \"console-78cbb6bb89-nw5sx\" (UID: \"527dcd95-98da-4412-930f-06a3feaf4cc2\") " pod="openshift-console/console-78cbb6bb89-nw5sx" Jul 14 09:28:10.486175 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:10.486040 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/527dcd95-98da-4412-930f-06a3feaf4cc2-console-serving-cert\") pod \"console-78cbb6bb89-nw5sx\" (UID: \"527dcd95-98da-4412-930f-06a3feaf4cc2\") " pod="openshift-console/console-78cbb6bb89-nw5sx" Jul 14 09:28:10.486175 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:10.486069 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/527dcd95-98da-4412-930f-06a3feaf4cc2-service-ca\") pod \"console-78cbb6bb89-nw5sx\" (UID: \"527dcd95-98da-4412-930f-06a3feaf4cc2\") " pod="openshift-console/console-78cbb6bb89-nw5sx" Jul 14 09:28:10.486175 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:10.486093 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/527dcd95-98da-4412-930f-06a3feaf4cc2-oauth-serving-cert\") pod \"console-78cbb6bb89-nw5sx\" (UID: \"527dcd95-98da-4412-930f-06a3feaf4cc2\") " pod="openshift-console/console-78cbb6bb89-nw5sx" Jul 14 09:28:10.486175 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:10.486110 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/527dcd95-98da-4412-930f-06a3feaf4cc2-console-oauth-config\") pod \"console-78cbb6bb89-nw5sx\" (UID: \"527dcd95-98da-4412-930f-06a3feaf4cc2\") " pod="openshift-console/console-78cbb6bb89-nw5sx" Jul 14 09:28:10.587045 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:10.587011 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/527dcd95-98da-4412-930f-06a3feaf4cc2-service-ca\") pod \"console-78cbb6bb89-nw5sx\" (UID: \"527dcd95-98da-4412-930f-06a3feaf4cc2\") " pod="openshift-console/console-78cbb6bb89-nw5sx" Jul 14 09:28:10.587228 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:10.587072 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/527dcd95-98da-4412-930f-06a3feaf4cc2-oauth-serving-cert\") pod \"console-78cbb6bb89-nw5sx\" (UID: \"527dcd95-98da-4412-930f-06a3feaf4cc2\") " pod="openshift-console/console-78cbb6bb89-nw5sx" Jul 14 09:28:10.587228 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:10.587102 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/527dcd95-98da-4412-930f-06a3feaf4cc2-console-oauth-config\") pod \"console-78cbb6bb89-nw5sx\" (UID: \"527dcd95-98da-4412-930f-06a3feaf4cc2\") " pod="openshift-console/console-78cbb6bb89-nw5sx" Jul 14 09:28:10.588975 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:10.587866 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/527dcd95-98da-4412-930f-06a3feaf4cc2-oauth-serving-cert\") pod \"console-78cbb6bb89-nw5sx\" (UID: \"527dcd95-98da-4412-930f-06a3feaf4cc2\") " pod="openshift-console/console-78cbb6bb89-nw5sx" Jul 14 09:28:10.588975 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:10.587966 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59sgt\" (UniqueName: \"kubernetes.io/projected/527dcd95-98da-4412-930f-06a3feaf4cc2-kube-api-access-59sgt\") pod \"console-78cbb6bb89-nw5sx\" (UID: \"527dcd95-98da-4412-930f-06a3feaf4cc2\") " pod="openshift-console/console-78cbb6bb89-nw5sx" Jul 14 09:28:10.588975 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:10.588283 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/527dcd95-98da-4412-930f-06a3feaf4cc2-console-config\") pod \"console-78cbb6bb89-nw5sx\" (UID: \"527dcd95-98da-4412-930f-06a3feaf4cc2\") " pod="openshift-console/console-78cbb6bb89-nw5sx" Jul 14 09:28:10.588975 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:10.588401 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/527dcd95-98da-4412-930f-06a3feaf4cc2-trusted-ca-bundle\") pod \"console-78cbb6bb89-nw5sx\" (UID: \"527dcd95-98da-4412-930f-06a3feaf4cc2\") " pod="openshift-console/console-78cbb6bb89-nw5sx" Jul 14 09:28:10.588975 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:10.588466 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/527dcd95-98da-4412-930f-06a3feaf4cc2-service-ca\") pod \"console-78cbb6bb89-nw5sx\" (UID: \"527dcd95-98da-4412-930f-06a3feaf4cc2\") " pod="openshift-console/console-78cbb6bb89-nw5sx" Jul 14 09:28:10.588975 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:10.588640 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/527dcd95-98da-4412-930f-06a3feaf4cc2-console-serving-cert\") pod \"console-78cbb6bb89-nw5sx\" (UID: \"527dcd95-98da-4412-930f-06a3feaf4cc2\") " pod="openshift-console/console-78cbb6bb89-nw5sx" Jul 14 09:28:10.589417 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:10.589009 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/527dcd95-98da-4412-930f-06a3feaf4cc2-console-config\") pod \"console-78cbb6bb89-nw5sx\" (UID: \"527dcd95-98da-4412-930f-06a3feaf4cc2\") " pod="openshift-console/console-78cbb6bb89-nw5sx" Jul 14 09:28:10.589855 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:10.589834 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/527dcd95-98da-4412-930f-06a3feaf4cc2-trusted-ca-bundle\") pod \"console-78cbb6bb89-nw5sx\" (UID: \"527dcd95-98da-4412-930f-06a3feaf4cc2\") " pod="openshift-console/console-78cbb6bb89-nw5sx" Jul 14 09:28:10.590143 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:10.590121 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/527dcd95-98da-4412-930f-06a3feaf4cc2-console-oauth-config\") pod \"console-78cbb6bb89-nw5sx\" (UID: \"527dcd95-98da-4412-930f-06a3feaf4cc2\") " pod="openshift-console/console-78cbb6bb89-nw5sx" Jul 14 09:28:10.591378 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:10.591351 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/527dcd95-98da-4412-930f-06a3feaf4cc2-console-serving-cert\") pod \"console-78cbb6bb89-nw5sx\" (UID: \"527dcd95-98da-4412-930f-06a3feaf4cc2\") " pod="openshift-console/console-78cbb6bb89-nw5sx" Jul 14 09:28:10.602780 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:10.602709 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59sgt\" (UniqueName: \"kubernetes.io/projected/527dcd95-98da-4412-930f-06a3feaf4cc2-kube-api-access-59sgt\") pod \"console-78cbb6bb89-nw5sx\" (UID: \"527dcd95-98da-4412-930f-06a3feaf4cc2\") " pod="openshift-console/console-78cbb6bb89-nw5sx" Jul 14 09:28:10.684699 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:10.684660 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78cbb6bb89-nw5sx" Jul 14 09:28:10.838889 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:10.838848 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78cbb6bb89-nw5sx"] Jul 14 09:28:10.842199 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:28:10.842171 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod527dcd95_98da_4412_930f_06a3feaf4cc2.slice/crio-fba30d5ad83dac9a7b8f5e229e732838dcb4a89c8abfd31903332f9ce6f94079 WatchSource:0}: Error finding container fba30d5ad83dac9a7b8f5e229e732838dcb4a89c8abfd31903332f9ce6f94079: Status 404 returned error can't find the container with id fba30d5ad83dac9a7b8f5e229e732838dcb4a89c8abfd31903332f9ce6f94079 Jul 14 09:28:11.471656 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:11.471618 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78cbb6bb89-nw5sx" event={"ID":"527dcd95-98da-4412-930f-06a3feaf4cc2","Type":"ContainerStarted","Data":"992a5744ff17e9dc208c3eea12327f1d18b8b3cc166a2105876a4d98a4fe1a2d"} Jul 14 09:28:11.472092 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:11.471676 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78cbb6bb89-nw5sx" event={"ID":"527dcd95-98da-4412-930f-06a3feaf4cc2","Type":"ContainerStarted","Data":"fba30d5ad83dac9a7b8f5e229e732838dcb4a89c8abfd31903332f9ce6f94079"} Jul 14 09:28:11.491035 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:11.490986 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-78cbb6bb89-nw5sx" podStartSLOduration=1.490972127 podStartE2EDuration="1.490972127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 09:28:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:28:11.4889576 +0000 UTC m=+80.998189049" watchObservedRunningTime="2026-07-14 09:28:11.490972127 +0000 UTC m=+81.000203619" Jul 14 09:28:14.692589 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:14.692460 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-cdcbcdb48-r2pqc" Jul 14 09:28:20.685049 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:20.685014 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-78cbb6bb89-nw5sx" Jul 14 09:28:20.685453 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:20.685084 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-78cbb6bb89-nw5sx" Jul 14 09:28:20.689879 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:20.689857 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-78cbb6bb89-nw5sx" Jul 14 09:28:21.515919 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:21.515891 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-78cbb6bb89-nw5sx" Jul 14 09:28:21.563681 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:21.563643 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7555476c54-7f4b2"] Jul 14 09:28:24.116409 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:24.116370 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6cdc69fb5d-nqq6h" Jul 14 09:28:24.116773 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:24.116422 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6cdc69fb5d-nqq6h" Jul 14 09:28:29.344997 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:29.344932 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-948c58bdb-g9g2g" podUID="394653f7-9641-443f-b1e2-8ea222d9ae02" containerName="console" containerID="cri-o://7dfc67eca4620e1462f07c58aa943d8fbaa8b393c85927db9ce6edbb0f36d546" gracePeriod=15 Jul 14 09:28:29.546166 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:29.546131 2565 generic.go:356] "Generic (PLEG): container finished" podID="394653f7-9641-443f-b1e2-8ea222d9ae02" containerID="7dfc67eca4620e1462f07c58aa943d8fbaa8b393c85927db9ce6edbb0f36d546" exitCode=0 Jul 14 09:28:29.546295 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:29.546202 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-948c58bdb-g9g2g" event={"ID":"394653f7-9641-443f-b1e2-8ea222d9ae02","Type":"ContainerDied","Data":"7dfc67eca4620e1462f07c58aa943d8fbaa8b393c85927db9ce6edbb0f36d546"} Jul 14 09:28:30.118008 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:30.117978 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-948c58bdb-g9g2g" Jul 14 09:28:30.156748 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:30.156719 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/394653f7-9641-443f-b1e2-8ea222d9ae02-trusted-ca-bundle\") pod \"394653f7-9641-443f-b1e2-8ea222d9ae02\" (UID: \"394653f7-9641-443f-b1e2-8ea222d9ae02\") " Jul 14 09:28:30.156911 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:30.156794 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/394653f7-9641-443f-b1e2-8ea222d9ae02-oauth-serving-cert\") pod \"394653f7-9641-443f-b1e2-8ea222d9ae02\" (UID: \"394653f7-9641-443f-b1e2-8ea222d9ae02\") " Jul 14 09:28:30.156911 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:30.156862 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5bj78\" (UniqueName: \"kubernetes.io/projected/394653f7-9641-443f-b1e2-8ea222d9ae02-kube-api-access-5bj78\") pod \"394653f7-9641-443f-b1e2-8ea222d9ae02\" (UID: \"394653f7-9641-443f-b1e2-8ea222d9ae02\") " Jul 14 09:28:30.156911 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:30.156884 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/394653f7-9641-443f-b1e2-8ea222d9ae02-console-oauth-config\") pod \"394653f7-9641-443f-b1e2-8ea222d9ae02\" (UID: \"394653f7-9641-443f-b1e2-8ea222d9ae02\") " Jul 14 09:28:30.156911 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:30.156908 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/394653f7-9641-443f-b1e2-8ea222d9ae02-service-ca\") pod \"394653f7-9641-443f-b1e2-8ea222d9ae02\" (UID: \"394653f7-9641-443f-b1e2-8ea222d9ae02\") " Jul 14 09:28:30.157111 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:30.156926 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/394653f7-9641-443f-b1e2-8ea222d9ae02-console-serving-cert\") pod \"394653f7-9641-443f-b1e2-8ea222d9ae02\" (UID: \"394653f7-9641-443f-b1e2-8ea222d9ae02\") " Jul 14 09:28:30.157111 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:30.156959 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/394653f7-9641-443f-b1e2-8ea222d9ae02-console-config\") pod \"394653f7-9641-443f-b1e2-8ea222d9ae02\" (UID: \"394653f7-9641-443f-b1e2-8ea222d9ae02\") " Jul 14 09:28:30.157269 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:30.157244 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/394653f7-9641-443f-b1e2-8ea222d9ae02-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "394653f7-9641-443f-b1e2-8ea222d9ae02" (UID: "394653f7-9641-443f-b1e2-8ea222d9ae02"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 09:28:30.157361 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:30.157269 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/394653f7-9641-443f-b1e2-8ea222d9ae02-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "394653f7-9641-443f-b1e2-8ea222d9ae02" (UID: "394653f7-9641-443f-b1e2-8ea222d9ae02"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 09:28:30.157448 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:30.157426 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/394653f7-9641-443f-b1e2-8ea222d9ae02-service-ca" (OuterVolumeSpecName: "service-ca") pod "394653f7-9641-443f-b1e2-8ea222d9ae02" (UID: "394653f7-9641-443f-b1e2-8ea222d9ae02"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 09:28:30.157488 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:30.157465 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/394653f7-9641-443f-b1e2-8ea222d9ae02-console-config" (OuterVolumeSpecName: "console-config") pod "394653f7-9641-443f-b1e2-8ea222d9ae02" (UID: "394653f7-9641-443f-b1e2-8ea222d9ae02"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 09:28:30.159248 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:30.159215 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/394653f7-9641-443f-b1e2-8ea222d9ae02-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "394653f7-9641-443f-b1e2-8ea222d9ae02" (UID: "394653f7-9641-443f-b1e2-8ea222d9ae02"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 09:28:30.159355 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:30.159248 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/394653f7-9641-443f-b1e2-8ea222d9ae02-kube-api-access-5bj78" (OuterVolumeSpecName: "kube-api-access-5bj78") pod "394653f7-9641-443f-b1e2-8ea222d9ae02" (UID: "394653f7-9641-443f-b1e2-8ea222d9ae02"). InnerVolumeSpecName "kube-api-access-5bj78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 09:28:30.159355 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:30.159253 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/394653f7-9641-443f-b1e2-8ea222d9ae02-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "394653f7-9641-443f-b1e2-8ea222d9ae02" (UID: "394653f7-9641-443f-b1e2-8ea222d9ae02"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 09:28:30.258459 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:30.258381 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/394653f7-9641-443f-b1e2-8ea222d9ae02-trusted-ca-bundle\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:28:30.258459 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:30.258410 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/394653f7-9641-443f-b1e2-8ea222d9ae02-oauth-serving-cert\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:28:30.258459 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:30.258421 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bj78\" (UniqueName: \"kubernetes.io/projected/394653f7-9641-443f-b1e2-8ea222d9ae02-kube-api-access-5bj78\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:28:30.258459 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:30.258429 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/394653f7-9641-443f-b1e2-8ea222d9ae02-console-oauth-config\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:28:30.258459 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:30.258463 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/394653f7-9641-443f-b1e2-8ea222d9ae02-service-ca\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:28:30.258759 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:30.258473 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/394653f7-9641-443f-b1e2-8ea222d9ae02-console-serving-cert\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:28:30.258759 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:30.258482 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/394653f7-9641-443f-b1e2-8ea222d9ae02-console-config\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:28:30.552031 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:30.551998 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-948c58bdb-g9g2g" event={"ID":"394653f7-9641-443f-b1e2-8ea222d9ae02","Type":"ContainerDied","Data":"df565999602e1101590270cbe027035ca9cb1ac7b31463b4730e94a77d9a13e0"} Jul 14 09:28:30.552031 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:30.552022 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-948c58bdb-g9g2g" Jul 14 09:28:30.552504 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:30.552071 2565 scope.go:117] "RemoveContainer" containerID="7dfc67eca4620e1462f07c58aa943d8fbaa8b393c85927db9ce6edbb0f36d546" Jul 14 09:28:30.575044 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:30.575017 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-948c58bdb-g9g2g"] Jul 14 09:28:30.585753 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:30.585723 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-948c58bdb-g9g2g"] Jul 14 09:28:30.967284 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:30.967239 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="394653f7-9641-443f-b1e2-8ea222d9ae02" path="/var/lib/kubelet/pods/394653f7-9641-443f-b1e2-8ea222d9ae02/volumes" Jul 14 09:28:34.378873 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:34.378839 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-n8c7q" Jul 14 09:28:35.029989 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:35.029960 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zlzfk_8f2133f7-7ee2-48c5-bc8f-1f17049b6c46/serve-healthcheck-canary/0.log" Jul 14 09:28:35.368711 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:35.368599 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-cdcbcdb48-r2pqc" podUID="74493a16-c3ff-4158-bd5a-7a6a4b691817" containerName="console" containerID="cri-o://d51902194ae879e6855e7cd420b2b2582642501a37d954564937ff2ab4e41c55" gracePeriod=15 Jul 14 09:28:35.575184 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:35.575145 2565 generic.go:356] "Generic (PLEG): container finished" podID="74493a16-c3ff-4158-bd5a-7a6a4b691817" containerID="d51902194ae879e6855e7cd420b2b2582642501a37d954564937ff2ab4e41c55" exitCode=0 Jul 14 09:28:35.575538 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:35.575210 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cdcbcdb48-r2pqc" event={"ID":"74493a16-c3ff-4158-bd5a-7a6a4b691817","Type":"ContainerDied","Data":"d51902194ae879e6855e7cd420b2b2582642501a37d954564937ff2ab4e41c55"} Jul 14 09:28:35.652412 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:35.652383 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cdcbcdb48-r2pqc" Jul 14 09:28:35.704134 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:35.704103 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/74493a16-c3ff-4158-bd5a-7a6a4b691817-oauth-serving-cert\") pod \"74493a16-c3ff-4158-bd5a-7a6a4b691817\" (UID: \"74493a16-c3ff-4158-bd5a-7a6a4b691817\") " Jul 14 09:28:35.704292 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:35.704145 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/74493a16-c3ff-4158-bd5a-7a6a4b691817-service-ca\") pod \"74493a16-c3ff-4158-bd5a-7a6a4b691817\" (UID: \"74493a16-c3ff-4158-bd5a-7a6a4b691817\") " Jul 14 09:28:35.704292 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:35.704197 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74493a16-c3ff-4158-bd5a-7a6a4b691817-trusted-ca-bundle\") pod \"74493a16-c3ff-4158-bd5a-7a6a4b691817\" (UID: \"74493a16-c3ff-4158-bd5a-7a6a4b691817\") " Jul 14 09:28:35.704292 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:35.704232 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/74493a16-c3ff-4158-bd5a-7a6a4b691817-console-config\") pod \"74493a16-c3ff-4158-bd5a-7a6a4b691817\" (UID: \"74493a16-c3ff-4158-bd5a-7a6a4b691817\") " Jul 14 09:28:35.704292 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:35.704262 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/74493a16-c3ff-4158-bd5a-7a6a4b691817-console-oauth-config\") pod \"74493a16-c3ff-4158-bd5a-7a6a4b691817\" (UID: \"74493a16-c3ff-4158-bd5a-7a6a4b691817\") " Jul 14 09:28:35.704292 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:35.704277 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/74493a16-c3ff-4158-bd5a-7a6a4b691817-console-serving-cert\") pod \"74493a16-c3ff-4158-bd5a-7a6a4b691817\" (UID: \"74493a16-c3ff-4158-bd5a-7a6a4b691817\") " Jul 14 09:28:35.704521 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:35.704315 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mrn7k\" (UniqueName: \"kubernetes.io/projected/74493a16-c3ff-4158-bd5a-7a6a4b691817-kube-api-access-mrn7k\") pod \"74493a16-c3ff-4158-bd5a-7a6a4b691817\" (UID: \"74493a16-c3ff-4158-bd5a-7a6a4b691817\") " Jul 14 09:28:35.704579 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:35.704549 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74493a16-c3ff-4158-bd5a-7a6a4b691817-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "74493a16-c3ff-4158-bd5a-7a6a4b691817" (UID: "74493a16-c3ff-4158-bd5a-7a6a4b691817"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 09:28:35.704692 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:35.704665 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74493a16-c3ff-4158-bd5a-7a6a4b691817-service-ca" (OuterVolumeSpecName: "service-ca") pod "74493a16-c3ff-4158-bd5a-7a6a4b691817" (UID: "74493a16-c3ff-4158-bd5a-7a6a4b691817"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 09:28:35.704747 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:35.704683 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74493a16-c3ff-4158-bd5a-7a6a4b691817-console-config" (OuterVolumeSpecName: "console-config") pod "74493a16-c3ff-4158-bd5a-7a6a4b691817" (UID: "74493a16-c3ff-4158-bd5a-7a6a4b691817"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 09:28:35.704747 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:35.704694 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74493a16-c3ff-4158-bd5a-7a6a4b691817-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "74493a16-c3ff-4158-bd5a-7a6a4b691817" (UID: "74493a16-c3ff-4158-bd5a-7a6a4b691817"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 09:28:35.706488 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:35.706456 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74493a16-c3ff-4158-bd5a-7a6a4b691817-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "74493a16-c3ff-4158-bd5a-7a6a4b691817" (UID: "74493a16-c3ff-4158-bd5a-7a6a4b691817"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 09:28:35.706601 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:35.706491 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74493a16-c3ff-4158-bd5a-7a6a4b691817-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "74493a16-c3ff-4158-bd5a-7a6a4b691817" (UID: "74493a16-c3ff-4158-bd5a-7a6a4b691817"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 09:28:35.706601 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:35.706563 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74493a16-c3ff-4158-bd5a-7a6a4b691817-kube-api-access-mrn7k" (OuterVolumeSpecName: "kube-api-access-mrn7k") pod "74493a16-c3ff-4158-bd5a-7a6a4b691817" (UID: "74493a16-c3ff-4158-bd5a-7a6a4b691817"). InnerVolumeSpecName "kube-api-access-mrn7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 09:28:35.805698 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:35.805660 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/74493a16-c3ff-4158-bd5a-7a6a4b691817-oauth-serving-cert\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:28:35.805698 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:35.805691 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/74493a16-c3ff-4158-bd5a-7a6a4b691817-service-ca\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:28:35.805698 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:35.805702 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74493a16-c3ff-4158-bd5a-7a6a4b691817-trusted-ca-bundle\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:28:35.805952 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:35.805738 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/74493a16-c3ff-4158-bd5a-7a6a4b691817-console-config\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:28:35.805952 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:35.805746 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/74493a16-c3ff-4158-bd5a-7a6a4b691817-console-oauth-config\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:28:35.805952 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:35.805755 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/74493a16-c3ff-4158-bd5a-7a6a4b691817-console-serving-cert\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:28:35.805952 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:35.805763 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrn7k\" (UniqueName: \"kubernetes.io/projected/74493a16-c3ff-4158-bd5a-7a6a4b691817-kube-api-access-mrn7k\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:28:36.580143 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:36.580100 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cdcbcdb48-r2pqc" Jul 14 09:28:36.580143 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:36.580134 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cdcbcdb48-r2pqc" event={"ID":"74493a16-c3ff-4158-bd5a-7a6a4b691817","Type":"ContainerDied","Data":"fb5474e9a7f613f23c11f8251e7764aa35756cc0b94a56a961fd265b113bee0f"} Jul 14 09:28:36.580672 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:36.580195 2565 scope.go:117] "RemoveContainer" containerID="d51902194ae879e6855e7cd420b2b2582642501a37d954564937ff2ab4e41c55" Jul 14 09:28:36.601939 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:36.601912 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cdcbcdb48-r2pqc"] Jul 14 09:28:36.607788 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:36.607762 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-cdcbcdb48-r2pqc"] Jul 14 09:28:36.967748 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:36.967717 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74493a16-c3ff-4158-bd5a-7a6a4b691817" path="/var/lib/kubelet/pods/74493a16-c3ff-4158-bd5a-7a6a4b691817/volumes" Jul 14 09:28:44.122051 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:44.122019 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6cdc69fb5d-nqq6h" Jul 14 09:28:44.125905 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:44.125879 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6cdc69fb5d-nqq6h" Jul 14 09:28:46.587006 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:46.586957 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7555476c54-7f4b2" podUID="1d088b26-558e-44e9-a8c3-5e836a0936e2" containerName="console" containerID="cri-o://f5b5e9584e5529cbd49e78300ba4444448aa702b7daeba57183261a4d3fe8dd5" gracePeriod=15 Jul 14 09:28:46.848937 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:46.848879 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7555476c54-7f4b2" Jul 14 09:28:46.903248 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:46.903213 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d088b26-558e-44e9-a8c3-5e836a0936e2-console-serving-cert\") pod \"1d088b26-558e-44e9-a8c3-5e836a0936e2\" (UID: \"1d088b26-558e-44e9-a8c3-5e836a0936e2\") " Jul 14 09:28:46.903248 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:46.903245 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d088b26-558e-44e9-a8c3-5e836a0936e2-console-oauth-config\") pod \"1d088b26-558e-44e9-a8c3-5e836a0936e2\" (UID: \"1d088b26-558e-44e9-a8c3-5e836a0936e2\") " Jul 14 09:28:46.903495 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:46.903291 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d088b26-558e-44e9-a8c3-5e836a0936e2-service-ca\") pod \"1d088b26-558e-44e9-a8c3-5e836a0936e2\" (UID: \"1d088b26-558e-44e9-a8c3-5e836a0936e2\") " Jul 14 09:28:46.903495 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:46.903399 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d088b26-558e-44e9-a8c3-5e836a0936e2-oauth-serving-cert\") pod \"1d088b26-558e-44e9-a8c3-5e836a0936e2\" (UID: \"1d088b26-558e-44e9-a8c3-5e836a0936e2\") " Jul 14 09:28:46.903495 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:46.903458 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d088b26-558e-44e9-a8c3-5e836a0936e2-console-config\") pod \"1d088b26-558e-44e9-a8c3-5e836a0936e2\" (UID: \"1d088b26-558e-44e9-a8c3-5e836a0936e2\") " Jul 14 09:28:46.903634 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:46.903515 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hntz\" (UniqueName: \"kubernetes.io/projected/1d088b26-558e-44e9-a8c3-5e836a0936e2-kube-api-access-8hntz\") pod \"1d088b26-558e-44e9-a8c3-5e836a0936e2\" (UID: \"1d088b26-558e-44e9-a8c3-5e836a0936e2\") " Jul 14 09:28:46.903683 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:46.903634 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d088b26-558e-44e9-a8c3-5e836a0936e2-service-ca" (OuterVolumeSpecName: "service-ca") pod "1d088b26-558e-44e9-a8c3-5e836a0936e2" (UID: "1d088b26-558e-44e9-a8c3-5e836a0936e2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 09:28:46.903683 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:46.903657 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d088b26-558e-44e9-a8c3-5e836a0936e2-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1d088b26-558e-44e9-a8c3-5e836a0936e2" (UID: "1d088b26-558e-44e9-a8c3-5e836a0936e2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 09:28:46.903872 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:46.903852 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d088b26-558e-44e9-a8c3-5e836a0936e2-service-ca\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:28:46.903945 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:46.903876 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d088b26-558e-44e9-a8c3-5e836a0936e2-oauth-serving-cert\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:28:46.904030 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:46.904009 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d088b26-558e-44e9-a8c3-5e836a0936e2-console-config" (OuterVolumeSpecName: "console-config") pod "1d088b26-558e-44e9-a8c3-5e836a0936e2" (UID: "1d088b26-558e-44e9-a8c3-5e836a0936e2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 09:28:46.905725 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:46.905700 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d088b26-558e-44e9-a8c3-5e836a0936e2-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1d088b26-558e-44e9-a8c3-5e836a0936e2" (UID: "1d088b26-558e-44e9-a8c3-5e836a0936e2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 09:28:46.906056 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:46.906033 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d088b26-558e-44e9-a8c3-5e836a0936e2-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1d088b26-558e-44e9-a8c3-5e836a0936e2" (UID: "1d088b26-558e-44e9-a8c3-5e836a0936e2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 09:28:46.906154 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:46.906097 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d088b26-558e-44e9-a8c3-5e836a0936e2-kube-api-access-8hntz" (OuterVolumeSpecName: "kube-api-access-8hntz") pod "1d088b26-558e-44e9-a8c3-5e836a0936e2" (UID: "1d088b26-558e-44e9-a8c3-5e836a0936e2"). InnerVolumeSpecName "kube-api-access-8hntz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 09:28:47.005284 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:47.005249 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d088b26-558e-44e9-a8c3-5e836a0936e2-console-config\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:28:47.005284 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:47.005276 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hntz\" (UniqueName: \"kubernetes.io/projected/1d088b26-558e-44e9-a8c3-5e836a0936e2-kube-api-access-8hntz\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:28:47.005284 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:47.005285 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d088b26-558e-44e9-a8c3-5e836a0936e2-console-serving-cert\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:28:47.005284 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:47.005294 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d088b26-558e-44e9-a8c3-5e836a0936e2-console-oauth-config\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:28:47.634879 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:47.634847 2565 generic.go:356] "Generic (PLEG): container finished" podID="1d088b26-558e-44e9-a8c3-5e836a0936e2" containerID="f5b5e9584e5529cbd49e78300ba4444448aa702b7daeba57183261a4d3fe8dd5" exitCode=0 Jul 14 09:28:47.635324 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:47.634887 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7555476c54-7f4b2" Jul 14 09:28:47.635324 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:47.634925 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7555476c54-7f4b2" event={"ID":"1d088b26-558e-44e9-a8c3-5e836a0936e2","Type":"ContainerDied","Data":"f5b5e9584e5529cbd49e78300ba4444448aa702b7daeba57183261a4d3fe8dd5"} Jul 14 09:28:47.635324 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:47.634980 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7555476c54-7f4b2" event={"ID":"1d088b26-558e-44e9-a8c3-5e836a0936e2","Type":"ContainerDied","Data":"d2f3ae8567aae739c1a8f4f01a2ff474ec762e03a0886ed1e7ac06e6459e5de6"} Jul 14 09:28:47.635324 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:47.635000 2565 scope.go:117] "RemoveContainer" containerID="f5b5e9584e5529cbd49e78300ba4444448aa702b7daeba57183261a4d3fe8dd5" Jul 14 09:28:47.650891 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:47.650774 2565 scope.go:117] "RemoveContainer" containerID="f5b5e9584e5529cbd49e78300ba4444448aa702b7daeba57183261a4d3fe8dd5" Jul 14 09:28:47.651361 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:28:47.651262 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5b5e9584e5529cbd49e78300ba4444448aa702b7daeba57183261a4d3fe8dd5\": container with ID starting with f5b5e9584e5529cbd49e78300ba4444448aa702b7daeba57183261a4d3fe8dd5 not found: ID does not exist" containerID="f5b5e9584e5529cbd49e78300ba4444448aa702b7daeba57183261a4d3fe8dd5" Jul 14 09:28:47.651361 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:47.651303 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5b5e9584e5529cbd49e78300ba4444448aa702b7daeba57183261a4d3fe8dd5"} err="failed to get container status \"f5b5e9584e5529cbd49e78300ba4444448aa702b7daeba57183261a4d3fe8dd5\": rpc error: code = NotFound desc = could not find container \"f5b5e9584e5529cbd49e78300ba4444448aa702b7daeba57183261a4d3fe8dd5\": container with ID starting with f5b5e9584e5529cbd49e78300ba4444448aa702b7daeba57183261a4d3fe8dd5 not found: ID does not exist" Jul 14 09:28:47.666603 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:47.666573 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7555476c54-7f4b2"] Jul 14 09:28:47.671277 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:47.671249 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7555476c54-7f4b2"] Jul 14 09:28:48.967308 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:28:48.967268 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d088b26-558e-44e9-a8c3-5e836a0936e2" path="/var/lib/kubelet/pods/1d088b26-558e-44e9-a8c3-5e836a0936e2/volumes" Jul 14 09:29:24.554670 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:29:24.554637 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6cdd8bc874-bztpx"] Jul 14 09:29:24.555154 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:29:24.555026 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="394653f7-9641-443f-b1e2-8ea222d9ae02" containerName="console" Jul 14 09:29:24.555154 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:29:24.555039 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="394653f7-9641-443f-b1e2-8ea222d9ae02" containerName="console" Jul 14 09:29:24.555154 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:29:24.555050 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74493a16-c3ff-4158-bd5a-7a6a4b691817" containerName="console" Jul 14 09:29:24.555154 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:29:24.555055 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="74493a16-c3ff-4158-bd5a-7a6a4b691817" containerName="console" Jul 14 09:29:24.555154 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:29:24.555061 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d088b26-558e-44e9-a8c3-5e836a0936e2" containerName="console" Jul 14 09:29:24.555154 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:29:24.555066 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d088b26-558e-44e9-a8c3-5e836a0936e2" containerName="console" Jul 14 09:29:24.555154 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:29:24.555123 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="394653f7-9641-443f-b1e2-8ea222d9ae02" containerName="console" Jul 14 09:29:24.555154 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:29:24.555131 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="74493a16-c3ff-4158-bd5a-7a6a4b691817" containerName="console" Jul 14 09:29:24.555154 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:29:24.555138 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d088b26-558e-44e9-a8c3-5e836a0936e2" containerName="console" Jul 14 09:29:24.557054 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:29:24.557037 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cdd8bc874-bztpx" Jul 14 09:29:24.574452 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:29:24.574424 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cdd8bc874-bztpx"] Jul 14 09:29:24.632745 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:29:24.632712 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a1cd8e3b-7798-486f-89f1-d3522219c8ab-console-oauth-config\") pod \"console-6cdd8bc874-bztpx\" (UID: \"a1cd8e3b-7798-486f-89f1-d3522219c8ab\") " pod="openshift-console/console-6cdd8bc874-bztpx" Jul 14 09:29:24.632968 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:29:24.632766 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a1cd8e3b-7798-486f-89f1-d3522219c8ab-console-config\") pod \"console-6cdd8bc874-bztpx\" (UID: \"a1cd8e3b-7798-486f-89f1-d3522219c8ab\") " pod="openshift-console/console-6cdd8bc874-bztpx" Jul 14 09:29:24.632968 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:29:24.632835 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a1cd8e3b-7798-486f-89f1-d3522219c8ab-service-ca\") pod \"console-6cdd8bc874-bztpx\" (UID: \"a1cd8e3b-7798-486f-89f1-d3522219c8ab\") " pod="openshift-console/console-6cdd8bc874-bztpx" Jul 14 09:29:24.632968 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:29:24.632881 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdblk\" (UniqueName: \"kubernetes.io/projected/a1cd8e3b-7798-486f-89f1-d3522219c8ab-kube-api-access-gdblk\") pod \"console-6cdd8bc874-bztpx\" (UID: \"a1cd8e3b-7798-486f-89f1-d3522219c8ab\") " pod="openshift-console/console-6cdd8bc874-bztpx" Jul 14 09:29:24.632968 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:29:24.632919 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a1cd8e3b-7798-486f-89f1-d3522219c8ab-oauth-serving-cert\") pod \"console-6cdd8bc874-bztpx\" (UID: \"a1cd8e3b-7798-486f-89f1-d3522219c8ab\") " pod="openshift-console/console-6cdd8bc874-bztpx" Jul 14 09:29:24.633137 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:29:24.632993 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1cd8e3b-7798-486f-89f1-d3522219c8ab-trusted-ca-bundle\") pod \"console-6cdd8bc874-bztpx\" (UID: \"a1cd8e3b-7798-486f-89f1-d3522219c8ab\") " pod="openshift-console/console-6cdd8bc874-bztpx" Jul 14 09:29:24.633137 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:29:24.633046 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1cd8e3b-7798-486f-89f1-d3522219c8ab-console-serving-cert\") pod \"console-6cdd8bc874-bztpx\" (UID: \"a1cd8e3b-7798-486f-89f1-d3522219c8ab\") " pod="openshift-console/console-6cdd8bc874-bztpx" Jul 14 09:29:24.734027 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:29:24.733984 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a1cd8e3b-7798-486f-89f1-d3522219c8ab-console-oauth-config\") pod \"console-6cdd8bc874-bztpx\" (UID: \"a1cd8e3b-7798-486f-89f1-d3522219c8ab\") " pod="openshift-console/console-6cdd8bc874-bztpx" Jul 14 09:29:24.734186 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:29:24.734042 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a1cd8e3b-7798-486f-89f1-d3522219c8ab-console-config\") pod \"console-6cdd8bc874-bztpx\" (UID: \"a1cd8e3b-7798-486f-89f1-d3522219c8ab\") " pod="openshift-console/console-6cdd8bc874-bztpx" Jul 14 09:29:24.734186 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:29:24.734071 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a1cd8e3b-7798-486f-89f1-d3522219c8ab-service-ca\") pod \"console-6cdd8bc874-bztpx\" (UID: \"a1cd8e3b-7798-486f-89f1-d3522219c8ab\") " pod="openshift-console/console-6cdd8bc874-bztpx" Jul 14 09:29:24.734186 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:29:24.734094 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdblk\" (UniqueName: \"kubernetes.io/projected/a1cd8e3b-7798-486f-89f1-d3522219c8ab-kube-api-access-gdblk\") pod \"console-6cdd8bc874-bztpx\" (UID: \"a1cd8e3b-7798-486f-89f1-d3522219c8ab\") " pod="openshift-console/console-6cdd8bc874-bztpx" Jul 14 09:29:24.734186 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:29:24.734124 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a1cd8e3b-7798-486f-89f1-d3522219c8ab-oauth-serving-cert\") pod \"console-6cdd8bc874-bztpx\" (UID: \"a1cd8e3b-7798-486f-89f1-d3522219c8ab\") " pod="openshift-console/console-6cdd8bc874-bztpx" Jul 14 09:29:24.734390 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:29:24.734337 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1cd8e3b-7798-486f-89f1-d3522219c8ab-trusted-ca-bundle\") pod \"console-6cdd8bc874-bztpx\" (UID: \"a1cd8e3b-7798-486f-89f1-d3522219c8ab\") " pod="openshift-console/console-6cdd8bc874-bztpx" Jul 14 09:29:24.734438 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:29:24.734424 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1cd8e3b-7798-486f-89f1-d3522219c8ab-console-serving-cert\") pod \"console-6cdd8bc874-bztpx\" (UID: \"a1cd8e3b-7798-486f-89f1-d3522219c8ab\") " pod="openshift-console/console-6cdd8bc874-bztpx" Jul 14 09:29:24.734941 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:29:24.734916 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a1cd8e3b-7798-486f-89f1-d3522219c8ab-service-ca\") pod \"console-6cdd8bc874-bztpx\" (UID: \"a1cd8e3b-7798-486f-89f1-d3522219c8ab\") " pod="openshift-console/console-6cdd8bc874-bztpx" Jul 14 09:29:24.735077 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:29:24.734969 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a1cd8e3b-7798-486f-89f1-d3522219c8ab-oauth-serving-cert\") pod \"console-6cdd8bc874-bztpx\" (UID: \"a1cd8e3b-7798-486f-89f1-d3522219c8ab\") " pod="openshift-console/console-6cdd8bc874-bztpx" Jul 14 09:29:24.735077 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:29:24.734991 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a1cd8e3b-7798-486f-89f1-d3522219c8ab-console-config\") pod \"console-6cdd8bc874-bztpx\" (UID: \"a1cd8e3b-7798-486f-89f1-d3522219c8ab\") " pod="openshift-console/console-6cdd8bc874-bztpx" Jul 14 09:29:24.735236 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:29:24.735216 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1cd8e3b-7798-486f-89f1-d3522219c8ab-trusted-ca-bundle\") pod \"console-6cdd8bc874-bztpx\" (UID: \"a1cd8e3b-7798-486f-89f1-d3522219c8ab\") " pod="openshift-console/console-6cdd8bc874-bztpx" Jul 14 09:29:24.736508 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:29:24.736485 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a1cd8e3b-7798-486f-89f1-d3522219c8ab-console-oauth-config\") pod \"console-6cdd8bc874-bztpx\" (UID: \"a1cd8e3b-7798-486f-89f1-d3522219c8ab\") " pod="openshift-console/console-6cdd8bc874-bztpx" Jul 14 09:29:24.736838 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:29:24.736821 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1cd8e3b-7798-486f-89f1-d3522219c8ab-console-serving-cert\") pod \"console-6cdd8bc874-bztpx\" (UID: \"a1cd8e3b-7798-486f-89f1-d3522219c8ab\") " pod="openshift-console/console-6cdd8bc874-bztpx" Jul 14 09:29:24.756266 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:29:24.756240 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdblk\" (UniqueName: \"kubernetes.io/projected/a1cd8e3b-7798-486f-89f1-d3522219c8ab-kube-api-access-gdblk\") pod \"console-6cdd8bc874-bztpx\" (UID: \"a1cd8e3b-7798-486f-89f1-d3522219c8ab\") " pod="openshift-console/console-6cdd8bc874-bztpx" Jul 14 09:29:24.866220 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:29:24.866120 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cdd8bc874-bztpx" Jul 14 09:29:24.995164 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:29:24.995140 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cdd8bc874-bztpx"] Jul 14 09:29:24.997583 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:29:24.997557 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1cd8e3b_7798_486f_89f1_d3522219c8ab.slice/crio-2b332db0503584efb8443e9b12ac71acdebf94901ded9ded266632022ec9c9ad WatchSource:0}: Error finding container 2b332db0503584efb8443e9b12ac71acdebf94901ded9ded266632022ec9c9ad: Status 404 returned error can't find the container with id 2b332db0503584efb8443e9b12ac71acdebf94901ded9ded266632022ec9c9ad Jul 14 09:29:25.783849 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:29:25.783795 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cdd8bc874-bztpx" event={"ID":"a1cd8e3b-7798-486f-89f1-d3522219c8ab","Type":"ContainerStarted","Data":"a3e4981065c37b4341c34c8bda0d655d27666e92af8fe82e0be60fe6029a076a"} Jul 14 09:29:25.784230 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:29:25.783865 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cdd8bc874-bztpx" event={"ID":"a1cd8e3b-7798-486f-89f1-d3522219c8ab","Type":"ContainerStarted","Data":"2b332db0503584efb8443e9b12ac71acdebf94901ded9ded266632022ec9c9ad"} Jul 14 09:29:25.802009 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:29:25.801956 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6cdd8bc874-bztpx" podStartSLOduration=1.801939766 podStartE2EDuration="1.801939766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 09:29:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:29:25.800031941 +0000 UTC m=+155.309263391" watchObservedRunningTime="2026-07-14 09:29:25.801939766 +0000 UTC m=+155.311171214" Jul 14 09:29:34.866616 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:29:34.866579 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6cdd8bc874-bztpx" Jul 14 09:29:34.867113 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:29:34.866671 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6cdd8bc874-bztpx" Jul 14 09:29:34.871452 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:29:34.871426 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6cdd8bc874-bztpx" Jul 14 09:29:35.826169 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:29:35.826142 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6cdd8bc874-bztpx" Jul 14 09:29:35.866952 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:29:35.866920 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78cbb6bb89-nw5sx"] Jul 14 09:30:00.888148 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:00.888032 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-78cbb6bb89-nw5sx" podUID="527dcd95-98da-4412-930f-06a3feaf4cc2" containerName="console" containerID="cri-o://992a5744ff17e9dc208c3eea12327f1d18b8b3cc166a2105876a4d98a4fe1a2d" gracePeriod=15 Jul 14 09:30:01.127718 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:01.127691 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78cbb6bb89-nw5sx" Jul 14 09:30:01.259031 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:01.258927 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/527dcd95-98da-4412-930f-06a3feaf4cc2-oauth-serving-cert\") pod \"527dcd95-98da-4412-930f-06a3feaf4cc2\" (UID: \"527dcd95-98da-4412-930f-06a3feaf4cc2\") " Jul 14 09:30:01.259031 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:01.259012 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/527dcd95-98da-4412-930f-06a3feaf4cc2-console-serving-cert\") pod \"527dcd95-98da-4412-930f-06a3feaf4cc2\" (UID: \"527dcd95-98da-4412-930f-06a3feaf4cc2\") " Jul 14 09:30:01.259265 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:01.259036 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/527dcd95-98da-4412-930f-06a3feaf4cc2-console-oauth-config\") pod \"527dcd95-98da-4412-930f-06a3feaf4cc2\" (UID: \"527dcd95-98da-4412-930f-06a3feaf4cc2\") " Jul 14 09:30:01.259265 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:01.259060 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/527dcd95-98da-4412-930f-06a3feaf4cc2-console-config\") pod \"527dcd95-98da-4412-930f-06a3feaf4cc2\" (UID: \"527dcd95-98da-4412-930f-06a3feaf4cc2\") " Jul 14 09:30:01.259265 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:01.259218 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/527dcd95-98da-4412-930f-06a3feaf4cc2-trusted-ca-bundle\") pod \"527dcd95-98da-4412-930f-06a3feaf4cc2\" (UID: \"527dcd95-98da-4412-930f-06a3feaf4cc2\") " Jul 14 09:30:01.259423 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:01.259271 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59sgt\" (UniqueName: \"kubernetes.io/projected/527dcd95-98da-4412-930f-06a3feaf4cc2-kube-api-access-59sgt\") pod \"527dcd95-98da-4412-930f-06a3feaf4cc2\" (UID: \"527dcd95-98da-4412-930f-06a3feaf4cc2\") " Jul 14 09:30:01.259423 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:01.259370 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/527dcd95-98da-4412-930f-06a3feaf4cc2-service-ca\") pod \"527dcd95-98da-4412-930f-06a3feaf4cc2\" (UID: \"527dcd95-98da-4412-930f-06a3feaf4cc2\") " Jul 14 09:30:01.259423 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:01.259305 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/527dcd95-98da-4412-930f-06a3feaf4cc2-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "527dcd95-98da-4412-930f-06a3feaf4cc2" (UID: "527dcd95-98da-4412-930f-06a3feaf4cc2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 09:30:01.259559 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:01.259494 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/527dcd95-98da-4412-930f-06a3feaf4cc2-console-config" (OuterVolumeSpecName: "console-config") pod "527dcd95-98da-4412-930f-06a3feaf4cc2" (UID: "527dcd95-98da-4412-930f-06a3feaf4cc2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 09:30:01.259703 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:01.259673 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/527dcd95-98da-4412-930f-06a3feaf4cc2-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "527dcd95-98da-4412-930f-06a3feaf4cc2" (UID: "527dcd95-98da-4412-930f-06a3feaf4cc2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 09:30:01.259703 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:01.259703 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/527dcd95-98da-4412-930f-06a3feaf4cc2-oauth-serving-cert\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:30:01.259923 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:01.259724 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/527dcd95-98da-4412-930f-06a3feaf4cc2-console-config\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:30:01.259923 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:01.259734 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/527dcd95-98da-4412-930f-06a3feaf4cc2-service-ca" (OuterVolumeSpecName: "service-ca") pod "527dcd95-98da-4412-930f-06a3feaf4cc2" (UID: "527dcd95-98da-4412-930f-06a3feaf4cc2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 09:30:01.261304 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:01.261280 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/527dcd95-98da-4412-930f-06a3feaf4cc2-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "527dcd95-98da-4412-930f-06a3feaf4cc2" (UID: "527dcd95-98da-4412-930f-06a3feaf4cc2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 09:30:01.261394 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:01.261282 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/527dcd95-98da-4412-930f-06a3feaf4cc2-kube-api-access-59sgt" (OuterVolumeSpecName: "kube-api-access-59sgt") pod "527dcd95-98da-4412-930f-06a3feaf4cc2" (UID: "527dcd95-98da-4412-930f-06a3feaf4cc2"). InnerVolumeSpecName "kube-api-access-59sgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 09:30:01.261394 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:01.261301 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/527dcd95-98da-4412-930f-06a3feaf4cc2-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "527dcd95-98da-4412-930f-06a3feaf4cc2" (UID: "527dcd95-98da-4412-930f-06a3feaf4cc2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 09:30:01.360126 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:01.360089 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/527dcd95-98da-4412-930f-06a3feaf4cc2-console-serving-cert\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:30:01.360126 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:01.360119 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/527dcd95-98da-4412-930f-06a3feaf4cc2-console-oauth-config\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:30:01.360126 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:01.360128 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/527dcd95-98da-4412-930f-06a3feaf4cc2-trusted-ca-bundle\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:30:01.360126 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:01.360137 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59sgt\" (UniqueName: \"kubernetes.io/projected/527dcd95-98da-4412-930f-06a3feaf4cc2-kube-api-access-59sgt\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:30:01.360379 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:01.360147 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/527dcd95-98da-4412-930f-06a3feaf4cc2-service-ca\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:30:01.924869 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:01.924799 2565 generic.go:356] "Generic (PLEG): container finished" podID="527dcd95-98da-4412-930f-06a3feaf4cc2" containerID="992a5744ff17e9dc208c3eea12327f1d18b8b3cc166a2105876a4d98a4fe1a2d" exitCode=0 Jul 14 09:30:01.925352 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:01.924884 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78cbb6bb89-nw5sx" Jul 14 09:30:01.925352 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:01.924888 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78cbb6bb89-nw5sx" event={"ID":"527dcd95-98da-4412-930f-06a3feaf4cc2","Type":"ContainerDied","Data":"992a5744ff17e9dc208c3eea12327f1d18b8b3cc166a2105876a4d98a4fe1a2d"} Jul 14 09:30:01.925352 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:01.924987 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78cbb6bb89-nw5sx" event={"ID":"527dcd95-98da-4412-930f-06a3feaf4cc2","Type":"ContainerDied","Data":"fba30d5ad83dac9a7b8f5e229e732838dcb4a89c8abfd31903332f9ce6f94079"} Jul 14 09:30:01.925352 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:01.925021 2565 scope.go:117] "RemoveContainer" containerID="992a5744ff17e9dc208c3eea12327f1d18b8b3cc166a2105876a4d98a4fe1a2d" Jul 14 09:30:01.933565 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:01.933547 2565 scope.go:117] "RemoveContainer" containerID="992a5744ff17e9dc208c3eea12327f1d18b8b3cc166a2105876a4d98a4fe1a2d" Jul 14 09:30:01.933888 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:30:01.933868 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"992a5744ff17e9dc208c3eea12327f1d18b8b3cc166a2105876a4d98a4fe1a2d\": container with ID starting with 992a5744ff17e9dc208c3eea12327f1d18b8b3cc166a2105876a4d98a4fe1a2d not found: ID does not exist" containerID="992a5744ff17e9dc208c3eea12327f1d18b8b3cc166a2105876a4d98a4fe1a2d" Jul 14 09:30:01.933950 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:01.933898 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"992a5744ff17e9dc208c3eea12327f1d18b8b3cc166a2105876a4d98a4fe1a2d"} err="failed to get container status \"992a5744ff17e9dc208c3eea12327f1d18b8b3cc166a2105876a4d98a4fe1a2d\": rpc error: code = NotFound desc = could not find container \"992a5744ff17e9dc208c3eea12327f1d18b8b3cc166a2105876a4d98a4fe1a2d\": container with ID starting with 992a5744ff17e9dc208c3eea12327f1d18b8b3cc166a2105876a4d98a4fe1a2d not found: ID does not exist" Jul 14 09:30:01.950020 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:01.949989 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78cbb6bb89-nw5sx"] Jul 14 09:30:01.955125 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:01.955101 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-78cbb6bb89-nw5sx"] Jul 14 09:30:02.967276 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:02.967238 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="527dcd95-98da-4412-930f-06a3feaf4cc2" path="/var/lib/kubelet/pods/527dcd95-98da-4412-930f-06a3feaf4cc2/volumes" Jul 14 09:30:09.855866 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:09.855828 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vfzwx"] Jul 14 09:30:09.856242 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:09.856218 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="527dcd95-98da-4412-930f-06a3feaf4cc2" containerName="console" Jul 14 09:30:09.856242 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:09.856234 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="527dcd95-98da-4412-930f-06a3feaf4cc2" containerName="console" Jul 14 09:30:09.856316 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:09.856285 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="527dcd95-98da-4412-930f-06a3feaf4cc2" containerName="console" Jul 14 09:30:09.859172 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:09.859156 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vfzwx" Jul 14 09:30:09.861914 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:09.861894 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 14 09:30:09.862725 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:09.862709 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 14 09:30:09.862821 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:09.862713 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-nqd2l\"" Jul 14 09:30:09.869710 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:09.869685 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vfzwx"] Jul 14 09:30:09.932270 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:09.932234 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6b767ee4-6d53-49b3-9b99-6116c4c39c01-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vfzwx\" (UID: \"6b767ee4-6d53-49b3-9b99-6116c4c39c01\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vfzwx" Jul 14 09:30:09.932439 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:09.932276 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6b767ee4-6d53-49b3-9b99-6116c4c39c01-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vfzwx\" (UID: \"6b767ee4-6d53-49b3-9b99-6116c4c39c01\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vfzwx" Jul 14 09:30:09.932439 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:09.932358 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k66r4\" (UniqueName: \"kubernetes.io/projected/6b767ee4-6d53-49b3-9b99-6116c4c39c01-kube-api-access-k66r4\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vfzwx\" (UID: \"6b767ee4-6d53-49b3-9b99-6116c4c39c01\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vfzwx" Jul 14 09:30:10.033445 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:10.033411 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k66r4\" (UniqueName: \"kubernetes.io/projected/6b767ee4-6d53-49b3-9b99-6116c4c39c01-kube-api-access-k66r4\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vfzwx\" (UID: \"6b767ee4-6d53-49b3-9b99-6116c4c39c01\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vfzwx" Jul 14 09:30:10.033611 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:10.033476 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6b767ee4-6d53-49b3-9b99-6116c4c39c01-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vfzwx\" (UID: \"6b767ee4-6d53-49b3-9b99-6116c4c39c01\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vfzwx" Jul 14 09:30:10.033611 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:10.033502 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6b767ee4-6d53-49b3-9b99-6116c4c39c01-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vfzwx\" (UID: \"6b767ee4-6d53-49b3-9b99-6116c4c39c01\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vfzwx" Jul 14 09:30:10.033842 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:10.033827 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6b767ee4-6d53-49b3-9b99-6116c4c39c01-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vfzwx\" (UID: \"6b767ee4-6d53-49b3-9b99-6116c4c39c01\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vfzwx" Jul 14 09:30:10.033933 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:10.033915 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6b767ee4-6d53-49b3-9b99-6116c4c39c01-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vfzwx\" (UID: \"6b767ee4-6d53-49b3-9b99-6116c4c39c01\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vfzwx" Jul 14 09:30:10.042315 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:10.042281 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k66r4\" (UniqueName: \"kubernetes.io/projected/6b767ee4-6d53-49b3-9b99-6116c4c39c01-kube-api-access-k66r4\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vfzwx\" (UID: \"6b767ee4-6d53-49b3-9b99-6116c4c39c01\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vfzwx" Jul 14 09:30:10.167716 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:10.167689 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vfzwx" Jul 14 09:30:10.295878 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:10.295852 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vfzwx"] Jul 14 09:30:10.298166 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:30:10.298123 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b767ee4_6d53_49b3_9b99_6116c4c39c01.slice/crio-18d3977c394617ed7fe8da76a695c2412570ac22df29ced329d29e1776207431 WatchSource:0}: Error finding container 18d3977c394617ed7fe8da76a695c2412570ac22df29ced329d29e1776207431: Status 404 returned error can't find the container with id 18d3977c394617ed7fe8da76a695c2412570ac22df29ced329d29e1776207431 Jul 14 09:30:10.961546 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:10.961512 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vfzwx" event={"ID":"6b767ee4-6d53-49b3-9b99-6116c4c39c01","Type":"ContainerStarted","Data":"18d3977c394617ed7fe8da76a695c2412570ac22df29ced329d29e1776207431"} Jul 14 09:30:16.995156 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:16.995117 2565 generic.go:356] "Generic (PLEG): container finished" podID="6b767ee4-6d53-49b3-9b99-6116c4c39c01" containerID="dbe95c44c1a8120ea57ae901e6c6befe115756365863293c984dda07e517cf7e" exitCode=0 Jul 14 09:30:16.995518 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:16.995175 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vfzwx" event={"ID":"6b767ee4-6d53-49b3-9b99-6116c4c39c01","Type":"ContainerDied","Data":"dbe95c44c1a8120ea57ae901e6c6befe115756365863293c984dda07e517cf7e"} Jul 14 09:30:34.063793 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:34.063708 2565 generic.go:356] "Generic (PLEG): container finished" podID="6b767ee4-6d53-49b3-9b99-6116c4c39c01" containerID="534221de5385a79ed2dc7cbd825410f0d1fe5d6c9f8aeb65390c4f5c150f27ac" exitCode=0 Jul 14 09:30:34.063793 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:34.063751 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vfzwx" event={"ID":"6b767ee4-6d53-49b3-9b99-6116c4c39c01","Type":"ContainerDied","Data":"534221de5385a79ed2dc7cbd825410f0d1fe5d6c9f8aeb65390c4f5c150f27ac"} Jul 14 09:30:42.097419 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:42.097384 2565 generic.go:356] "Generic (PLEG): container finished" podID="6b767ee4-6d53-49b3-9b99-6116c4c39c01" containerID="09052e72109829efa01e17dad3e170cc57db6a027b1aff6b90802d5f59c0fdab" exitCode=0 Jul 14 09:30:42.097791 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:42.097439 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vfzwx" event={"ID":"6b767ee4-6d53-49b3-9b99-6116c4c39c01","Type":"ContainerDied","Data":"09052e72109829efa01e17dad3e170cc57db6a027b1aff6b90802d5f59c0fdab"} Jul 14 09:30:43.227438 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:43.227415 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vfzwx" Jul 14 09:30:43.299774 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:43.299745 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6b767ee4-6d53-49b3-9b99-6116c4c39c01-util\") pod \"6b767ee4-6d53-49b3-9b99-6116c4c39c01\" (UID: \"6b767ee4-6d53-49b3-9b99-6116c4c39c01\") " Jul 14 09:30:43.299967 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:43.299830 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6b767ee4-6d53-49b3-9b99-6116c4c39c01-bundle\") pod \"6b767ee4-6d53-49b3-9b99-6116c4c39c01\" (UID: \"6b767ee4-6d53-49b3-9b99-6116c4c39c01\") " Jul 14 09:30:43.299967 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:43.299874 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k66r4\" (UniqueName: \"kubernetes.io/projected/6b767ee4-6d53-49b3-9b99-6116c4c39c01-kube-api-access-k66r4\") pod \"6b767ee4-6d53-49b3-9b99-6116c4c39c01\" (UID: \"6b767ee4-6d53-49b3-9b99-6116c4c39c01\") " Jul 14 09:30:43.300741 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:43.300711 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b767ee4-6d53-49b3-9b99-6116c4c39c01-bundle" (OuterVolumeSpecName: "bundle") pod "6b767ee4-6d53-49b3-9b99-6116c4c39c01" (UID: "6b767ee4-6d53-49b3-9b99-6116c4c39c01"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 09:30:43.302437 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:43.302411 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b767ee4-6d53-49b3-9b99-6116c4c39c01-kube-api-access-k66r4" (OuterVolumeSpecName: "kube-api-access-k66r4") pod "6b767ee4-6d53-49b3-9b99-6116c4c39c01" (UID: "6b767ee4-6d53-49b3-9b99-6116c4c39c01"). InnerVolumeSpecName "kube-api-access-k66r4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 09:30:43.305697 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:43.305671 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b767ee4-6d53-49b3-9b99-6116c4c39c01-util" (OuterVolumeSpecName: "util") pod "6b767ee4-6d53-49b3-9b99-6116c4c39c01" (UID: "6b767ee4-6d53-49b3-9b99-6116c4c39c01"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 09:30:43.401058 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:43.401026 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6b767ee4-6d53-49b3-9b99-6116c4c39c01-util\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:30:43.401058 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:43.401058 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6b767ee4-6d53-49b3-9b99-6116c4c39c01-bundle\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:30:43.401245 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:43.401067 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k66r4\" (UniqueName: \"kubernetes.io/projected/6b767ee4-6d53-49b3-9b99-6116c4c39c01-kube-api-access-k66r4\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:30:44.107431 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:44.107404 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vfzwx" Jul 14 09:30:44.107603 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:44.107403 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vfzwx" event={"ID":"6b767ee4-6d53-49b3-9b99-6116c4c39c01","Type":"ContainerDied","Data":"18d3977c394617ed7fe8da76a695c2412570ac22df29ced329d29e1776207431"} Jul 14 09:30:44.107603 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:44.107518 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18d3977c394617ed7fe8da76a695c2412570ac22df29ced329d29e1776207431" Jul 14 09:30:48.388319 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:48.388286 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x8fj8"] Jul 14 09:30:48.388682 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:48.388609 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b767ee4-6d53-49b3-9b99-6116c4c39c01" containerName="pull" Jul 14 09:30:48.388682 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:48.388620 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b767ee4-6d53-49b3-9b99-6116c4c39c01" containerName="pull" Jul 14 09:30:48.388682 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:48.388628 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b767ee4-6d53-49b3-9b99-6116c4c39c01" containerName="extract" Jul 14 09:30:48.388682 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:48.388633 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b767ee4-6d53-49b3-9b99-6116c4c39c01" containerName="extract" Jul 14 09:30:48.388682 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:48.388643 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b767ee4-6d53-49b3-9b99-6116c4c39c01" containerName="util" Jul 14 09:30:48.388682 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:48.388649 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b767ee4-6d53-49b3-9b99-6116c4c39c01" containerName="util" Jul 14 09:30:48.388885 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:48.388708 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b767ee4-6d53-49b3-9b99-6116c4c39c01" containerName="extract" Jul 14 09:30:48.390789 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:48.390774 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x8fj8" Jul 14 09:30:48.397615 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:48.397592 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 14 09:30:48.397770 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:48.397752 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 14 09:30:48.397893 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:48.397754 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-sct57\"" Jul 14 09:30:48.419447 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:48.419420 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x8fj8"] Jul 14 09:30:48.445940 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:48.445905 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4542f867-35ee-4e6b-8819-bd70458e5e57-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-x8fj8\" (UID: \"4542f867-35ee-4e6b-8819-bd70458e5e57\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x8fj8" Jul 14 09:30:48.446062 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:48.445953 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8cgv\" (UniqueName: \"kubernetes.io/projected/4542f867-35ee-4e6b-8819-bd70458e5e57-kube-api-access-x8cgv\") pod \"cert-manager-operator-controller-manager-55c444bf86-x8fj8\" (UID: \"4542f867-35ee-4e6b-8819-bd70458e5e57\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x8fj8" Jul 14 09:30:48.546373 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:48.546341 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4542f867-35ee-4e6b-8819-bd70458e5e57-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-x8fj8\" (UID: \"4542f867-35ee-4e6b-8819-bd70458e5e57\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x8fj8" Jul 14 09:30:48.546543 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:48.546383 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8cgv\" (UniqueName: \"kubernetes.io/projected/4542f867-35ee-4e6b-8819-bd70458e5e57-kube-api-access-x8cgv\") pod \"cert-manager-operator-controller-manager-55c444bf86-x8fj8\" (UID: \"4542f867-35ee-4e6b-8819-bd70458e5e57\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x8fj8" Jul 14 09:30:48.546768 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:48.546748 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4542f867-35ee-4e6b-8819-bd70458e5e57-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-x8fj8\" (UID: \"4542f867-35ee-4e6b-8819-bd70458e5e57\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x8fj8" Jul 14 09:30:48.566675 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:48.566648 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8cgv\" (UniqueName: \"kubernetes.io/projected/4542f867-35ee-4e6b-8819-bd70458e5e57-kube-api-access-x8cgv\") pod \"cert-manager-operator-controller-manager-55c444bf86-x8fj8\" (UID: \"4542f867-35ee-4e6b-8819-bd70458e5e57\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x8fj8" Jul 14 09:30:48.699653 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:48.699565 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x8fj8" Jul 14 09:30:48.849887 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:48.849724 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x8fj8"] Jul 14 09:30:48.852921 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:30:48.852887 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4542f867_35ee_4e6b_8819_bd70458e5e57.slice/crio-f50b02bb95c963277ad38f6f34948aeedd918123fa9e4076d4d9f06df64ea345 WatchSource:0}: Error finding container f50b02bb95c963277ad38f6f34948aeedd918123fa9e4076d4d9f06df64ea345: Status 404 returned error can't find the container with id f50b02bb95c963277ad38f6f34948aeedd918123fa9e4076d4d9f06df64ea345 Jul 14 09:30:49.128486 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:49.128406 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x8fj8" event={"ID":"4542f867-35ee-4e6b-8819-bd70458e5e57","Type":"ContainerStarted","Data":"f50b02bb95c963277ad38f6f34948aeedd918123fa9e4076d4d9f06df64ea345"} Jul 14 09:30:54.152486 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:54.152451 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x8fj8" event={"ID":"4542f867-35ee-4e6b-8819-bd70458e5e57","Type":"ContainerStarted","Data":"454a0f29da07003093e93bdb35e81dbd94bd9b5bf52ccc01a8f06ce4a5d782aa"} Jul 14 09:30:54.188964 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:54.188898 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x8fj8" podStartSLOduration=0.988686937 podStartE2EDuration="6.188877669s" totalImagesPullingTime="5.200190732s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 09:30:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:30:54.184451616 +0000 UTC m=+243.693683065" watchObservedRunningTime="2026-07-14 09:30:54.188877669 +0000 UTC m=+243.698109119" Jul 14 09:30:55.847706 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:55.847675 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmzm9z"] Jul 14 09:30:55.851090 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:55.851062 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmzm9z" Jul 14 09:30:55.855119 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:55.855096 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 14 09:30:55.855242 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:55.855229 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-nqd2l\"" Jul 14 09:30:55.856077 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:55.856058 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 14 09:30:55.865235 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:55.865212 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmzm9z"] Jul 14 09:30:55.913695 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:55.913664 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mgxx\" (UniqueName: \"kubernetes.io/projected/52757dab-82b1-4040-a8fd-1e60254ec093-kube-api-access-5mgxx\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmzm9z\" (UID: \"52757dab-82b1-4040-a8fd-1e60254ec093\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmzm9z" Jul 14 09:30:55.913923 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:55.913705 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/52757dab-82b1-4040-a8fd-1e60254ec093-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmzm9z\" (UID: \"52757dab-82b1-4040-a8fd-1e60254ec093\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmzm9z" Jul 14 09:30:55.913923 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:55.913882 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/52757dab-82b1-4040-a8fd-1e60254ec093-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmzm9z\" (UID: \"52757dab-82b1-4040-a8fd-1e60254ec093\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmzm9z" Jul 14 09:30:56.015106 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:56.015070 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mgxx\" (UniqueName: \"kubernetes.io/projected/52757dab-82b1-4040-a8fd-1e60254ec093-kube-api-access-5mgxx\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmzm9z\" (UID: \"52757dab-82b1-4040-a8fd-1e60254ec093\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmzm9z" Jul 14 09:30:56.015106 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:56.015114 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/52757dab-82b1-4040-a8fd-1e60254ec093-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmzm9z\" (UID: \"52757dab-82b1-4040-a8fd-1e60254ec093\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmzm9z" Jul 14 09:30:56.015356 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:56.015169 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/52757dab-82b1-4040-a8fd-1e60254ec093-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmzm9z\" (UID: \"52757dab-82b1-4040-a8fd-1e60254ec093\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmzm9z" Jul 14 09:30:56.015499 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:56.015480 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/52757dab-82b1-4040-a8fd-1e60254ec093-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmzm9z\" (UID: \"52757dab-82b1-4040-a8fd-1e60254ec093\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmzm9z" Jul 14 09:30:56.015551 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:56.015514 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/52757dab-82b1-4040-a8fd-1e60254ec093-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmzm9z\" (UID: \"52757dab-82b1-4040-a8fd-1e60254ec093\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmzm9z" Jul 14 09:30:56.025195 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:56.025164 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mgxx\" (UniqueName: \"kubernetes.io/projected/52757dab-82b1-4040-a8fd-1e60254ec093-kube-api-access-5mgxx\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmzm9z\" (UID: \"52757dab-82b1-4040-a8fd-1e60254ec093\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmzm9z" Jul 14 09:30:56.160978 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:56.160953 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmzm9z" Jul 14 09:30:56.308569 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:56.308545 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmzm9z"] Jul 14 09:30:56.312629 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:30:56.312599 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52757dab_82b1_4040_a8fd_1e60254ec093.slice/crio-29ef3816baebaa42dc1d824d88cfa00a84439671a7cee37d14e51624e2e31077 WatchSource:0}: Error finding container 29ef3816baebaa42dc1d824d88cfa00a84439671a7cee37d14e51624e2e31077: Status 404 returned error can't find the container with id 29ef3816baebaa42dc1d824d88cfa00a84439671a7cee37d14e51624e2e31077 Jul 14 09:30:57.166459 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:57.166421 2565 generic.go:356] "Generic (PLEG): container finished" podID="52757dab-82b1-4040-a8fd-1e60254ec093" containerID="558e1b3c852ec6beb21a87e55f7824bd5ffc5713c09f1ec2567e82823bbb6601" exitCode=0 Jul 14 09:30:57.166939 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:57.166478 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmzm9z" event={"ID":"52757dab-82b1-4040-a8fd-1e60254ec093","Type":"ContainerDied","Data":"558e1b3c852ec6beb21a87e55f7824bd5ffc5713c09f1ec2567e82823bbb6601"} Jul 14 09:30:57.166939 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:57.166515 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmzm9z" event={"ID":"52757dab-82b1-4040-a8fd-1e60254ec093","Type":"ContainerStarted","Data":"29ef3816baebaa42dc1d824d88cfa00a84439671a7cee37d14e51624e2e31077"} Jul 14 09:30:57.768120 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:57.768086 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-fnbsw"] Jul 14 09:30:57.771281 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:57.771263 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-fnbsw" Jul 14 09:30:57.778659 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:57.778626 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 14 09:30:57.779331 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:57.779303 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-sv5b7\"" Jul 14 09:30:57.789879 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:57.789857 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 14 09:30:57.798295 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:57.798271 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-fnbsw"] Jul 14 09:30:57.833213 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:57.833171 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2ce0dac4-27ff-49e0-8b04-cc274dcc4170-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-fnbsw\" (UID: \"2ce0dac4-27ff-49e0-8b04-cc274dcc4170\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fnbsw" Jul 14 09:30:57.833385 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:57.833258 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmdgv\" (UniqueName: \"kubernetes.io/projected/2ce0dac4-27ff-49e0-8b04-cc274dcc4170-kube-api-access-zmdgv\") pod \"cert-manager-cainjector-559cdd8c87-fnbsw\" (UID: \"2ce0dac4-27ff-49e0-8b04-cc274dcc4170\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fnbsw" Jul 14 09:30:57.934413 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:57.934358 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2ce0dac4-27ff-49e0-8b04-cc274dcc4170-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-fnbsw\" (UID: \"2ce0dac4-27ff-49e0-8b04-cc274dcc4170\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fnbsw" Jul 14 09:30:57.934584 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:57.934426 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmdgv\" (UniqueName: \"kubernetes.io/projected/2ce0dac4-27ff-49e0-8b04-cc274dcc4170-kube-api-access-zmdgv\") pod \"cert-manager-cainjector-559cdd8c87-fnbsw\" (UID: \"2ce0dac4-27ff-49e0-8b04-cc274dcc4170\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fnbsw" Jul 14 09:30:57.947628 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:57.947593 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2ce0dac4-27ff-49e0-8b04-cc274dcc4170-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-fnbsw\" (UID: \"2ce0dac4-27ff-49e0-8b04-cc274dcc4170\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fnbsw" Jul 14 09:30:57.949238 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:57.949212 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmdgv\" (UniqueName: \"kubernetes.io/projected/2ce0dac4-27ff-49e0-8b04-cc274dcc4170-kube-api-access-zmdgv\") pod \"cert-manager-cainjector-559cdd8c87-fnbsw\" (UID: \"2ce0dac4-27ff-49e0-8b04-cc274dcc4170\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fnbsw" Jul 14 09:30:58.089095 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:58.089005 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-fnbsw" Jul 14 09:30:58.247235 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:58.247183 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-fnbsw"] Jul 14 09:30:58.249963 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:30:58.249929 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ce0dac4_27ff_49e0_8b04_cc274dcc4170.slice/crio-6be8ac5ccbe3a12a3597b124bfba3c57b54c37ac082f433f036c47a74930e03f WatchSource:0}: Error finding container 6be8ac5ccbe3a12a3597b124bfba3c57b54c37ac082f433f036c47a74930e03f: Status 404 returned error can't find the container with id 6be8ac5ccbe3a12a3597b124bfba3c57b54c37ac082f433f036c47a74930e03f Jul 14 09:30:59.179146 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:30:59.179110 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-fnbsw" event={"ID":"2ce0dac4-27ff-49e0-8b04-cc274dcc4170","Type":"ContainerStarted","Data":"6be8ac5ccbe3a12a3597b124bfba3c57b54c37ac082f433f036c47a74930e03f"} Jul 14 09:31:00.187351 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:00.187313 2565 generic.go:356] "Generic (PLEG): container finished" podID="52757dab-82b1-4040-a8fd-1e60254ec093" containerID="74d1f1975a3cd08a845be3db27a9c115f2e6be3184f13a0d2d1c80423c74cd82" exitCode=0 Jul 14 09:31:00.187799 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:00.187406 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmzm9z" event={"ID":"52757dab-82b1-4040-a8fd-1e60254ec093","Type":"ContainerDied","Data":"74d1f1975a3cd08a845be3db27a9c115f2e6be3184f13a0d2d1c80423c74cd82"} Jul 14 09:31:01.193536 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:01.193505 2565 generic.go:356] "Generic (PLEG): container finished" podID="52757dab-82b1-4040-a8fd-1e60254ec093" containerID="2776e54580b0d370735056842af6dded24599c661cb6acbfd2d5edefcb66db38" exitCode=0 Jul 14 09:31:01.193914 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:01.193586 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmzm9z" event={"ID":"52757dab-82b1-4040-a8fd-1e60254ec093","Type":"ContainerDied","Data":"2776e54580b0d370735056842af6dded24599c661cb6acbfd2d5edefcb66db38"} Jul 14 09:31:02.199246 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:02.199208 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-fnbsw" event={"ID":"2ce0dac4-27ff-49e0-8b04-cc274dcc4170","Type":"ContainerStarted","Data":"f035e8e122b8e684f6c718538b343936312872884ac413cb3e2eb35130d43e60"} Jul 14 09:31:02.218402 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:02.218345 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-fnbsw" podStartSLOduration=2.2728685730000002 podStartE2EDuration="5.218327587s" totalImagesPullingTime="2.945459014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 09:30:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:31:02.216960557 +0000 UTC m=+251.726192031" watchObservedRunningTime="2026-07-14 09:31:02.218327587 +0000 UTC m=+251.727559035" Jul 14 09:31:02.325654 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:02.325626 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmzm9z" Jul 14 09:31:02.474512 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:02.474418 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5mgxx\" (UniqueName: \"kubernetes.io/projected/52757dab-82b1-4040-a8fd-1e60254ec093-kube-api-access-5mgxx\") pod \"52757dab-82b1-4040-a8fd-1e60254ec093\" (UID: \"52757dab-82b1-4040-a8fd-1e60254ec093\") " Jul 14 09:31:02.474512 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:02.474478 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/52757dab-82b1-4040-a8fd-1e60254ec093-util\") pod \"52757dab-82b1-4040-a8fd-1e60254ec093\" (UID: \"52757dab-82b1-4040-a8fd-1e60254ec093\") " Jul 14 09:31:02.474512 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:02.474513 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/52757dab-82b1-4040-a8fd-1e60254ec093-bundle\") pod \"52757dab-82b1-4040-a8fd-1e60254ec093\" (UID: \"52757dab-82b1-4040-a8fd-1e60254ec093\") " Jul 14 09:31:02.474957 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:02.474921 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52757dab-82b1-4040-a8fd-1e60254ec093-bundle" (OuterVolumeSpecName: "bundle") pod "52757dab-82b1-4040-a8fd-1e60254ec093" (UID: "52757dab-82b1-4040-a8fd-1e60254ec093"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 09:31:02.476499 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:02.476474 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52757dab-82b1-4040-a8fd-1e60254ec093-kube-api-access-5mgxx" (OuterVolumeSpecName: "kube-api-access-5mgxx") pod "52757dab-82b1-4040-a8fd-1e60254ec093" (UID: "52757dab-82b1-4040-a8fd-1e60254ec093"). InnerVolumeSpecName "kube-api-access-5mgxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 09:31:02.478786 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:02.478752 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52757dab-82b1-4040-a8fd-1e60254ec093-util" (OuterVolumeSpecName: "util") pod "52757dab-82b1-4040-a8fd-1e60254ec093" (UID: "52757dab-82b1-4040-a8fd-1e60254ec093"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 09:31:02.575635 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:02.575593 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mgxx\" (UniqueName: \"kubernetes.io/projected/52757dab-82b1-4040-a8fd-1e60254ec093-kube-api-access-5mgxx\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:31:02.575862 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:02.575674 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/52757dab-82b1-4040-a8fd-1e60254ec093-util\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:31:02.575862 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:02.575700 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/52757dab-82b1-4040-a8fd-1e60254ec093-bundle\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:31:03.121642 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:03.121608 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-vvlj9"] Jul 14 09:31:03.121957 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:03.121945 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52757dab-82b1-4040-a8fd-1e60254ec093" containerName="pull" Jul 14 09:31:03.122007 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:03.121959 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="52757dab-82b1-4040-a8fd-1e60254ec093" containerName="pull" Jul 14 09:31:03.122007 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:03.121970 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52757dab-82b1-4040-a8fd-1e60254ec093" containerName="util" Jul 14 09:31:03.122007 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:03.121976 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="52757dab-82b1-4040-a8fd-1e60254ec093" containerName="util" Jul 14 09:31:03.122007 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:03.121987 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52757dab-82b1-4040-a8fd-1e60254ec093" containerName="extract" Jul 14 09:31:03.122007 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:03.121993 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="52757dab-82b1-4040-a8fd-1e60254ec093" containerName="extract" Jul 14 09:31:03.122194 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:03.122043 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="52757dab-82b1-4040-a8fd-1e60254ec093" containerName="extract" Jul 14 09:31:03.124259 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:03.124236 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-vvlj9" Jul 14 09:31:03.126926 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:03.126905 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-fvg9q\"" Jul 14 09:31:03.150172 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:03.150145 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-vvlj9"] Jul 14 09:31:03.180476 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:03.180439 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5836f4c7-f419-4222-aa36-fc7a5decea1e-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-vvlj9\" (UID: \"5836f4c7-f419-4222-aa36-fc7a5decea1e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-vvlj9" Jul 14 09:31:03.180629 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:03.180504 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjd6m\" (UniqueName: \"kubernetes.io/projected/5836f4c7-f419-4222-aa36-fc7a5decea1e-kube-api-access-gjd6m\") pod \"cert-manager-webhook-6486ffd997-vvlj9\" (UID: \"5836f4c7-f419-4222-aa36-fc7a5decea1e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-vvlj9" Jul 14 09:31:03.204953 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:03.204928 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmzm9z" Jul 14 09:31:03.205333 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:03.204930 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmzm9z" event={"ID":"52757dab-82b1-4040-a8fd-1e60254ec093","Type":"ContainerDied","Data":"29ef3816baebaa42dc1d824d88cfa00a84439671a7cee37d14e51624e2e31077"} Jul 14 09:31:03.205333 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:03.205048 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29ef3816baebaa42dc1d824d88cfa00a84439671a7cee37d14e51624e2e31077" Jul 14 09:31:03.281582 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:03.281488 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5836f4c7-f419-4222-aa36-fc7a5decea1e-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-vvlj9\" (UID: \"5836f4c7-f419-4222-aa36-fc7a5decea1e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-vvlj9" Jul 14 09:31:03.281582 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:03.281549 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjd6m\" (UniqueName: \"kubernetes.io/projected/5836f4c7-f419-4222-aa36-fc7a5decea1e-kube-api-access-gjd6m\") pod \"cert-manager-webhook-6486ffd997-vvlj9\" (UID: \"5836f4c7-f419-4222-aa36-fc7a5decea1e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-vvlj9" Jul 14 09:31:03.290725 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:03.290690 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5836f4c7-f419-4222-aa36-fc7a5decea1e-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-vvlj9\" (UID: \"5836f4c7-f419-4222-aa36-fc7a5decea1e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-vvlj9" Jul 14 09:31:03.290883 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:03.290865 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjd6m\" (UniqueName: \"kubernetes.io/projected/5836f4c7-f419-4222-aa36-fc7a5decea1e-kube-api-access-gjd6m\") pod \"cert-manager-webhook-6486ffd997-vvlj9\" (UID: \"5836f4c7-f419-4222-aa36-fc7a5decea1e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-vvlj9" Jul 14 09:31:03.433109 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:03.433077 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-vvlj9" Jul 14 09:31:03.560584 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:03.560503 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-vvlj9"] Jul 14 09:31:03.563506 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:31:03.563468 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5836f4c7_f419_4222_aa36_fc7a5decea1e.slice/crio-3c587e41b176fca2a91171dc0075344cf949223164a515f41d2874f2cc14796f WatchSource:0}: Error finding container 3c587e41b176fca2a91171dc0075344cf949223164a515f41d2874f2cc14796f: Status 404 returned error can't find the container with id 3c587e41b176fca2a91171dc0075344cf949223164a515f41d2874f2cc14796f Jul 14 09:31:04.211273 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:04.211241 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-vvlj9" event={"ID":"5836f4c7-f419-4222-aa36-fc7a5decea1e","Type":"ContainerStarted","Data":"e863d8af8532054756f601e5fc24249b71ef5d3979bcd027a53633560c54f96b"} Jul 14 09:31:04.211652 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:04.211283 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-vvlj9" event={"ID":"5836f4c7-f419-4222-aa36-fc7a5decea1e","Type":"ContainerStarted","Data":"3c587e41b176fca2a91171dc0075344cf949223164a515f41d2874f2cc14796f"} Jul 14 09:31:04.211652 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:04.211309 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-vvlj9" Jul 14 09:31:04.230497 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:04.230441 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-vvlj9" podStartSLOduration=1.230423436 podStartE2EDuration="1.230423436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 09:31:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:31:04.229430771 +0000 UTC m=+253.738662219" watchObservedRunningTime="2026-07-14 09:31:04.230423436 +0000 UTC m=+253.739654883" Jul 14 09:31:06.236671 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:06.236640 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-4h2jb"] Jul 14 09:31:06.243894 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:06.243874 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-4h2jb" Jul 14 09:31:06.246785 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:06.246766 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-9mz2z\"" Jul 14 09:31:06.253461 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:06.253441 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-4h2jb"] Jul 14 09:31:06.306672 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:06.306636 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9983f211-f6cf-41a4-9165-ba08d1323444-bound-sa-token\") pod \"cert-manager-55b4d954d5-4h2jb\" (UID: \"9983f211-f6cf-41a4-9165-ba08d1323444\") " pod="cert-manager/cert-manager-55b4d954d5-4h2jb" Jul 14 09:31:06.306908 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:06.306699 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbmj9\" (UniqueName: \"kubernetes.io/projected/9983f211-f6cf-41a4-9165-ba08d1323444-kube-api-access-cbmj9\") pod \"cert-manager-55b4d954d5-4h2jb\" (UID: \"9983f211-f6cf-41a4-9165-ba08d1323444\") " pod="cert-manager/cert-manager-55b4d954d5-4h2jb" Jul 14 09:31:06.407991 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:06.407954 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbmj9\" (UniqueName: \"kubernetes.io/projected/9983f211-f6cf-41a4-9165-ba08d1323444-kube-api-access-cbmj9\") pod \"cert-manager-55b4d954d5-4h2jb\" (UID: \"9983f211-f6cf-41a4-9165-ba08d1323444\") " pod="cert-manager/cert-manager-55b4d954d5-4h2jb" Jul 14 09:31:06.408126 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:06.408045 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9983f211-f6cf-41a4-9165-ba08d1323444-bound-sa-token\") pod \"cert-manager-55b4d954d5-4h2jb\" (UID: \"9983f211-f6cf-41a4-9165-ba08d1323444\") " pod="cert-manager/cert-manager-55b4d954d5-4h2jb" Jul 14 09:31:06.421501 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:06.421474 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbmj9\" (UniqueName: \"kubernetes.io/projected/9983f211-f6cf-41a4-9165-ba08d1323444-kube-api-access-cbmj9\") pod \"cert-manager-55b4d954d5-4h2jb\" (UID: \"9983f211-f6cf-41a4-9165-ba08d1323444\") " pod="cert-manager/cert-manager-55b4d954d5-4h2jb" Jul 14 09:31:06.422944 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:06.422917 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9983f211-f6cf-41a4-9165-ba08d1323444-bound-sa-token\") pod \"cert-manager-55b4d954d5-4h2jb\" (UID: \"9983f211-f6cf-41a4-9165-ba08d1323444\") " pod="cert-manager/cert-manager-55b4d954d5-4h2jb" Jul 14 09:31:06.553970 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:06.553881 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-4h2jb" Jul 14 09:31:06.687977 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:06.687932 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-4h2jb"] Jul 14 09:31:06.690677 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:31:06.690650 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9983f211_f6cf_41a4_9165_ba08d1323444.slice/crio-4c9fcbd68ac4528c65a57695563cfb9d2871aa279744cb5663a7867bcbac6b48 WatchSource:0}: Error finding container 4c9fcbd68ac4528c65a57695563cfb9d2871aa279744cb5663a7867bcbac6b48: Status 404 returned error can't find the container with id 4c9fcbd68ac4528c65a57695563cfb9d2871aa279744cb5663a7867bcbac6b48 Jul 14 09:31:07.225503 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:07.225466 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-4h2jb" event={"ID":"9983f211-f6cf-41a4-9165-ba08d1323444","Type":"ContainerStarted","Data":"ac31a3cb2c8893d264ffc48510d6c8f95514efc0a478a371c90989e427b7c3cf"} Jul 14 09:31:07.225503 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:07.225510 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-4h2jb" event={"ID":"9983f211-f6cf-41a4-9165-ba08d1323444","Type":"ContainerStarted","Data":"4c9fcbd68ac4528c65a57695563cfb9d2871aa279744cb5663a7867bcbac6b48"} Jul 14 09:31:07.243565 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:07.243511 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-4h2jb" podStartSLOduration=1.243496241 podStartE2EDuration="1.243496241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 09:31:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:31:07.241497913 +0000 UTC m=+256.750729362" watchObservedRunningTime="2026-07-14 09:31:07.243496241 +0000 UTC m=+256.752727691" Jul 14 09:31:09.937760 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:09.937726 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xcrns"] Jul 14 09:31:09.941423 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:09.941401 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xcrns" Jul 14 09:31:09.945053 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:09.945034 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 14 09:31:09.946238 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:09.946224 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 14 09:31:09.946299 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:09.946258 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-fcm4z\"" Jul 14 09:31:09.959588 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:09.959563 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xcrns"] Jul 14 09:31:10.040649 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:10.040617 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thv75\" (UniqueName: \"kubernetes.io/projected/80022556-10a2-464a-8b8a-a1798cb32ae9-kube-api-access-thv75\") pod \"openshift-lws-operator-fd8ccff4c-xcrns\" (UID: \"80022556-10a2-464a-8b8a-a1798cb32ae9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xcrns" Jul 14 09:31:10.040840 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:10.040675 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80022556-10a2-464a-8b8a-a1798cb32ae9-tmp\") pod \"openshift-lws-operator-fd8ccff4c-xcrns\" (UID: \"80022556-10a2-464a-8b8a-a1798cb32ae9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xcrns" Jul 14 09:31:10.141988 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:10.141953 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thv75\" (UniqueName: \"kubernetes.io/projected/80022556-10a2-464a-8b8a-a1798cb32ae9-kube-api-access-thv75\") pod \"openshift-lws-operator-fd8ccff4c-xcrns\" (UID: \"80022556-10a2-464a-8b8a-a1798cb32ae9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xcrns" Jul 14 09:31:10.142143 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:10.142013 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80022556-10a2-464a-8b8a-a1798cb32ae9-tmp\") pod \"openshift-lws-operator-fd8ccff4c-xcrns\" (UID: \"80022556-10a2-464a-8b8a-a1798cb32ae9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xcrns" Jul 14 09:31:10.142420 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:10.142404 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80022556-10a2-464a-8b8a-a1798cb32ae9-tmp\") pod \"openshift-lws-operator-fd8ccff4c-xcrns\" (UID: \"80022556-10a2-464a-8b8a-a1798cb32ae9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xcrns" Jul 14 09:31:10.151581 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:10.151549 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thv75\" (UniqueName: \"kubernetes.io/projected/80022556-10a2-464a-8b8a-a1798cb32ae9-kube-api-access-thv75\") pod \"openshift-lws-operator-fd8ccff4c-xcrns\" (UID: \"80022556-10a2-464a-8b8a-a1798cb32ae9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xcrns" Jul 14 09:31:10.218421 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:10.218338 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-vvlj9" Jul 14 09:31:10.251095 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:10.251065 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xcrns" Jul 14 09:31:10.396967 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:10.396944 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xcrns"] Jul 14 09:31:10.399182 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:31:10.399154 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod80022556_10a2_464a_8b8a_a1798cb32ae9.slice/crio-71e0b5f404846ca2b8afea4c0f271bc5fd1d9de3bf36d4c94e7e1392c07b801c WatchSource:0}: Error finding container 71e0b5f404846ca2b8afea4c0f271bc5fd1d9de3bf36d4c94e7e1392c07b801c: Status 404 returned error can't find the container with id 71e0b5f404846ca2b8afea4c0f271bc5fd1d9de3bf36d4c94e7e1392c07b801c Jul 14 09:31:11.248449 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:11.248406 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xcrns" event={"ID":"80022556-10a2-464a-8b8a-a1798cb32ae9","Type":"ContainerStarted","Data":"71e0b5f404846ca2b8afea4c0f271bc5fd1d9de3bf36d4c94e7e1392c07b801c"} Jul 14 09:31:13.259196 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:13.259163 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xcrns" event={"ID":"80022556-10a2-464a-8b8a-a1798cb32ae9","Type":"ContainerStarted","Data":"bb89de7f4883443eaf0346dfefe8c0ab05549b49fd481e3c3dd6e3e8e1dc2c74"} Jul 14 09:31:13.276433 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:13.276364 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xcrns" podStartSLOduration=2.361012137 podStartE2EDuration="4.276346819s" totalImagesPullingTime="1.915334682s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 09:31:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:31:13.275241198 +0000 UTC m=+262.784472671" watchObservedRunningTime="2026-07-14 09:31:13.276346819 +0000 UTC m=+262.785578267" Jul 14 09:31:16.030220 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:16.030188 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qlzz"] Jul 14 09:31:16.033003 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:16.032982 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qlzz" Jul 14 09:31:16.035769 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:16.035749 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 14 09:31:16.035894 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:16.035784 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-nqd2l\"" Jul 14 09:31:16.036714 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:16.036698 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 14 09:31:16.043754 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:16.043722 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qlzz"] Jul 14 09:31:16.196058 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:16.196005 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fq2v\" (UniqueName: \"kubernetes.io/projected/614db725-aec8-40d9-b425-8adf0949255f-kube-api-access-6fq2v\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qlzz\" (UID: \"614db725-aec8-40d9-b425-8adf0949255f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qlzz" Jul 14 09:31:16.196251 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:16.196115 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/614db725-aec8-40d9-b425-8adf0949255f-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qlzz\" (UID: \"614db725-aec8-40d9-b425-8adf0949255f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qlzz" Jul 14 09:31:16.196251 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:16.196220 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/614db725-aec8-40d9-b425-8adf0949255f-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qlzz\" (UID: \"614db725-aec8-40d9-b425-8adf0949255f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qlzz" Jul 14 09:31:16.297135 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:16.297041 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/614db725-aec8-40d9-b425-8adf0949255f-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qlzz\" (UID: \"614db725-aec8-40d9-b425-8adf0949255f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qlzz" Jul 14 09:31:16.297135 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:16.297105 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fq2v\" (UniqueName: \"kubernetes.io/projected/614db725-aec8-40d9-b425-8adf0949255f-kube-api-access-6fq2v\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qlzz\" (UID: \"614db725-aec8-40d9-b425-8adf0949255f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qlzz" Jul 14 09:31:16.297135 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:16.297134 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/614db725-aec8-40d9-b425-8adf0949255f-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qlzz\" (UID: \"614db725-aec8-40d9-b425-8adf0949255f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qlzz" Jul 14 09:31:16.297441 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:16.297423 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/614db725-aec8-40d9-b425-8adf0949255f-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qlzz\" (UID: \"614db725-aec8-40d9-b425-8adf0949255f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qlzz" Jul 14 09:31:16.297494 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:16.297474 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/614db725-aec8-40d9-b425-8adf0949255f-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qlzz\" (UID: \"614db725-aec8-40d9-b425-8adf0949255f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qlzz" Jul 14 09:31:16.305572 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:16.305540 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fq2v\" (UniqueName: \"kubernetes.io/projected/614db725-aec8-40d9-b425-8adf0949255f-kube-api-access-6fq2v\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qlzz\" (UID: \"614db725-aec8-40d9-b425-8adf0949255f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qlzz" Jul 14 09:31:16.342445 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:16.342411 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qlzz" Jul 14 09:31:16.473766 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:16.473700 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qlzz"] Jul 14 09:31:16.476327 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:31:16.476300 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod614db725_aec8_40d9_b425_8adf0949255f.slice/crio-93cded408c49f249acee5d9f7c52f3c4e9530d0dfa9dfc1a34be8f1a824f72ae WatchSource:0}: Error finding container 93cded408c49f249acee5d9f7c52f3c4e9530d0dfa9dfc1a34be8f1a824f72ae: Status 404 returned error can't find the container with id 93cded408c49f249acee5d9f7c52f3c4e9530d0dfa9dfc1a34be8f1a824f72ae Jul 14 09:31:17.278982 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:17.278947 2565 generic.go:356] "Generic (PLEG): container finished" podID="614db725-aec8-40d9-b425-8adf0949255f" containerID="c3490a3e4dce500e732ac01199b298f0efac8a912325c4370ffe8509bc5af0a0" exitCode=0 Jul 14 09:31:17.279417 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:17.279023 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qlzz" event={"ID":"614db725-aec8-40d9-b425-8adf0949255f","Type":"ContainerDied","Data":"c3490a3e4dce500e732ac01199b298f0efac8a912325c4370ffe8509bc5af0a0"} Jul 14 09:31:17.279417 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:17.279062 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qlzz" event={"ID":"614db725-aec8-40d9-b425-8adf0949255f","Type":"ContainerStarted","Data":"93cded408c49f249acee5d9f7c52f3c4e9530d0dfa9dfc1a34be8f1a824f72ae"} Jul 14 09:31:18.285837 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:18.285730 2565 generic.go:356] "Generic (PLEG): container finished" podID="614db725-aec8-40d9-b425-8adf0949255f" containerID="e2ff9eb2d92fb27033a98ca8dd1672e5c6755bbaeffdb103cf095a7bf422f61b" exitCode=0 Jul 14 09:31:18.286176 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:18.285841 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qlzz" event={"ID":"614db725-aec8-40d9-b425-8adf0949255f","Type":"ContainerDied","Data":"e2ff9eb2d92fb27033a98ca8dd1672e5c6755bbaeffdb103cf095a7bf422f61b"} Jul 14 09:31:19.293120 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:19.293081 2565 generic.go:356] "Generic (PLEG): container finished" podID="614db725-aec8-40d9-b425-8adf0949255f" containerID="77239ded98df862f0737a50a9ad0e66904d24d4a09b19ce03ec25741f429990a" exitCode=0 Jul 14 09:31:19.293489 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:19.293171 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qlzz" event={"ID":"614db725-aec8-40d9-b425-8adf0949255f","Type":"ContainerDied","Data":"77239ded98df862f0737a50a9ad0e66904d24d4a09b19ce03ec25741f429990a"} Jul 14 09:31:20.421785 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:20.421758 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qlzz" Jul 14 09:31:20.535044 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:20.535014 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/614db725-aec8-40d9-b425-8adf0949255f-util\") pod \"614db725-aec8-40d9-b425-8adf0949255f\" (UID: \"614db725-aec8-40d9-b425-8adf0949255f\") " Jul 14 09:31:20.535227 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:20.535106 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6fq2v\" (UniqueName: \"kubernetes.io/projected/614db725-aec8-40d9-b425-8adf0949255f-kube-api-access-6fq2v\") pod \"614db725-aec8-40d9-b425-8adf0949255f\" (UID: \"614db725-aec8-40d9-b425-8adf0949255f\") " Jul 14 09:31:20.535227 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:20.535170 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/614db725-aec8-40d9-b425-8adf0949255f-bundle\") pod \"614db725-aec8-40d9-b425-8adf0949255f\" (UID: \"614db725-aec8-40d9-b425-8adf0949255f\") " Jul 14 09:31:20.535884 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:20.535845 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/614db725-aec8-40d9-b425-8adf0949255f-bundle" (OuterVolumeSpecName: "bundle") pod "614db725-aec8-40d9-b425-8adf0949255f" (UID: "614db725-aec8-40d9-b425-8adf0949255f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 09:31:20.537227 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:20.537196 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/614db725-aec8-40d9-b425-8adf0949255f-kube-api-access-6fq2v" (OuterVolumeSpecName: "kube-api-access-6fq2v") pod "614db725-aec8-40d9-b425-8adf0949255f" (UID: "614db725-aec8-40d9-b425-8adf0949255f"). InnerVolumeSpecName "kube-api-access-6fq2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 09:31:20.540469 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:20.540450 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/614db725-aec8-40d9-b425-8adf0949255f-util" (OuterVolumeSpecName: "util") pod "614db725-aec8-40d9-b425-8adf0949255f" (UID: "614db725-aec8-40d9-b425-8adf0949255f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 09:31:20.636286 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:20.636256 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/614db725-aec8-40d9-b425-8adf0949255f-util\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:31:20.636286 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:20.636286 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fq2v\" (UniqueName: \"kubernetes.io/projected/614db725-aec8-40d9-b425-8adf0949255f-kube-api-access-6fq2v\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:31:20.636456 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:20.636296 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/614db725-aec8-40d9-b425-8adf0949255f-bundle\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:31:21.303781 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:21.303703 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qlzz" event={"ID":"614db725-aec8-40d9-b425-8adf0949255f","Type":"ContainerDied","Data":"93cded408c49f249acee5d9f7c52f3c4e9530d0dfa9dfc1a34be8f1a824f72ae"} Jul 14 09:31:21.303781 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:21.303752 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93cded408c49f249acee5d9f7c52f3c4e9530d0dfa9dfc1a34be8f1a824f72ae" Jul 14 09:31:21.303781 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:21.303709 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qlzz" Jul 14 09:31:33.150599 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:33.150558 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-644f4b6bc9-9r548"] Jul 14 09:31:33.151063 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:33.150930 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="614db725-aec8-40d9-b425-8adf0949255f" containerName="util" Jul 14 09:31:33.151063 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:33.150941 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="614db725-aec8-40d9-b425-8adf0949255f" containerName="util" Jul 14 09:31:33.151063 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:33.150948 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="614db725-aec8-40d9-b425-8adf0949255f" containerName="extract" Jul 14 09:31:33.151063 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:33.150955 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="614db725-aec8-40d9-b425-8adf0949255f" containerName="extract" Jul 14 09:31:33.151063 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:33.150971 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="614db725-aec8-40d9-b425-8adf0949255f" containerName="pull" Jul 14 09:31:33.151063 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:33.150978 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="614db725-aec8-40d9-b425-8adf0949255f" containerName="pull" Jul 14 09:31:33.151063 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:33.151034 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="614db725-aec8-40d9-b425-8adf0949255f" containerName="extract" Jul 14 09:31:33.155340 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:33.155323 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-644f4b6bc9-9r548" Jul 14 09:31:33.158123 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:33.158097 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 14 09:31:33.158280 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:33.158196 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 14 09:31:33.160111 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:33.160089 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 14 09:31:33.160235 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:33.160182 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 14 09:31:33.160475 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:33.160455 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-vjhmj\"" Jul 14 09:31:33.171152 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:33.171131 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-644f4b6bc9-9r548"] Jul 14 09:31:33.248442 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:33.248411 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/59b87710-b331-4ad8-bc72-bc9f35c2d4f5-manifests\") pod \"opendatahub-operator-controller-manager-644f4b6bc9-9r548\" (UID: \"59b87710-b331-4ad8-bc72-bc9f35c2d4f5\") " pod="opendatahub/opendatahub-operator-controller-manager-644f4b6bc9-9r548" Jul 14 09:31:33.248442 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:33.248443 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/59b87710-b331-4ad8-bc72-bc9f35c2d4f5-webhook-cert\") pod \"opendatahub-operator-controller-manager-644f4b6bc9-9r548\" (UID: \"59b87710-b331-4ad8-bc72-bc9f35c2d4f5\") " pod="opendatahub/opendatahub-operator-controller-manager-644f4b6bc9-9r548" Jul 14 09:31:33.248679 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:33.248466 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqjvb\" (UniqueName: \"kubernetes.io/projected/59b87710-b331-4ad8-bc72-bc9f35c2d4f5-kube-api-access-fqjvb\") pod \"opendatahub-operator-controller-manager-644f4b6bc9-9r548\" (UID: \"59b87710-b331-4ad8-bc72-bc9f35c2d4f5\") " pod="opendatahub/opendatahub-operator-controller-manager-644f4b6bc9-9r548" Jul 14 09:31:33.248679 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:33.248517 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59b87710-b331-4ad8-bc72-bc9f35c2d4f5-tmp\") pod \"opendatahub-operator-controller-manager-644f4b6bc9-9r548\" (UID: \"59b87710-b331-4ad8-bc72-bc9f35c2d4f5\") " pod="opendatahub/opendatahub-operator-controller-manager-644f4b6bc9-9r548" Jul 14 09:31:33.248679 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:33.248619 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/59b87710-b331-4ad8-bc72-bc9f35c2d4f5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-644f4b6bc9-9r548\" (UID: \"59b87710-b331-4ad8-bc72-bc9f35c2d4f5\") " pod="opendatahub/opendatahub-operator-controller-manager-644f4b6bc9-9r548" Jul 14 09:31:33.349974 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:33.349946 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/59b87710-b331-4ad8-bc72-bc9f35c2d4f5-manifests\") pod \"opendatahub-operator-controller-manager-644f4b6bc9-9r548\" (UID: \"59b87710-b331-4ad8-bc72-bc9f35c2d4f5\") " pod="opendatahub/opendatahub-operator-controller-manager-644f4b6bc9-9r548" Jul 14 09:31:33.349974 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:33.349977 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/59b87710-b331-4ad8-bc72-bc9f35c2d4f5-webhook-cert\") pod \"opendatahub-operator-controller-manager-644f4b6bc9-9r548\" (UID: \"59b87710-b331-4ad8-bc72-bc9f35c2d4f5\") " pod="opendatahub/opendatahub-operator-controller-manager-644f4b6bc9-9r548" Jul 14 09:31:33.350233 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:33.349998 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqjvb\" (UniqueName: \"kubernetes.io/projected/59b87710-b331-4ad8-bc72-bc9f35c2d4f5-kube-api-access-fqjvb\") pod \"opendatahub-operator-controller-manager-644f4b6bc9-9r548\" (UID: \"59b87710-b331-4ad8-bc72-bc9f35c2d4f5\") " pod="opendatahub/opendatahub-operator-controller-manager-644f4b6bc9-9r548" Jul 14 09:31:33.350233 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:33.350138 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59b87710-b331-4ad8-bc72-bc9f35c2d4f5-tmp\") pod \"opendatahub-operator-controller-manager-644f4b6bc9-9r548\" (UID: \"59b87710-b331-4ad8-bc72-bc9f35c2d4f5\") " pod="opendatahub/opendatahub-operator-controller-manager-644f4b6bc9-9r548" Jul 14 09:31:33.350349 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:33.350248 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/59b87710-b331-4ad8-bc72-bc9f35c2d4f5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-644f4b6bc9-9r548\" (UID: \"59b87710-b331-4ad8-bc72-bc9f35c2d4f5\") " pod="opendatahub/opendatahub-operator-controller-manager-644f4b6bc9-9r548" Jul 14 09:31:33.350445 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:33.350425 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/59b87710-b331-4ad8-bc72-bc9f35c2d4f5-manifests\") pod \"opendatahub-operator-controller-manager-644f4b6bc9-9r548\" (UID: \"59b87710-b331-4ad8-bc72-bc9f35c2d4f5\") " pod="opendatahub/opendatahub-operator-controller-manager-644f4b6bc9-9r548" Jul 14 09:31:33.352541 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:33.352518 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/59b87710-b331-4ad8-bc72-bc9f35c2d4f5-webhook-cert\") pod \"opendatahub-operator-controller-manager-644f4b6bc9-9r548\" (UID: \"59b87710-b331-4ad8-bc72-bc9f35c2d4f5\") " pod="opendatahub/opendatahub-operator-controller-manager-644f4b6bc9-9r548" Jul 14 09:31:33.352676 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:33.352656 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59b87710-b331-4ad8-bc72-bc9f35c2d4f5-tmp\") pod \"opendatahub-operator-controller-manager-644f4b6bc9-9r548\" (UID: \"59b87710-b331-4ad8-bc72-bc9f35c2d4f5\") " pod="opendatahub/opendatahub-operator-controller-manager-644f4b6bc9-9r548" Jul 14 09:31:33.352725 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:33.352656 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/59b87710-b331-4ad8-bc72-bc9f35c2d4f5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-644f4b6bc9-9r548\" (UID: \"59b87710-b331-4ad8-bc72-bc9f35c2d4f5\") " pod="opendatahub/opendatahub-operator-controller-manager-644f4b6bc9-9r548" Jul 14 09:31:33.367503 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:33.367476 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqjvb\" (UniqueName: \"kubernetes.io/projected/59b87710-b331-4ad8-bc72-bc9f35c2d4f5-kube-api-access-fqjvb\") pod \"opendatahub-operator-controller-manager-644f4b6bc9-9r548\" (UID: \"59b87710-b331-4ad8-bc72-bc9f35c2d4f5\") " pod="opendatahub/opendatahub-operator-controller-manager-644f4b6bc9-9r548" Jul 14 09:31:33.466837 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:33.466727 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-644f4b6bc9-9r548" Jul 14 09:31:33.605102 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:33.605065 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-644f4b6bc9-9r548"] Jul 14 09:31:33.608470 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:31:33.608446 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59b87710_b331_4ad8_bc72_bc9f35c2d4f5.slice/crio-92cc78e00c90b4effd7f298452fab18c4cbc38ad93d6272b10de811fba98be8d WatchSource:0}: Error finding container 92cc78e00c90b4effd7f298452fab18c4cbc38ad93d6272b10de811fba98be8d: Status 404 returned error can't find the container with id 92cc78e00c90b4effd7f298452fab18c4cbc38ad93d6272b10de811fba98be8d Jul 14 09:31:34.364477 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:34.364435 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-644f4b6bc9-9r548" event={"ID":"59b87710-b331-4ad8-bc72-bc9f35c2d4f5","Type":"ContainerStarted","Data":"92cc78e00c90b4effd7f298452fab18c4cbc38ad93d6272b10de811fba98be8d"} Jul 14 09:31:37.382977 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:37.382924 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-644f4b6bc9-9r548" event={"ID":"59b87710-b331-4ad8-bc72-bc9f35c2d4f5","Type":"ContainerStarted","Data":"edbefbee6a496ea5f9de229077201a99d3256eaa6e0b9238df9cd3099041aca2"} Jul 14 09:31:38.389220 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:38.389185 2565 generic.go:356] "Generic (PLEG): container finished" podID="59b87710-b331-4ad8-bc72-bc9f35c2d4f5" containerID="edbefbee6a496ea5f9de229077201a99d3256eaa6e0b9238df9cd3099041aca2" exitCode=0 Jul 14 09:31:38.389595 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:38.389238 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-644f4b6bc9-9r548" event={"ID":"59b87710-b331-4ad8-bc72-bc9f35c2d4f5","Type":"ContainerDied","Data":"edbefbee6a496ea5f9de229077201a99d3256eaa6e0b9238df9cd3099041aca2"} Jul 14 09:31:39.398967 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:39.398933 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-644f4b6bc9-9r548" event={"ID":"59b87710-b331-4ad8-bc72-bc9f35c2d4f5","Type":"ContainerStarted","Data":"4bd831d6ca37a8aa62115ec4f0b7b82f90d0d2d6e803ecb9c40b1ae3ac6c0ebd"} Jul 14 09:31:39.399357 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:39.399055 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-644f4b6bc9-9r548" Jul 14 09:31:39.441555 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:39.441499 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-644f4b6bc9-9r548" podStartSLOduration=2.09254835 podStartE2EDuration="6.441481582s" totalImagesPullingTime="3.284773972s" totalInitContainerRuntime="1.06415926s" isStatefulPod=false podCreationTimestamp="2026-07-14 09:31:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:31:39.431565424 +0000 UTC m=+288.940796871" watchObservedRunningTime="2026-07-14 09:31:39.441481582 +0000 UTC m=+288.950713029" Jul 14 09:31:50.407919 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:50.407886 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-644f4b6bc9-9r548" Jul 14 09:31:50.852468 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:50.852397 2565 kubelet.go:1648] "Image garbage collection succeeded" Jul 14 09:31:54.476909 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:54.476876 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-b5d5m"] Jul 14 09:31:54.480014 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:54.479996 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b5d5m" Jul 14 09:31:54.484486 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:54.484463 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 14 09:31:54.484486 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:54.484501 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 14 09:31:54.484694 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:54.484667 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 14 09:31:54.484694 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:54.484678 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-dz5h5\"" Jul 14 09:31:54.484768 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:54.484700 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 14 09:31:54.484821 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:54.484763 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 14 09:31:54.484959 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:54.484947 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 14 09:31:54.494370 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:54.494346 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-b5d5m"] Jul 14 09:31:54.639099 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:54.639066 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddd5cc17-497b-4938-bdd5-eff7052e964a-istio-token\") pod \"istiod-openshift-gateway-7574459656-b5d5m\" (UID: \"ddd5cc17-497b-4938-bdd5-eff7052e964a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b5d5m" Jul 14 09:31:54.639274 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:54.639106 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddd5cc17-497b-4938-bdd5-eff7052e964a-local-certs\") pod \"istiod-openshift-gateway-7574459656-b5d5m\" (UID: \"ddd5cc17-497b-4938-bdd5-eff7052e964a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b5d5m" Jul 14 09:31:54.639274 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:54.639128 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ddd5cc17-497b-4938-bdd5-eff7052e964a-cacerts\") pod \"istiod-openshift-gateway-7574459656-b5d5m\" (UID: \"ddd5cc17-497b-4938-bdd5-eff7052e964a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b5d5m" Jul 14 09:31:54.639274 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:54.639194 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ddd5cc17-497b-4938-bdd5-eff7052e964a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-b5d5m\" (UID: \"ddd5cc17-497b-4938-bdd5-eff7052e964a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b5d5m" Jul 14 09:31:54.639274 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:54.639230 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ddd5cc17-497b-4938-bdd5-eff7052e964a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-b5d5m\" (UID: \"ddd5cc17-497b-4938-bdd5-eff7052e964a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b5d5m" Jul 14 09:31:54.639274 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:54.639254 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxbqn\" (UniqueName: \"kubernetes.io/projected/ddd5cc17-497b-4938-bdd5-eff7052e964a-kube-api-access-fxbqn\") pod \"istiod-openshift-gateway-7574459656-b5d5m\" (UID: \"ddd5cc17-497b-4938-bdd5-eff7052e964a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b5d5m" Jul 14 09:31:54.639274 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:54.639270 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ddd5cc17-497b-4938-bdd5-eff7052e964a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-b5d5m\" (UID: \"ddd5cc17-497b-4938-bdd5-eff7052e964a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b5d5m" Jul 14 09:31:54.740043 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:54.739953 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddd5cc17-497b-4938-bdd5-eff7052e964a-local-certs\") pod \"istiod-openshift-gateway-7574459656-b5d5m\" (UID: \"ddd5cc17-497b-4938-bdd5-eff7052e964a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b5d5m" Jul 14 09:31:54.740043 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:54.739998 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ddd5cc17-497b-4938-bdd5-eff7052e964a-cacerts\") pod \"istiod-openshift-gateway-7574459656-b5d5m\" (UID: \"ddd5cc17-497b-4938-bdd5-eff7052e964a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b5d5m" Jul 14 09:31:54.740043 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:54.740038 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ddd5cc17-497b-4938-bdd5-eff7052e964a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-b5d5m\" (UID: \"ddd5cc17-497b-4938-bdd5-eff7052e964a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b5d5m" Jul 14 09:31:54.740316 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:54.740076 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ddd5cc17-497b-4938-bdd5-eff7052e964a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-b5d5m\" (UID: \"ddd5cc17-497b-4938-bdd5-eff7052e964a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b5d5m" Jul 14 09:31:54.740316 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:54.740209 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxbqn\" (UniqueName: \"kubernetes.io/projected/ddd5cc17-497b-4938-bdd5-eff7052e964a-kube-api-access-fxbqn\") pod \"istiod-openshift-gateway-7574459656-b5d5m\" (UID: \"ddd5cc17-497b-4938-bdd5-eff7052e964a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b5d5m" Jul 14 09:31:54.740316 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:54.740260 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ddd5cc17-497b-4938-bdd5-eff7052e964a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-b5d5m\" (UID: \"ddd5cc17-497b-4938-bdd5-eff7052e964a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b5d5m" Jul 14 09:31:54.740463 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:54.740317 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddd5cc17-497b-4938-bdd5-eff7052e964a-istio-token\") pod \"istiod-openshift-gateway-7574459656-b5d5m\" (UID: \"ddd5cc17-497b-4938-bdd5-eff7052e964a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b5d5m" Jul 14 09:31:54.741012 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:54.740988 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ddd5cc17-497b-4938-bdd5-eff7052e964a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-b5d5m\" (UID: \"ddd5cc17-497b-4938-bdd5-eff7052e964a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b5d5m" Jul 14 09:31:54.742404 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:54.742384 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ddd5cc17-497b-4938-bdd5-eff7052e964a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-b5d5m\" (UID: \"ddd5cc17-497b-4938-bdd5-eff7052e964a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b5d5m" Jul 14 09:31:54.742519 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:54.742451 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ddd5cc17-497b-4938-bdd5-eff7052e964a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-b5d5m\" (UID: \"ddd5cc17-497b-4938-bdd5-eff7052e964a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b5d5m" Jul 14 09:31:54.742644 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:54.742629 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddd5cc17-497b-4938-bdd5-eff7052e964a-local-certs\") pod \"istiod-openshift-gateway-7574459656-b5d5m\" (UID: \"ddd5cc17-497b-4938-bdd5-eff7052e964a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b5d5m" Jul 14 09:31:54.742868 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:54.742849 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ddd5cc17-497b-4938-bdd5-eff7052e964a-cacerts\") pod \"istiod-openshift-gateway-7574459656-b5d5m\" (UID: \"ddd5cc17-497b-4938-bdd5-eff7052e964a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b5d5m" Jul 14 09:31:54.748971 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:54.748950 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxbqn\" (UniqueName: \"kubernetes.io/projected/ddd5cc17-497b-4938-bdd5-eff7052e964a-kube-api-access-fxbqn\") pod \"istiod-openshift-gateway-7574459656-b5d5m\" (UID: \"ddd5cc17-497b-4938-bdd5-eff7052e964a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b5d5m" Jul 14 09:31:54.749050 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:54.749036 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddd5cc17-497b-4938-bdd5-eff7052e964a-istio-token\") pod \"istiod-openshift-gateway-7574459656-b5d5m\" (UID: \"ddd5cc17-497b-4938-bdd5-eff7052e964a\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-b5d5m" Jul 14 09:31:54.789240 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:54.789205 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b5d5m" Jul 14 09:31:54.922125 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:54.922029 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-b5d5m"] Jul 14 09:31:54.925271 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:31:54.925231 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddd5cc17_497b_4938_bdd5_eff7052e964a.slice/crio-5979c05d89c81e431800a414b811b1cd34aff2349797785d2991bf3af30bc535 WatchSource:0}: Error finding container 5979c05d89c81e431800a414b811b1cd34aff2349797785d2991bf3af30bc535: Status 404 returned error can't find the container with id 5979c05d89c81e431800a414b811b1cd34aff2349797785d2991bf3af30bc535 Jul 14 09:31:54.927284 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:54.927267 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 09:31:55.475948 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:55.475917 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b5d5m" event={"ID":"ddd5cc17-497b-4938-bdd5-eff7052e964a","Type":"ContainerStarted","Data":"5979c05d89c81e431800a414b811b1cd34aff2349797785d2991bf3af30bc535"} Jul 14 09:31:57.572854 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:57.572792 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 09:31:57.573149 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:57.572891 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 09:31:58.496006 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:58.495958 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b5d5m" event={"ID":"ddd5cc17-497b-4938-bdd5-eff7052e964a","Type":"ContainerStarted","Data":"dbca7e479085c897e543088f766ada0914c39ed055f9bb59510b7340cff89ea6"} Jul 14 09:31:58.496220 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:58.496142 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b5d5m" Jul 14 09:31:58.497841 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:58.497798 2565 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-b5d5m container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 09:31:58.497943 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:58.497862 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b5d5m" podUID="ddd5cc17-497b-4938-bdd5-eff7052e964a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 09:31:58.544796 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:58.544734 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b5d5m" podStartSLOduration=1.899579369 podStartE2EDuration="4.544714303s" totalImagesPullingTime="2.645134934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 09:31:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:31:58.540015096 +0000 UTC m=+308.049246574" watchObservedRunningTime="2026-07-14 09:31:58.544714303 +0000 UTC m=+308.053945752" Jul 14 09:31:59.502081 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:59.502048 2565 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-b5d5m container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 09:31:59.502467 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:31:59.502104 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b5d5m" podUID="ddd5cc17-497b-4938-bdd5-eff7052e964a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 09:32:02.502879 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:02.502844 2565 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-b5d5m container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 09:32:02.503346 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:02.502926 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b5d5m" podUID="ddd5cc17-497b-4938-bdd5-eff7052e964a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 09:32:02.589195 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:02.589155 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-56954f6f7d-5zgsf"] Jul 14 09:32:02.592956 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:02.592933 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-56954f6f7d-5zgsf" Jul 14 09:32:02.607889 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:02.607866 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 14 09:32:02.609442 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:02.609420 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 14 09:32:02.609588 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:02.609566 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-glml4\"" Jul 14 09:32:02.609693 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:02.609491 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 14 09:32:02.611993 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:02.611971 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-56954f6f7d-5zgsf"] Jul 14 09:32:02.715826 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:02.715777 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7vlg\" (UniqueName: \"kubernetes.io/projected/5cf780ae-cba9-467b-a3b0-32928bbb4c57-kube-api-access-c7vlg\") pod \"lws-controller-manager-56954f6f7d-5zgsf\" (UID: \"5cf780ae-cba9-467b-a3b0-32928bbb4c57\") " pod="openshift-lws-operator/lws-controller-manager-56954f6f7d-5zgsf" Jul 14 09:32:02.716024 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:02.715880 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5cf780ae-cba9-467b-a3b0-32928bbb4c57-cert\") pod \"lws-controller-manager-56954f6f7d-5zgsf\" (UID: \"5cf780ae-cba9-467b-a3b0-32928bbb4c57\") " pod="openshift-lws-operator/lws-controller-manager-56954f6f7d-5zgsf" Jul 14 09:32:02.716024 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:02.715938 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5cf780ae-cba9-467b-a3b0-32928bbb4c57-manager-config\") pod \"lws-controller-manager-56954f6f7d-5zgsf\" (UID: \"5cf780ae-cba9-467b-a3b0-32928bbb4c57\") " pod="openshift-lws-operator/lws-controller-manager-56954f6f7d-5zgsf" Jul 14 09:32:02.716024 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:02.715988 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5cf780ae-cba9-467b-a3b0-32928bbb4c57-metrics-cert\") pod \"lws-controller-manager-56954f6f7d-5zgsf\" (UID: \"5cf780ae-cba9-467b-a3b0-32928bbb4c57\") " pod="openshift-lws-operator/lws-controller-manager-56954f6f7d-5zgsf" Jul 14 09:32:02.817508 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:02.817420 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7vlg\" (UniqueName: \"kubernetes.io/projected/5cf780ae-cba9-467b-a3b0-32928bbb4c57-kube-api-access-c7vlg\") pod \"lws-controller-manager-56954f6f7d-5zgsf\" (UID: \"5cf780ae-cba9-467b-a3b0-32928bbb4c57\") " pod="openshift-lws-operator/lws-controller-manager-56954f6f7d-5zgsf" Jul 14 09:32:02.817508 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:02.817473 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5cf780ae-cba9-467b-a3b0-32928bbb4c57-cert\") pod \"lws-controller-manager-56954f6f7d-5zgsf\" (UID: \"5cf780ae-cba9-467b-a3b0-32928bbb4c57\") " pod="openshift-lws-operator/lws-controller-manager-56954f6f7d-5zgsf" Jul 14 09:32:02.817735 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:02.817515 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5cf780ae-cba9-467b-a3b0-32928bbb4c57-manager-config\") pod \"lws-controller-manager-56954f6f7d-5zgsf\" (UID: \"5cf780ae-cba9-467b-a3b0-32928bbb4c57\") " pod="openshift-lws-operator/lws-controller-manager-56954f6f7d-5zgsf" Jul 14 09:32:02.817735 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:02.817537 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5cf780ae-cba9-467b-a3b0-32928bbb4c57-metrics-cert\") pod \"lws-controller-manager-56954f6f7d-5zgsf\" (UID: \"5cf780ae-cba9-467b-a3b0-32928bbb4c57\") " pod="openshift-lws-operator/lws-controller-manager-56954f6f7d-5zgsf" Jul 14 09:32:02.818312 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:02.818285 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5cf780ae-cba9-467b-a3b0-32928bbb4c57-manager-config\") pod \"lws-controller-manager-56954f6f7d-5zgsf\" (UID: \"5cf780ae-cba9-467b-a3b0-32928bbb4c57\") " pod="openshift-lws-operator/lws-controller-manager-56954f6f7d-5zgsf" Jul 14 09:32:02.819946 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:02.819918 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5cf780ae-cba9-467b-a3b0-32928bbb4c57-cert\") pod \"lws-controller-manager-56954f6f7d-5zgsf\" (UID: \"5cf780ae-cba9-467b-a3b0-32928bbb4c57\") " pod="openshift-lws-operator/lws-controller-manager-56954f6f7d-5zgsf" Jul 14 09:32:02.820033 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:02.820008 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5cf780ae-cba9-467b-a3b0-32928bbb4c57-metrics-cert\") pod \"lws-controller-manager-56954f6f7d-5zgsf\" (UID: \"5cf780ae-cba9-467b-a3b0-32928bbb4c57\") " pod="openshift-lws-operator/lws-controller-manager-56954f6f7d-5zgsf" Jul 14 09:32:02.841483 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:02.841453 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7vlg\" (UniqueName: \"kubernetes.io/projected/5cf780ae-cba9-467b-a3b0-32928bbb4c57-kube-api-access-c7vlg\") pod \"lws-controller-manager-56954f6f7d-5zgsf\" (UID: \"5cf780ae-cba9-467b-a3b0-32928bbb4c57\") " pod="openshift-lws-operator/lws-controller-manager-56954f6f7d-5zgsf" Jul 14 09:32:02.903190 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:02.903149 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-56954f6f7d-5zgsf" Jul 14 09:32:03.041898 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:32:03.041868 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cf780ae_cba9_467b_a3b0_32928bbb4c57.slice/crio-69158b4ceabc58300cc2e0d4d5c962c198182ea5734c8a2f8d2e6950f6ceb3d1 WatchSource:0}: Error finding container 69158b4ceabc58300cc2e0d4d5c962c198182ea5734c8a2f8d2e6950f6ceb3d1: Status 404 returned error can't find the container with id 69158b4ceabc58300cc2e0d4d5c962c198182ea5734c8a2f8d2e6950f6ceb3d1 Jul 14 09:32:03.048790 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:03.048771 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-56954f6f7d-5zgsf"] Jul 14 09:32:03.520853 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:03.520799 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-56954f6f7d-5zgsf" event={"ID":"5cf780ae-cba9-467b-a3b0-32928bbb4c57","Type":"ContainerStarted","Data":"69158b4ceabc58300cc2e0d4d5c962c198182ea5734c8a2f8d2e6950f6ceb3d1"} Jul 14 09:32:05.502570 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:05.502534 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b5d5m" Jul 14 09:32:05.533741 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:05.533704 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-56954f6f7d-5zgsf" event={"ID":"5cf780ae-cba9-467b-a3b0-32928bbb4c57","Type":"ContainerStarted","Data":"2bd7158b958900172df0554f38b051356d50444239f929f4e6f437a09cec8175"} Jul 14 09:32:05.533947 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:05.533841 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-56954f6f7d-5zgsf" Jul 14 09:32:05.622524 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:05.622463 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-56954f6f7d-5zgsf" podStartSLOduration=2.069301323 podStartE2EDuration="3.622439011s" totalImagesPullingTime="1.553137688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 09:32:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:32:05.617710886 +0000 UTC m=+315.126942359" watchObservedRunningTime="2026-07-14 09:32:05.622439011 +0000 UTC m=+315.131670460" Jul 14 09:32:09.383199 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:09.383164 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-b5d5m"] Jul 14 09:32:09.383687 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:09.383382 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b5d5m" podUID="ddd5cc17-497b-4938-bdd5-eff7052e964a" containerName="discovery" containerID="cri-o://dbca7e479085c897e543088f766ada0914c39ed055f9bb59510b7340cff89ea6" gracePeriod=30 Jul 14 09:32:09.554275 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:09.554241 2565 generic.go:356] "Generic (PLEG): container finished" podID="ddd5cc17-497b-4938-bdd5-eff7052e964a" containerID="dbca7e479085c897e543088f766ada0914c39ed055f9bb59510b7340cff89ea6" exitCode=0 Jul 14 09:32:09.554432 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:09.554284 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b5d5m" event={"ID":"ddd5cc17-497b-4938-bdd5-eff7052e964a","Type":"ContainerDied","Data":"dbca7e479085c897e543088f766ada0914c39ed055f9bb59510b7340cff89ea6"} Jul 14 09:32:10.139679 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:10.139654 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b5d5m" Jul 14 09:32:10.181603 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:10.181569 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ddd5cc17-497b-4938-bdd5-eff7052e964a-istio-csr-dns-cert\") pod \"ddd5cc17-497b-4938-bdd5-eff7052e964a\" (UID: \"ddd5cc17-497b-4938-bdd5-eff7052e964a\") " Jul 14 09:32:10.181765 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:10.181629 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ddd5cc17-497b-4938-bdd5-eff7052e964a-cacerts\") pod \"ddd5cc17-497b-4938-bdd5-eff7052e964a\" (UID: \"ddd5cc17-497b-4938-bdd5-eff7052e964a\") " Jul 14 09:32:10.181765 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:10.181697 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddd5cc17-497b-4938-bdd5-eff7052e964a-istio-token\") pod \"ddd5cc17-497b-4938-bdd5-eff7052e964a\" (UID: \"ddd5cc17-497b-4938-bdd5-eff7052e964a\") " Jul 14 09:32:10.181765 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:10.181725 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddd5cc17-497b-4938-bdd5-eff7052e964a-local-certs\") pod \"ddd5cc17-497b-4938-bdd5-eff7052e964a\" (UID: \"ddd5cc17-497b-4938-bdd5-eff7052e964a\") " Jul 14 09:32:10.181946 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:10.181772 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ddd5cc17-497b-4938-bdd5-eff7052e964a-istio-csr-ca-configmap\") pod \"ddd5cc17-497b-4938-bdd5-eff7052e964a\" (UID: \"ddd5cc17-497b-4938-bdd5-eff7052e964a\") " Jul 14 09:32:10.181946 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:10.181797 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fxbqn\" (UniqueName: \"kubernetes.io/projected/ddd5cc17-497b-4938-bdd5-eff7052e964a-kube-api-access-fxbqn\") pod \"ddd5cc17-497b-4938-bdd5-eff7052e964a\" (UID: \"ddd5cc17-497b-4938-bdd5-eff7052e964a\") " Jul 14 09:32:10.181946 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:10.181860 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ddd5cc17-497b-4938-bdd5-eff7052e964a-istio-kubeconfig\") pod \"ddd5cc17-497b-4938-bdd5-eff7052e964a\" (UID: \"ddd5cc17-497b-4938-bdd5-eff7052e964a\") " Jul 14 09:32:10.182470 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:10.182411 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddd5cc17-497b-4938-bdd5-eff7052e964a-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "ddd5cc17-497b-4938-bdd5-eff7052e964a" (UID: "ddd5cc17-497b-4938-bdd5-eff7052e964a"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 09:32:10.184213 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:10.184182 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddd5cc17-497b-4938-bdd5-eff7052e964a-kube-api-access-fxbqn" (OuterVolumeSpecName: "kube-api-access-fxbqn") pod "ddd5cc17-497b-4938-bdd5-eff7052e964a" (UID: "ddd5cc17-497b-4938-bdd5-eff7052e964a"). InnerVolumeSpecName "kube-api-access-fxbqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 09:32:10.184574 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:10.184551 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddd5cc17-497b-4938-bdd5-eff7052e964a-istio-token" (OuterVolumeSpecName: "istio-token") pod "ddd5cc17-497b-4938-bdd5-eff7052e964a" (UID: "ddd5cc17-497b-4938-bdd5-eff7052e964a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 09:32:10.184664 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:10.184591 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddd5cc17-497b-4938-bdd5-eff7052e964a-cacerts" (OuterVolumeSpecName: "cacerts") pod "ddd5cc17-497b-4938-bdd5-eff7052e964a" (UID: "ddd5cc17-497b-4938-bdd5-eff7052e964a"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 09:32:10.184664 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:10.184617 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddd5cc17-497b-4938-bdd5-eff7052e964a-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "ddd5cc17-497b-4938-bdd5-eff7052e964a" (UID: "ddd5cc17-497b-4938-bdd5-eff7052e964a"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 09:32:10.184664 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:10.184633 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddd5cc17-497b-4938-bdd5-eff7052e964a-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "ddd5cc17-497b-4938-bdd5-eff7052e964a" (UID: "ddd5cc17-497b-4938-bdd5-eff7052e964a"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 09:32:10.184664 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:10.184629 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddd5cc17-497b-4938-bdd5-eff7052e964a-local-certs" (OuterVolumeSpecName: "local-certs") pod "ddd5cc17-497b-4938-bdd5-eff7052e964a" (UID: "ddd5cc17-497b-4938-bdd5-eff7052e964a"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 09:32:10.283250 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:10.283159 2565 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ddd5cc17-497b-4938-bdd5-eff7052e964a-istio-csr-ca-configmap\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:32:10.283250 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:10.283186 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxbqn\" (UniqueName: \"kubernetes.io/projected/ddd5cc17-497b-4938-bdd5-eff7052e964a-kube-api-access-fxbqn\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:32:10.283250 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:10.283196 2565 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ddd5cc17-497b-4938-bdd5-eff7052e964a-istio-kubeconfig\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:32:10.283250 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:10.283205 2565 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ddd5cc17-497b-4938-bdd5-eff7052e964a-istio-csr-dns-cert\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:32:10.283250 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:10.283212 2565 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ddd5cc17-497b-4938-bdd5-eff7052e964a-cacerts\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:32:10.283250 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:10.283220 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddd5cc17-497b-4938-bdd5-eff7052e964a-istio-token\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:32:10.283250 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:10.283230 2565 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddd5cc17-497b-4938-bdd5-eff7052e964a-local-certs\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:32:10.560708 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:10.560619 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b5d5m" Jul 14 09:32:10.561182 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:10.560611 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-b5d5m" event={"ID":"ddd5cc17-497b-4938-bdd5-eff7052e964a","Type":"ContainerDied","Data":"5979c05d89c81e431800a414b811b1cd34aff2349797785d2991bf3af30bc535"} Jul 14 09:32:10.561182 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:10.560775 2565 scope.go:117] "RemoveContainer" containerID="dbca7e479085c897e543088f766ada0914c39ed055f9bb59510b7340cff89ea6" Jul 14 09:32:10.647036 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:10.646998 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-b5d5m"] Jul 14 09:32:10.658228 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:10.658194 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-b5d5m"] Jul 14 09:32:10.969215 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:10.969176 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddd5cc17-497b-4938-bdd5-eff7052e964a" path="/var/lib/kubelet/pods/ddd5cc17-497b-4938-bdd5-eff7052e964a/volumes" Jul 14 09:32:16.540310 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:16.540274 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-56954f6f7d-5zgsf" Jul 14 09:32:53.921284 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:53.921244 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-twfkw"] Jul 14 09:32:53.921669 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:53.921622 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddd5cc17-497b-4938-bdd5-eff7052e964a" containerName="discovery" Jul 14 09:32:53.921669 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:53.921633 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddd5cc17-497b-4938-bdd5-eff7052e964a" containerName="discovery" Jul 14 09:32:53.921739 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:53.921704 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddd5cc17-497b-4938-bdd5-eff7052e964a" containerName="discovery" Jul 14 09:32:53.928775 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:53.928747 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-twfkw" Jul 14 09:32:53.935301 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:53.935264 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-twfkw"] Jul 14 09:32:53.937365 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:53.937342 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 14 09:32:53.942870 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:53.942850 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-wxbzd\"" Jul 14 09:32:53.943416 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:53.943398 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 14 09:32:53.972661 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:53.972634 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mlmj\" (UniqueName: \"kubernetes.io/projected/dbfa0afb-3b17-4b47-91f8-03628e4e9b82-kube-api-access-2mlmj\") pod \"kuadrant-operator-catalog-twfkw\" (UID: \"dbfa0afb-3b17-4b47-91f8-03628e4e9b82\") " pod="kuadrant-system/kuadrant-operator-catalog-twfkw" Jul 14 09:32:54.073195 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:54.073163 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mlmj\" (UniqueName: \"kubernetes.io/projected/dbfa0afb-3b17-4b47-91f8-03628e4e9b82-kube-api-access-2mlmj\") pod \"kuadrant-operator-catalog-twfkw\" (UID: \"dbfa0afb-3b17-4b47-91f8-03628e4e9b82\") " pod="kuadrant-system/kuadrant-operator-catalog-twfkw" Jul 14 09:32:54.090691 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:54.090667 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mlmj\" (UniqueName: \"kubernetes.io/projected/dbfa0afb-3b17-4b47-91f8-03628e4e9b82-kube-api-access-2mlmj\") pod \"kuadrant-operator-catalog-twfkw\" (UID: \"dbfa0afb-3b17-4b47-91f8-03628e4e9b82\") " pod="kuadrant-system/kuadrant-operator-catalog-twfkw" Jul 14 09:32:54.239612 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:54.239522 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-twfkw" Jul 14 09:32:54.270776 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:54.270744 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-twfkw"] Jul 14 09:32:54.374234 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:54.374212 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-twfkw"] Jul 14 09:32:54.375987 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:32:54.375950 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbfa0afb_3b17_4b47_91f8_03628e4e9b82.slice/crio-2ec29ef88d480fccee99c991166320a9fbb3d4077733a7e1cee1902423082d8c WatchSource:0}: Error finding container 2ec29ef88d480fccee99c991166320a9fbb3d4077733a7e1cee1902423082d8c: Status 404 returned error can't find the container with id 2ec29ef88d480fccee99c991166320a9fbb3d4077733a7e1cee1902423082d8c Jul 14 09:32:54.486750 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:54.486714 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wlmn2"] Jul 14 09:32:54.491263 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:54.491196 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-wlmn2" Jul 14 09:32:54.501138 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:54.501110 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wlmn2"] Jul 14 09:32:54.576372 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:54.576336 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q22wf\" (UniqueName: \"kubernetes.io/projected/16802750-6790-487f-95ac-cc5acd349214-kube-api-access-q22wf\") pod \"kuadrant-operator-catalog-wlmn2\" (UID: \"16802750-6790-487f-95ac-cc5acd349214\") " pod="kuadrant-system/kuadrant-operator-catalog-wlmn2" Jul 14 09:32:54.677219 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:54.677182 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q22wf\" (UniqueName: \"kubernetes.io/projected/16802750-6790-487f-95ac-cc5acd349214-kube-api-access-q22wf\") pod \"kuadrant-operator-catalog-wlmn2\" (UID: \"16802750-6790-487f-95ac-cc5acd349214\") " pod="kuadrant-system/kuadrant-operator-catalog-wlmn2" Jul 14 09:32:54.694990 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:54.694965 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q22wf\" (UniqueName: \"kubernetes.io/projected/16802750-6790-487f-95ac-cc5acd349214-kube-api-access-q22wf\") pod \"kuadrant-operator-catalog-wlmn2\" (UID: \"16802750-6790-487f-95ac-cc5acd349214\") " pod="kuadrant-system/kuadrant-operator-catalog-wlmn2" Jul 14 09:32:54.781291 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:54.781206 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-twfkw" event={"ID":"dbfa0afb-3b17-4b47-91f8-03628e4e9b82","Type":"ContainerStarted","Data":"2ec29ef88d480fccee99c991166320a9fbb3d4077733a7e1cee1902423082d8c"} Jul 14 09:32:54.802351 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:54.802319 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-wlmn2" Jul 14 09:32:54.939263 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:32:54.939235 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16802750_6790_487f_95ac_cc5acd349214.slice/crio-5bddb4e3f4bde99a58447410dd4dbba339e89e7f9094d4661365adc2bce48be6 WatchSource:0}: Error finding container 5bddb4e3f4bde99a58447410dd4dbba339e89e7f9094d4661365adc2bce48be6: Status 404 returned error can't find the container with id 5bddb4e3f4bde99a58447410dd4dbba339e89e7f9094d4661365adc2bce48be6 Jul 14 09:32:54.942536 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:54.942518 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wlmn2"] Jul 14 09:32:55.787746 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:55.787715 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-wlmn2" event={"ID":"16802750-6790-487f-95ac-cc5acd349214","Type":"ContainerStarted","Data":"5bddb4e3f4bde99a58447410dd4dbba339e89e7f9094d4661365adc2bce48be6"} Jul 14 09:32:57.798487 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:57.798456 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-wlmn2" event={"ID":"16802750-6790-487f-95ac-cc5acd349214","Type":"ContainerStarted","Data":"1d3bedeabdae25ca3bad90ba53fe88c22ec0511c55b0562630bbb0619a860fe1"} Jul 14 09:32:57.800011 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:57.799988 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-twfkw" event={"ID":"dbfa0afb-3b17-4b47-91f8-03628e4e9b82","Type":"ContainerStarted","Data":"3b03625c16755e03825dd12bb1b145d5063283796c36c0b5b298d34ad28ad4ce"} Jul 14 09:32:57.800135 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:57.800064 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-twfkw" podUID="dbfa0afb-3b17-4b47-91f8-03628e4e9b82" containerName="registry-server" containerID="cri-o://3b03625c16755e03825dd12bb1b145d5063283796c36c0b5b298d34ad28ad4ce" gracePeriod=2 Jul 14 09:32:57.889183 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:57.889131 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-twfkw" podStartSLOduration=2.414175635 podStartE2EDuration="4.889115895s" totalImagesPullingTime="2.47494026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 09:32:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:32:57.887669324 +0000 UTC m=+367.396900790" watchObservedRunningTime="2026-07-14 09:32:57.889115895 +0000 UTC m=+367.398347342" Jul 14 09:32:57.889647 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:57.889626 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-wlmn2" podStartSLOduration=1.974878097 podStartE2EDuration="3.889620061s" totalImagesPullingTime="1.914741964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 09:32:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:32:57.84830555 +0000 UTC m=+367.357536998" watchObservedRunningTime="2026-07-14 09:32:57.889620061 +0000 UTC m=+367.398851508" Jul 14 09:32:58.046052 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:58.046026 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-twfkw" Jul 14 09:32:58.109396 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:58.109310 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2mlmj\" (UniqueName: \"kubernetes.io/projected/dbfa0afb-3b17-4b47-91f8-03628e4e9b82-kube-api-access-2mlmj\") pod \"dbfa0afb-3b17-4b47-91f8-03628e4e9b82\" (UID: \"dbfa0afb-3b17-4b47-91f8-03628e4e9b82\") " Jul 14 09:32:58.111426 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:58.111394 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbfa0afb-3b17-4b47-91f8-03628e4e9b82-kube-api-access-2mlmj" (OuterVolumeSpecName: "kube-api-access-2mlmj") pod "dbfa0afb-3b17-4b47-91f8-03628e4e9b82" (UID: "dbfa0afb-3b17-4b47-91f8-03628e4e9b82"). InnerVolumeSpecName "kube-api-access-2mlmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 09:32:58.210055 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:58.210007 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mlmj\" (UniqueName: \"kubernetes.io/projected/dbfa0afb-3b17-4b47-91f8-03628e4e9b82-kube-api-access-2mlmj\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:32:58.806345 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:58.806307 2565 generic.go:356] "Generic (PLEG): container finished" podID="dbfa0afb-3b17-4b47-91f8-03628e4e9b82" containerID="3b03625c16755e03825dd12bb1b145d5063283796c36c0b5b298d34ad28ad4ce" exitCode=0 Jul 14 09:32:58.806714 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:58.806354 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-twfkw" Jul 14 09:32:58.806714 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:58.806390 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-twfkw" event={"ID":"dbfa0afb-3b17-4b47-91f8-03628e4e9b82","Type":"ContainerDied","Data":"3b03625c16755e03825dd12bb1b145d5063283796c36c0b5b298d34ad28ad4ce"} Jul 14 09:32:58.806714 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:58.806439 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-twfkw" event={"ID":"dbfa0afb-3b17-4b47-91f8-03628e4e9b82","Type":"ContainerDied","Data":"2ec29ef88d480fccee99c991166320a9fbb3d4077733a7e1cee1902423082d8c"} Jul 14 09:32:58.806714 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:58.806459 2565 scope.go:117] "RemoveContainer" containerID="3b03625c16755e03825dd12bb1b145d5063283796c36c0b5b298d34ad28ad4ce" Jul 14 09:32:58.816328 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:58.816307 2565 scope.go:117] "RemoveContainer" containerID="3b03625c16755e03825dd12bb1b145d5063283796c36c0b5b298d34ad28ad4ce" Jul 14 09:32:58.816592 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:32:58.816575 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b03625c16755e03825dd12bb1b145d5063283796c36c0b5b298d34ad28ad4ce\": container with ID starting with 3b03625c16755e03825dd12bb1b145d5063283796c36c0b5b298d34ad28ad4ce not found: ID does not exist" containerID="3b03625c16755e03825dd12bb1b145d5063283796c36c0b5b298d34ad28ad4ce" Jul 14 09:32:58.816643 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:58.816603 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b03625c16755e03825dd12bb1b145d5063283796c36c0b5b298d34ad28ad4ce"} err="failed to get container status \"3b03625c16755e03825dd12bb1b145d5063283796c36c0b5b298d34ad28ad4ce\": rpc error: code = NotFound desc = could not find container \"3b03625c16755e03825dd12bb1b145d5063283796c36c0b5b298d34ad28ad4ce\": container with ID starting with 3b03625c16755e03825dd12bb1b145d5063283796c36c0b5b298d34ad28ad4ce not found: ID does not exist" Jul 14 09:32:58.833507 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:58.833477 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-twfkw"] Jul 14 09:32:58.844419 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:58.844390 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-twfkw"] Jul 14 09:32:58.969612 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:32:58.969581 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbfa0afb-3b17-4b47-91f8-03628e4e9b82" path="/var/lib/kubelet/pods/dbfa0afb-3b17-4b47-91f8-03628e4e9b82/volumes" Jul 14 09:33:04.803352 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:04.803315 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-wlmn2" Jul 14 09:33:04.803760 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:04.803391 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-wlmn2" Jul 14 09:33:04.826878 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:04.826851 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-wlmn2" Jul 14 09:33:04.858377 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:04.858354 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-wlmn2" Jul 14 09:33:09.301001 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:09.300969 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14cnhz"] Jul 14 09:33:09.301384 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:09.301356 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbfa0afb-3b17-4b47-91f8-03628e4e9b82" containerName="registry-server" Jul 14 09:33:09.301384 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:09.301369 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbfa0afb-3b17-4b47-91f8-03628e4e9b82" containerName="registry-server" Jul 14 09:33:09.301451 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:09.301432 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbfa0afb-3b17-4b47-91f8-03628e4e9b82" containerName="registry-server" Jul 14 09:33:09.305881 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:09.305864 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14cnhz" Jul 14 09:33:09.308853 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:09.308834 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-qtnl9\"" Jul 14 09:33:09.313262 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:09.312988 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14cnhz"] Jul 14 09:33:09.416051 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:09.416015 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1cfca97-329a-4ce6-bca8-d711c05cfe2c-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14cnhz\" (UID: \"c1cfca97-329a-4ce6-bca8-d711c05cfe2c\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14cnhz" Jul 14 09:33:09.416217 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:09.416093 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frlt2\" (UniqueName: \"kubernetes.io/projected/c1cfca97-329a-4ce6-bca8-d711c05cfe2c-kube-api-access-frlt2\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14cnhz\" (UID: \"c1cfca97-329a-4ce6-bca8-d711c05cfe2c\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14cnhz" Jul 14 09:33:09.416217 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:09.416125 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1cfca97-329a-4ce6-bca8-d711c05cfe2c-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14cnhz\" (UID: \"c1cfca97-329a-4ce6-bca8-d711c05cfe2c\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14cnhz" Jul 14 09:33:09.516891 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:09.516850 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frlt2\" (UniqueName: \"kubernetes.io/projected/c1cfca97-329a-4ce6-bca8-d711c05cfe2c-kube-api-access-frlt2\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14cnhz\" (UID: \"c1cfca97-329a-4ce6-bca8-d711c05cfe2c\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14cnhz" Jul 14 09:33:09.517065 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:09.516899 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1cfca97-329a-4ce6-bca8-d711c05cfe2c-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14cnhz\" (UID: \"c1cfca97-329a-4ce6-bca8-d711c05cfe2c\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14cnhz" Jul 14 09:33:09.517065 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:09.516949 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1cfca97-329a-4ce6-bca8-d711c05cfe2c-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14cnhz\" (UID: \"c1cfca97-329a-4ce6-bca8-d711c05cfe2c\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14cnhz" Jul 14 09:33:09.517327 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:09.517311 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1cfca97-329a-4ce6-bca8-d711c05cfe2c-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14cnhz\" (UID: \"c1cfca97-329a-4ce6-bca8-d711c05cfe2c\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14cnhz" Jul 14 09:33:09.517363 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:09.517313 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1cfca97-329a-4ce6-bca8-d711c05cfe2c-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14cnhz\" (UID: \"c1cfca97-329a-4ce6-bca8-d711c05cfe2c\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14cnhz" Jul 14 09:33:09.525839 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:09.525797 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frlt2\" (UniqueName: \"kubernetes.io/projected/c1cfca97-329a-4ce6-bca8-d711c05cfe2c-kube-api-access-frlt2\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14cnhz\" (UID: \"c1cfca97-329a-4ce6-bca8-d711c05cfe2c\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14cnhz" Jul 14 09:33:09.615443 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:09.615358 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14cnhz" Jul 14 09:33:09.765214 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:09.765189 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14cnhz"] Jul 14 09:33:09.768177 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:33:09.768142 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1cfca97_329a_4ce6_bca8_d711c05cfe2c.slice/crio-ed9cc5d96564d5d6b14d40d79f02044dbff4a60c9e19f5223181cb12ece6be50 WatchSource:0}: Error finding container ed9cc5d96564d5d6b14d40d79f02044dbff4a60c9e19f5223181cb12ece6be50: Status 404 returned error can't find the container with id ed9cc5d96564d5d6b14d40d79f02044dbff4a60c9e19f5223181cb12ece6be50 Jul 14 09:33:09.861547 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:09.861508 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14cnhz" event={"ID":"c1cfca97-329a-4ce6-bca8-d711c05cfe2c","Type":"ContainerStarted","Data":"8869b7ef67989277ea5e788d188f3cf74c0d24b94ec41a7fb1c4083a56530d78"} Jul 14 09:33:09.861799 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:09.861572 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14cnhz" event={"ID":"c1cfca97-329a-4ce6-bca8-d711c05cfe2c","Type":"ContainerStarted","Data":"ed9cc5d96564d5d6b14d40d79f02044dbff4a60c9e19f5223181cb12ece6be50"} Jul 14 09:33:09.900108 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:09.900074 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7597xwn5"] Jul 14 09:33:09.903525 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:09.903509 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7597xwn5" Jul 14 09:33:09.910943 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:09.910918 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7597xwn5"] Jul 14 09:33:10.020786 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.020748 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d1f66f44-3ca4-4d76-bb20-4c7e830ac11e-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7597xwn5\" (UID: \"d1f66f44-3ca4-4d76-bb20-4c7e830ac11e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7597xwn5" Jul 14 09:33:10.020786 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.020783 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d1f66f44-3ca4-4d76-bb20-4c7e830ac11e-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7597xwn5\" (UID: \"d1f66f44-3ca4-4d76-bb20-4c7e830ac11e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7597xwn5" Jul 14 09:33:10.021003 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.020838 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh5gk\" (UniqueName: \"kubernetes.io/projected/d1f66f44-3ca4-4d76-bb20-4c7e830ac11e-kube-api-access-xh5gk\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7597xwn5\" (UID: \"d1f66f44-3ca4-4d76-bb20-4c7e830ac11e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7597xwn5" Jul 14 09:33:10.122353 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.122314 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh5gk\" (UniqueName: \"kubernetes.io/projected/d1f66f44-3ca4-4d76-bb20-4c7e830ac11e-kube-api-access-xh5gk\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7597xwn5\" (UID: \"d1f66f44-3ca4-4d76-bb20-4c7e830ac11e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7597xwn5" Jul 14 09:33:10.122521 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.122425 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d1f66f44-3ca4-4d76-bb20-4c7e830ac11e-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7597xwn5\" (UID: \"d1f66f44-3ca4-4d76-bb20-4c7e830ac11e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7597xwn5" Jul 14 09:33:10.122521 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.122455 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d1f66f44-3ca4-4d76-bb20-4c7e830ac11e-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7597xwn5\" (UID: \"d1f66f44-3ca4-4d76-bb20-4c7e830ac11e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7597xwn5" Jul 14 09:33:10.122939 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.122918 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d1f66f44-3ca4-4d76-bb20-4c7e830ac11e-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7597xwn5\" (UID: \"d1f66f44-3ca4-4d76-bb20-4c7e830ac11e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7597xwn5" Jul 14 09:33:10.122990 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.122935 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d1f66f44-3ca4-4d76-bb20-4c7e830ac11e-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7597xwn5\" (UID: \"d1f66f44-3ca4-4d76-bb20-4c7e830ac11e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7597xwn5" Jul 14 09:33:10.132217 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.132157 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh5gk\" (UniqueName: \"kubernetes.io/projected/d1f66f44-3ca4-4d76-bb20-4c7e830ac11e-kube-api-access-xh5gk\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7597xwn5\" (UID: \"d1f66f44-3ca4-4d76-bb20-4c7e830ac11e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7597xwn5" Jul 14 09:33:10.240783 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.240742 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7597xwn5" Jul 14 09:33:10.308441 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.308410 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf59x"] Jul 14 09:33:10.313575 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.313549 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf59x" Jul 14 09:33:10.323280 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.323250 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf59x"] Jul 14 09:33:10.373032 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.372993 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-56894c8c45-vc4p9"] Jul 14 09:33:10.376526 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.376499 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56894c8c45-vc4p9" Jul 14 09:33:10.384089 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.384050 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7597xwn5"] Jul 14 09:33:10.386890 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:33:10.386865 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1f66f44_3ca4_4d76_bb20_4c7e830ac11e.slice/crio-5f1344be67162f6744c04ed12f7034ed396fc08b1f80790500d3571e0e3966f3 WatchSource:0}: Error finding container 5f1344be67162f6744c04ed12f7034ed396fc08b1f80790500d3571e0e3966f3: Status 404 returned error can't find the container with id 5f1344be67162f6744c04ed12f7034ed396fc08b1f80790500d3571e0e3966f3 Jul 14 09:33:10.390332 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.390309 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56894c8c45-vc4p9"] Jul 14 09:33:10.425695 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.425661 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/45a87a74-8b57-4bf1-87ec-891cc41b76ef-console-config\") pod \"console-56894c8c45-vc4p9\" (UID: \"45a87a74-8b57-4bf1-87ec-891cc41b76ef\") " pod="openshift-console/console-56894c8c45-vc4p9" Jul 14 09:33:10.425878 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.425758 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf59x\" (UID: \"9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf59x" Jul 14 09:33:10.425878 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.425817 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/45a87a74-8b57-4bf1-87ec-891cc41b76ef-console-oauth-config\") pod \"console-56894c8c45-vc4p9\" (UID: \"45a87a74-8b57-4bf1-87ec-891cc41b76ef\") " pod="openshift-console/console-56894c8c45-vc4p9" Jul 14 09:33:10.425878 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.425853 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/45a87a74-8b57-4bf1-87ec-891cc41b76ef-console-serving-cert\") pod \"console-56894c8c45-vc4p9\" (UID: \"45a87a74-8b57-4bf1-87ec-891cc41b76ef\") " pod="openshift-console/console-56894c8c45-vc4p9" Jul 14 09:33:10.425878 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.425878 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/45a87a74-8b57-4bf1-87ec-891cc41b76ef-oauth-serving-cert\") pod \"console-56894c8c45-vc4p9\" (UID: \"45a87a74-8b57-4bf1-87ec-891cc41b76ef\") " pod="openshift-console/console-56894c8c45-vc4p9" Jul 14 09:33:10.426271 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.425896 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flwfs\" (UniqueName: \"kubernetes.io/projected/45a87a74-8b57-4bf1-87ec-891cc41b76ef-kube-api-access-flwfs\") pod \"console-56894c8c45-vc4p9\" (UID: \"45a87a74-8b57-4bf1-87ec-891cc41b76ef\") " pod="openshift-console/console-56894c8c45-vc4p9" Jul 14 09:33:10.426271 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.425961 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbz76\" (UniqueName: \"kubernetes.io/projected/9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e-kube-api-access-nbz76\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf59x\" (UID: \"9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf59x" Jul 14 09:33:10.426271 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.426003 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45a87a74-8b57-4bf1-87ec-891cc41b76ef-trusted-ca-bundle\") pod \"console-56894c8c45-vc4p9\" (UID: \"45a87a74-8b57-4bf1-87ec-891cc41b76ef\") " pod="openshift-console/console-56894c8c45-vc4p9" Jul 14 09:33:10.426271 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.426080 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/45a87a74-8b57-4bf1-87ec-891cc41b76ef-service-ca\") pod \"console-56894c8c45-vc4p9\" (UID: \"45a87a74-8b57-4bf1-87ec-891cc41b76ef\") " pod="openshift-console/console-56894c8c45-vc4p9" Jul 14 09:33:10.426271 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.426120 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf59x\" (UID: \"9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf59x" Jul 14 09:33:10.527240 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.527203 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/45a87a74-8b57-4bf1-87ec-891cc41b76ef-oauth-serving-cert\") pod \"console-56894c8c45-vc4p9\" (UID: \"45a87a74-8b57-4bf1-87ec-891cc41b76ef\") " pod="openshift-console/console-56894c8c45-vc4p9" Jul 14 09:33:10.527240 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.527236 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flwfs\" (UniqueName: \"kubernetes.io/projected/45a87a74-8b57-4bf1-87ec-891cc41b76ef-kube-api-access-flwfs\") pod \"console-56894c8c45-vc4p9\" (UID: \"45a87a74-8b57-4bf1-87ec-891cc41b76ef\") " pod="openshift-console/console-56894c8c45-vc4p9" Jul 14 09:33:10.527441 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.527256 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbz76\" (UniqueName: \"kubernetes.io/projected/9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e-kube-api-access-nbz76\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf59x\" (UID: \"9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf59x" Jul 14 09:33:10.527480 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.527452 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45a87a74-8b57-4bf1-87ec-891cc41b76ef-trusted-ca-bundle\") pod \"console-56894c8c45-vc4p9\" (UID: \"45a87a74-8b57-4bf1-87ec-891cc41b76ef\") " pod="openshift-console/console-56894c8c45-vc4p9" Jul 14 09:33:10.527526 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.527511 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/45a87a74-8b57-4bf1-87ec-891cc41b76ef-service-ca\") pod \"console-56894c8c45-vc4p9\" (UID: \"45a87a74-8b57-4bf1-87ec-891cc41b76ef\") " pod="openshift-console/console-56894c8c45-vc4p9" Jul 14 09:33:10.527564 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.527549 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf59x\" (UID: \"9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf59x" Jul 14 09:33:10.527609 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.527595 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/45a87a74-8b57-4bf1-87ec-891cc41b76ef-console-config\") pod \"console-56894c8c45-vc4p9\" (UID: \"45a87a74-8b57-4bf1-87ec-891cc41b76ef\") " pod="openshift-console/console-56894c8c45-vc4p9" Jul 14 09:33:10.527698 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.527681 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf59x\" (UID: \"9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf59x" Jul 14 09:33:10.527745 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.527728 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/45a87a74-8b57-4bf1-87ec-891cc41b76ef-console-oauth-config\") pod \"console-56894c8c45-vc4p9\" (UID: \"45a87a74-8b57-4bf1-87ec-891cc41b76ef\") " pod="openshift-console/console-56894c8c45-vc4p9" Jul 14 09:33:10.527797 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.527771 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/45a87a74-8b57-4bf1-87ec-891cc41b76ef-console-serving-cert\") pod \"console-56894c8c45-vc4p9\" (UID: \"45a87a74-8b57-4bf1-87ec-891cc41b76ef\") " pod="openshift-console/console-56894c8c45-vc4p9" Jul 14 09:33:10.528236 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.528035 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf59x\" (UID: \"9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf59x" Jul 14 09:33:10.528236 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.528040 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/45a87a74-8b57-4bf1-87ec-891cc41b76ef-oauth-serving-cert\") pod \"console-56894c8c45-vc4p9\" (UID: \"45a87a74-8b57-4bf1-87ec-891cc41b76ef\") " pod="openshift-console/console-56894c8c45-vc4p9" Jul 14 09:33:10.528236 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.528166 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf59x\" (UID: \"9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf59x" Jul 14 09:33:10.528432 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.528267 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/45a87a74-8b57-4bf1-87ec-891cc41b76ef-service-ca\") pod \"console-56894c8c45-vc4p9\" (UID: \"45a87a74-8b57-4bf1-87ec-891cc41b76ef\") " pod="openshift-console/console-56894c8c45-vc4p9" Jul 14 09:33:10.528432 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.528393 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/45a87a74-8b57-4bf1-87ec-891cc41b76ef-console-config\") pod \"console-56894c8c45-vc4p9\" (UID: \"45a87a74-8b57-4bf1-87ec-891cc41b76ef\") " pod="openshift-console/console-56894c8c45-vc4p9" Jul 14 09:33:10.528432 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.528401 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45a87a74-8b57-4bf1-87ec-891cc41b76ef-trusted-ca-bundle\") pod \"console-56894c8c45-vc4p9\" (UID: \"45a87a74-8b57-4bf1-87ec-891cc41b76ef\") " pod="openshift-console/console-56894c8c45-vc4p9" Jul 14 09:33:10.530220 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.530197 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/45a87a74-8b57-4bf1-87ec-891cc41b76ef-console-oauth-config\") pod \"console-56894c8c45-vc4p9\" (UID: \"45a87a74-8b57-4bf1-87ec-891cc41b76ef\") " pod="openshift-console/console-56894c8c45-vc4p9" Jul 14 09:33:10.530316 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.530301 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/45a87a74-8b57-4bf1-87ec-891cc41b76ef-console-serving-cert\") pod \"console-56894c8c45-vc4p9\" (UID: \"45a87a74-8b57-4bf1-87ec-891cc41b76ef\") " pod="openshift-console/console-56894c8c45-vc4p9" Jul 14 09:33:10.538659 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.538631 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbz76\" (UniqueName: \"kubernetes.io/projected/9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e-kube-api-access-nbz76\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf59x\" (UID: \"9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf59x" Jul 14 09:33:10.538767 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.538707 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flwfs\" (UniqueName: \"kubernetes.io/projected/45a87a74-8b57-4bf1-87ec-891cc41b76ef-kube-api-access-flwfs\") pod \"console-56894c8c45-vc4p9\" (UID: \"45a87a74-8b57-4bf1-87ec-891cc41b76ef\") " pod="openshift-console/console-56894c8c45-vc4p9" Jul 14 09:33:10.623834 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.623773 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf59x" Jul 14 09:33:10.687401 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.687369 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56894c8c45-vc4p9" Jul 14 09:33:10.707709 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.707663 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739h42f"] Jul 14 09:33:10.713549 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.713523 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739h42f" Jul 14 09:33:10.729238 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.729208 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739h42f"] Jul 14 09:33:10.770543 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.770523 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf59x"] Jul 14 09:33:10.773005 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:33:10.772977 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c77c9bc_4bbb_4d23_8ebf_1b552a5cdf4e.slice/crio-686cf9e669e18d24a5130170d8ed52e400c507e998811bf62219bb50a3df1824 WatchSource:0}: Error finding container 686cf9e669e18d24a5130170d8ed52e400c507e998811bf62219bb50a3df1824: Status 404 returned error can't find the container with id 686cf9e669e18d24a5130170d8ed52e400c507e998811bf62219bb50a3df1824 Jul 14 09:33:10.829825 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.829785 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrk6h\" (UniqueName: \"kubernetes.io/projected/b51c5501-6d06-4b37-b989-08b6016bf4e5-kube-api-access-mrk6h\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739h42f\" (UID: \"b51c5501-6d06-4b37-b989-08b6016bf4e5\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739h42f" Jul 14 09:33:10.829932 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.829874 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b51c5501-6d06-4b37-b989-08b6016bf4e5-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739h42f\" (UID: \"b51c5501-6d06-4b37-b989-08b6016bf4e5\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739h42f" Jul 14 09:33:10.829932 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.829902 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b51c5501-6d06-4b37-b989-08b6016bf4e5-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739h42f\" (UID: \"b51c5501-6d06-4b37-b989-08b6016bf4e5\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739h42f" Jul 14 09:33:10.842420 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.842401 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56894c8c45-vc4p9"] Jul 14 09:33:10.844576 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:33:10.844548 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45a87a74_8b57_4bf1_87ec_891cc41b76ef.slice/crio-627b023b084596f2b9d4dcaff83d21604838fbe5bc3e99c0abf5854a5d62e74f WatchSource:0}: Error finding container 627b023b084596f2b9d4dcaff83d21604838fbe5bc3e99c0abf5854a5d62e74f: Status 404 returned error can't find the container with id 627b023b084596f2b9d4dcaff83d21604838fbe5bc3e99c0abf5854a5d62e74f Jul 14 09:33:10.869849 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.869774 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf59x" event={"ID":"9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e","Type":"ContainerStarted","Data":"2a8cdd0ec2149f96238016f50b3469de667f06271a3b4bd0f5e78984314410cf"} Jul 14 09:33:10.869984 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.869877 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf59x" event={"ID":"9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e","Type":"ContainerStarted","Data":"686cf9e669e18d24a5130170d8ed52e400c507e998811bf62219bb50a3df1824"} Jul 14 09:33:10.871405 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.871376 2565 generic.go:356] "Generic (PLEG): container finished" podID="d1f66f44-3ca4-4d76-bb20-4c7e830ac11e" containerID="f1d3b5cf36f938f3f0a0785f89aa001691cfdf465734738dbb75cea7be694498" exitCode=0 Jul 14 09:33:10.871537 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.871510 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7597xwn5" event={"ID":"d1f66f44-3ca4-4d76-bb20-4c7e830ac11e","Type":"ContainerDied","Data":"f1d3b5cf36f938f3f0a0785f89aa001691cfdf465734738dbb75cea7be694498"} Jul 14 09:33:10.871601 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.871576 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7597xwn5" event={"ID":"d1f66f44-3ca4-4d76-bb20-4c7e830ac11e","Type":"ContainerStarted","Data":"5f1344be67162f6744c04ed12f7034ed396fc08b1f80790500d3571e0e3966f3"} Jul 14 09:33:10.872772 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.872746 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56894c8c45-vc4p9" event={"ID":"45a87a74-8b57-4bf1-87ec-891cc41b76ef","Type":"ContainerStarted","Data":"627b023b084596f2b9d4dcaff83d21604838fbe5bc3e99c0abf5854a5d62e74f"} Jul 14 09:33:10.874254 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.874232 2565 generic.go:356] "Generic (PLEG): container finished" podID="c1cfca97-329a-4ce6-bca8-d711c05cfe2c" containerID="8869b7ef67989277ea5e788d188f3cf74c0d24b94ec41a7fb1c4083a56530d78" exitCode=0 Jul 14 09:33:10.874384 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.874267 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14cnhz" event={"ID":"c1cfca97-329a-4ce6-bca8-d711c05cfe2c","Type":"ContainerDied","Data":"8869b7ef67989277ea5e788d188f3cf74c0d24b94ec41a7fb1c4083a56530d78"} Jul 14 09:33:10.930985 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.930931 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrk6h\" (UniqueName: \"kubernetes.io/projected/b51c5501-6d06-4b37-b989-08b6016bf4e5-kube-api-access-mrk6h\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739h42f\" (UID: \"b51c5501-6d06-4b37-b989-08b6016bf4e5\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739h42f" Jul 14 09:33:10.931143 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.931053 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b51c5501-6d06-4b37-b989-08b6016bf4e5-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739h42f\" (UID: \"b51c5501-6d06-4b37-b989-08b6016bf4e5\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739h42f" Jul 14 09:33:10.931143 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.931092 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b51c5501-6d06-4b37-b989-08b6016bf4e5-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739h42f\" (UID: \"b51c5501-6d06-4b37-b989-08b6016bf4e5\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739h42f" Jul 14 09:33:10.931422 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.931400 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b51c5501-6d06-4b37-b989-08b6016bf4e5-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739h42f\" (UID: \"b51c5501-6d06-4b37-b989-08b6016bf4e5\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739h42f" Jul 14 09:33:10.931494 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.931463 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b51c5501-6d06-4b37-b989-08b6016bf4e5-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739h42f\" (UID: \"b51c5501-6d06-4b37-b989-08b6016bf4e5\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739h42f" Jul 14 09:33:10.942024 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:10.942001 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrk6h\" (UniqueName: \"kubernetes.io/projected/b51c5501-6d06-4b37-b989-08b6016bf4e5-kube-api-access-mrk6h\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739h42f\" (UID: \"b51c5501-6d06-4b37-b989-08b6016bf4e5\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739h42f" Jul 14 09:33:11.026291 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:11.026257 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739h42f" Jul 14 09:33:11.162623 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:11.162597 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739h42f"] Jul 14 09:33:11.164712 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:33:11.164687 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb51c5501_6d06_4b37_b989_08b6016bf4e5.slice/crio-e64671e9115e9cf4a83d67935b9152262db42ef8920af1f0563022aab9ddd3e0 WatchSource:0}: Error finding container e64671e9115e9cf4a83d67935b9152262db42ef8920af1f0563022aab9ddd3e0: Status 404 returned error can't find the container with id e64671e9115e9cf4a83d67935b9152262db42ef8920af1f0563022aab9ddd3e0 Jul 14 09:33:11.881171 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:11.881141 2565 generic.go:356] "Generic (PLEG): container finished" podID="9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e" containerID="2a8cdd0ec2149f96238016f50b3469de667f06271a3b4bd0f5e78984314410cf" exitCode=0 Jul 14 09:33:11.881543 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:11.881226 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf59x" event={"ID":"9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e","Type":"ContainerDied","Data":"2a8cdd0ec2149f96238016f50b3469de667f06271a3b4bd0f5e78984314410cf"} Jul 14 09:33:11.882995 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:11.882975 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56894c8c45-vc4p9" event={"ID":"45a87a74-8b57-4bf1-87ec-891cc41b76ef","Type":"ContainerStarted","Data":"6dc36a54494ea62ed980e95f948770b162b5bad7e084665b04ec9d32d756030a"} Jul 14 09:33:11.884450 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:11.884430 2565 generic.go:356] "Generic (PLEG): container finished" podID="b51c5501-6d06-4b37-b989-08b6016bf4e5" containerID="c284c885cd63cd821f5217e518b44ae95a2fa6239541a1a3b075e174b990fe8e" exitCode=0 Jul 14 09:33:11.884529 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:11.884475 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739h42f" event={"ID":"b51c5501-6d06-4b37-b989-08b6016bf4e5","Type":"ContainerDied","Data":"c284c885cd63cd821f5217e518b44ae95a2fa6239541a1a3b075e174b990fe8e"} Jul 14 09:33:11.884529 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:11.884511 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739h42f" event={"ID":"b51c5501-6d06-4b37-b989-08b6016bf4e5","Type":"ContainerStarted","Data":"e64671e9115e9cf4a83d67935b9152262db42ef8920af1f0563022aab9ddd3e0"} Jul 14 09:33:11.918369 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:11.918327 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-56894c8c45-vc4p9" podStartSLOduration=1.9183103209999999 podStartE2EDuration="1.918310321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 09:33:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:33:11.916153612 +0000 UTC m=+381.425385061" watchObservedRunningTime="2026-07-14 09:33:11.918310321 +0000 UTC m=+381.427541839" Jul 14 09:33:12.892689 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:12.892655 2565 generic.go:356] "Generic (PLEG): container finished" podID="c1cfca97-329a-4ce6-bca8-d711c05cfe2c" containerID="b7d85bcc048d08894250980ae662e3009b9aeb73fd5d148a304aa5e61f14bcaa" exitCode=0 Jul 14 09:33:12.893110 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:12.892720 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14cnhz" event={"ID":"c1cfca97-329a-4ce6-bca8-d711c05cfe2c","Type":"ContainerDied","Data":"b7d85bcc048d08894250980ae662e3009b9aeb73fd5d148a304aa5e61f14bcaa"} Jul 14 09:33:12.894625 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:12.894607 2565 generic.go:356] "Generic (PLEG): container finished" podID="9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e" containerID="ddd5bff21ff017d55a0738ba519f7a9cc0c26aa9b6a4461de5f16e8a302a530d" exitCode=0 Jul 14 09:33:12.894710 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:12.894647 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf59x" event={"ID":"9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e","Type":"ContainerDied","Data":"ddd5bff21ff017d55a0738ba519f7a9cc0c26aa9b6a4461de5f16e8a302a530d"} Jul 14 09:33:12.896595 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:12.896570 2565 generic.go:356] "Generic (PLEG): container finished" podID="d1f66f44-3ca4-4d76-bb20-4c7e830ac11e" containerID="f1a43129b2e2737c56b458cd4a554cdd5b6c3c71299a452c570035ad1e08cce3" exitCode=0 Jul 14 09:33:12.896692 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:12.896645 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7597xwn5" event={"ID":"d1f66f44-3ca4-4d76-bb20-4c7e830ac11e","Type":"ContainerDied","Data":"f1a43129b2e2737c56b458cd4a554cdd5b6c3c71299a452c570035ad1e08cce3"} Jul 14 09:33:12.898772 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:12.898755 2565 generic.go:356] "Generic (PLEG): container finished" podID="b51c5501-6d06-4b37-b989-08b6016bf4e5" containerID="4fe7e8500c15b19ac8037fa3e50c1f4817bd21f6c95184bd50f7707f22b17ca3" exitCode=0 Jul 14 09:33:12.898867 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:12.898787 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739h42f" event={"ID":"b51c5501-6d06-4b37-b989-08b6016bf4e5","Type":"ContainerDied","Data":"4fe7e8500c15b19ac8037fa3e50c1f4817bd21f6c95184bd50f7707f22b17ca3"} Jul 14 09:33:13.906679 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:13.906639 2565 generic.go:356] "Generic (PLEG): container finished" podID="b51c5501-6d06-4b37-b989-08b6016bf4e5" containerID="06e2688c9cbfb2223730dfe61cc38407e0da56347bb97b01787302af6b764cb7" exitCode=0 Jul 14 09:33:13.907115 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:13.906722 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739h42f" event={"ID":"b51c5501-6d06-4b37-b989-08b6016bf4e5","Type":"ContainerDied","Data":"06e2688c9cbfb2223730dfe61cc38407e0da56347bb97b01787302af6b764cb7"} Jul 14 09:33:13.908690 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:13.908670 2565 generic.go:356] "Generic (PLEG): container finished" podID="c1cfca97-329a-4ce6-bca8-d711c05cfe2c" containerID="9e640eadc4959222d6c9cd6acabd248f31099f57f37644e96c22310937669f0b" exitCode=0 Jul 14 09:33:13.908763 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:13.908718 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14cnhz" event={"ID":"c1cfca97-329a-4ce6-bca8-d711c05cfe2c","Type":"ContainerDied","Data":"9e640eadc4959222d6c9cd6acabd248f31099f57f37644e96c22310937669f0b"} Jul 14 09:33:13.910639 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:13.910617 2565 generic.go:356] "Generic (PLEG): container finished" podID="9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e" containerID="4dd5fce950d7c0f98382a64336caec6f2f60b9a5e2bf8533468d2542f3a492a1" exitCode=0 Jul 14 09:33:13.910758 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:13.910693 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf59x" event={"ID":"9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e","Type":"ContainerDied","Data":"4dd5fce950d7c0f98382a64336caec6f2f60b9a5e2bf8533468d2542f3a492a1"} Jul 14 09:33:13.919009 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:13.918982 2565 generic.go:356] "Generic (PLEG): container finished" podID="d1f66f44-3ca4-4d76-bb20-4c7e830ac11e" containerID="fd9dcbb140c851d24c5603a9c0945bcfcb08cb939392e0fe358754e04715f43f" exitCode=0 Jul 14 09:33:13.919111 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:13.919024 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7597xwn5" event={"ID":"d1f66f44-3ca4-4d76-bb20-4c7e830ac11e","Type":"ContainerDied","Data":"fd9dcbb140c851d24c5603a9c0945bcfcb08cb939392e0fe358754e04715f43f"} Jul 14 09:33:15.108458 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:15.108422 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf59x" Jul 14 09:33:15.139387 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:15.139360 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739h42f" Jul 14 09:33:15.142527 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:15.142503 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7597xwn5" Jul 14 09:33:15.146319 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:15.146302 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14cnhz" Jul 14 09:33:15.176441 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:15.176370 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nbz76\" (UniqueName: \"kubernetes.io/projected/9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e-kube-api-access-nbz76\") pod \"9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e\" (UID: \"9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e\") " Jul 14 09:33:15.176592 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:15.176576 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e-bundle\") pod \"9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e\" (UID: \"9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e\") " Jul 14 09:33:15.176667 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:15.176623 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e-util\") pod \"9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e\" (UID: \"9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e\") " Jul 14 09:33:15.177232 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:15.177193 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e-bundle" (OuterVolumeSpecName: "bundle") pod "9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e" (UID: "9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 09:33:15.179128 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:15.179096 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e-kube-api-access-nbz76" (OuterVolumeSpecName: "kube-api-access-nbz76") pod "9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e" (UID: "9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e"). InnerVolumeSpecName "kube-api-access-nbz76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 09:33:15.182876 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:15.182852 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e-util" (OuterVolumeSpecName: "util") pod "9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e" (UID: "9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 09:33:15.277984 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:15.277946 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xh5gk\" (UniqueName: \"kubernetes.io/projected/d1f66f44-3ca4-4d76-bb20-4c7e830ac11e-kube-api-access-xh5gk\") pod \"d1f66f44-3ca4-4d76-bb20-4c7e830ac11e\" (UID: \"d1f66f44-3ca4-4d76-bb20-4c7e830ac11e\") " Jul 14 09:33:15.278177 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:15.278019 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1cfca97-329a-4ce6-bca8-d711c05cfe2c-util\") pod \"c1cfca97-329a-4ce6-bca8-d711c05cfe2c\" (UID: \"c1cfca97-329a-4ce6-bca8-d711c05cfe2c\") " Jul 14 09:33:15.278177 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:15.278043 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b51c5501-6d06-4b37-b989-08b6016bf4e5-util\") pod \"b51c5501-6d06-4b37-b989-08b6016bf4e5\" (UID: \"b51c5501-6d06-4b37-b989-08b6016bf4e5\") " Jul 14 09:33:15.278177 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:15.278077 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-frlt2\" (UniqueName: \"kubernetes.io/projected/c1cfca97-329a-4ce6-bca8-d711c05cfe2c-kube-api-access-frlt2\") pod \"c1cfca97-329a-4ce6-bca8-d711c05cfe2c\" (UID: \"c1cfca97-329a-4ce6-bca8-d711c05cfe2c\") " Jul 14 09:33:15.278177 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:15.278124 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1cfca97-329a-4ce6-bca8-d711c05cfe2c-bundle\") pod \"c1cfca97-329a-4ce6-bca8-d711c05cfe2c\" (UID: \"c1cfca97-329a-4ce6-bca8-d711c05cfe2c\") " Jul 14 09:33:15.278177 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:15.278148 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d1f66f44-3ca4-4d76-bb20-4c7e830ac11e-util\") pod \"d1f66f44-3ca4-4d76-bb20-4c7e830ac11e\" (UID: \"d1f66f44-3ca4-4d76-bb20-4c7e830ac11e\") " Jul 14 09:33:15.278177 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:15.278178 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d1f66f44-3ca4-4d76-bb20-4c7e830ac11e-bundle\") pod \"d1f66f44-3ca4-4d76-bb20-4c7e830ac11e\" (UID: \"d1f66f44-3ca4-4d76-bb20-4c7e830ac11e\") " Jul 14 09:33:15.278478 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:15.278233 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mrk6h\" (UniqueName: \"kubernetes.io/projected/b51c5501-6d06-4b37-b989-08b6016bf4e5-kube-api-access-mrk6h\") pod \"b51c5501-6d06-4b37-b989-08b6016bf4e5\" (UID: \"b51c5501-6d06-4b37-b989-08b6016bf4e5\") " Jul 14 09:33:15.278478 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:15.278259 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b51c5501-6d06-4b37-b989-08b6016bf4e5-bundle\") pod \"b51c5501-6d06-4b37-b989-08b6016bf4e5\" (UID: \"b51c5501-6d06-4b37-b989-08b6016bf4e5\") " Jul 14 09:33:15.278578 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:15.278546 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e-bundle\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:33:15.278578 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:15.278562 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e-util\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:33:15.278578 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:15.278574 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbz76\" (UniqueName: \"kubernetes.io/projected/9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e-kube-api-access-nbz76\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:33:15.279336 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:15.279028 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1f66f44-3ca4-4d76-bb20-4c7e830ac11e-bundle" (OuterVolumeSpecName: "bundle") pod "d1f66f44-3ca4-4d76-bb20-4c7e830ac11e" (UID: "d1f66f44-3ca4-4d76-bb20-4c7e830ac11e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 09:33:15.279336 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:15.279264 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b51c5501-6d06-4b37-b989-08b6016bf4e5-bundle" (OuterVolumeSpecName: "bundle") pod "b51c5501-6d06-4b37-b989-08b6016bf4e5" (UID: "b51c5501-6d06-4b37-b989-08b6016bf4e5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 09:33:15.279505 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:15.279437 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1cfca97-329a-4ce6-bca8-d711c05cfe2c-bundle" (OuterVolumeSpecName: "bundle") pod "c1cfca97-329a-4ce6-bca8-d711c05cfe2c" (UID: "c1cfca97-329a-4ce6-bca8-d711c05cfe2c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 09:33:15.280460 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:15.280436 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1f66f44-3ca4-4d76-bb20-4c7e830ac11e-kube-api-access-xh5gk" (OuterVolumeSpecName: "kube-api-access-xh5gk") pod "d1f66f44-3ca4-4d76-bb20-4c7e830ac11e" (UID: "d1f66f44-3ca4-4d76-bb20-4c7e830ac11e"). InnerVolumeSpecName "kube-api-access-xh5gk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 09:33:15.280661 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:15.280639 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1cfca97-329a-4ce6-bca8-d711c05cfe2c-kube-api-access-frlt2" (OuterVolumeSpecName: "kube-api-access-frlt2") pod "c1cfca97-329a-4ce6-bca8-d711c05cfe2c" (UID: "c1cfca97-329a-4ce6-bca8-d711c05cfe2c"). InnerVolumeSpecName "kube-api-access-frlt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 09:33:15.281166 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:15.281146 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b51c5501-6d06-4b37-b989-08b6016bf4e5-kube-api-access-mrk6h" (OuterVolumeSpecName: "kube-api-access-mrk6h") pod "b51c5501-6d06-4b37-b989-08b6016bf4e5" (UID: "b51c5501-6d06-4b37-b989-08b6016bf4e5"). InnerVolumeSpecName "kube-api-access-mrk6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 09:33:15.283641 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:15.283608 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1cfca97-329a-4ce6-bca8-d711c05cfe2c-util" (OuterVolumeSpecName: "util") pod "c1cfca97-329a-4ce6-bca8-d711c05cfe2c" (UID: "c1cfca97-329a-4ce6-bca8-d711c05cfe2c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 09:33:15.284254 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:15.284237 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b51c5501-6d06-4b37-b989-08b6016bf4e5-util" (OuterVolumeSpecName: "util") pod "b51c5501-6d06-4b37-b989-08b6016bf4e5" (UID: "b51c5501-6d06-4b37-b989-08b6016bf4e5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 09:33:15.284606 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:15.284585 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1f66f44-3ca4-4d76-bb20-4c7e830ac11e-util" (OuterVolumeSpecName: "util") pod "d1f66f44-3ca4-4d76-bb20-4c7e830ac11e" (UID: "d1f66f44-3ca4-4d76-bb20-4c7e830ac11e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 09:33:15.379653 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:15.379615 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh5gk\" (UniqueName: \"kubernetes.io/projected/d1f66f44-3ca4-4d76-bb20-4c7e830ac11e-kube-api-access-xh5gk\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:33:15.379653 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:15.379645 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1cfca97-329a-4ce6-bca8-d711c05cfe2c-util\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:33:15.379653 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:15.379657 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b51c5501-6d06-4b37-b989-08b6016bf4e5-util\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:33:15.379905 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:15.379669 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frlt2\" (UniqueName: \"kubernetes.io/projected/c1cfca97-329a-4ce6-bca8-d711c05cfe2c-kube-api-access-frlt2\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:33:15.379905 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:15.379681 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1cfca97-329a-4ce6-bca8-d711c05cfe2c-bundle\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:33:15.379905 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:15.379692 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d1f66f44-3ca4-4d76-bb20-4c7e830ac11e-util\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:33:15.379905 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:15.379701 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d1f66f44-3ca4-4d76-bb20-4c7e830ac11e-bundle\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:33:15.379905 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:15.379711 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrk6h\" (UniqueName: \"kubernetes.io/projected/b51c5501-6d06-4b37-b989-08b6016bf4e5-kube-api-access-mrk6h\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:33:15.379905 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:15.379722 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b51c5501-6d06-4b37-b989-08b6016bf4e5-bundle\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:33:15.931467 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:15.931442 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739h42f" Jul 14 09:33:15.931669 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:15.931439 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739h42f" event={"ID":"b51c5501-6d06-4b37-b989-08b6016bf4e5","Type":"ContainerDied","Data":"e64671e9115e9cf4a83d67935b9152262db42ef8920af1f0563022aab9ddd3e0"} Jul 14 09:33:15.931669 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:15.931562 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e64671e9115e9cf4a83d67935b9152262db42ef8920af1f0563022aab9ddd3e0" Jul 14 09:33:15.933385 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:15.933365 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14cnhz" Jul 14 09:33:15.933514 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:15.933385 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14cnhz" event={"ID":"c1cfca97-329a-4ce6-bca8-d711c05cfe2c","Type":"ContainerDied","Data":"ed9cc5d96564d5d6b14d40d79f02044dbff4a60c9e19f5223181cb12ece6be50"} Jul 14 09:33:15.933514 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:15.933424 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed9cc5d96564d5d6b14d40d79f02044dbff4a60c9e19f5223181cb12ece6be50" Jul 14 09:33:15.935283 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:15.935266 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf59x" Jul 14 09:33:15.935422 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:15.935290 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf59x" event={"ID":"9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e","Type":"ContainerDied","Data":"686cf9e669e18d24a5130170d8ed52e400c507e998811bf62219bb50a3df1824"} Jul 14 09:33:15.935422 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:15.935342 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="686cf9e669e18d24a5130170d8ed52e400c507e998811bf62219bb50a3df1824" Jul 14 09:33:15.937383 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:15.937366 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7597xwn5" Jul 14 09:33:15.937473 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:15.937367 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7597xwn5" event={"ID":"d1f66f44-3ca4-4d76-bb20-4c7e830ac11e","Type":"ContainerDied","Data":"5f1344be67162f6744c04ed12f7034ed396fc08b1f80790500d3571e0e3966f3"} Jul 14 09:33:15.937539 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:15.937486 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f1344be67162f6744c04ed12f7034ed396fc08b1f80790500d3571e0e3966f3" Jul 14 09:33:20.688012 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:20.687951 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-56894c8c45-vc4p9" Jul 14 09:33:20.688415 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:20.688043 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-56894c8c45-vc4p9" Jul 14 09:33:20.694048 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:20.694025 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-56894c8c45-vc4p9" Jul 14 09:33:20.970451 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:20.970380 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-56894c8c45-vc4p9" Jul 14 09:33:21.026069 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:21.026037 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cdd8bc874-bztpx"] Jul 14 09:33:27.163042 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:27.163004 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vl9gm"] Jul 14 09:33:27.163485 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:27.163368 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1f66f44-3ca4-4d76-bb20-4c7e830ac11e" containerName="pull" Jul 14 09:33:27.163485 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:27.163379 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1f66f44-3ca4-4d76-bb20-4c7e830ac11e" containerName="pull" Jul 14 09:33:27.163485 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:27.163388 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b51c5501-6d06-4b37-b989-08b6016bf4e5" containerName="pull" Jul 14 09:33:27.163485 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:27.163394 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b51c5501-6d06-4b37-b989-08b6016bf4e5" containerName="pull" Jul 14 09:33:27.163485 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:27.163402 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1f66f44-3ca4-4d76-bb20-4c7e830ac11e" containerName="util" Jul 14 09:33:27.163485 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:27.163407 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1f66f44-3ca4-4d76-bb20-4c7e830ac11e" containerName="util" Jul 14 09:33:27.163485 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:27.163413 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b51c5501-6d06-4b37-b989-08b6016bf4e5" containerName="util" Jul 14 09:33:27.163485 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:27.163420 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b51c5501-6d06-4b37-b989-08b6016bf4e5" containerName="util" Jul 14 09:33:27.163485 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:27.163430 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e" containerName="pull" Jul 14 09:33:27.163485 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:27.163435 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e" containerName="pull" Jul 14 09:33:27.163485 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:27.163440 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e" containerName="extract" Jul 14 09:33:27.163485 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:27.163445 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e" containerName="extract" Jul 14 09:33:27.163485 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:27.163455 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1cfca97-329a-4ce6-bca8-d711c05cfe2c" containerName="util" Jul 14 09:33:27.163485 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:27.163460 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1cfca97-329a-4ce6-bca8-d711c05cfe2c" containerName="util" Jul 14 09:33:27.163485 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:27.163466 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b51c5501-6d06-4b37-b989-08b6016bf4e5" containerName="extract" Jul 14 09:33:27.163485 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:27.163471 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b51c5501-6d06-4b37-b989-08b6016bf4e5" containerName="extract" Jul 14 09:33:27.163485 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:27.163479 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e" containerName="util" Jul 14 09:33:27.163485 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:27.163484 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e" containerName="util" Jul 14 09:33:27.164037 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:27.163496 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1cfca97-329a-4ce6-bca8-d711c05cfe2c" containerName="pull" Jul 14 09:33:27.164037 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:27.163501 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1cfca97-329a-4ce6-bca8-d711c05cfe2c" containerName="pull" Jul 14 09:33:27.164037 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:27.163508 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1f66f44-3ca4-4d76-bb20-4c7e830ac11e" containerName="extract" Jul 14 09:33:27.164037 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:27.163513 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1f66f44-3ca4-4d76-bb20-4c7e830ac11e" containerName="extract" Jul 14 09:33:27.164037 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:27.163519 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1cfca97-329a-4ce6-bca8-d711c05cfe2c" containerName="extract" Jul 14 09:33:27.164037 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:27.163523 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1cfca97-329a-4ce6-bca8-d711c05cfe2c" containerName="extract" Jul 14 09:33:27.164037 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:27.163578 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e" containerName="extract" Jul 14 09:33:27.164037 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:27.163587 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b51c5501-6d06-4b37-b989-08b6016bf4e5" containerName="extract" Jul 14 09:33:27.164037 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:27.163592 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1cfca97-329a-4ce6-bca8-d711c05cfe2c" containerName="extract" Jul 14 09:33:27.164037 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:27.163600 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1f66f44-3ca4-4d76-bb20-4c7e830ac11e" containerName="extract" Jul 14 09:33:27.165926 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:27.165908 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vl9gm" Jul 14 09:33:27.169415 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:27.169393 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-l4nht\"" Jul 14 09:33:27.182459 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:27.182432 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vl9gm"] Jul 14 09:33:27.285706 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:27.285665 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptlzb\" (UniqueName: \"kubernetes.io/projected/9745acfb-c8a2-48a5-ac27-cbb5e992be42-kube-api-access-ptlzb\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vl9gm\" (UID: \"9745acfb-c8a2-48a5-ac27-cbb5e992be42\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vl9gm" Jul 14 09:33:27.285706 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:27.285713 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9745acfb-c8a2-48a5-ac27-cbb5e992be42-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vl9gm\" (UID: \"9745acfb-c8a2-48a5-ac27-cbb5e992be42\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vl9gm" Jul 14 09:33:27.386780 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:27.386717 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptlzb\" (UniqueName: \"kubernetes.io/projected/9745acfb-c8a2-48a5-ac27-cbb5e992be42-kube-api-access-ptlzb\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vl9gm\" (UID: \"9745acfb-c8a2-48a5-ac27-cbb5e992be42\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vl9gm" Jul 14 09:33:27.386780 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:27.386756 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9745acfb-c8a2-48a5-ac27-cbb5e992be42-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vl9gm\" (UID: \"9745acfb-c8a2-48a5-ac27-cbb5e992be42\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vl9gm" Jul 14 09:33:27.387156 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:27.387139 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9745acfb-c8a2-48a5-ac27-cbb5e992be42-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vl9gm\" (UID: \"9745acfb-c8a2-48a5-ac27-cbb5e992be42\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vl9gm" Jul 14 09:33:27.396254 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:27.396222 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptlzb\" (UniqueName: \"kubernetes.io/projected/9745acfb-c8a2-48a5-ac27-cbb5e992be42-kube-api-access-ptlzb\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vl9gm\" (UID: \"9745acfb-c8a2-48a5-ac27-cbb5e992be42\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vl9gm" Jul 14 09:33:27.476909 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:27.476799 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vl9gm" Jul 14 09:33:27.614857 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:27.614833 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vl9gm"] Jul 14 09:33:27.616915 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:33:27.616880 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9745acfb_c8a2_48a5_ac27_cbb5e992be42.slice/crio-09ee44a4e2673f26dd927e80eb28c6b43eb68b4671b739fca6889f1148562638 WatchSource:0}: Error finding container 09ee44a4e2673f26dd927e80eb28c6b43eb68b4671b739fca6889f1148562638: Status 404 returned error can't find the container with id 09ee44a4e2673f26dd927e80eb28c6b43eb68b4671b739fca6889f1148562638 Jul 14 09:33:28.002825 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:28.002768 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vl9gm" event={"ID":"9745acfb-c8a2-48a5-ac27-cbb5e992be42","Type":"ContainerStarted","Data":"09ee44a4e2673f26dd927e80eb28c6b43eb68b4671b739fca6889f1148562638"} Jul 14 09:33:29.731253 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:29.731213 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-829mm"] Jul 14 09:33:29.733919 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:29.733896 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-829mm" Jul 14 09:33:29.736690 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:29.736651 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-wldgn\"" Jul 14 09:33:29.748373 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:29.748346 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-829mm"] Jul 14 09:33:29.810535 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:29.810497 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv6tb\" (UniqueName: \"kubernetes.io/projected/4efb1cb8-0291-4753-a575-dfbb9aebf533-kube-api-access-cv6tb\") pod \"authorino-operator-75cc8d5cb7-829mm\" (UID: \"4efb1cb8-0291-4753-a575-dfbb9aebf533\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-829mm" Jul 14 09:33:29.911367 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:29.911333 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv6tb\" (UniqueName: \"kubernetes.io/projected/4efb1cb8-0291-4753-a575-dfbb9aebf533-kube-api-access-cv6tb\") pod \"authorino-operator-75cc8d5cb7-829mm\" (UID: \"4efb1cb8-0291-4753-a575-dfbb9aebf533\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-829mm" Jul 14 09:33:29.924748 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:29.924708 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv6tb\" (UniqueName: \"kubernetes.io/projected/4efb1cb8-0291-4753-a575-dfbb9aebf533-kube-api-access-cv6tb\") pod \"authorino-operator-75cc8d5cb7-829mm\" (UID: \"4efb1cb8-0291-4753-a575-dfbb9aebf533\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-829mm" Jul 14 09:33:30.047901 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:30.047800 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-829mm" Jul 14 09:33:30.191783 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:30.191755 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-829mm"] Jul 14 09:33:30.193702 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:33:30.193671 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4efb1cb8_0291_4753_a575_dfbb9aebf533.slice/crio-5efff6698aa1e351a05fbe901391a4086db8c97a1458f448728de748e73547a8 WatchSource:0}: Error finding container 5efff6698aa1e351a05fbe901391a4086db8c97a1458f448728de748e73547a8: Status 404 returned error can't find the container with id 5efff6698aa1e351a05fbe901391a4086db8c97a1458f448728de748e73547a8 Jul 14 09:33:31.019991 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:31.019953 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-829mm" event={"ID":"4efb1cb8-0291-4753-a575-dfbb9aebf533","Type":"ContainerStarted","Data":"5efff6698aa1e351a05fbe901391a4086db8c97a1458f448728de748e73547a8"} Jul 14 09:33:33.037175 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:33.037130 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-829mm" event={"ID":"4efb1cb8-0291-4753-a575-dfbb9aebf533","Type":"ContainerStarted","Data":"1ce1a56cd707c1d248ebc41bc1f647c511752c8fb44724a90c0957f7d733e5e0"} Jul 14 09:33:33.037648 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:33.037255 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-829mm" Jul 14 09:33:33.057233 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:33.057155 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-829mm" podStartSLOduration=1.6380532730000001 podStartE2EDuration="4.057133395s" totalImagesPullingTime="2.419080122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 09:33:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:33:33.054571426 +0000 UTC m=+402.563802869" watchObservedRunningTime="2026-07-14 09:33:33.057133395 +0000 UTC m=+402.566364848" Jul 14 09:33:35.049383 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:35.049347 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vl9gm" event={"ID":"9745acfb-c8a2-48a5-ac27-cbb5e992be42","Type":"ContainerStarted","Data":"fc35341e085836497118d577953cc0a2c10bd0d6492de84d7ef8de390a05ff9c"} Jul 14 09:33:35.049898 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:35.049554 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vl9gm" Jul 14 09:33:35.084577 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:35.084526 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vl9gm" podStartSLOduration=1.6990621849999998 podStartE2EDuration="8.084505384s" totalImagesPullingTime="6.385443199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 09:33:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:33:35.082976273 +0000 UTC m=+404.592207734" watchObservedRunningTime="2026-07-14 09:33:35.084505384 +0000 UTC m=+404.593736851" Jul 14 09:33:36.908878 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:36.908842 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-stf5s"] Jul 14 09:33:36.912678 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:36.912659 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-stf5s" Jul 14 09:33:36.915322 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:36.915275 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-qtnl9\"" Jul 14 09:33:36.915322 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:36.915291 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 14 09:33:36.915629 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:36.915610 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 14 09:33:36.922955 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:36.922935 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-stf5s"] Jul 14 09:33:36.973174 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:36.973143 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5ww7\" (UniqueName: \"kubernetes.io/projected/851849d6-2fb4-4aa7-ac07-bbd0c3daaedd-kube-api-access-v5ww7\") pod \"kuadrant-console-plugin-66b5774985-stf5s\" (UID: \"851849d6-2fb4-4aa7-ac07-bbd0c3daaedd\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-stf5s" Jul 14 09:33:36.973349 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:36.973216 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/851849d6-2fb4-4aa7-ac07-bbd0c3daaedd-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-stf5s\" (UID: \"851849d6-2fb4-4aa7-ac07-bbd0c3daaedd\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-stf5s" Jul 14 09:33:36.973349 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:36.973290 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/851849d6-2fb4-4aa7-ac07-bbd0c3daaedd-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-stf5s\" (UID: \"851849d6-2fb4-4aa7-ac07-bbd0c3daaedd\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-stf5s" Jul 14 09:33:37.074169 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:37.074138 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5ww7\" (UniqueName: \"kubernetes.io/projected/851849d6-2fb4-4aa7-ac07-bbd0c3daaedd-kube-api-access-v5ww7\") pod \"kuadrant-console-plugin-66b5774985-stf5s\" (UID: \"851849d6-2fb4-4aa7-ac07-bbd0c3daaedd\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-stf5s" Jul 14 09:33:37.074385 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:37.074199 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/851849d6-2fb4-4aa7-ac07-bbd0c3daaedd-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-stf5s\" (UID: \"851849d6-2fb4-4aa7-ac07-bbd0c3daaedd\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-stf5s" Jul 14 09:33:37.074385 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:37.074256 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/851849d6-2fb4-4aa7-ac07-bbd0c3daaedd-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-stf5s\" (UID: \"851849d6-2fb4-4aa7-ac07-bbd0c3daaedd\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-stf5s" Jul 14 09:33:37.074771 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:37.074751 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/851849d6-2fb4-4aa7-ac07-bbd0c3daaedd-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-stf5s\" (UID: \"851849d6-2fb4-4aa7-ac07-bbd0c3daaedd\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-stf5s" Jul 14 09:33:37.076647 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:37.076616 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/851849d6-2fb4-4aa7-ac07-bbd0c3daaedd-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-stf5s\" (UID: \"851849d6-2fb4-4aa7-ac07-bbd0c3daaedd\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-stf5s" Jul 14 09:33:37.091764 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:37.091731 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5ww7\" (UniqueName: \"kubernetes.io/projected/851849d6-2fb4-4aa7-ac07-bbd0c3daaedd-kube-api-access-v5ww7\") pod \"kuadrant-console-plugin-66b5774985-stf5s\" (UID: \"851849d6-2fb4-4aa7-ac07-bbd0c3daaedd\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-stf5s" Jul 14 09:33:37.223399 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:37.223314 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-stf5s" Jul 14 09:33:37.352257 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:37.352230 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-stf5s"] Jul 14 09:33:37.354004 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:33:37.353977 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod851849d6_2fb4_4aa7_ac07_bbd0c3daaedd.slice/crio-36c77a2ace24c89fd3d429d8d258a343e6a97840bca41c2c3e9b62350f7b9348 WatchSource:0}: Error finding container 36c77a2ace24c89fd3d429d8d258a343e6a97840bca41c2c3e9b62350f7b9348: Status 404 returned error can't find the container with id 36c77a2ace24c89fd3d429d8d258a343e6a97840bca41c2c3e9b62350f7b9348 Jul 14 09:33:38.066445 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:38.066410 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-stf5s" event={"ID":"851849d6-2fb4-4aa7-ac07-bbd0c3daaedd","Type":"ContainerStarted","Data":"36c77a2ace24c89fd3d429d8d258a343e6a97840bca41c2c3e9b62350f7b9348"} Jul 14 09:33:44.045145 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:44.045112 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-829mm" Jul 14 09:33:46.051908 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:46.051873 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6cdd8bc874-bztpx" podUID="a1cd8e3b-7798-486f-89f1-d3522219c8ab" containerName="console" containerID="cri-o://a3e4981065c37b4341c34c8bda0d655d27666e92af8fe82e0be60fe6029a076a" gracePeriod=15 Jul 14 09:33:46.057695 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:46.057670 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vl9gm" Jul 14 09:33:48.112838 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:48.112786 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dz746"] Jul 14 09:33:48.134899 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:48.133445 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dz746" Jul 14 09:33:48.137367 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:48.137336 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dz746"] Jul 14 09:33:48.178586 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:48.178561 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgstl\" (UniqueName: \"kubernetes.io/projected/27a70d43-c911-43fe-98b9-1e8b7b938952-kube-api-access-jgstl\") pod \"kuadrant-operator-controller-manager-55467d9f7-dz746\" (UID: \"27a70d43-c911-43fe-98b9-1e8b7b938952\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dz746" Jul 14 09:33:48.178757 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:48.178603 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/27a70d43-c911-43fe-98b9-1e8b7b938952-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-dz746\" (UID: \"27a70d43-c911-43fe-98b9-1e8b7b938952\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dz746" Jul 14 09:33:48.256051 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:48.256018 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dz746"] Jul 14 09:33:48.256406 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:33:48.256377 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-jgstl], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dz746" podUID="27a70d43-c911-43fe-98b9-1e8b7b938952" Jul 14 09:33:48.263009 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:48.262979 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dz746"] Jul 14 09:33:48.279530 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:48.279483 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vl9gm"] Jul 14 09:33:48.279790 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:48.279740 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vl9gm" podUID="9745acfb-c8a2-48a5-ac27-cbb5e992be42" containerName="manager" containerID="cri-o://fc35341e085836497118d577953cc0a2c10bd0d6492de84d7ef8de390a05ff9c" gracePeriod=2 Jul 14 09:33:48.279934 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:48.279911 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgstl\" (UniqueName: \"kubernetes.io/projected/27a70d43-c911-43fe-98b9-1e8b7b938952-kube-api-access-jgstl\") pod \"kuadrant-operator-controller-manager-55467d9f7-dz746\" (UID: \"27a70d43-c911-43fe-98b9-1e8b7b938952\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dz746" Jul 14 09:33:48.279991 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:48.279974 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/27a70d43-c911-43fe-98b9-1e8b7b938952-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-dz746\" (UID: \"27a70d43-c911-43fe-98b9-1e8b7b938952\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dz746" Jul 14 09:33:48.280705 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:48.280367 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/27a70d43-c911-43fe-98b9-1e8b7b938952-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-dz746\" (UID: \"27a70d43-c911-43fe-98b9-1e8b7b938952\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dz746" Jul 14 09:33:48.287652 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:33:48.287631 2565 projected.go:196] Error preparing data for projected volume kube-api-access-jgstl for pod kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dz746: failed to fetch token: pod "kuadrant-operator-controller-manager-55467d9f7-dz746" not found Jul 14 09:33:48.287764 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:33:48.287712 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/27a70d43-c911-43fe-98b9-1e8b7b938952-kube-api-access-jgstl podName:27a70d43-c911-43fe-98b9-1e8b7b938952 nodeName:}" failed. No retries permitted until 2026-07-14 09:33:48.787689719 +0000 UTC m=+418.296921151 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jgstl" (UniqueName: "kubernetes.io/projected/27a70d43-c911-43fe-98b9-1e8b7b938952-kube-api-access-jgstl") pod "kuadrant-operator-controller-manager-55467d9f7-dz746" (UID: "27a70d43-c911-43fe-98b9-1e8b7b938952") : failed to fetch token: pod "kuadrant-operator-controller-manager-55467d9f7-dz746" not found Jul 14 09:33:48.294351 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:48.294321 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vl9gm"] Jul 14 09:33:48.295835 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:33:48.295779 2565 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"9745acfb-c8a2-48a5-ac27-cbb5e992be42\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-vl9gm\": pods \"kuadrant-operator-controller-manager-8684c9ff49-vl9gm\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vl9gm" Jul 14 09:33:48.297913 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:48.297759 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-hxwd4"] Jul 14 09:33:48.298432 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:48.298413 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9745acfb-c8a2-48a5-ac27-cbb5e992be42" containerName="manager" Jul 14 09:33:48.298545 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:48.298533 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9745acfb-c8a2-48a5-ac27-cbb5e992be42" containerName="manager" Jul 14 09:33:48.298769 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:48.298745 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9745acfb-c8a2-48a5-ac27-cbb5e992be42" containerName="manager" Jul 14 09:33:48.304755 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:48.304207 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-hxwd4" Jul 14 09:33:48.306846 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:33:48.306785 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-vl9gm\" is forbidden: User \"system:node:ip-10-0-132-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-2.ec2.internal' and this object" podUID="9745acfb-c8a2-48a5-ac27-cbb5e992be42" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vl9gm" Jul 14 09:33:48.316178 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:48.316152 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2twp9"] Jul 14 09:33:48.322402 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:48.322381 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2twp9" Jul 14 09:33:48.323734 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:48.323681 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-hxwd4"] Jul 14 09:33:48.353026 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:48.352986 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2twp9"] Jul 14 09:33:48.377732 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:33:48.377657 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-vl9gm\" is forbidden: User \"system:node:ip-10-0-132-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-2.ec2.internal' and this object" podUID="9745acfb-c8a2-48a5-ac27-cbb5e992be42" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vl9gm" Jul 14 09:33:48.381155 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:48.381131 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzdkh\" (UniqueName: \"kubernetes.io/projected/9369f51c-9787-4ccd-a017-3c041d461d6d-kube-api-access-vzdkh\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2twp9\" (UID: \"9369f51c-9787-4ccd-a017-3c041d461d6d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2twp9" Jul 14 09:33:48.381284 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:48.381256 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/03f20d58-13b7-48df-944f-ea50847d6128-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-hxwd4\" (UID: \"03f20d58-13b7-48df-944f-ea50847d6128\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-hxwd4" Jul 14 09:33:48.381343 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:48.381283 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9psrs\" (UniqueName: \"kubernetes.io/projected/03f20d58-13b7-48df-944f-ea50847d6128-kube-api-access-9psrs\") pod \"kuadrant-operator-controller-manager-55467d9f7-hxwd4\" (UID: \"03f20d58-13b7-48df-944f-ea50847d6128\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-hxwd4" Jul 14 09:33:48.381343 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:48.381317 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9369f51c-9787-4ccd-a017-3c041d461d6d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2twp9\" (UID: \"9369f51c-9787-4ccd-a017-3c041d461d6d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2twp9" Jul 14 09:33:48.435861 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:48.435824 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2twp9"] Jul 14 09:33:48.436274 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:33:48.436246 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-vzdkh], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2twp9" podUID="9369f51c-9787-4ccd-a017-3c041d461d6d" Jul 14 09:33:48.482371 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:48.482329 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/03f20d58-13b7-48df-944f-ea50847d6128-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-hxwd4\" (UID: \"03f20d58-13b7-48df-944f-ea50847d6128\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-hxwd4" Jul 14 09:33:48.482550 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:48.482380 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9psrs\" (UniqueName: \"kubernetes.io/projected/03f20d58-13b7-48df-944f-ea50847d6128-kube-api-access-9psrs\") pod \"kuadrant-operator-controller-manager-55467d9f7-hxwd4\" (UID: \"03f20d58-13b7-48df-944f-ea50847d6128\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-hxwd4" Jul 14 09:33:48.482550 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:48.482415 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9369f51c-9787-4ccd-a017-3c041d461d6d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2twp9\" (UID: \"9369f51c-9787-4ccd-a017-3c041d461d6d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2twp9" Jul 14 09:33:48.482550 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:48.482520 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzdkh\" (UniqueName: \"kubernetes.io/projected/9369f51c-9787-4ccd-a017-3c041d461d6d-kube-api-access-vzdkh\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2twp9\" (UID: \"9369f51c-9787-4ccd-a017-3c041d461d6d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2twp9" Jul 14 09:33:48.482763 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:48.482736 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/03f20d58-13b7-48df-944f-ea50847d6128-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-hxwd4\" (UID: \"03f20d58-13b7-48df-944f-ea50847d6128\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-hxwd4" Jul 14 09:33:48.483034 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:48.483015 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9369f51c-9787-4ccd-a017-3c041d461d6d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2twp9\" (UID: \"9369f51c-9787-4ccd-a017-3c041d461d6d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2twp9" Jul 14 09:33:48.498208 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:48.498186 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzdkh\" (UniqueName: \"kubernetes.io/projected/9369f51c-9787-4ccd-a017-3c041d461d6d-kube-api-access-vzdkh\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2twp9\" (UID: \"9369f51c-9787-4ccd-a017-3c041d461d6d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2twp9" Jul 14 09:33:48.508764 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:48.508712 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9psrs\" (UniqueName: \"kubernetes.io/projected/03f20d58-13b7-48df-944f-ea50847d6128-kube-api-access-9psrs\") pod \"kuadrant-operator-controller-manager-55467d9f7-hxwd4\" (UID: \"03f20d58-13b7-48df-944f-ea50847d6128\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-hxwd4" Jul 14 09:33:48.660605 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:48.660538 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-hxwd4" Jul 14 09:33:48.885970 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:48.885934 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgstl\" (UniqueName: \"kubernetes.io/projected/27a70d43-c911-43fe-98b9-1e8b7b938952-kube-api-access-jgstl\") pod \"kuadrant-operator-controller-manager-55467d9f7-dz746\" (UID: \"27a70d43-c911-43fe-98b9-1e8b7b938952\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dz746" Jul 14 09:33:48.888859 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:33:48.888836 2565 projected.go:196] Error preparing data for projected volume kube-api-access-jgstl for pod kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dz746: failed to fetch token: pod "kuadrant-operator-controller-manager-55467d9f7-dz746" not found Jul 14 09:33:48.889016 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:33:48.888912 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/27a70d43-c911-43fe-98b9-1e8b7b938952-kube-api-access-jgstl podName:27a70d43-c911-43fe-98b9-1e8b7b938952 nodeName:}" failed. No retries permitted until 2026-07-14 09:33:49.888892303 +0000 UTC m=+419.398123729 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jgstl" (UniqueName: "kubernetes.io/projected/27a70d43-c911-43fe-98b9-1e8b7b938952-kube-api-access-jgstl") pod "kuadrant-operator-controller-manager-55467d9f7-dz746" (UID: "27a70d43-c911-43fe-98b9-1e8b7b938952") : failed to fetch token: pod "kuadrant-operator-controller-manager-55467d9f7-dz746" not found Jul 14 09:33:49.151212 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:49.151184 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2twp9" Jul 14 09:33:49.151634 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:49.151189 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dz746" Jul 14 09:33:49.166987 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:49.166959 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2twp9" Jul 14 09:33:49.170574 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:49.170550 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dz746" Jul 14 09:33:49.179491 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:33:49.179461 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-dz746\" is forbidden: User \"system:node:ip-10-0-132-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-2.ec2.internal' and this object" podUID="27a70d43-c911-43fe-98b9-1e8b7b938952" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dz746" Jul 14 09:33:49.289786 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:49.289751 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vzdkh\" (UniqueName: \"kubernetes.io/projected/9369f51c-9787-4ccd-a017-3c041d461d6d-kube-api-access-vzdkh\") pod \"9369f51c-9787-4ccd-a017-3c041d461d6d\" (UID: \"9369f51c-9787-4ccd-a017-3c041d461d6d\") " Jul 14 09:33:49.289968 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:49.289928 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9369f51c-9787-4ccd-a017-3c041d461d6d-extensions-socket-volume\") pod \"9369f51c-9787-4ccd-a017-3c041d461d6d\" (UID: \"9369f51c-9787-4ccd-a017-3c041d461d6d\") " Jul 14 09:33:49.290034 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:49.289967 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/27a70d43-c911-43fe-98b9-1e8b7b938952-extensions-socket-volume\") pod \"27a70d43-c911-43fe-98b9-1e8b7b938952\" (UID: \"27a70d43-c911-43fe-98b9-1e8b7b938952\") " Jul 14 09:33:49.290207 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:49.290175 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9369f51c-9787-4ccd-a017-3c041d461d6d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9369f51c-9787-4ccd-a017-3c041d461d6d" (UID: "9369f51c-9787-4ccd-a017-3c041d461d6d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 09:33:49.290294 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:49.290207 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgstl\" (UniqueName: \"kubernetes.io/projected/27a70d43-c911-43fe-98b9-1e8b7b938952-kube-api-access-jgstl\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:33:49.290346 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:49.290315 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27a70d43-c911-43fe-98b9-1e8b7b938952-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "27a70d43-c911-43fe-98b9-1e8b7b938952" (UID: "27a70d43-c911-43fe-98b9-1e8b7b938952"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 09:33:49.292515 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:49.292492 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9369f51c-9787-4ccd-a017-3c041d461d6d-kube-api-access-vzdkh" (OuterVolumeSpecName: "kube-api-access-vzdkh") pod "9369f51c-9787-4ccd-a017-3c041d461d6d" (UID: "9369f51c-9787-4ccd-a017-3c041d461d6d"). InnerVolumeSpecName "kube-api-access-vzdkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 09:33:49.391511 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:49.391479 2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9369f51c-9787-4ccd-a017-3c041d461d6d-extensions-socket-volume\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:33:49.391511 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:49.391511 2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/27a70d43-c911-43fe-98b9-1e8b7b938952-extensions-socket-volume\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:33:49.391704 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:49.391525 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzdkh\" (UniqueName: \"kubernetes.io/projected/9369f51c-9787-4ccd-a017-3c041d461d6d-kube-api-access-vzdkh\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:33:50.157234 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:50.157197 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dz746" Jul 14 09:33:50.157731 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:50.157366 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2twp9" Jul 14 09:33:50.159712 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:33:50.159675 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-dz746\" is forbidden: User \"system:node:ip-10-0-132-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-2.ec2.internal' and this object" podUID="27a70d43-c911-43fe-98b9-1e8b7b938952" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dz746" Jul 14 09:33:50.168994 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:33:50.168969 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-dz746\" is forbidden: User \"system:node:ip-10-0-132-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-2.ec2.internal' and this object" podUID="27a70d43-c911-43fe-98b9-1e8b7b938952" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dz746" Jul 14 09:33:50.170786 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:33:50.170761 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-dz746\" is forbidden: User \"system:node:ip-10-0-132-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-2.ec2.internal' and this object" podUID="27a70d43-c911-43fe-98b9-1e8b7b938952" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dz746" Jul 14 09:33:50.201490 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:50.201425 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2twp9"] Jul 14 09:33:50.204003 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:50.203976 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2twp9"] Jul 14 09:33:50.968394 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:33:50.968349 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-dz746\" is forbidden: User \"system:node:ip-10-0-132-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-2.ec2.internal' and this object" podUID="27a70d43-c911-43fe-98b9-1e8b7b938952" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-dz746" Jul 14 09:33:50.971128 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:50.971098 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27a70d43-c911-43fe-98b9-1e8b7b938952" path="/var/lib/kubelet/pods/27a70d43-c911-43fe-98b9-1e8b7b938952/volumes" Jul 14 09:33:50.971543 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:50.971521 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9369f51c-9787-4ccd-a017-3c041d461d6d" path="/var/lib/kubelet/pods/9369f51c-9787-4ccd-a017-3c041d461d6d/volumes" Jul 14 09:33:55.823531 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:55.823489 2565 patch_prober.go:28] interesting pod/console-6cdd8bc874-bztpx container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.134.0.22:8443/health\": dial tcp 10.134.0.22:8443: connect: connection refused" start-of-body= Jul 14 09:33:55.823973 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:33:55.823558 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-6cdd8bc874-bztpx" podUID="a1cd8e3b-7798-486f-89f1-d3522219c8ab" containerName="console" probeResult="failure" output="Get \"https://10.134.0.22:8443/health\": dial tcp 10.134.0.22:8443: connect: connection refused" Jul 14 09:34:00.225879 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:00.225736 2565 generic.go:356] "Generic (PLEG): container finished" podID="a1cd8e3b-7798-486f-89f1-d3522219c8ab" containerID="a3e4981065c37b4341c34c8bda0d655d27666e92af8fe82e0be60fe6029a076a" exitCode=0 Jul 14 09:34:00.226480 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:00.225989 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cdd8bc874-bztpx" event={"ID":"a1cd8e3b-7798-486f-89f1-d3522219c8ab","Type":"ContainerDied","Data":"a3e4981065c37b4341c34c8bda0d655d27666e92af8fe82e0be60fe6029a076a"} Jul 14 09:34:00.230032 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:00.229961 2565 generic.go:356] "Generic (PLEG): container finished" podID="9745acfb-c8a2-48a5-ac27-cbb5e992be42" containerID="fc35341e085836497118d577953cc0a2c10bd0d6492de84d7ef8de390a05ff9c" exitCode=0 Jul 14 09:34:00.336078 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:00.336041 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vl9gm" Jul 14 09:34:00.338539 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:34:00.338510 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-vl9gm\" is forbidden: User \"system:node:ip-10-0-132-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-2.ec2.internal' and this object" podUID="9745acfb-c8a2-48a5-ac27-cbb5e992be42" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vl9gm" Jul 14 09:34:00.339454 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:00.339436 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cdd8bc874-bztpx" Jul 14 09:34:00.341786 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:34:00.341759 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-vl9gm\" is forbidden: User \"system:node:ip-10-0-132-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-2.ec2.internal' and this object" podUID="9745acfb-c8a2-48a5-ac27-cbb5e992be42" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vl9gm" Jul 14 09:34:00.347631 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:00.347608 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-hxwd4"] Jul 14 09:34:00.348750 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:34:00.348727 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03f20d58_13b7_48df_944f_ea50847d6128.slice/crio-0d0f2580178e86dcc095947ad4591a3e22525fb3201fc78ba9ec2d471255f7f0 WatchSource:0}: Error finding container 0d0f2580178e86dcc095947ad4591a3e22525fb3201fc78ba9ec2d471255f7f0: Status 404 returned error can't find the container with id 0d0f2580178e86dcc095947ad4591a3e22525fb3201fc78ba9ec2d471255f7f0 Jul 14 09:34:00.503360 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:00.503273 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdblk\" (UniqueName: \"kubernetes.io/projected/a1cd8e3b-7798-486f-89f1-d3522219c8ab-kube-api-access-gdblk\") pod \"a1cd8e3b-7798-486f-89f1-d3522219c8ab\" (UID: \"a1cd8e3b-7798-486f-89f1-d3522219c8ab\") " Jul 14 09:34:00.503360 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:00.503337 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1cd8e3b-7798-486f-89f1-d3522219c8ab-trusted-ca-bundle\") pod \"a1cd8e3b-7798-486f-89f1-d3522219c8ab\" (UID: \"a1cd8e3b-7798-486f-89f1-d3522219c8ab\") " Jul 14 09:34:00.503553 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:00.503367 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a1cd8e3b-7798-486f-89f1-d3522219c8ab-service-ca\") pod \"a1cd8e3b-7798-486f-89f1-d3522219c8ab\" (UID: \"a1cd8e3b-7798-486f-89f1-d3522219c8ab\") " Jul 14 09:34:00.503553 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:00.503392 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a1cd8e3b-7798-486f-89f1-d3522219c8ab-console-oauth-config\") pod \"a1cd8e3b-7798-486f-89f1-d3522219c8ab\" (UID: \"a1cd8e3b-7798-486f-89f1-d3522219c8ab\") " Jul 14 09:34:00.503553 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:00.503406 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1cd8e3b-7798-486f-89f1-d3522219c8ab-console-serving-cert\") pod \"a1cd8e3b-7798-486f-89f1-d3522219c8ab\" (UID: \"a1cd8e3b-7798-486f-89f1-d3522219c8ab\") " Jul 14 09:34:00.503553 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:00.503447 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a1cd8e3b-7798-486f-89f1-d3522219c8ab-oauth-serving-cert\") pod \"a1cd8e3b-7798-486f-89f1-d3522219c8ab\" (UID: \"a1cd8e3b-7798-486f-89f1-d3522219c8ab\") " Jul 14 09:34:00.503553 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:00.503511 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9745acfb-c8a2-48a5-ac27-cbb5e992be42-extensions-socket-volume\") pod \"9745acfb-c8a2-48a5-ac27-cbb5e992be42\" (UID: \"9745acfb-c8a2-48a5-ac27-cbb5e992be42\") " Jul 14 09:34:00.503553 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:00.503548 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a1cd8e3b-7798-486f-89f1-d3522219c8ab-console-config\") pod \"a1cd8e3b-7798-486f-89f1-d3522219c8ab\" (UID: \"a1cd8e3b-7798-486f-89f1-d3522219c8ab\") " Jul 14 09:34:00.503885 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:00.503825 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1cd8e3b-7798-486f-89f1-d3522219c8ab-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a1cd8e3b-7798-486f-89f1-d3522219c8ab" (UID: "a1cd8e3b-7798-486f-89f1-d3522219c8ab"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 09:34:00.503885 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:00.503832 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1cd8e3b-7798-486f-89f1-d3522219c8ab-service-ca" (OuterVolumeSpecName: "service-ca") pod "a1cd8e3b-7798-486f-89f1-d3522219c8ab" (UID: "a1cd8e3b-7798-486f-89f1-d3522219c8ab"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 09:34:00.503995 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:00.503894 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptlzb\" (UniqueName: \"kubernetes.io/projected/9745acfb-c8a2-48a5-ac27-cbb5e992be42-kube-api-access-ptlzb\") pod \"9745acfb-c8a2-48a5-ac27-cbb5e992be42\" (UID: \"9745acfb-c8a2-48a5-ac27-cbb5e992be42\") " Jul 14 09:34:00.504076 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:00.503951 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1cd8e3b-7798-486f-89f1-d3522219c8ab-console-config" (OuterVolumeSpecName: "console-config") pod "a1cd8e3b-7798-486f-89f1-d3522219c8ab" (UID: "a1cd8e3b-7798-486f-89f1-d3522219c8ab"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 09:34:00.504152 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:00.504059 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1cd8e3b-7798-486f-89f1-d3522219c8ab-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a1cd8e3b-7798-486f-89f1-d3522219c8ab" (UID: "a1cd8e3b-7798-486f-89f1-d3522219c8ab"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 09:34:00.504259 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:00.504242 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1cd8e3b-7798-486f-89f1-d3522219c8ab-trusted-ca-bundle\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:34:00.504329 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:00.504264 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a1cd8e3b-7798-486f-89f1-d3522219c8ab-service-ca\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:34:00.504329 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:00.504278 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a1cd8e3b-7798-486f-89f1-d3522219c8ab-oauth-serving-cert\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:34:00.504329 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:00.504292 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a1cd8e3b-7798-486f-89f1-d3522219c8ab-console-config\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:34:00.504422 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:00.504360 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9745acfb-c8a2-48a5-ac27-cbb5e992be42-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9745acfb-c8a2-48a5-ac27-cbb5e992be42" (UID: "9745acfb-c8a2-48a5-ac27-cbb5e992be42"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 09:34:00.505670 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:00.505651 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1cd8e3b-7798-486f-89f1-d3522219c8ab-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a1cd8e3b-7798-486f-89f1-d3522219c8ab" (UID: "a1cd8e3b-7798-486f-89f1-d3522219c8ab"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 09:34:00.505845 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:00.505829 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9745acfb-c8a2-48a5-ac27-cbb5e992be42-kube-api-access-ptlzb" (OuterVolumeSpecName: "kube-api-access-ptlzb") pod "9745acfb-c8a2-48a5-ac27-cbb5e992be42" (UID: "9745acfb-c8a2-48a5-ac27-cbb5e992be42"). InnerVolumeSpecName "kube-api-access-ptlzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 09:34:00.505996 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:00.505971 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1cd8e3b-7798-486f-89f1-d3522219c8ab-kube-api-access-gdblk" (OuterVolumeSpecName: "kube-api-access-gdblk") pod "a1cd8e3b-7798-486f-89f1-d3522219c8ab" (UID: "a1cd8e3b-7798-486f-89f1-d3522219c8ab"). InnerVolumeSpecName "kube-api-access-gdblk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 09:34:00.505996 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:00.505971 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1cd8e3b-7798-486f-89f1-d3522219c8ab-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a1cd8e3b-7798-486f-89f1-d3522219c8ab" (UID: "a1cd8e3b-7798-486f-89f1-d3522219c8ab"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 09:34:00.604840 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:00.604784 2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9745acfb-c8a2-48a5-ac27-cbb5e992be42-extensions-socket-volume\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:34:00.604840 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:00.604829 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptlzb\" (UniqueName: \"kubernetes.io/projected/9745acfb-c8a2-48a5-ac27-cbb5e992be42-kube-api-access-ptlzb\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:34:00.604840 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:00.604839 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdblk\" (UniqueName: \"kubernetes.io/projected/a1cd8e3b-7798-486f-89f1-d3522219c8ab-kube-api-access-gdblk\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:34:00.604840 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:00.604848 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a1cd8e3b-7798-486f-89f1-d3522219c8ab-console-oauth-config\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:34:00.605200 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:00.604856 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1cd8e3b-7798-486f-89f1-d3522219c8ab-console-serving-cert\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:34:00.968033 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:34:00.967989 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-vl9gm\" is forbidden: User \"system:node:ip-10-0-132-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-2.ec2.internal' and this object" podUID="9745acfb-c8a2-48a5-ac27-cbb5e992be42" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vl9gm" Jul 14 09:34:00.968961 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:00.968940 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9745acfb-c8a2-48a5-ac27-cbb5e992be42" path="/var/lib/kubelet/pods/9745acfb-c8a2-48a5-ac27-cbb5e992be42/volumes" Jul 14 09:34:01.237603 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:01.237505 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-hxwd4" event={"ID":"03f20d58-13b7-48df-944f-ea50847d6128","Type":"ContainerStarted","Data":"1f923aecf8466601f56fef58ab088e7e18a6d8d8f8be8802144237d3434cb76e"} Jul 14 09:34:01.237603 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:01.237562 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-hxwd4" event={"ID":"03f20d58-13b7-48df-944f-ea50847d6128","Type":"ContainerStarted","Data":"0d0f2580178e86dcc095947ad4591a3e22525fb3201fc78ba9ec2d471255f7f0"} Jul 14 09:34:01.238112 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:01.237625 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-hxwd4" Jul 14 09:34:01.239266 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:01.239242 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-stf5s" event={"ID":"851849d6-2fb4-4aa7-ac07-bbd0c3daaedd","Type":"ContainerStarted","Data":"4732369787d1e6247a730c9312fb724e825e60c179829bfc009ae58e5ec86d64"} Jul 14 09:34:01.240455 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:01.240441 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vl9gm" Jul 14 09:34:01.240549 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:01.240462 2565 scope.go:117] "RemoveContainer" containerID="fc35341e085836497118d577953cc0a2c10bd0d6492de84d7ef8de390a05ff9c" Jul 14 09:34:01.241728 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:01.241711 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cdd8bc874-bztpx" Jul 14 09:34:01.241839 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:01.241713 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cdd8bc874-bztpx" event={"ID":"a1cd8e3b-7798-486f-89f1-d3522219c8ab","Type":"ContainerDied","Data":"2b332db0503584efb8443e9b12ac71acdebf94901ded9ded266632022ec9c9ad"} Jul 14 09:34:01.249618 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:01.249604 2565 scope.go:117] "RemoveContainer" containerID="a3e4981065c37b4341c34c8bda0d655d27666e92af8fe82e0be60fe6029a076a" Jul 14 09:34:01.267766 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:01.267717 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-hxwd4" podStartSLOduration=13.267704104 podStartE2EDuration="13.267704104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 09:33:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:34:01.266258157 +0000 UTC m=+430.775489623" watchObservedRunningTime="2026-07-14 09:34:01.267704104 +0000 UTC m=+430.776935551" Jul 14 09:34:01.292457 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:34:01.292422 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-vl9gm\" is forbidden: User \"system:node:ip-10-0-132-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-2.ec2.internal' and this object" podUID="9745acfb-c8a2-48a5-ac27-cbb5e992be42" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vl9gm" Jul 14 09:34:01.313506 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:01.313455 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-stf5s" podStartSLOduration=2.399696753 podStartE2EDuration="25.313439378s" totalImagesPullingTime="22.913742625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 09:33:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:34:01.290229777 +0000 UTC m=+430.799461226" watchObservedRunningTime="2026-07-14 09:34:01.313439378 +0000 UTC m=+430.822670869" Jul 14 09:34:01.313992 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:01.313976 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cdd8bc874-bztpx"] Jul 14 09:34:01.323904 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:01.323878 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6cdd8bc874-bztpx"] Jul 14 09:34:02.968784 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:02.968746 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1cd8e3b-7798-486f-89f1-d3522219c8ab" path="/var/lib/kubelet/pods/a1cd8e3b-7798-486f-89f1-d3522219c8ab/volumes" Jul 14 09:34:05.849100 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:05.849067 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-hxwd4"] Jul 14 09:34:05.849514 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:05.849267 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-hxwd4" podUID="03f20d58-13b7-48df-944f-ea50847d6128" containerName="manager" containerID="cri-o://1f923aecf8466601f56fef58ab088e7e18a6d8d8f8be8802144237d3434cb76e" gracePeriod=10 Jul 14 09:34:05.851266 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:05.851246 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-hxwd4" Jul 14 09:34:06.274013 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:06.273979 2565 generic.go:356] "Generic (PLEG): container finished" podID="03f20d58-13b7-48df-944f-ea50847d6128" containerID="1f923aecf8466601f56fef58ab088e7e18a6d8d8f8be8802144237d3434cb76e" exitCode=0 Jul 14 09:34:06.274198 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:06.274053 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-hxwd4" event={"ID":"03f20d58-13b7-48df-944f-ea50847d6128","Type":"ContainerDied","Data":"1f923aecf8466601f56fef58ab088e7e18a6d8d8f8be8802144237d3434cb76e"} Jul 14 09:34:06.646997 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:06.646970 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-hxwd4" Jul 14 09:34:06.762947 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:06.762911 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/03f20d58-13b7-48df-944f-ea50847d6128-extensions-socket-volume\") pod \"03f20d58-13b7-48df-944f-ea50847d6128\" (UID: \"03f20d58-13b7-48df-944f-ea50847d6128\") " Jul 14 09:34:06.763142 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:06.762976 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9psrs\" (UniqueName: \"kubernetes.io/projected/03f20d58-13b7-48df-944f-ea50847d6128-kube-api-access-9psrs\") pod \"03f20d58-13b7-48df-944f-ea50847d6128\" (UID: \"03f20d58-13b7-48df-944f-ea50847d6128\") " Jul 14 09:34:06.763365 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:06.763340 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03f20d58-13b7-48df-944f-ea50847d6128-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "03f20d58-13b7-48df-944f-ea50847d6128" (UID: "03f20d58-13b7-48df-944f-ea50847d6128"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 09:34:06.765072 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:06.765053 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03f20d58-13b7-48df-944f-ea50847d6128-kube-api-access-9psrs" (OuterVolumeSpecName: "kube-api-access-9psrs") pod "03f20d58-13b7-48df-944f-ea50847d6128" (UID: "03f20d58-13b7-48df-944f-ea50847d6128"). InnerVolumeSpecName "kube-api-access-9psrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 09:34:06.864594 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:06.864514 2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/03f20d58-13b7-48df-944f-ea50847d6128-extensions-socket-volume\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:34:06.864594 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:06.864545 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9psrs\" (UniqueName: \"kubernetes.io/projected/03f20d58-13b7-48df-944f-ea50847d6128-kube-api-access-9psrs\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:34:07.280214 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:07.280184 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-hxwd4" Jul 14 09:34:07.280380 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:07.280222 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-hxwd4" event={"ID":"03f20d58-13b7-48df-944f-ea50847d6128","Type":"ContainerDied","Data":"0d0f2580178e86dcc095947ad4591a3e22525fb3201fc78ba9ec2d471255f7f0"} Jul 14 09:34:07.280380 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:07.280278 2565 scope.go:117] "RemoveContainer" containerID="1f923aecf8466601f56fef58ab088e7e18a6d8d8f8be8802144237d3434cb76e" Jul 14 09:34:07.300069 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:07.300035 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-hxwd4"] Jul 14 09:34:07.306943 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:07.306919 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-hxwd4"] Jul 14 09:34:08.969403 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:08.969368 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03f20d58-13b7-48df-944f-ea50847d6128" path="/var/lib/kubelet/pods/03f20d58-13b7-48df-944f-ea50847d6128/volumes" Jul 14 09:34:22.273698 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:22.273664 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6c9dg"] Jul 14 09:34:22.274300 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:22.274077 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03f20d58-13b7-48df-944f-ea50847d6128" containerName="manager" Jul 14 09:34:22.274300 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:22.274092 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="03f20d58-13b7-48df-944f-ea50847d6128" containerName="manager" Jul 14 09:34:22.274300 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:22.274110 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1cd8e3b-7798-486f-89f1-d3522219c8ab" containerName="console" Jul 14 09:34:22.274300 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:22.274116 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1cd8e3b-7798-486f-89f1-d3522219c8ab" containerName="console" Jul 14 09:34:22.274300 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:22.274200 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1cd8e3b-7798-486f-89f1-d3522219c8ab" containerName="console" Jul 14 09:34:22.274300 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:22.274217 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="03f20d58-13b7-48df-944f-ea50847d6128" containerName="manager" Jul 14 09:34:22.322290 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:22.322251 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6c9dg"] Jul 14 09:34:22.322489 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:22.322363 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6c9dg" Jul 14 09:34:22.325278 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:22.325245 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 14 09:34:22.325278 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:22.325274 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 14 09:34:22.325466 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:22.325289 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 14 09:34:22.325466 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:22.325245 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-9mjs7\"" Jul 14 09:34:22.404159 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:22.404129 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-6c9dg\" (UID: \"5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6c9dg" Jul 14 09:34:22.404159 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:22.404159 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-6c9dg\" (UID: \"5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6c9dg" Jul 14 09:34:22.404429 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:22.404189 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf855\" (UniqueName: \"kubernetes.io/projected/5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a-kube-api-access-sf855\") pod \"maas-default-gateway-openshift-default-56654dcf7-6c9dg\" (UID: \"5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6c9dg" Jul 14 09:34:22.404429 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:22.404271 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-6c9dg\" (UID: \"5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6c9dg" Jul 14 09:34:22.404429 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:22.404318 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-6c9dg\" (UID: \"5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6c9dg" Jul 14 09:34:22.404429 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:22.404373 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-6c9dg\" (UID: \"5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6c9dg" Jul 14 09:34:22.404658 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:22.404441 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-6c9dg\" (UID: \"5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6c9dg" Jul 14 09:34:22.404658 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:22.404479 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-6c9dg\" (UID: \"5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6c9dg" Jul 14 09:34:22.404658 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:22.404508 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-6c9dg\" (UID: \"5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6c9dg" Jul 14 09:34:22.505577 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:22.505538 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-6c9dg\" (UID: \"5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6c9dg" Jul 14 09:34:22.505759 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:22.505594 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-6c9dg\" (UID: \"5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6c9dg" Jul 14 09:34:22.505759 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:22.505615 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-6c9dg\" (UID: \"5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6c9dg" Jul 14 09:34:22.505759 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:22.505636 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-6c9dg\" (UID: \"5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6c9dg" Jul 14 09:34:22.505759 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:22.505669 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-6c9dg\" (UID: \"5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6c9dg" Jul 14 09:34:22.505759 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:22.505686 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-6c9dg\" (UID: \"5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6c9dg" Jul 14 09:34:22.505759 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:22.505706 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf855\" (UniqueName: \"kubernetes.io/projected/5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a-kube-api-access-sf855\") pod \"maas-default-gateway-openshift-default-56654dcf7-6c9dg\" (UID: \"5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6c9dg" Jul 14 09:34:22.505759 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:22.505739 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-6c9dg\" (UID: \"5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6c9dg" Jul 14 09:34:22.506134 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:22.505769 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-6c9dg\" (UID: \"5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6c9dg" Jul 14 09:34:22.506134 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:22.506120 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-6c9dg\" (UID: \"5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6c9dg" Jul 14 09:34:22.506239 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:22.506130 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-6c9dg\" (UID: \"5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6c9dg" Jul 14 09:34:22.506239 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:22.506187 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-6c9dg\" (UID: \"5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6c9dg" Jul 14 09:34:22.506342 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:22.506327 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-6c9dg\" (UID: \"5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6c9dg" Jul 14 09:34:22.506528 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:22.506513 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-6c9dg\" (UID: \"5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6c9dg" Jul 14 09:34:22.507952 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:22.507929 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-6c9dg\" (UID: \"5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6c9dg" Jul 14 09:34:22.508392 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:22.508375 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-6c9dg\" (UID: \"5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6c9dg" Jul 14 09:34:22.513862 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:22.513841 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-6c9dg\" (UID: \"5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6c9dg" Jul 14 09:34:22.514079 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:22.514060 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf855\" (UniqueName: \"kubernetes.io/projected/5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a-kube-api-access-sf855\") pod \"maas-default-gateway-openshift-default-56654dcf7-6c9dg\" (UID: \"5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6c9dg" Jul 14 09:34:22.633610 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:22.633517 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6c9dg" Jul 14 09:34:22.772872 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:22.772841 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6c9dg"] Jul 14 09:34:22.774331 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:34:22.774305 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5506cd2a_20fa_48d2_a5bb_4e2b81d2c35a.slice/crio-6c9f0251228889acb1e3f4059b01e2ce497c91bb3dc90609064342d24e94d29c WatchSource:0}: Error finding container 6c9f0251228889acb1e3f4059b01e2ce497c91bb3dc90609064342d24e94d29c: Status 404 returned error can't find the container with id 6c9f0251228889acb1e3f4059b01e2ce497c91bb3dc90609064342d24e94d29c Jul 14 09:34:23.373777 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:23.373741 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6c9dg" event={"ID":"5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a","Type":"ContainerStarted","Data":"6c9f0251228889acb1e3f4059b01e2ce497c91bb3dc90609064342d24e94d29c"} Jul 14 09:34:25.284038 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:25.283994 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 09:34:25.284305 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:25.284085 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 09:34:25.284305 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:25.284115 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 09:34:25.388628 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:25.388589 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6c9dg" event={"ID":"5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a","Type":"ContainerStarted","Data":"645c14f3394e752b1a1594d5aac47966ccde101bf7ca7be4c448938b654da883"} Jul 14 09:34:25.412930 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:25.412879 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6c9dg" podStartSLOduration=0.905545259 podStartE2EDuration="3.412862752s" totalImagesPullingTime="2.507317493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 09:34:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:34:25.410212562 +0000 UTC m=+454.919444011" watchObservedRunningTime="2026-07-14 09:34:25.412862752 +0000 UTC m=+454.922094200" Jul 14 09:34:25.634711 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:25.634636 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6c9dg" Jul 14 09:34:26.489481 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:26.489442 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-jqbx4"] Jul 14 09:34:26.492877 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:26.492854 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-jqbx4" Jul 14 09:34:26.495412 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:26.495389 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 14 09:34:26.502836 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:26.502772 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-jqbx4"] Jul 14 09:34:26.545154 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:26.545112 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4761a03d-b3b2-40c7-925d-387bf34bc755-config-file\") pod \"limitador-limitador-64c94fd64f-jqbx4\" (UID: \"4761a03d-b3b2-40c7-925d-387bf34bc755\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-jqbx4" Jul 14 09:34:26.545339 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:26.545167 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db8sv\" (UniqueName: \"kubernetes.io/projected/4761a03d-b3b2-40c7-925d-387bf34bc755-kube-api-access-db8sv\") pod \"limitador-limitador-64c94fd64f-jqbx4\" (UID: \"4761a03d-b3b2-40c7-925d-387bf34bc755\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-jqbx4" Jul 14 09:34:26.593987 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:26.593940 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-jqbx4"] Jul 14 09:34:26.639145 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:26.639116 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6c9dg" Jul 14 09:34:26.645795 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:26.645774 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4761a03d-b3b2-40c7-925d-387bf34bc755-config-file\") pod \"limitador-limitador-64c94fd64f-jqbx4\" (UID: \"4761a03d-b3b2-40c7-925d-387bf34bc755\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-jqbx4" Jul 14 09:34:26.645930 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:26.645826 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db8sv\" (UniqueName: \"kubernetes.io/projected/4761a03d-b3b2-40c7-925d-387bf34bc755-kube-api-access-db8sv\") pod \"limitador-limitador-64c94fd64f-jqbx4\" (UID: \"4761a03d-b3b2-40c7-925d-387bf34bc755\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-jqbx4" Jul 14 09:34:26.646431 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:26.646412 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4761a03d-b3b2-40c7-925d-387bf34bc755-config-file\") pod \"limitador-limitador-64c94fd64f-jqbx4\" (UID: \"4761a03d-b3b2-40c7-925d-387bf34bc755\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-jqbx4" Jul 14 09:34:26.654296 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:26.654278 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-db8sv\" (UniqueName: \"kubernetes.io/projected/4761a03d-b3b2-40c7-925d-387bf34bc755-kube-api-access-db8sv\") pod \"limitador-limitador-64c94fd64f-jqbx4\" (UID: \"4761a03d-b3b2-40c7-925d-387bf34bc755\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-jqbx4" Jul 14 09:34:26.803711 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:26.803613 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-jqbx4" Jul 14 09:34:26.946504 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:26.946471 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-jqbx4"] Jul 14 09:34:26.948020 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:34:26.947991 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4761a03d_b3b2_40c7_925d_387bf34bc755.slice/crio-2cbc08f1051266ab1103f8204d0d440a5dd9c656f49335ea3ade4a316c053b20 WatchSource:0}: Error finding container 2cbc08f1051266ab1103f8204d0d440a5dd9c656f49335ea3ade4a316c053b20: Status 404 returned error can't find the container with id 2cbc08f1051266ab1103f8204d0d440a5dd9c656f49335ea3ade4a316c053b20 Jul 14 09:34:27.400447 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:27.400413 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-jqbx4" event={"ID":"4761a03d-b3b2-40c7-925d-387bf34bc755","Type":"ContainerStarted","Data":"2cbc08f1051266ab1103f8204d0d440a5dd9c656f49335ea3ade4a316c053b20"} Jul 14 09:34:27.400626 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:27.400600 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6c9dg" Jul 14 09:34:27.401582 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:27.401561 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6c9dg" Jul 14 09:34:27.639484 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:27.639454 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-9fz49"] Jul 14 09:34:27.642902 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:27.642870 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-9fz49" Jul 14 09:34:27.645631 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:27.645600 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-nm889\"" Jul 14 09:34:27.650385 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:27.650355 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-9fz49"] Jul 14 09:34:27.756542 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:27.756388 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwxqx\" (UniqueName: \"kubernetes.io/projected/fe74cc74-df63-4a0e-b716-2c738a01afce-kube-api-access-nwxqx\") pod \"authorino-75d556bd77-9fz49\" (UID: \"fe74cc74-df63-4a0e-b716-2c738a01afce\") " pod="kuadrant-system/authorino-75d556bd77-9fz49" Jul 14 09:34:27.857673 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:27.857634 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwxqx\" (UniqueName: \"kubernetes.io/projected/fe74cc74-df63-4a0e-b716-2c738a01afce-kube-api-access-nwxqx\") pod \"authorino-75d556bd77-9fz49\" (UID: \"fe74cc74-df63-4a0e-b716-2c738a01afce\") " pod="kuadrant-system/authorino-75d556bd77-9fz49" Jul 14 09:34:27.868359 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:27.868298 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwxqx\" (UniqueName: \"kubernetes.io/projected/fe74cc74-df63-4a0e-b716-2c738a01afce-kube-api-access-nwxqx\") pod \"authorino-75d556bd77-9fz49\" (UID: \"fe74cc74-df63-4a0e-b716-2c738a01afce\") " pod="kuadrant-system/authorino-75d556bd77-9fz49" Jul 14 09:34:27.956104 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:27.956001 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-9fz49" Jul 14 09:34:28.137716 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:28.137686 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-9fz49"] Jul 14 09:34:28.139156 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:34:28.139116 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe74cc74_df63_4a0e_b716_2c738a01afce.slice/crio-1741037327e85b500b99ae70f59a53943f986bc6d3e6879080c1f747b7d5740b WatchSource:0}: Error finding container 1741037327e85b500b99ae70f59a53943f986bc6d3e6879080c1f747b7d5740b: Status 404 returned error can't find the container with id 1741037327e85b500b99ae70f59a53943f986bc6d3e6879080c1f747b7d5740b Jul 14 09:34:28.408055 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:28.408010 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-9fz49" event={"ID":"fe74cc74-df63-4a0e-b716-2c738a01afce","Type":"ContainerStarted","Data":"1741037327e85b500b99ae70f59a53943f986bc6d3e6879080c1f747b7d5740b"} Jul 14 09:34:30.426428 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:30.426384 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-jqbx4" event={"ID":"4761a03d-b3b2-40c7-925d-387bf34bc755","Type":"ContainerStarted","Data":"8c3e0dcd86dffa496e85b97b9ba22d8d697723b814fa7d5990164c75e5c33a67"} Jul 14 09:34:30.426926 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:30.426497 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-jqbx4" Jul 14 09:34:30.447386 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:30.447325 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-jqbx4" podStartSLOduration=1.5819310739999999 podStartE2EDuration="4.447302055s" totalImagesPullingTime="2.865370981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 09:34:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:34:30.445097069 +0000 UTC m=+459.954328519" watchObservedRunningTime="2026-07-14 09:34:30.447302055 +0000 UTC m=+459.956533505" Jul 14 09:34:32.442657 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:32.442574 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-9fz49" event={"ID":"fe74cc74-df63-4a0e-b716-2c738a01afce","Type":"ContainerStarted","Data":"37abeab0fdbf40b7f57b9069ecd9a783598e8daf846e5d1bdb71844662f558bb"} Jul 14 09:34:32.458993 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:32.458946 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-9fz49" podStartSLOduration=1.434276497 podStartE2EDuration="5.458928103s" totalImagesPullingTime="4.024651606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 09:34:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:34:32.457075543 +0000 UTC m=+461.966307011" watchObservedRunningTime="2026-07-14 09:34:32.458928103 +0000 UTC m=+461.968159552" Jul 14 09:34:41.434101 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:41.434073 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-jqbx4" Jul 14 09:34:42.029474 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:42.029439 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-jqbx4"] Jul 14 09:34:42.029692 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:42.029636 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-jqbx4" podUID="4761a03d-b3b2-40c7-925d-387bf34bc755" containerName="limitador" containerID="cri-o://8c3e0dcd86dffa496e85b97b9ba22d8d697723b814fa7d5990164c75e5c33a67" gracePeriod=30 Jul 14 09:34:42.497642 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:42.497606 2565 generic.go:356] "Generic (PLEG): container finished" podID="4761a03d-b3b2-40c7-925d-387bf34bc755" containerID="8c3e0dcd86dffa496e85b97b9ba22d8d697723b814fa7d5990164c75e5c33a67" exitCode=0 Jul 14 09:34:42.498004 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:42.497661 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-jqbx4" event={"ID":"4761a03d-b3b2-40c7-925d-387bf34bc755","Type":"ContainerDied","Data":"8c3e0dcd86dffa496e85b97b9ba22d8d697723b814fa7d5990164c75e5c33a67"} Jul 14 09:34:42.580647 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:42.580625 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-jqbx4" Jul 14 09:34:42.701081 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:42.701045 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-db8sv\" (UniqueName: \"kubernetes.io/projected/4761a03d-b3b2-40c7-925d-387bf34bc755-kube-api-access-db8sv\") pod \"4761a03d-b3b2-40c7-925d-387bf34bc755\" (UID: \"4761a03d-b3b2-40c7-925d-387bf34bc755\") " Jul 14 09:34:42.701254 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:42.701146 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4761a03d-b3b2-40c7-925d-387bf34bc755-config-file\") pod \"4761a03d-b3b2-40c7-925d-387bf34bc755\" (UID: \"4761a03d-b3b2-40c7-925d-387bf34bc755\") " Jul 14 09:34:42.701485 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:42.701460 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4761a03d-b3b2-40c7-925d-387bf34bc755-config-file" (OuterVolumeSpecName: "config-file") pod "4761a03d-b3b2-40c7-925d-387bf34bc755" (UID: "4761a03d-b3b2-40c7-925d-387bf34bc755"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 09:34:42.703131 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:42.703102 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4761a03d-b3b2-40c7-925d-387bf34bc755-kube-api-access-db8sv" (OuterVolumeSpecName: "kube-api-access-db8sv") pod "4761a03d-b3b2-40c7-925d-387bf34bc755" (UID: "4761a03d-b3b2-40c7-925d-387bf34bc755"). InnerVolumeSpecName "kube-api-access-db8sv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 09:34:42.802312 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:42.802276 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db8sv\" (UniqueName: \"kubernetes.io/projected/4761a03d-b3b2-40c7-925d-387bf34bc755-kube-api-access-db8sv\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:34:42.802312 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:42.802305 2565 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4761a03d-b3b2-40c7-925d-387bf34bc755-config-file\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:34:43.504345 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:43.504320 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-jqbx4" Jul 14 09:34:43.504747 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:43.504315 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-jqbx4" event={"ID":"4761a03d-b3b2-40c7-925d-387bf34bc755","Type":"ContainerDied","Data":"2cbc08f1051266ab1103f8204d0d440a5dd9c656f49335ea3ade4a316c053b20"} Jul 14 09:34:43.504747 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:43.504450 2565 scope.go:117] "RemoveContainer" containerID="8c3e0dcd86dffa496e85b97b9ba22d8d697723b814fa7d5990164c75e5c33a67" Jul 14 09:34:43.528433 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:43.528403 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-jqbx4"] Jul 14 09:34:43.545689 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:43.545649 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-jqbx4"] Jul 14 09:34:44.969056 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:44.969023 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4761a03d-b3b2-40c7-925d-387bf34bc755" path="/var/lib/kubelet/pods/4761a03d-b3b2-40c7-925d-387bf34bc755/volumes" Jul 14 09:34:48.208116 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:48.208081 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-wtrvz"] Jul 14 09:34:48.208482 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:48.208449 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4761a03d-b3b2-40c7-925d-387bf34bc755" containerName="limitador" Jul 14 09:34:48.208482 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:48.208460 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4761a03d-b3b2-40c7-925d-387bf34bc755" containerName="limitador" Jul 14 09:34:48.208561 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:48.208545 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4761a03d-b3b2-40c7-925d-387bf34bc755" containerName="limitador" Jul 14 09:34:48.210652 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:48.210633 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wtrvz" Jul 14 09:34:48.213275 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:48.213255 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 14 09:34:48.213407 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:48.213254 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 14 09:34:48.214231 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:48.214214 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-lkttw\"" Jul 14 09:34:48.214348 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:48.214251 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 09:34:48.223920 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:48.223889 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-wtrvz"] Jul 14 09:34:48.356671 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:48.356631 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f681359f-dde2-4f16-ba74-7b13cd6b6a26-data\") pod \"postgres-69fb4594cf-wtrvz\" (UID: \"f681359f-dde2-4f16-ba74-7b13cd6b6a26\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wtrvz" Jul 14 09:34:48.356917 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:48.356686 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mb79\" (UniqueName: \"kubernetes.io/projected/f681359f-dde2-4f16-ba74-7b13cd6b6a26-kube-api-access-7mb79\") pod \"postgres-69fb4594cf-wtrvz\" (UID: \"f681359f-dde2-4f16-ba74-7b13cd6b6a26\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wtrvz" Jul 14 09:34:48.457624 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:48.457587 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f681359f-dde2-4f16-ba74-7b13cd6b6a26-data\") pod \"postgres-69fb4594cf-wtrvz\" (UID: \"f681359f-dde2-4f16-ba74-7b13cd6b6a26\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wtrvz" Jul 14 09:34:48.457792 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:48.457645 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mb79\" (UniqueName: \"kubernetes.io/projected/f681359f-dde2-4f16-ba74-7b13cd6b6a26-kube-api-access-7mb79\") pod \"postgres-69fb4594cf-wtrvz\" (UID: \"f681359f-dde2-4f16-ba74-7b13cd6b6a26\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wtrvz" Jul 14 09:34:48.457987 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:48.457966 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f681359f-dde2-4f16-ba74-7b13cd6b6a26-data\") pod \"postgres-69fb4594cf-wtrvz\" (UID: \"f681359f-dde2-4f16-ba74-7b13cd6b6a26\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wtrvz" Jul 14 09:34:48.466513 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:48.466449 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mb79\" (UniqueName: \"kubernetes.io/projected/f681359f-dde2-4f16-ba74-7b13cd6b6a26-kube-api-access-7mb79\") pod \"postgres-69fb4594cf-wtrvz\" (UID: \"f681359f-dde2-4f16-ba74-7b13cd6b6a26\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wtrvz" Jul 14 09:34:48.520436 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:48.520407 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wtrvz" Jul 14 09:34:48.674364 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:48.674339 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-wtrvz"] Jul 14 09:34:48.675462 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:34:48.675437 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf681359f_dde2_4f16_ba74_7b13cd6b6a26.slice/crio-2f6388993ff8476324e662cf0f5d5e3944a8de0ed6c520517670757622624c14 WatchSource:0}: Error finding container 2f6388993ff8476324e662cf0f5d5e3944a8de0ed6c520517670757622624c14: Status 404 returned error can't find the container with id 2f6388993ff8476324e662cf0f5d5e3944a8de0ed6c520517670757622624c14 Jul 14 09:34:49.539588 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:34:49.539547 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wtrvz" event={"ID":"f681359f-dde2-4f16-ba74-7b13cd6b6a26","Type":"ContainerStarted","Data":"2f6388993ff8476324e662cf0f5d5e3944a8de0ed6c520517670757622624c14"} Jul 14 09:35:01.935767 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:01.935733 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 14 09:35:02.618841 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:02.618770 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wtrvz" event={"ID":"f681359f-dde2-4f16-ba74-7b13cd6b6a26","Type":"ContainerStarted","Data":"6ce0da429fe5f5219f628afaf854a6c01b4038aed02889aceec6004aea127f18"} Jul 14 09:35:02.619043 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:02.618931 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wtrvz" Jul 14 09:35:02.636376 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:02.636321 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wtrvz" podStartSLOduration=1.380292563 podStartE2EDuration="14.636305708s" totalImagesPullingTime="13.256013145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 09:34:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:35:02.634719517 +0000 UTC m=+492.143950963" watchObservedRunningTime="2026-07-14 09:35:02.636305708 +0000 UTC m=+492.145537155" Jul 14 09:35:08.655828 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:08.655781 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wtrvz" Jul 14 09:35:09.941734 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:09.941662 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-gplgl"] Jul 14 09:35:09.944213 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:09.944195 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-gplgl" Jul 14 09:35:09.952588 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:09.952561 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-gplgl"] Jul 14 09:35:10.044738 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:10.044699 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cv7h\" (UniqueName: \"kubernetes.io/projected/7d16e320-9ae9-4f4e-bc11-c875240e5353-kube-api-access-4cv7h\") pod \"authorino-5d4b67c446-gplgl\" (UID: \"7d16e320-9ae9-4f4e-bc11-c875240e5353\") " pod="kuadrant-system/authorino-5d4b67c446-gplgl" Jul 14 09:35:10.135182 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:10.135148 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-gplgl"] Jul 14 09:35:10.135502 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:35:10.135478 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-4cv7h], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-gplgl" podUID="7d16e320-9ae9-4f4e-bc11-c875240e5353" Jul 14 09:35:10.145871 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:10.145840 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cv7h\" (UniqueName: \"kubernetes.io/projected/7d16e320-9ae9-4f4e-bc11-c875240e5353-kube-api-access-4cv7h\") pod \"authorino-5d4b67c446-gplgl\" (UID: \"7d16e320-9ae9-4f4e-bc11-c875240e5353\") " pod="kuadrant-system/authorino-5d4b67c446-gplgl" Jul 14 09:35:10.161004 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:10.160981 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cv7h\" (UniqueName: \"kubernetes.io/projected/7d16e320-9ae9-4f4e-bc11-c875240e5353-kube-api-access-4cv7h\") pod \"authorino-5d4b67c446-gplgl\" (UID: \"7d16e320-9ae9-4f4e-bc11-c875240e5353\") " pod="kuadrant-system/authorino-5d4b67c446-gplgl" Jul 14 09:35:10.164566 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:10.164541 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76877994-txhsd"] Jul 14 09:35:10.167055 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:10.167031 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-txhsd" Jul 14 09:35:10.169885 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:10.169865 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 14 09:35:10.178272 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:10.178247 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76877994-txhsd"] Jul 14 09:35:10.247075 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:10.246987 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/65f84bb4-247a-4604-8398-7469a4c53040-tls-cert\") pod \"authorino-76877994-txhsd\" (UID: \"65f84bb4-247a-4604-8398-7469a4c53040\") " pod="kuadrant-system/authorino-76877994-txhsd" Jul 14 09:35:10.247228 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:10.247118 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzfm7\" (UniqueName: \"kubernetes.io/projected/65f84bb4-247a-4604-8398-7469a4c53040-kube-api-access-vzfm7\") pod \"authorino-76877994-txhsd\" (UID: \"65f84bb4-247a-4604-8398-7469a4c53040\") " pod="kuadrant-system/authorino-76877994-txhsd" Jul 14 09:35:10.347656 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:10.347611 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzfm7\" (UniqueName: \"kubernetes.io/projected/65f84bb4-247a-4604-8398-7469a4c53040-kube-api-access-vzfm7\") pod \"authorino-76877994-txhsd\" (UID: \"65f84bb4-247a-4604-8398-7469a4c53040\") " pod="kuadrant-system/authorino-76877994-txhsd" Jul 14 09:35:10.347892 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:10.347674 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/65f84bb4-247a-4604-8398-7469a4c53040-tls-cert\") pod \"authorino-76877994-txhsd\" (UID: \"65f84bb4-247a-4604-8398-7469a4c53040\") " pod="kuadrant-system/authorino-76877994-txhsd" Jul 14 09:35:10.350179 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:10.350158 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/65f84bb4-247a-4604-8398-7469a4c53040-tls-cert\") pod \"authorino-76877994-txhsd\" (UID: \"65f84bb4-247a-4604-8398-7469a4c53040\") " pod="kuadrant-system/authorino-76877994-txhsd" Jul 14 09:35:10.359440 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:10.359415 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzfm7\" (UniqueName: \"kubernetes.io/projected/65f84bb4-247a-4604-8398-7469a4c53040-kube-api-access-vzfm7\") pod \"authorino-76877994-txhsd\" (UID: \"65f84bb4-247a-4604-8398-7469a4c53040\") " pod="kuadrant-system/authorino-76877994-txhsd" Jul 14 09:35:10.390273 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:10.390238 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-txhsd"] Jul 14 09:35:10.390600 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:10.390587 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-txhsd" Jul 14 09:35:10.415317 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:10.415286 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-fbb7696bf-nlbgh"] Jul 14 09:35:10.418371 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:10.418349 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-fbb7696bf-nlbgh" Jul 14 09:35:10.426656 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:10.426628 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-fbb7696bf-nlbgh"] Jul 14 09:35:10.448593 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:10.448563 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9f9c9bdb-daa8-42f1-bead-0a45fd18bf93-tls-cert\") pod \"authorino-fbb7696bf-nlbgh\" (UID: \"9f9c9bdb-daa8-42f1-bead-0a45fd18bf93\") " pod="kuadrant-system/authorino-fbb7696bf-nlbgh" Jul 14 09:35:10.448741 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:10.448721 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v646w\" (UniqueName: \"kubernetes.io/projected/9f9c9bdb-daa8-42f1-bead-0a45fd18bf93-kube-api-access-v646w\") pod \"authorino-fbb7696bf-nlbgh\" (UID: \"9f9c9bdb-daa8-42f1-bead-0a45fd18bf93\") " pod="kuadrant-system/authorino-fbb7696bf-nlbgh" Jul 14 09:35:10.531694 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:10.531664 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-txhsd"] Jul 14 09:35:10.533291 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:35:10.533267 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65f84bb4_247a_4604_8398_7469a4c53040.slice/crio-4678d47c76d8dbb5de76da1ed37490321b9fc062e68460dfbc7fa547b5f6740d WatchSource:0}: Error finding container 4678d47c76d8dbb5de76da1ed37490321b9fc062e68460dfbc7fa547b5f6740d: Status 404 returned error can't find the container with id 4678d47c76d8dbb5de76da1ed37490321b9fc062e68460dfbc7fa547b5f6740d Jul 14 09:35:10.549839 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:10.549790 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v646w\" (UniqueName: \"kubernetes.io/projected/9f9c9bdb-daa8-42f1-bead-0a45fd18bf93-kube-api-access-v646w\") pod \"authorino-fbb7696bf-nlbgh\" (UID: \"9f9c9bdb-daa8-42f1-bead-0a45fd18bf93\") " pod="kuadrant-system/authorino-fbb7696bf-nlbgh" Jul 14 09:35:10.549953 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:10.549936 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9f9c9bdb-daa8-42f1-bead-0a45fd18bf93-tls-cert\") pod \"authorino-fbb7696bf-nlbgh\" (UID: \"9f9c9bdb-daa8-42f1-bead-0a45fd18bf93\") " pod="kuadrant-system/authorino-fbb7696bf-nlbgh" Jul 14 09:35:10.552328 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:10.552308 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9f9c9bdb-daa8-42f1-bead-0a45fd18bf93-tls-cert\") pod \"authorino-fbb7696bf-nlbgh\" (UID: \"9f9c9bdb-daa8-42f1-bead-0a45fd18bf93\") " pod="kuadrant-system/authorino-fbb7696bf-nlbgh" Jul 14 09:35:10.558452 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:10.558423 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v646w\" (UniqueName: \"kubernetes.io/projected/9f9c9bdb-daa8-42f1-bead-0a45fd18bf93-kube-api-access-v646w\") pod \"authorino-fbb7696bf-nlbgh\" (UID: \"9f9c9bdb-daa8-42f1-bead-0a45fd18bf93\") " pod="kuadrant-system/authorino-fbb7696bf-nlbgh" Jul 14 09:35:10.667796 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:10.667757 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76877994-txhsd" event={"ID":"65f84bb4-247a-4604-8398-7469a4c53040","Type":"ContainerStarted","Data":"4678d47c76d8dbb5de76da1ed37490321b9fc062e68460dfbc7fa547b5f6740d"} Jul 14 09:35:10.667982 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:10.667826 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-gplgl" Jul 14 09:35:10.674224 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:10.674199 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-gplgl" Jul 14 09:35:10.730799 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:10.730766 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-fbb7696bf-nlbgh" Jul 14 09:35:10.751982 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:10.751936 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4cv7h\" (UniqueName: \"kubernetes.io/projected/7d16e320-9ae9-4f4e-bc11-c875240e5353-kube-api-access-4cv7h\") pod \"7d16e320-9ae9-4f4e-bc11-c875240e5353\" (UID: \"7d16e320-9ae9-4f4e-bc11-c875240e5353\") " Jul 14 09:35:10.758820 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:10.754926 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d16e320-9ae9-4f4e-bc11-c875240e5353-kube-api-access-4cv7h" (OuterVolumeSpecName: "kube-api-access-4cv7h") pod "7d16e320-9ae9-4f4e-bc11-c875240e5353" (UID: "7d16e320-9ae9-4f4e-bc11-c875240e5353"). InnerVolumeSpecName "kube-api-access-4cv7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 09:35:10.853618 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:10.853587 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cv7h\" (UniqueName: \"kubernetes.io/projected/7d16e320-9ae9-4f4e-bc11-c875240e5353-kube-api-access-4cv7h\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:35:10.861100 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:10.861079 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-fbb7696bf-nlbgh"] Jul 14 09:35:10.862665 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:35:10.862644 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f9c9bdb_daa8_42f1_bead_0a45fd18bf93.slice/crio-81564ba159849fb31d21e081a2e903a50d61eeddfef99ccc54d4ee054b45de96 WatchSource:0}: Error finding container 81564ba159849fb31d21e081a2e903a50d61eeddfef99ccc54d4ee054b45de96: Status 404 returned error can't find the container with id 81564ba159849fb31d21e081a2e903a50d61eeddfef99ccc54d4ee054b45de96 Jul 14 09:35:11.675287 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:11.675247 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-fbb7696bf-nlbgh" event={"ID":"9f9c9bdb-daa8-42f1-bead-0a45fd18bf93","Type":"ContainerStarted","Data":"0b50ba825dcadf7e798f300170823ad63fc9e5e5dcfa03370ea089f8c26d20d1"} Jul 14 09:35:11.675287 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:11.675297 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-fbb7696bf-nlbgh" event={"ID":"9f9c9bdb-daa8-42f1-bead-0a45fd18bf93","Type":"ContainerStarted","Data":"81564ba159849fb31d21e081a2e903a50d61eeddfef99ccc54d4ee054b45de96"} Jul 14 09:35:11.676849 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:11.676823 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76877994-txhsd" event={"ID":"65f84bb4-247a-4604-8398-7469a4c53040","Type":"ContainerStarted","Data":"8b8211d7f04665a23e4bfe997e1c04122f0024e10f61017ccefaf89460b09fa7"} Jul 14 09:35:11.676963 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:11.676859 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-76877994-txhsd" podUID="65f84bb4-247a-4604-8398-7469a4c53040" containerName="authorino" containerID="cri-o://8b8211d7f04665a23e4bfe997e1c04122f0024e10f61017ccefaf89460b09fa7" gracePeriod=30 Jul 14 09:35:11.676963 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:11.676876 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-gplgl" Jul 14 09:35:11.695129 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:11.695085 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-fbb7696bf-nlbgh" podStartSLOduration=1.350932271 podStartE2EDuration="1.695070795s" totalImagesPullingTime="344.138524ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 09:35:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:35:11.692546931 +0000 UTC m=+501.201778396" watchObservedRunningTime="2026-07-14 09:35:11.695070795 +0000 UTC m=+501.204302254" Jul 14 09:35:11.719285 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:11.719247 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-gplgl"] Jul 14 09:35:11.724132 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:11.723970 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-9fz49"] Jul 14 09:35:11.724302 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:11.724189 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-9fz49" podUID="fe74cc74-df63-4a0e-b716-2c738a01afce" containerName="authorino" containerID="cri-o://37abeab0fdbf40b7f57b9069ecd9a783598e8daf846e5d1bdb71844662f558bb" gracePeriod=30 Jul 14 09:35:11.727325 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:11.727131 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-gplgl"] Jul 14 09:35:11.745692 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:11.745632 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-76877994-txhsd" podStartSLOduration=1.394908504 podStartE2EDuration="1.745612166s" totalImagesPullingTime="350.703662ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 09:35:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:35:11.744919177 +0000 UTC m=+501.254150651" watchObservedRunningTime="2026-07-14 09:35:11.745612166 +0000 UTC m=+501.254843617" Jul 14 09:35:11.977660 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:11.977639 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-txhsd" Jul 14 09:35:11.983500 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:11.983477 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-9fz49" Jul 14 09:35:12.067161 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:12.067132 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/65f84bb4-247a-4604-8398-7469a4c53040-tls-cert\") pod \"65f84bb4-247a-4604-8398-7469a4c53040\" (UID: \"65f84bb4-247a-4604-8398-7469a4c53040\") " Jul 14 09:35:12.067322 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:12.067181 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vzfm7\" (UniqueName: \"kubernetes.io/projected/65f84bb4-247a-4604-8398-7469a4c53040-kube-api-access-vzfm7\") pod \"65f84bb4-247a-4604-8398-7469a4c53040\" (UID: \"65f84bb4-247a-4604-8398-7469a4c53040\") " Jul 14 09:35:12.067322 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:12.067235 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nwxqx\" (UniqueName: \"kubernetes.io/projected/fe74cc74-df63-4a0e-b716-2c738a01afce-kube-api-access-nwxqx\") pod \"fe74cc74-df63-4a0e-b716-2c738a01afce\" (UID: \"fe74cc74-df63-4a0e-b716-2c738a01afce\") " Jul 14 09:35:12.069223 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:12.069198 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65f84bb4-247a-4604-8398-7469a4c53040-kube-api-access-vzfm7" (OuterVolumeSpecName: "kube-api-access-vzfm7") pod "65f84bb4-247a-4604-8398-7469a4c53040" (UID: "65f84bb4-247a-4604-8398-7469a4c53040"). InnerVolumeSpecName "kube-api-access-vzfm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 09:35:12.069306 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:12.069261 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe74cc74-df63-4a0e-b716-2c738a01afce-kube-api-access-nwxqx" (OuterVolumeSpecName: "kube-api-access-nwxqx") pod "fe74cc74-df63-4a0e-b716-2c738a01afce" (UID: "fe74cc74-df63-4a0e-b716-2c738a01afce"). InnerVolumeSpecName "kube-api-access-nwxqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 09:35:12.078337 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:12.078311 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65f84bb4-247a-4604-8398-7469a4c53040-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "65f84bb4-247a-4604-8398-7469a4c53040" (UID: "65f84bb4-247a-4604-8398-7469a4c53040"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 09:35:12.168008 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:12.167983 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzfm7\" (UniqueName: \"kubernetes.io/projected/65f84bb4-247a-4604-8398-7469a4c53040-kube-api-access-vzfm7\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:35:12.168099 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:12.168014 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwxqx\" (UniqueName: \"kubernetes.io/projected/fe74cc74-df63-4a0e-b716-2c738a01afce-kube-api-access-nwxqx\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:35:12.168099 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:12.168028 2565 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/65f84bb4-247a-4604-8398-7469a4c53040-tls-cert\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:35:12.683737 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:12.683707 2565 generic.go:356] "Generic (PLEG): container finished" podID="fe74cc74-df63-4a0e-b716-2c738a01afce" containerID="37abeab0fdbf40b7f57b9069ecd9a783598e8daf846e5d1bdb71844662f558bb" exitCode=0 Jul 14 09:35:12.684201 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:12.683752 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-9fz49" Jul 14 09:35:12.684201 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:12.683796 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-9fz49" event={"ID":"fe74cc74-df63-4a0e-b716-2c738a01afce","Type":"ContainerDied","Data":"37abeab0fdbf40b7f57b9069ecd9a783598e8daf846e5d1bdb71844662f558bb"} Jul 14 09:35:12.684201 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:12.683851 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-9fz49" event={"ID":"fe74cc74-df63-4a0e-b716-2c738a01afce","Type":"ContainerDied","Data":"1741037327e85b500b99ae70f59a53943f986bc6d3e6879080c1f747b7d5740b"} Jul 14 09:35:12.684201 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:12.683874 2565 scope.go:117] "RemoveContainer" containerID="37abeab0fdbf40b7f57b9069ecd9a783598e8daf846e5d1bdb71844662f558bb" Jul 14 09:35:12.685143 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:12.685118 2565 generic.go:356] "Generic (PLEG): container finished" podID="65f84bb4-247a-4604-8398-7469a4c53040" containerID="8b8211d7f04665a23e4bfe997e1c04122f0024e10f61017ccefaf89460b09fa7" exitCode=0 Jul 14 09:35:12.685242 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:12.685168 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-txhsd" Jul 14 09:35:12.685242 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:12.685203 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76877994-txhsd" event={"ID":"65f84bb4-247a-4604-8398-7469a4c53040","Type":"ContainerDied","Data":"8b8211d7f04665a23e4bfe997e1c04122f0024e10f61017ccefaf89460b09fa7"} Jul 14 09:35:12.685242 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:12.685236 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76877994-txhsd" event={"ID":"65f84bb4-247a-4604-8398-7469a4c53040","Type":"ContainerDied","Data":"4678d47c76d8dbb5de76da1ed37490321b9fc062e68460dfbc7fa547b5f6740d"} Jul 14 09:35:12.693180 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:12.693161 2565 scope.go:117] "RemoveContainer" containerID="37abeab0fdbf40b7f57b9069ecd9a783598e8daf846e5d1bdb71844662f558bb" Jul 14 09:35:12.693425 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:35:12.693408 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37abeab0fdbf40b7f57b9069ecd9a783598e8daf846e5d1bdb71844662f558bb\": container with ID starting with 37abeab0fdbf40b7f57b9069ecd9a783598e8daf846e5d1bdb71844662f558bb not found: ID does not exist" containerID="37abeab0fdbf40b7f57b9069ecd9a783598e8daf846e5d1bdb71844662f558bb" Jul 14 09:35:12.693511 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:12.693437 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37abeab0fdbf40b7f57b9069ecd9a783598e8daf846e5d1bdb71844662f558bb"} err="failed to get container status \"37abeab0fdbf40b7f57b9069ecd9a783598e8daf846e5d1bdb71844662f558bb\": rpc error: code = NotFound desc = could not find container \"37abeab0fdbf40b7f57b9069ecd9a783598e8daf846e5d1bdb71844662f558bb\": container with ID starting with 37abeab0fdbf40b7f57b9069ecd9a783598e8daf846e5d1bdb71844662f558bb not found: ID does not exist" Jul 14 09:35:12.693511 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:12.693459 2565 scope.go:117] "RemoveContainer" containerID="8b8211d7f04665a23e4bfe997e1c04122f0024e10f61017ccefaf89460b09fa7" Jul 14 09:35:12.701571 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:12.701554 2565 scope.go:117] "RemoveContainer" containerID="8b8211d7f04665a23e4bfe997e1c04122f0024e10f61017ccefaf89460b09fa7" Jul 14 09:35:12.701815 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:35:12.701787 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b8211d7f04665a23e4bfe997e1c04122f0024e10f61017ccefaf89460b09fa7\": container with ID starting with 8b8211d7f04665a23e4bfe997e1c04122f0024e10f61017ccefaf89460b09fa7 not found: ID does not exist" containerID="8b8211d7f04665a23e4bfe997e1c04122f0024e10f61017ccefaf89460b09fa7" Jul 14 09:35:12.701868 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:12.701825 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b8211d7f04665a23e4bfe997e1c04122f0024e10f61017ccefaf89460b09fa7"} err="failed to get container status \"8b8211d7f04665a23e4bfe997e1c04122f0024e10f61017ccefaf89460b09fa7\": rpc error: code = NotFound desc = could not find container \"8b8211d7f04665a23e4bfe997e1c04122f0024e10f61017ccefaf89460b09fa7\": container with ID starting with 8b8211d7f04665a23e4bfe997e1c04122f0024e10f61017ccefaf89460b09fa7 not found: ID does not exist" Jul 14 09:35:12.708081 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:12.708057 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-txhsd"] Jul 14 09:35:12.711920 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:12.711900 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76877994-txhsd"] Jul 14 09:35:12.723544 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:12.723524 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-9fz49"] Jul 14 09:35:12.729775 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:12.729751 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-9fz49"] Jul 14 09:35:12.974697 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:12.974610 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65f84bb4-247a-4604-8398-7469a4c53040" path="/var/lib/kubelet/pods/65f84bb4-247a-4604-8398-7469a4c53040/volumes" Jul 14 09:35:12.974976 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:12.974962 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d16e320-9ae9-4f4e-bc11-c875240e5353" path="/var/lib/kubelet/pods/7d16e320-9ae9-4f4e-bc11-c875240e5353/volumes" Jul 14 09:35:12.975176 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:12.975164 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe74cc74-df63-4a0e-b716-2c738a01afce" path="/var/lib/kubelet/pods/fe74cc74-df63-4a0e-b716-2c738a01afce/volumes" Jul 14 09:35:37.767474 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:37.767432 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-85d65f5d9f-xjxbs"] Jul 14 09:35:37.768070 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:37.768021 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe74cc74-df63-4a0e-b716-2c738a01afce" containerName="authorino" Jul 14 09:35:37.768070 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:37.768043 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe74cc74-df63-4a0e-b716-2c738a01afce" containerName="authorino" Jul 14 09:35:37.768070 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:37.768079 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65f84bb4-247a-4604-8398-7469a4c53040" containerName="authorino" Jul 14 09:35:37.768246 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:37.768088 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="65f84bb4-247a-4604-8398-7469a4c53040" containerName="authorino" Jul 14 09:35:37.768246 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:37.768205 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="65f84bb4-247a-4604-8398-7469a4c53040" containerName="authorino" Jul 14 09:35:37.768246 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:37.768218 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe74cc74-df63-4a0e-b716-2c738a01afce" containerName="authorino" Jul 14 09:35:37.790050 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:37.790019 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-85d65f5d9f-xjxbs" Jul 14 09:35:37.791478 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:37.791442 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-85d65f5d9f-xjxbs"] Jul 14 09:35:37.793311 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:37.793289 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 14 09:35:37.793454 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:37.793311 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9fctp\"" Jul 14 09:35:37.903818 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:37.903782 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4036b4fc-90ec-43de-a369-ce999774cc3d-maas-api-tls\") pod \"maas-api-85d65f5d9f-xjxbs\" (UID: \"4036b4fc-90ec-43de-a369-ce999774cc3d\") " pod="odh-ai-gateway-infra/maas-api-85d65f5d9f-xjxbs" Jul 14 09:35:37.904033 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:37.903904 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b99pv\" (UniqueName: \"kubernetes.io/projected/4036b4fc-90ec-43de-a369-ce999774cc3d-kube-api-access-b99pv\") pod \"maas-api-85d65f5d9f-xjxbs\" (UID: \"4036b4fc-90ec-43de-a369-ce999774cc3d\") " pod="odh-ai-gateway-infra/maas-api-85d65f5d9f-xjxbs" Jul 14 09:35:37.904033 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:37.903991 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4036b4fc-90ec-43de-a369-ce999774cc3d-openshift-service-ca\") pod \"maas-api-85d65f5d9f-xjxbs\" (UID: \"4036b4fc-90ec-43de-a369-ce999774cc3d\") " pod="odh-ai-gateway-infra/maas-api-85d65f5d9f-xjxbs" Jul 14 09:35:38.004999 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:38.004960 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4036b4fc-90ec-43de-a369-ce999774cc3d-maas-api-tls\") pod \"maas-api-85d65f5d9f-xjxbs\" (UID: \"4036b4fc-90ec-43de-a369-ce999774cc3d\") " pod="odh-ai-gateway-infra/maas-api-85d65f5d9f-xjxbs" Jul 14 09:35:38.005189 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:38.005040 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b99pv\" (UniqueName: \"kubernetes.io/projected/4036b4fc-90ec-43de-a369-ce999774cc3d-kube-api-access-b99pv\") pod \"maas-api-85d65f5d9f-xjxbs\" (UID: \"4036b4fc-90ec-43de-a369-ce999774cc3d\") " pod="odh-ai-gateway-infra/maas-api-85d65f5d9f-xjxbs" Jul 14 09:35:38.005189 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:38.005088 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4036b4fc-90ec-43de-a369-ce999774cc3d-openshift-service-ca\") pod \"maas-api-85d65f5d9f-xjxbs\" (UID: \"4036b4fc-90ec-43de-a369-ce999774cc3d\") " pod="odh-ai-gateway-infra/maas-api-85d65f5d9f-xjxbs" Jul 14 09:35:38.005775 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:38.005757 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4036b4fc-90ec-43de-a369-ce999774cc3d-openshift-service-ca\") pod \"maas-api-85d65f5d9f-xjxbs\" (UID: \"4036b4fc-90ec-43de-a369-ce999774cc3d\") " pod="odh-ai-gateway-infra/maas-api-85d65f5d9f-xjxbs" Jul 14 09:35:38.007452 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:38.007432 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4036b4fc-90ec-43de-a369-ce999774cc3d-maas-api-tls\") pod \"maas-api-85d65f5d9f-xjxbs\" (UID: \"4036b4fc-90ec-43de-a369-ce999774cc3d\") " pod="odh-ai-gateway-infra/maas-api-85d65f5d9f-xjxbs" Jul 14 09:35:38.016923 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:38.016899 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b99pv\" (UniqueName: \"kubernetes.io/projected/4036b4fc-90ec-43de-a369-ce999774cc3d-kube-api-access-b99pv\") pod \"maas-api-85d65f5d9f-xjxbs\" (UID: \"4036b4fc-90ec-43de-a369-ce999774cc3d\") " pod="odh-ai-gateway-infra/maas-api-85d65f5d9f-xjxbs" Jul 14 09:35:38.100728 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:38.100641 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-85d65f5d9f-xjxbs" Jul 14 09:35:38.254556 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:38.254516 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-85d65f5d9f-xjxbs"] Jul 14 09:35:38.256070 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:35:38.256038 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4036b4fc_90ec_43de_a369_ce999774cc3d.slice/crio-97cd41c42220623e6b41b038d273f6997f8ba74fb19aa0800d9a4d0232b6c202 WatchSource:0}: Error finding container 97cd41c42220623e6b41b038d273f6997f8ba74fb19aa0800d9a4d0232b6c202: Status 404 returned error can't find the container with id 97cd41c42220623e6b41b038d273f6997f8ba74fb19aa0800d9a4d0232b6c202 Jul 14 09:35:38.322406 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:38.322370 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-24r2v"] Jul 14 09:35:38.327698 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:38.327675 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-24r2v" Jul 14 09:35:38.330530 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:38.330508 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-9x5vz\"" Jul 14 09:35:38.330649 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:38.330595 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 14 09:35:38.339548 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:38.339522 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-24r2v"] Jul 14 09:35:38.510824 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:38.510770 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqhzz\" (UniqueName: \"kubernetes.io/projected/56e18c92-abdf-47ad-98da-0c68f6c38e81-kube-api-access-qqhzz\") pod \"payload-pre-processing-74584bb9f8-24r2v\" (UID: \"56e18c92-abdf-47ad-98da-0c68f6c38e81\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-24r2v" Jul 14 09:35:38.510998 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:38.510873 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56e18c92-abdf-47ad-98da-0c68f6c38e81-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-24r2v\" (UID: \"56e18c92-abdf-47ad-98da-0c68f6c38e81\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-24r2v" Jul 14 09:35:38.611826 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:38.611767 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqhzz\" (UniqueName: \"kubernetes.io/projected/56e18c92-abdf-47ad-98da-0c68f6c38e81-kube-api-access-qqhzz\") pod \"payload-pre-processing-74584bb9f8-24r2v\" (UID: \"56e18c92-abdf-47ad-98da-0c68f6c38e81\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-24r2v" Jul 14 09:35:38.612011 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:38.611870 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56e18c92-abdf-47ad-98da-0c68f6c38e81-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-24r2v\" (UID: \"56e18c92-abdf-47ad-98da-0c68f6c38e81\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-24r2v" Jul 14 09:35:38.612556 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:38.612536 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56e18c92-abdf-47ad-98da-0c68f6c38e81-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-24r2v\" (UID: \"56e18c92-abdf-47ad-98da-0c68f6c38e81\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-24r2v" Jul 14 09:35:38.624994 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:38.624967 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqhzz\" (UniqueName: \"kubernetes.io/projected/56e18c92-abdf-47ad-98da-0c68f6c38e81-kube-api-access-qqhzz\") pod \"payload-pre-processing-74584bb9f8-24r2v\" (UID: \"56e18c92-abdf-47ad-98da-0c68f6c38e81\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-24r2v" Jul 14 09:35:38.639691 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:38.639667 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-24r2v" Jul 14 09:35:38.773478 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:38.773455 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-24r2v"] Jul 14 09:35:38.775227 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:35:38.775200 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56e18c92_abdf_47ad_98da_0c68f6c38e81.slice/crio-0c003476aaf57be59b4df0745d7830be82a69564f63245166c7baaea4c27d9a3 WatchSource:0}: Error finding container 0c003476aaf57be59b4df0745d7830be82a69564f63245166c7baaea4c27d9a3: Status 404 returned error can't find the container with id 0c003476aaf57be59b4df0745d7830be82a69564f63245166c7baaea4c27d9a3 Jul 14 09:35:38.831684 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:38.831654 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-85d65f5d9f-xjxbs" event={"ID":"4036b4fc-90ec-43de-a369-ce999774cc3d","Type":"ContainerStarted","Data":"97cd41c42220623e6b41b038d273f6997f8ba74fb19aa0800d9a4d0232b6c202"} Jul 14 09:35:38.832963 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:38.832931 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-24r2v" event={"ID":"56e18c92-abdf-47ad-98da-0c68f6c38e81","Type":"ContainerStarted","Data":"0c003476aaf57be59b4df0745d7830be82a69564f63245166c7baaea4c27d9a3"} Jul 14 09:35:41.854786 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:41.854752 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-85d65f5d9f-xjxbs" event={"ID":"4036b4fc-90ec-43de-a369-ce999774cc3d","Type":"ContainerStarted","Data":"bc3197f9493c9237477f55cc93741a6356aab1f23ee49c862d3944eee8530dc8"} Jul 14 09:35:41.855237 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:41.854825 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-85d65f5d9f-xjxbs" Jul 14 09:35:41.856362 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:41.856338 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-24r2v" event={"ID":"56e18c92-abdf-47ad-98da-0c68f6c38e81","Type":"ContainerStarted","Data":"ae2f80e76a4e90299a965ab620e8004369022d239f5aa8abeb952f6b8c89495a"} Jul 14 09:35:41.856470 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:41.856419 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-24r2v" Jul 14 09:35:41.874147 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:41.874100 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-85d65f5d9f-xjxbs" podStartSLOduration=2.195718431 podStartE2EDuration="4.874084191s" totalImagesPullingTime="2.67836576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 09:35:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:35:41.872120336 +0000 UTC m=+531.381351785" watchObservedRunningTime="2026-07-14 09:35:41.874084191 +0000 UTC m=+531.383315639" Jul 14 09:35:46.885305 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:46.885272 2565 generic.go:356] "Generic (PLEG): container finished" podID="5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a" containerID="645c14f3394e752b1a1594d5aac47966ccde101bf7ca7be4c448938b654da883" exitCode=0 Jul 14 09:35:46.885750 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:46.885334 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6c9dg" event={"ID":"5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a","Type":"ContainerDied","Data":"645c14f3394e752b1a1594d5aac47966ccde101bf7ca7be4c448938b654da883"} Jul 14 09:35:46.885750 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:46.885686 2565 scope.go:117] "RemoveContainer" containerID="645c14f3394e752b1a1594d5aac47966ccde101bf7ca7be4c448938b654da883" Jul 14 09:35:46.886712 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:46.886677 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 09:35:46.886838 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:46.886768 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 09:35:46.886911 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:46.886843 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 14 09:35:46.908657 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:46.908610 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-74584bb9f8-24r2v" podStartSLOduration=6.678118739 podStartE2EDuration="8.908593301s" totalImagesPullingTime="2.230474562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 09:35:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:35:41.892324419 +0000 UTC m=+531.401555881" watchObservedRunningTime="2026-07-14 09:35:46.908593301 +0000 UTC m=+536.417824748" Jul 14 09:35:47.868320 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:47.868293 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-85d65f5d9f-xjxbs" Jul 14 09:35:47.894066 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:47.894018 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6c9dg" event={"ID":"5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a","Type":"ContainerStarted","Data":"cd7fc525941aac71fd5228a3f0505b5401198fd2578e0c83c7859b4c4e571d84"} Jul 14 09:35:48.634183 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:48.634155 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6c9dg" Jul 14 09:35:48.634417 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:48.634394 2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-6c9dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 14 09:35:48.634482 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:48.634445 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6c9dg" podUID="5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 14 09:35:49.634732 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:49.634690 2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-6c9dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 14 09:35:49.635166 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:49.634754 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6c9dg" podUID="5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 14 09:35:50.634915 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:50.634876 2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-6c9dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 14 09:35:50.635308 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:50.634935 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6c9dg" podUID="5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 14 09:35:51.634049 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:51.634015 2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-6c9dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 14 09:35:51.634236 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:51.634096 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6c9dg" podUID="5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 14 09:35:52.638121 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:52.638092 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6c9dg" Jul 14 09:35:52.638604 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:52.638485 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6c9dg" Jul 14 09:35:52.639300 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:52.639281 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6c9dg" Jul 14 09:35:52.864709 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:35:52.864666 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-74584bb9f8-24r2v" podUID="56e18c92-abdf-47ad-98da-0c68f6c38e81" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.55:9004: connect: connection refused" Jul 14 09:36:02.863358 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:02.863330 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-24r2v" Jul 14 09:36:04.027292 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:04.027256 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4jhsg"] Jul 14 09:36:04.032929 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:04.032901 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4jhsg" Jul 14 09:36:04.037209 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:04.037161 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-zfwqb\"" Jul 14 09:36:04.037640 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:04.037194 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 14 09:36:04.037766 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:04.037722 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 14 09:36:04.037978 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:04.037198 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 14 09:36:04.040613 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:04.040591 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4jhsg"] Jul 14 09:36:04.146874 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:04.146839 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d181d90d-3be0-4c6e-aeb0-7f458819d582-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4jhsg\" (UID: \"d181d90d-3be0-4c6e-aeb0-7f458819d582\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4jhsg" Jul 14 09:36:04.147076 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:04.146879 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7cfn\" (UniqueName: \"kubernetes.io/projected/d181d90d-3be0-4c6e-aeb0-7f458819d582-kube-api-access-r7cfn\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4jhsg\" (UID: \"d181d90d-3be0-4c6e-aeb0-7f458819d582\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4jhsg" Jul 14 09:36:04.147076 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:04.146964 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d181d90d-3be0-4c6e-aeb0-7f458819d582-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4jhsg\" (UID: \"d181d90d-3be0-4c6e-aeb0-7f458819d582\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4jhsg" Jul 14 09:36:04.147076 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:04.147027 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d181d90d-3be0-4c6e-aeb0-7f458819d582-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4jhsg\" (UID: \"d181d90d-3be0-4c6e-aeb0-7f458819d582\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4jhsg" Jul 14 09:36:04.147076 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:04.147060 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d181d90d-3be0-4c6e-aeb0-7f458819d582-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4jhsg\" (UID: \"d181d90d-3be0-4c6e-aeb0-7f458819d582\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4jhsg" Jul 14 09:36:04.147316 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:04.147089 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d181d90d-3be0-4c6e-aeb0-7f458819d582-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4jhsg\" (UID: \"d181d90d-3be0-4c6e-aeb0-7f458819d582\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4jhsg" Jul 14 09:36:04.248492 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:04.248454 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d181d90d-3be0-4c6e-aeb0-7f458819d582-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4jhsg\" (UID: \"d181d90d-3be0-4c6e-aeb0-7f458819d582\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4jhsg" Jul 14 09:36:04.248492 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:04.248494 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7cfn\" (UniqueName: \"kubernetes.io/projected/d181d90d-3be0-4c6e-aeb0-7f458819d582-kube-api-access-r7cfn\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4jhsg\" (UID: \"d181d90d-3be0-4c6e-aeb0-7f458819d582\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4jhsg" Jul 14 09:36:04.248759 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:04.248549 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d181d90d-3be0-4c6e-aeb0-7f458819d582-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4jhsg\" (UID: \"d181d90d-3be0-4c6e-aeb0-7f458819d582\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4jhsg" Jul 14 09:36:04.248759 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:04.248567 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d181d90d-3be0-4c6e-aeb0-7f458819d582-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4jhsg\" (UID: \"d181d90d-3be0-4c6e-aeb0-7f458819d582\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4jhsg" Jul 14 09:36:04.248759 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:04.248590 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d181d90d-3be0-4c6e-aeb0-7f458819d582-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4jhsg\" (UID: \"d181d90d-3be0-4c6e-aeb0-7f458819d582\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4jhsg" Jul 14 09:36:04.248759 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:04.248620 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d181d90d-3be0-4c6e-aeb0-7f458819d582-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4jhsg\" (UID: \"d181d90d-3be0-4c6e-aeb0-7f458819d582\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4jhsg" Jul 14 09:36:04.249028 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:04.249005 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d181d90d-3be0-4c6e-aeb0-7f458819d582-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4jhsg\" (UID: \"d181d90d-3be0-4c6e-aeb0-7f458819d582\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4jhsg" Jul 14 09:36:04.249028 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:04.249016 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d181d90d-3be0-4c6e-aeb0-7f458819d582-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4jhsg\" (UID: \"d181d90d-3be0-4c6e-aeb0-7f458819d582\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4jhsg" Jul 14 09:36:04.249113 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:04.249031 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d181d90d-3be0-4c6e-aeb0-7f458819d582-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4jhsg\" (UID: \"d181d90d-3be0-4c6e-aeb0-7f458819d582\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4jhsg" Jul 14 09:36:04.250994 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:04.250968 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d181d90d-3be0-4c6e-aeb0-7f458819d582-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4jhsg\" (UID: \"d181d90d-3be0-4c6e-aeb0-7f458819d582\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4jhsg" Jul 14 09:36:04.251221 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:04.251206 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d181d90d-3be0-4c6e-aeb0-7f458819d582-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4jhsg\" (UID: \"d181d90d-3be0-4c6e-aeb0-7f458819d582\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4jhsg" Jul 14 09:36:04.256690 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:04.256657 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7cfn\" (UniqueName: \"kubernetes.io/projected/d181d90d-3be0-4c6e-aeb0-7f458819d582-kube-api-access-r7cfn\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4jhsg\" (UID: \"d181d90d-3be0-4c6e-aeb0-7f458819d582\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4jhsg" Jul 14 09:36:04.346128 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:04.346034 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4jhsg" Jul 14 09:36:04.483309 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:04.483281 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4jhsg"] Jul 14 09:36:04.485277 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:36:04.485252 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd181d90d_3be0_4c6e_aeb0_7f458819d582.slice/crio-9b55aedc9f6071ddfe1d1dbdbf99b9c64554a67d97a506222cc084bfad526bd6 WatchSource:0}: Error finding container 9b55aedc9f6071ddfe1d1dbdbf99b9c64554a67d97a506222cc084bfad526bd6: Status 404 returned error can't find the container with id 9b55aedc9f6071ddfe1d1dbdbf99b9c64554a67d97a506222cc084bfad526bd6 Jul 14 09:36:05.001654 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:05.001613 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4jhsg" event={"ID":"d181d90d-3be0-4c6e-aeb0-7f458819d582","Type":"ContainerStarted","Data":"9b55aedc9f6071ddfe1d1dbdbf99b9c64554a67d97a506222cc084bfad526bd6"} Jul 14 09:36:06.008196 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:06.008155 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4jhsg" event={"ID":"d181d90d-3be0-4c6e-aeb0-7f458819d582","Type":"ContainerStarted","Data":"74f31a3d537bea80f2cf9f112cad01820c5993b8fdc648e91de380b6d27337ec"} Jul 14 09:36:06.028298 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:06.028242 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4jhsg" podStartSLOduration=0.647362372 podStartE2EDuration="2.028227197s" totalImagesPullingTime="1.380864825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 09:36:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:36:06.027110259 +0000 UTC m=+555.536341722" watchObservedRunningTime="2026-07-14 09:36:06.028227197 +0000 UTC m=+555.537458684" Jul 14 09:36:14.346733 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:14.346683 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4jhsg" Jul 14 09:36:14.347230 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:14.346768 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4jhsg" Jul 14 09:36:14.359312 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:14.359280 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4jhsg" Jul 14 09:36:15.070079 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:15.070046 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4jhsg" Jul 14 09:36:28.432240 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:28.432209 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-hz9cw"] Jul 14 09:36:28.435858 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:28.435841 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hz9cw" Jul 14 09:36:28.439027 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:28.439006 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 14 09:36:28.445939 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:28.445910 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-hz9cw"] Jul 14 09:36:28.489386 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:28.489340 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62a6b10d-5e6a-46f1-9eb9-3174ea1a264d-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hz9cw\" (UID: \"62a6b10d-5e6a-46f1-9eb9-3174ea1a264d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hz9cw" Jul 14 09:36:28.489386 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:28.489383 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62a6b10d-5e6a-46f1-9eb9-3174ea1a264d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hz9cw\" (UID: \"62a6b10d-5e6a-46f1-9eb9-3174ea1a264d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hz9cw" Jul 14 09:36:28.489652 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:28.489423 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62a6b10d-5e6a-46f1-9eb9-3174ea1a264d-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hz9cw\" (UID: \"62a6b10d-5e6a-46f1-9eb9-3174ea1a264d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hz9cw" Jul 14 09:36:28.489652 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:28.489508 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx2kk\" (UniqueName: \"kubernetes.io/projected/62a6b10d-5e6a-46f1-9eb9-3174ea1a264d-kube-api-access-vx2kk\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hz9cw\" (UID: \"62a6b10d-5e6a-46f1-9eb9-3174ea1a264d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hz9cw" Jul 14 09:36:28.489652 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:28.489637 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62a6b10d-5e6a-46f1-9eb9-3174ea1a264d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hz9cw\" (UID: \"62a6b10d-5e6a-46f1-9eb9-3174ea1a264d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hz9cw" Jul 14 09:36:28.489843 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:28.489664 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62a6b10d-5e6a-46f1-9eb9-3174ea1a264d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hz9cw\" (UID: \"62a6b10d-5e6a-46f1-9eb9-3174ea1a264d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hz9cw" Jul 14 09:36:28.591162 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:28.591130 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62a6b10d-5e6a-46f1-9eb9-3174ea1a264d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hz9cw\" (UID: \"62a6b10d-5e6a-46f1-9eb9-3174ea1a264d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hz9cw" Jul 14 09:36:28.591162 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:28.591165 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62a6b10d-5e6a-46f1-9eb9-3174ea1a264d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hz9cw\" (UID: \"62a6b10d-5e6a-46f1-9eb9-3174ea1a264d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hz9cw" Jul 14 09:36:28.591417 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:28.591201 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62a6b10d-5e6a-46f1-9eb9-3174ea1a264d-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hz9cw\" (UID: \"62a6b10d-5e6a-46f1-9eb9-3174ea1a264d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hz9cw" Jul 14 09:36:28.591417 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:28.591216 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62a6b10d-5e6a-46f1-9eb9-3174ea1a264d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hz9cw\" (UID: \"62a6b10d-5e6a-46f1-9eb9-3174ea1a264d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hz9cw" Jul 14 09:36:28.591417 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:28.591237 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62a6b10d-5e6a-46f1-9eb9-3174ea1a264d-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hz9cw\" (UID: \"62a6b10d-5e6a-46f1-9eb9-3174ea1a264d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hz9cw" Jul 14 09:36:28.591417 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:28.591260 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx2kk\" (UniqueName: \"kubernetes.io/projected/62a6b10d-5e6a-46f1-9eb9-3174ea1a264d-kube-api-access-vx2kk\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hz9cw\" (UID: \"62a6b10d-5e6a-46f1-9eb9-3174ea1a264d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hz9cw" Jul 14 09:36:28.591630 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:28.591576 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62a6b10d-5e6a-46f1-9eb9-3174ea1a264d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hz9cw\" (UID: \"62a6b10d-5e6a-46f1-9eb9-3174ea1a264d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hz9cw" Jul 14 09:36:28.591630 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:28.591612 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62a6b10d-5e6a-46f1-9eb9-3174ea1a264d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hz9cw\" (UID: \"62a6b10d-5e6a-46f1-9eb9-3174ea1a264d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hz9cw" Jul 14 09:36:28.591739 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:28.591683 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62a6b10d-5e6a-46f1-9eb9-3174ea1a264d-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hz9cw\" (UID: \"62a6b10d-5e6a-46f1-9eb9-3174ea1a264d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hz9cw" Jul 14 09:36:28.593451 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:28.593425 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62a6b10d-5e6a-46f1-9eb9-3174ea1a264d-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hz9cw\" (UID: \"62a6b10d-5e6a-46f1-9eb9-3174ea1a264d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hz9cw" Jul 14 09:36:28.593736 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:28.593716 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62a6b10d-5e6a-46f1-9eb9-3174ea1a264d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hz9cw\" (UID: \"62a6b10d-5e6a-46f1-9eb9-3174ea1a264d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hz9cw" Jul 14 09:36:28.601392 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:28.601366 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx2kk\" (UniqueName: \"kubernetes.io/projected/62a6b10d-5e6a-46f1-9eb9-3174ea1a264d-kube-api-access-vx2kk\") pod \"e2e-trlp-test-simulated-kserve-598466548d-hz9cw\" (UID: \"62a6b10d-5e6a-46f1-9eb9-3174ea1a264d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hz9cw" Jul 14 09:36:28.747156 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:28.747059 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hz9cw" Jul 14 09:36:28.890537 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:28.890510 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-hz9cw"] Jul 14 09:36:28.891486 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:36:28.891460 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62a6b10d_5e6a_46f1_9eb9_3174ea1a264d.slice/crio-15699f5afe411b47c6fae906e56964a9125e1d629efec54e70adb54701ccf510 WatchSource:0}: Error finding container 15699f5afe411b47c6fae906e56964a9125e1d629efec54e70adb54701ccf510: Status 404 returned error can't find the container with id 15699f5afe411b47c6fae906e56964a9125e1d629efec54e70adb54701ccf510 Jul 14 09:36:29.140509 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:29.140475 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hz9cw" event={"ID":"62a6b10d-5e6a-46f1-9eb9-3174ea1a264d","Type":"ContainerStarted","Data":"15699f5afe411b47c6fae906e56964a9125e1d629efec54e70adb54701ccf510"} Jul 14 09:36:30.147936 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:30.147896 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hz9cw" event={"ID":"62a6b10d-5e6a-46f1-9eb9-3174ea1a264d","Type":"ContainerStarted","Data":"7200d13bdeea51b0f571f2d8484aab1d5038c4bf22a150d30fcf006cfad70c1f"} Jul 14 09:36:30.167758 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:30.167695 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hz9cw" podStartSLOduration=1.913495653 podStartE2EDuration="2.167674088s" totalImagesPullingTime="254.178435ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 09:36:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:36:30.166032695 +0000 UTC m=+579.675264144" watchObservedRunningTime="2026-07-14 09:36:30.167674088 +0000 UTC m=+579.676905537" Jul 14 09:36:37.828381 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:37.828341 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbl86wm"] Jul 14 09:36:37.835913 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:37.835887 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbl86wm" Jul 14 09:36:37.839022 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:37.839003 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 14 09:36:37.844059 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:37.844036 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbl86wm"] Jul 14 09:36:37.883145 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:37.883111 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff85a849-d87e-481d-a8fd-2dcff42e4a71-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbl86wm\" (UID: \"ff85a849-d87e-481d-a8fd-2dcff42e4a71\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbl86wm" Jul 14 09:36:37.883298 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:37.883154 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9pb9\" (UniqueName: \"kubernetes.io/projected/ff85a849-d87e-481d-a8fd-2dcff42e4a71-kube-api-access-f9pb9\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbl86wm\" (UID: \"ff85a849-d87e-481d-a8fd-2dcff42e4a71\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbl86wm" Jul 14 09:36:37.883298 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:37.883174 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff85a849-d87e-481d-a8fd-2dcff42e4a71-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbl86wm\" (UID: \"ff85a849-d87e-481d-a8fd-2dcff42e4a71\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbl86wm" Jul 14 09:36:37.883298 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:37.883202 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff85a849-d87e-481d-a8fd-2dcff42e4a71-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbl86wm\" (UID: \"ff85a849-d87e-481d-a8fd-2dcff42e4a71\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbl86wm" Jul 14 09:36:37.883448 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:37.883295 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff85a849-d87e-481d-a8fd-2dcff42e4a71-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbl86wm\" (UID: \"ff85a849-d87e-481d-a8fd-2dcff42e4a71\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbl86wm" Jul 14 09:36:37.883448 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:37.883394 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff85a849-d87e-481d-a8fd-2dcff42e4a71-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbl86wm\" (UID: \"ff85a849-d87e-481d-a8fd-2dcff42e4a71\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbl86wm" Jul 14 09:36:37.984454 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:37.984417 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff85a849-d87e-481d-a8fd-2dcff42e4a71-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbl86wm\" (UID: \"ff85a849-d87e-481d-a8fd-2dcff42e4a71\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbl86wm" Jul 14 09:36:37.984643 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:37.984464 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff85a849-d87e-481d-a8fd-2dcff42e4a71-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbl86wm\" (UID: \"ff85a849-d87e-481d-a8fd-2dcff42e4a71\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbl86wm" Jul 14 09:36:37.984643 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:37.984503 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9pb9\" (UniqueName: \"kubernetes.io/projected/ff85a849-d87e-481d-a8fd-2dcff42e4a71-kube-api-access-f9pb9\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbl86wm\" (UID: \"ff85a849-d87e-481d-a8fd-2dcff42e4a71\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbl86wm" Jul 14 09:36:37.984643 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:37.984531 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff85a849-d87e-481d-a8fd-2dcff42e4a71-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbl86wm\" (UID: \"ff85a849-d87e-481d-a8fd-2dcff42e4a71\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbl86wm" Jul 14 09:36:37.984643 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:37.984558 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff85a849-d87e-481d-a8fd-2dcff42e4a71-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbl86wm\" (UID: \"ff85a849-d87e-481d-a8fd-2dcff42e4a71\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbl86wm" Jul 14 09:36:37.984643 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:37.984615 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff85a849-d87e-481d-a8fd-2dcff42e4a71-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbl86wm\" (UID: \"ff85a849-d87e-481d-a8fd-2dcff42e4a71\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbl86wm" Jul 14 09:36:37.985014 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:37.984992 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff85a849-d87e-481d-a8fd-2dcff42e4a71-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbl86wm\" (UID: \"ff85a849-d87e-481d-a8fd-2dcff42e4a71\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbl86wm" Jul 14 09:36:37.985071 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:37.985011 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff85a849-d87e-481d-a8fd-2dcff42e4a71-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbl86wm\" (UID: \"ff85a849-d87e-481d-a8fd-2dcff42e4a71\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbl86wm" Jul 14 09:36:37.985071 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:37.985043 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff85a849-d87e-481d-a8fd-2dcff42e4a71-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbl86wm\" (UID: \"ff85a849-d87e-481d-a8fd-2dcff42e4a71\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbl86wm" Jul 14 09:36:37.986766 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:37.986745 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff85a849-d87e-481d-a8fd-2dcff42e4a71-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbl86wm\" (UID: \"ff85a849-d87e-481d-a8fd-2dcff42e4a71\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbl86wm" Jul 14 09:36:37.987013 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:37.986995 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff85a849-d87e-481d-a8fd-2dcff42e4a71-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbl86wm\" (UID: \"ff85a849-d87e-481d-a8fd-2dcff42e4a71\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbl86wm" Jul 14 09:36:37.993130 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:37.993107 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9pb9\" (UniqueName: \"kubernetes.io/projected/ff85a849-d87e-481d-a8fd-2dcff42e4a71-kube-api-access-f9pb9\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbl86wm\" (UID: \"ff85a849-d87e-481d-a8fd-2dcff42e4a71\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbl86wm" Jul 14 09:36:38.148129 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:38.148087 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbl86wm" Jul 14 09:36:38.288137 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:38.288096 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbl86wm"] Jul 14 09:36:38.289444 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:36:38.289419 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff85a849_d87e_481d_a8fd_2dcff42e4a71.slice/crio-fa86ce54c27296b31d2e2be99b52cf776f0488a20acd08210e516cb0e8b9c042 WatchSource:0}: Error finding container fa86ce54c27296b31d2e2be99b52cf776f0488a20acd08210e516cb0e8b9c042: Status 404 returned error can't find the container with id fa86ce54c27296b31d2e2be99b52cf776f0488a20acd08210e516cb0e8b9c042 Jul 14 09:36:38.747373 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:38.747337 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hz9cw" Jul 14 09:36:38.747567 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:38.747422 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hz9cw" Jul 14 09:36:38.759882 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:38.759785 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hz9cw" Jul 14 09:36:39.202099 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:39.202063 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbl86wm" event={"ID":"ff85a849-d87e-481d-a8fd-2dcff42e4a71","Type":"ContainerStarted","Data":"3502f48c9214fd0a19b3746f1e772074520a22db2386222463d60f6c05ad15c1"} Jul 14 09:36:39.202475 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:39.202119 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbl86wm" event={"ID":"ff85a849-d87e-481d-a8fd-2dcff42e4a71","Type":"ContainerStarted","Data":"fa86ce54c27296b31d2e2be99b52cf776f0488a20acd08210e516cb0e8b9c042"} Jul 14 09:36:39.213649 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:39.213627 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-hz9cw" Jul 14 09:36:39.222561 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:39.222512 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbl86wm" podStartSLOduration=2.004961855 podStartE2EDuration="2.222496025s" totalImagesPullingTime="217.53417ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 09:36:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:36:39.219909355 +0000 UTC m=+588.729140838" watchObservedRunningTime="2026-07-14 09:36:39.222496025 +0000 UTC m=+588.731727472" Jul 14 09:36:48.149096 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:48.149059 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbl86wm" Jul 14 09:36:48.149463 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:48.149119 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbl86wm" Jul 14 09:36:48.161500 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:48.161478 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbl86wm" Jul 14 09:36:48.266675 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:36:48.266643 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbl86wm" Jul 14 09:38:30.155348 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:38:30.155297 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-fbb7696bf-nlbgh"] Jul 14 09:38:30.156055 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:38:30.155514 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-fbb7696bf-nlbgh" podUID="9f9c9bdb-daa8-42f1-bead-0a45fd18bf93" containerName="authorino" containerID="cri-o://0b50ba825dcadf7e798f300170823ad63fc9e5e5dcfa03370ea089f8c26d20d1" gracePeriod=30 Jul 14 09:38:30.413015 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:38:30.412956 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-fbb7696bf-nlbgh" Jul 14 09:38:30.470410 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:38:30.470383 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v646w\" (UniqueName: \"kubernetes.io/projected/9f9c9bdb-daa8-42f1-bead-0a45fd18bf93-kube-api-access-v646w\") pod \"9f9c9bdb-daa8-42f1-bead-0a45fd18bf93\" (UID: \"9f9c9bdb-daa8-42f1-bead-0a45fd18bf93\") " Jul 14 09:38:30.470591 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:38:30.470502 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9f9c9bdb-daa8-42f1-bead-0a45fd18bf93-tls-cert\") pod \"9f9c9bdb-daa8-42f1-bead-0a45fd18bf93\" (UID: \"9f9c9bdb-daa8-42f1-bead-0a45fd18bf93\") " Jul 14 09:38:30.472448 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:38:30.472423 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f9c9bdb-daa8-42f1-bead-0a45fd18bf93-kube-api-access-v646w" (OuterVolumeSpecName: "kube-api-access-v646w") pod "9f9c9bdb-daa8-42f1-bead-0a45fd18bf93" (UID: "9f9c9bdb-daa8-42f1-bead-0a45fd18bf93"). InnerVolumeSpecName "kube-api-access-v646w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 09:38:30.481351 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:38:30.481321 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f9c9bdb-daa8-42f1-bead-0a45fd18bf93-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "9f9c9bdb-daa8-42f1-bead-0a45fd18bf93" (UID: "9f9c9bdb-daa8-42f1-bead-0a45fd18bf93"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 09:38:30.571972 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:38:30.571940 2565 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9f9c9bdb-daa8-42f1-bead-0a45fd18bf93-tls-cert\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:38:30.571972 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:38:30.571969 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v646w\" (UniqueName: \"kubernetes.io/projected/9f9c9bdb-daa8-42f1-bead-0a45fd18bf93-kube-api-access-v646w\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:38:30.839331 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:38:30.839237 2565 generic.go:356] "Generic (PLEG): container finished" podID="9f9c9bdb-daa8-42f1-bead-0a45fd18bf93" containerID="0b50ba825dcadf7e798f300170823ad63fc9e5e5dcfa03370ea089f8c26d20d1" exitCode=0 Jul 14 09:38:30.839331 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:38:30.839285 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-fbb7696bf-nlbgh" Jul 14 09:38:30.839331 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:38:30.839312 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-fbb7696bf-nlbgh" event={"ID":"9f9c9bdb-daa8-42f1-bead-0a45fd18bf93","Type":"ContainerDied","Data":"0b50ba825dcadf7e798f300170823ad63fc9e5e5dcfa03370ea089f8c26d20d1"} Jul 14 09:38:30.839575 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:38:30.839349 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-fbb7696bf-nlbgh" event={"ID":"9f9c9bdb-daa8-42f1-bead-0a45fd18bf93","Type":"ContainerDied","Data":"81564ba159849fb31d21e081a2e903a50d61eeddfef99ccc54d4ee054b45de96"} Jul 14 09:38:30.839575 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:38:30.839372 2565 scope.go:117] "RemoveContainer" containerID="0b50ba825dcadf7e798f300170823ad63fc9e5e5dcfa03370ea089f8c26d20d1" Jul 14 09:38:30.849380 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:38:30.849363 2565 scope.go:117] "RemoveContainer" containerID="0b50ba825dcadf7e798f300170823ad63fc9e5e5dcfa03370ea089f8c26d20d1" Jul 14 09:38:30.849624 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:38:30.849608 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b50ba825dcadf7e798f300170823ad63fc9e5e5dcfa03370ea089f8c26d20d1\": container with ID starting with 0b50ba825dcadf7e798f300170823ad63fc9e5e5dcfa03370ea089f8c26d20d1 not found: ID does not exist" containerID="0b50ba825dcadf7e798f300170823ad63fc9e5e5dcfa03370ea089f8c26d20d1" Jul 14 09:38:30.849680 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:38:30.849629 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b50ba825dcadf7e798f300170823ad63fc9e5e5dcfa03370ea089f8c26d20d1"} err="failed to get container status \"0b50ba825dcadf7e798f300170823ad63fc9e5e5dcfa03370ea089f8c26d20d1\": rpc error: code = NotFound desc = could not find container \"0b50ba825dcadf7e798f300170823ad63fc9e5e5dcfa03370ea089f8c26d20d1\": container with ID starting with 0b50ba825dcadf7e798f300170823ad63fc9e5e5dcfa03370ea089f8c26d20d1 not found: ID does not exist" Jul 14 09:38:30.864979 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:38:30.864946 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-fbb7696bf-nlbgh"] Jul 14 09:38:30.867886 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:38:30.867869 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-fbb7696bf-nlbgh"] Jul 14 09:38:30.973151 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:38:30.973120 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f9c9bdb-daa8-42f1-bead-0a45fd18bf93" path="/var/lib/kubelet/pods/9f9c9bdb-daa8-42f1-bead-0a45fd18bf93/volumes" Jul 14 09:52:27.273183 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:27.273142 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c9216ec8-66dddbbb79-c64pl"] Jul 14 09:52:27.273751 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:27.273731 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f9c9bdb-daa8-42f1-bead-0a45fd18bf93" containerName="authorino" Jul 14 09:52:27.273801 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:27.273756 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f9c9bdb-daa8-42f1-bead-0a45fd18bf93" containerName="authorino" Jul 14 09:52:27.273924 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:27.273912 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f9c9bdb-daa8-42f1-bead-0a45fd18bf93" containerName="authorino" Jul 14 09:52:27.277019 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:27.277002 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9216ec8-66dddbbb79-c64pl" Jul 14 09:52:27.280676 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:27.280650 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-c9216ec8-serving-cert\"" Jul 14 09:52:27.298618 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:27.298590 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c9216ec8-66dddbbb79-c64pl"] Jul 14 09:52:27.393513 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:27.393475 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/893d92e1-e3fd-4c8e-8b1e-b81c60011972-maas-api-tls\") pod \"maas-api-e2e-ait-c9216ec8-66dddbbb79-c64pl\" (UID: \"893d92e1-e3fd-4c8e-8b1e-b81c60011972\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9216ec8-66dddbbb79-c64pl" Jul 14 09:52:27.393681 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:27.393534 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/893d92e1-e3fd-4c8e-8b1e-b81c60011972-openshift-service-ca\") pod \"maas-api-e2e-ait-c9216ec8-66dddbbb79-c64pl\" (UID: \"893d92e1-e3fd-4c8e-8b1e-b81c60011972\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9216ec8-66dddbbb79-c64pl" Jul 14 09:52:27.393681 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:27.393606 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znxqw\" (UniqueName: \"kubernetes.io/projected/893d92e1-e3fd-4c8e-8b1e-b81c60011972-kube-api-access-znxqw\") pod \"maas-api-e2e-ait-c9216ec8-66dddbbb79-c64pl\" (UID: \"893d92e1-e3fd-4c8e-8b1e-b81c60011972\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9216ec8-66dddbbb79-c64pl" Jul 14 09:52:27.494670 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:27.494629 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/893d92e1-e3fd-4c8e-8b1e-b81c60011972-openshift-service-ca\") pod \"maas-api-e2e-ait-c9216ec8-66dddbbb79-c64pl\" (UID: \"893d92e1-e3fd-4c8e-8b1e-b81c60011972\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9216ec8-66dddbbb79-c64pl" Jul 14 09:52:27.494891 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:27.494681 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znxqw\" (UniqueName: \"kubernetes.io/projected/893d92e1-e3fd-4c8e-8b1e-b81c60011972-kube-api-access-znxqw\") pod \"maas-api-e2e-ait-c9216ec8-66dddbbb79-c64pl\" (UID: \"893d92e1-e3fd-4c8e-8b1e-b81c60011972\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9216ec8-66dddbbb79-c64pl" Jul 14 09:52:27.494891 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:27.494759 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/893d92e1-e3fd-4c8e-8b1e-b81c60011972-maas-api-tls\") pod \"maas-api-e2e-ait-c9216ec8-66dddbbb79-c64pl\" (UID: \"893d92e1-e3fd-4c8e-8b1e-b81c60011972\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9216ec8-66dddbbb79-c64pl" Jul 14 09:52:27.495327 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:27.495298 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/893d92e1-e3fd-4c8e-8b1e-b81c60011972-openshift-service-ca\") pod \"maas-api-e2e-ait-c9216ec8-66dddbbb79-c64pl\" (UID: \"893d92e1-e3fd-4c8e-8b1e-b81c60011972\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9216ec8-66dddbbb79-c64pl" Jul 14 09:52:27.497285 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:27.497262 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/893d92e1-e3fd-4c8e-8b1e-b81c60011972-maas-api-tls\") pod \"maas-api-e2e-ait-c9216ec8-66dddbbb79-c64pl\" (UID: \"893d92e1-e3fd-4c8e-8b1e-b81c60011972\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9216ec8-66dddbbb79-c64pl" Jul 14 09:52:27.510505 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:27.510475 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znxqw\" (UniqueName: \"kubernetes.io/projected/893d92e1-e3fd-4c8e-8b1e-b81c60011972-kube-api-access-znxqw\") pod \"maas-api-e2e-ait-c9216ec8-66dddbbb79-c64pl\" (UID: \"893d92e1-e3fd-4c8e-8b1e-b81c60011972\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9216ec8-66dddbbb79-c64pl" Jul 14 09:52:27.586763 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:27.586675 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9216ec8-66dddbbb79-c64pl" Jul 14 09:52:27.729358 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:27.729329 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c9216ec8-66dddbbb79-c64pl"] Jul 14 09:52:27.730238 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:52:27.730216 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod893d92e1_e3fd_4c8e_8b1e_b81c60011972.slice/crio-684209493b49f2a209fad578e455243510dd9932a223c9d52907f0441a475999 WatchSource:0}: Error finding container 684209493b49f2a209fad578e455243510dd9932a223c9d52907f0441a475999: Status 404 returned error can't find the container with id 684209493b49f2a209fad578e455243510dd9932a223c9d52907f0441a475999 Jul 14 09:52:27.731634 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:27.731613 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 09:52:28.555246 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:28.555208 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9216ec8-66dddbbb79-c64pl" event={"ID":"893d92e1-e3fd-4c8e-8b1e-b81c60011972","Type":"ContainerStarted","Data":"0e033551c246ae75a329d088e3efe3083faa4424cb931b69b57696aa21f70e8c"} Jul 14 09:52:28.555730 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:28.555264 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9216ec8-66dddbbb79-c64pl" event={"ID":"893d92e1-e3fd-4c8e-8b1e-b81c60011972","Type":"ContainerStarted","Data":"684209493b49f2a209fad578e455243510dd9932a223c9d52907f0441a475999"} Jul 14 09:52:28.555730 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:28.555302 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9216ec8-66dddbbb79-c64pl" Jul 14 09:52:28.575599 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:28.575542 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9216ec8-66dddbbb79-c64pl" podStartSLOduration=0.865297074 podStartE2EDuration="1.575498088s" totalImagesPullingTime="710.201014ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 09:52:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:52:28.574649433 +0000 UTC m=+1538.083880883" watchObservedRunningTime="2026-07-14 09:52:28.575498088 +0000 UTC m=+1538.084729536" Jul 14 09:52:34.567717 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:34.567687 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9216ec8-66dddbbb79-c64pl" Jul 14 09:52:42.654286 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:42.654244 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c9216ec8-66dddbbb79-c64pl"] Jul 14 09:52:42.654718 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:42.654483 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9216ec8-66dddbbb79-c64pl" podUID="893d92e1-e3fd-4c8e-8b1e-b81c60011972" containerName="maas-api" containerID="cri-o://0e033551c246ae75a329d088e3efe3083faa4424cb931b69b57696aa21f70e8c" gracePeriod=30 Jul 14 09:52:42.909615 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:42.909556 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9216ec8-66dddbbb79-c64pl" Jul 14 09:52:43.052758 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:43.052722 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/893d92e1-e3fd-4c8e-8b1e-b81c60011972-openshift-service-ca\") pod \"893d92e1-e3fd-4c8e-8b1e-b81c60011972\" (UID: \"893d92e1-e3fd-4c8e-8b1e-b81c60011972\") " Jul 14 09:52:43.052990 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:43.052878 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/893d92e1-e3fd-4c8e-8b1e-b81c60011972-maas-api-tls\") pod \"893d92e1-e3fd-4c8e-8b1e-b81c60011972\" (UID: \"893d92e1-e3fd-4c8e-8b1e-b81c60011972\") " Jul 14 09:52:43.052990 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:43.052919 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-znxqw\" (UniqueName: \"kubernetes.io/projected/893d92e1-e3fd-4c8e-8b1e-b81c60011972-kube-api-access-znxqw\") pod \"893d92e1-e3fd-4c8e-8b1e-b81c60011972\" (UID: \"893d92e1-e3fd-4c8e-8b1e-b81c60011972\") " Jul 14 09:52:43.053176 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:43.053146 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/893d92e1-e3fd-4c8e-8b1e-b81c60011972-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "893d92e1-e3fd-4c8e-8b1e-b81c60011972" (UID: "893d92e1-e3fd-4c8e-8b1e-b81c60011972"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 09:52:43.053293 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:43.053278 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/893d92e1-e3fd-4c8e-8b1e-b81c60011972-openshift-service-ca\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:52:43.055062 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:43.055036 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/893d92e1-e3fd-4c8e-8b1e-b81c60011972-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "893d92e1-e3fd-4c8e-8b1e-b81c60011972" (UID: "893d92e1-e3fd-4c8e-8b1e-b81c60011972"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 09:52:43.055135 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:43.055092 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/893d92e1-e3fd-4c8e-8b1e-b81c60011972-kube-api-access-znxqw" (OuterVolumeSpecName: "kube-api-access-znxqw") pod "893d92e1-e3fd-4c8e-8b1e-b81c60011972" (UID: "893d92e1-e3fd-4c8e-8b1e-b81c60011972"). InnerVolumeSpecName "kube-api-access-znxqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 09:52:43.154622 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:43.154588 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/893d92e1-e3fd-4c8e-8b1e-b81c60011972-maas-api-tls\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:52:43.154622 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:43.154616 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znxqw\" (UniqueName: \"kubernetes.io/projected/893d92e1-e3fd-4c8e-8b1e-b81c60011972-kube-api-access-znxqw\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:52:43.640324 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:43.640290 2565 generic.go:356] "Generic (PLEG): container finished" podID="893d92e1-e3fd-4c8e-8b1e-b81c60011972" containerID="0e033551c246ae75a329d088e3efe3083faa4424cb931b69b57696aa21f70e8c" exitCode=0 Jul 14 09:52:43.640502 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:43.640336 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9216ec8-66dddbbb79-c64pl" Jul 14 09:52:43.640502 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:43.640366 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9216ec8-66dddbbb79-c64pl" event={"ID":"893d92e1-e3fd-4c8e-8b1e-b81c60011972","Type":"ContainerDied","Data":"0e033551c246ae75a329d088e3efe3083faa4424cb931b69b57696aa21f70e8c"} Jul 14 09:52:43.640502 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:43.640420 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9216ec8-66dddbbb79-c64pl" event={"ID":"893d92e1-e3fd-4c8e-8b1e-b81c60011972","Type":"ContainerDied","Data":"684209493b49f2a209fad578e455243510dd9932a223c9d52907f0441a475999"} Jul 14 09:52:43.640502 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:43.640444 2565 scope.go:117] "RemoveContainer" containerID="0e033551c246ae75a329d088e3efe3083faa4424cb931b69b57696aa21f70e8c" Jul 14 09:52:43.651690 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:43.651672 2565 scope.go:117] "RemoveContainer" containerID="0e033551c246ae75a329d088e3efe3083faa4424cb931b69b57696aa21f70e8c" Jul 14 09:52:43.652036 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:52:43.652002 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e033551c246ae75a329d088e3efe3083faa4424cb931b69b57696aa21f70e8c\": container with ID starting with 0e033551c246ae75a329d088e3efe3083faa4424cb931b69b57696aa21f70e8c not found: ID does not exist" containerID="0e033551c246ae75a329d088e3efe3083faa4424cb931b69b57696aa21f70e8c" Jul 14 09:52:43.652189 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:43.652045 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e033551c246ae75a329d088e3efe3083faa4424cb931b69b57696aa21f70e8c"} err="failed to get container status \"0e033551c246ae75a329d088e3efe3083faa4424cb931b69b57696aa21f70e8c\": rpc error: code = NotFound desc = could not find container \"0e033551c246ae75a329d088e3efe3083faa4424cb931b69b57696aa21f70e8c\": container with ID starting with 0e033551c246ae75a329d088e3efe3083faa4424cb931b69b57696aa21f70e8c not found: ID does not exist" Jul 14 09:52:43.664378 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:43.664349 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c9216ec8-66dddbbb79-c64pl"] Jul 14 09:52:43.668926 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:43.668904 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c9216ec8-66dddbbb79-c64pl"] Jul 14 09:52:44.969697 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:44.969658 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="893d92e1-e3fd-4c8e-8b1e-b81c60011972" path="/var/lib/kubelet/pods/893d92e1-e3fd-4c8e-8b1e-b81c60011972/volumes" Jul 14 09:52:59.481260 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:59.481220 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-40d96159-69f54ccd7f-ncfjh"] Jul 14 09:52:59.481903 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:59.481883 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="893d92e1-e3fd-4c8e-8b1e-b81c60011972" containerName="maas-api" Jul 14 09:52:59.481903 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:59.481904 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="893d92e1-e3fd-4c8e-8b1e-b81c60011972" containerName="maas-api" Jul 14 09:52:59.482082 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:59.482015 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="893d92e1-e3fd-4c8e-8b1e-b81c60011972" containerName="maas-api" Jul 14 09:52:59.485540 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:59.485519 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-40d96159-69f54ccd7f-ncfjh" Jul 14 09:52:59.488040 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:59.488015 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-40d96159-serving-cert\"" Jul 14 09:52:59.494894 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:59.494869 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-40d96159-69f54ccd7f-ncfjh"] Jul 14 09:52:59.611005 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:59.610957 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/394ffed8-b1b3-4ee7-9b4a-89aaee7791c6-maas-api-tls\") pod \"maas-api-e2e-migrate-40d96159-69f54ccd7f-ncfjh\" (UID: \"394ffed8-b1b3-4ee7-9b4a-89aaee7791c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-40d96159-69f54ccd7f-ncfjh" Jul 14 09:52:59.611190 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:59.611013 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dpnn\" (UniqueName: \"kubernetes.io/projected/394ffed8-b1b3-4ee7-9b4a-89aaee7791c6-kube-api-access-8dpnn\") pod \"maas-api-e2e-migrate-40d96159-69f54ccd7f-ncfjh\" (UID: \"394ffed8-b1b3-4ee7-9b4a-89aaee7791c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-40d96159-69f54ccd7f-ncfjh" Jul 14 09:52:59.611190 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:59.611112 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/394ffed8-b1b3-4ee7-9b4a-89aaee7791c6-openshift-service-ca\") pod \"maas-api-e2e-migrate-40d96159-69f54ccd7f-ncfjh\" (UID: \"394ffed8-b1b3-4ee7-9b4a-89aaee7791c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-40d96159-69f54ccd7f-ncfjh" Jul 14 09:52:59.712585 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:59.712542 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/394ffed8-b1b3-4ee7-9b4a-89aaee7791c6-maas-api-tls\") pod \"maas-api-e2e-migrate-40d96159-69f54ccd7f-ncfjh\" (UID: \"394ffed8-b1b3-4ee7-9b4a-89aaee7791c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-40d96159-69f54ccd7f-ncfjh" Jul 14 09:52:59.712585 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:59.712589 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dpnn\" (UniqueName: \"kubernetes.io/projected/394ffed8-b1b3-4ee7-9b4a-89aaee7791c6-kube-api-access-8dpnn\") pod \"maas-api-e2e-migrate-40d96159-69f54ccd7f-ncfjh\" (UID: \"394ffed8-b1b3-4ee7-9b4a-89aaee7791c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-40d96159-69f54ccd7f-ncfjh" Jul 14 09:52:59.712868 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:59.712715 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/394ffed8-b1b3-4ee7-9b4a-89aaee7791c6-openshift-service-ca\") pod \"maas-api-e2e-migrate-40d96159-69f54ccd7f-ncfjh\" (UID: \"394ffed8-b1b3-4ee7-9b4a-89aaee7791c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-40d96159-69f54ccd7f-ncfjh" Jul 14 09:52:59.713359 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:59.713333 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/394ffed8-b1b3-4ee7-9b4a-89aaee7791c6-openshift-service-ca\") pod \"maas-api-e2e-migrate-40d96159-69f54ccd7f-ncfjh\" (UID: \"394ffed8-b1b3-4ee7-9b4a-89aaee7791c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-40d96159-69f54ccd7f-ncfjh" Jul 14 09:52:59.715011 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:59.714987 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/394ffed8-b1b3-4ee7-9b4a-89aaee7791c6-maas-api-tls\") pod \"maas-api-e2e-migrate-40d96159-69f54ccd7f-ncfjh\" (UID: \"394ffed8-b1b3-4ee7-9b4a-89aaee7791c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-40d96159-69f54ccd7f-ncfjh" Jul 14 09:52:59.721325 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:59.721303 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dpnn\" (UniqueName: \"kubernetes.io/projected/394ffed8-b1b3-4ee7-9b4a-89aaee7791c6-kube-api-access-8dpnn\") pod \"maas-api-e2e-migrate-40d96159-69f54ccd7f-ncfjh\" (UID: \"394ffed8-b1b3-4ee7-9b4a-89aaee7791c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-40d96159-69f54ccd7f-ncfjh" Jul 14 09:52:59.797327 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:59.797229 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-40d96159-69f54ccd7f-ncfjh" Jul 14 09:52:59.940510 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:52:59.940477 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-40d96159-69f54ccd7f-ncfjh"] Jul 14 09:52:59.941845 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:52:59.941818 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod394ffed8_b1b3_4ee7_9b4a_89aaee7791c6.slice/crio-5abf64f4fd1f2712e6895a9dc1e94541b6cda7bfa696ecdf8adf097df1db056f WatchSource:0}: Error finding container 5abf64f4fd1f2712e6895a9dc1e94541b6cda7bfa696ecdf8adf097df1db056f: Status 404 returned error can't find the container with id 5abf64f4fd1f2712e6895a9dc1e94541b6cda7bfa696ecdf8adf097df1db056f Jul 14 09:53:00.746674 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:53:00.746636 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-40d96159-69f54ccd7f-ncfjh" event={"ID":"394ffed8-b1b3-4ee7-9b4a-89aaee7791c6","Type":"ContainerStarted","Data":"0d61103b90b5e7f43440de8b17a3383ee6491a9d92c355870ea0055b7a5f631b"} Jul 14 09:53:00.747136 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:53:00.746696 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-40d96159-69f54ccd7f-ncfjh" event={"ID":"394ffed8-b1b3-4ee7-9b4a-89aaee7791c6","Type":"ContainerStarted","Data":"5abf64f4fd1f2712e6895a9dc1e94541b6cda7bfa696ecdf8adf097df1db056f"} Jul 14 09:53:00.747136 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:53:00.746848 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-40d96159-69f54ccd7f-ncfjh" Jul 14 09:53:00.765028 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:53:00.764973 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-40d96159-69f54ccd7f-ncfjh" podStartSLOduration=1.133157804 podStartE2EDuration="1.764956628s" totalImagesPullingTime="631.798824ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 09:52:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:53:00.763965132 +0000 UTC m=+1570.273196594" watchObservedRunningTime="2026-07-14 09:53:00.764956628 +0000 UTC m=+1570.274188408" Jul 14 09:53:06.757192 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:53:06.757163 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-40d96159-69f54ccd7f-ncfjh" Jul 14 09:54:04.268007 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:04.267974 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-40d96159-69f54ccd7f-ncfjh"] Jul 14 09:54:04.268535 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:04.268196 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-40d96159-69f54ccd7f-ncfjh" podUID="394ffed8-b1b3-4ee7-9b4a-89aaee7791c6" containerName="maas-api" containerID="cri-o://0d61103b90b5e7f43440de8b17a3383ee6491a9d92c355870ea0055b7a5f631b" gracePeriod=30 Jul 14 09:54:04.530272 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:04.530207 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-40d96159-69f54ccd7f-ncfjh" Jul 14 09:54:04.554934 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:04.554906 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/394ffed8-b1b3-4ee7-9b4a-89aaee7791c6-maas-api-tls\") pod \"394ffed8-b1b3-4ee7-9b4a-89aaee7791c6\" (UID: \"394ffed8-b1b3-4ee7-9b4a-89aaee7791c6\") " Jul 14 09:54:04.555079 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:04.554999 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/394ffed8-b1b3-4ee7-9b4a-89aaee7791c6-openshift-service-ca\") pod \"394ffed8-b1b3-4ee7-9b4a-89aaee7791c6\" (UID: \"394ffed8-b1b3-4ee7-9b4a-89aaee7791c6\") " Jul 14 09:54:04.555079 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:04.555034 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8dpnn\" (UniqueName: \"kubernetes.io/projected/394ffed8-b1b3-4ee7-9b4a-89aaee7791c6-kube-api-access-8dpnn\") pod \"394ffed8-b1b3-4ee7-9b4a-89aaee7791c6\" (UID: \"394ffed8-b1b3-4ee7-9b4a-89aaee7791c6\") " Jul 14 09:54:04.555364 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:04.555342 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/394ffed8-b1b3-4ee7-9b4a-89aaee7791c6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "394ffed8-b1b3-4ee7-9b4a-89aaee7791c6" (UID: "394ffed8-b1b3-4ee7-9b4a-89aaee7791c6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 09:54:04.557035 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:04.557004 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/394ffed8-b1b3-4ee7-9b4a-89aaee7791c6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "394ffed8-b1b3-4ee7-9b4a-89aaee7791c6" (UID: "394ffed8-b1b3-4ee7-9b4a-89aaee7791c6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 09:54:04.557176 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:04.557097 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/394ffed8-b1b3-4ee7-9b4a-89aaee7791c6-kube-api-access-8dpnn" (OuterVolumeSpecName: "kube-api-access-8dpnn") pod "394ffed8-b1b3-4ee7-9b4a-89aaee7791c6" (UID: "394ffed8-b1b3-4ee7-9b4a-89aaee7791c6"). InnerVolumeSpecName "kube-api-access-8dpnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 09:54:04.655894 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:04.655843 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/394ffed8-b1b3-4ee7-9b4a-89aaee7791c6-openshift-service-ca\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:54:04.655894 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:04.655887 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dpnn\" (UniqueName: \"kubernetes.io/projected/394ffed8-b1b3-4ee7-9b4a-89aaee7791c6-kube-api-access-8dpnn\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:54:04.655894 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:04.655898 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/394ffed8-b1b3-4ee7-9b4a-89aaee7791c6-maas-api-tls\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:54:05.124628 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:05.124587 2565 generic.go:356] "Generic (PLEG): container finished" podID="394ffed8-b1b3-4ee7-9b4a-89aaee7791c6" containerID="0d61103b90b5e7f43440de8b17a3383ee6491a9d92c355870ea0055b7a5f631b" exitCode=0 Jul 14 09:54:05.124797 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:05.124642 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-40d96159-69f54ccd7f-ncfjh" Jul 14 09:54:05.124797 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:05.124673 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-40d96159-69f54ccd7f-ncfjh" event={"ID":"394ffed8-b1b3-4ee7-9b4a-89aaee7791c6","Type":"ContainerDied","Data":"0d61103b90b5e7f43440de8b17a3383ee6491a9d92c355870ea0055b7a5f631b"} Jul 14 09:54:05.124797 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:05.124750 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-40d96159-69f54ccd7f-ncfjh" event={"ID":"394ffed8-b1b3-4ee7-9b4a-89aaee7791c6","Type":"ContainerDied","Data":"5abf64f4fd1f2712e6895a9dc1e94541b6cda7bfa696ecdf8adf097df1db056f"} Jul 14 09:54:05.124797 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:05.124773 2565 scope.go:117] "RemoveContainer" containerID="0d61103b90b5e7f43440de8b17a3383ee6491a9d92c355870ea0055b7a5f631b" Jul 14 09:54:05.133897 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:05.133877 2565 scope.go:117] "RemoveContainer" containerID="0d61103b90b5e7f43440de8b17a3383ee6491a9d92c355870ea0055b7a5f631b" Jul 14 09:54:05.134181 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:54:05.134161 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d61103b90b5e7f43440de8b17a3383ee6491a9d92c355870ea0055b7a5f631b\": container with ID starting with 0d61103b90b5e7f43440de8b17a3383ee6491a9d92c355870ea0055b7a5f631b not found: ID does not exist" containerID="0d61103b90b5e7f43440de8b17a3383ee6491a9d92c355870ea0055b7a5f631b" Jul 14 09:54:05.134254 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:05.134191 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d61103b90b5e7f43440de8b17a3383ee6491a9d92c355870ea0055b7a5f631b"} err="failed to get container status \"0d61103b90b5e7f43440de8b17a3383ee6491a9d92c355870ea0055b7a5f631b\": rpc error: code = NotFound desc = could not find container \"0d61103b90b5e7f43440de8b17a3383ee6491a9d92c355870ea0055b7a5f631b\": container with ID starting with 0d61103b90b5e7f43440de8b17a3383ee6491a9d92c355870ea0055b7a5f631b not found: ID does not exist" Jul 14 09:54:05.142428 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:05.142369 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-40d96159-69f54ccd7f-ncfjh"] Jul 14 09:54:05.145439 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:05.145416 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-40d96159-69f54ccd7f-ncfjh"] Jul 14 09:54:06.969617 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:06.969582 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="394ffed8-b1b3-4ee7-9b4a-89aaee7791c6" path="/var/lib/kubelet/pods/394ffed8-b1b3-4ee7-9b4a-89aaee7791c6/volumes" Jul 14 09:54:11.722069 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:11.722031 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ae87aa32-5cb54d99b-89kfg"] Jul 14 09:54:11.722657 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:11.722641 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="394ffed8-b1b3-4ee7-9b4a-89aaee7791c6" containerName="maas-api" Jul 14 09:54:11.722721 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:11.722661 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="394ffed8-b1b3-4ee7-9b4a-89aaee7791c6" containerName="maas-api" Jul 14 09:54:11.722842 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:11.722829 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="394ffed8-b1b3-4ee7-9b4a-89aaee7791c6" containerName="maas-api" Jul 14 09:54:11.727628 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:11.727610 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae87aa32-5cb54d99b-89kfg" Jul 14 09:54:11.730683 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:11.730658 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ae87aa32-serving-cert\"" Jul 14 09:54:11.737226 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:11.737203 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ae87aa32-5cb54d99b-89kfg"] Jul 14 09:54:11.826478 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:11.826440 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8b26100-7ea2-4641-b8be-fcc85cd8daaa-maas-api-tls\") pod \"maas-api-e2e-ait-ae87aa32-5cb54d99b-89kfg\" (UID: \"b8b26100-7ea2-4641-b8be-fcc85cd8daaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae87aa32-5cb54d99b-89kfg" Jul 14 09:54:11.826662 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:11.826506 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8b26100-7ea2-4641-b8be-fcc85cd8daaa-openshift-service-ca\") pod \"maas-api-e2e-ait-ae87aa32-5cb54d99b-89kfg\" (UID: \"b8b26100-7ea2-4641-b8be-fcc85cd8daaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae87aa32-5cb54d99b-89kfg" Jul 14 09:54:11.826662 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:11.826585 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gz96\" (UniqueName: \"kubernetes.io/projected/b8b26100-7ea2-4641-b8be-fcc85cd8daaa-kube-api-access-5gz96\") pod \"maas-api-e2e-ait-ae87aa32-5cb54d99b-89kfg\" (UID: \"b8b26100-7ea2-4641-b8be-fcc85cd8daaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae87aa32-5cb54d99b-89kfg" Jul 14 09:54:11.927271 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:11.927217 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8b26100-7ea2-4641-b8be-fcc85cd8daaa-openshift-service-ca\") pod \"maas-api-e2e-ait-ae87aa32-5cb54d99b-89kfg\" (UID: \"b8b26100-7ea2-4641-b8be-fcc85cd8daaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae87aa32-5cb54d99b-89kfg" Jul 14 09:54:11.927455 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:11.927282 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gz96\" (UniqueName: \"kubernetes.io/projected/b8b26100-7ea2-4641-b8be-fcc85cd8daaa-kube-api-access-5gz96\") pod \"maas-api-e2e-ait-ae87aa32-5cb54d99b-89kfg\" (UID: \"b8b26100-7ea2-4641-b8be-fcc85cd8daaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae87aa32-5cb54d99b-89kfg" Jul 14 09:54:11.927455 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:11.927365 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8b26100-7ea2-4641-b8be-fcc85cd8daaa-maas-api-tls\") pod \"maas-api-e2e-ait-ae87aa32-5cb54d99b-89kfg\" (UID: \"b8b26100-7ea2-4641-b8be-fcc85cd8daaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae87aa32-5cb54d99b-89kfg" Jul 14 09:54:11.927999 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:11.927972 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8b26100-7ea2-4641-b8be-fcc85cd8daaa-openshift-service-ca\") pod \"maas-api-e2e-ait-ae87aa32-5cb54d99b-89kfg\" (UID: \"b8b26100-7ea2-4641-b8be-fcc85cd8daaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae87aa32-5cb54d99b-89kfg" Jul 14 09:54:11.929946 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:11.929923 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8b26100-7ea2-4641-b8be-fcc85cd8daaa-maas-api-tls\") pod \"maas-api-e2e-ait-ae87aa32-5cb54d99b-89kfg\" (UID: \"b8b26100-7ea2-4641-b8be-fcc85cd8daaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae87aa32-5cb54d99b-89kfg" Jul 14 09:54:11.938899 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:11.938874 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gz96\" (UniqueName: \"kubernetes.io/projected/b8b26100-7ea2-4641-b8be-fcc85cd8daaa-kube-api-access-5gz96\") pod \"maas-api-e2e-ait-ae87aa32-5cb54d99b-89kfg\" (UID: \"b8b26100-7ea2-4641-b8be-fcc85cd8daaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae87aa32-5cb54d99b-89kfg" Jul 14 09:54:12.040215 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:12.040125 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae87aa32-5cb54d99b-89kfg" Jul 14 09:54:12.184169 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:12.184144 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ae87aa32-5cb54d99b-89kfg"] Jul 14 09:54:12.187194 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:54:12.187164 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8b26100_7ea2_4641_b8be_fcc85cd8daaa.slice/crio-56032c1f6c09ca08d67c12145edfdc5c851973d193e47728cf28929f2a20e433 WatchSource:0}: Error finding container 56032c1f6c09ca08d67c12145edfdc5c851973d193e47728cf28929f2a20e433: Status 404 returned error can't find the container with id 56032c1f6c09ca08d67c12145edfdc5c851973d193e47728cf28929f2a20e433 Jul 14 09:54:13.173105 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:13.173070 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae87aa32-5cb54d99b-89kfg" event={"ID":"b8b26100-7ea2-4641-b8be-fcc85cd8daaa","Type":"ContainerStarted","Data":"a4e60f530fe13eba7fadeebeca9a58b62fed4b3c852ef141b2c099461919300f"} Jul 14 09:54:13.173500 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:13.173123 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae87aa32-5cb54d99b-89kfg" event={"ID":"b8b26100-7ea2-4641-b8be-fcc85cd8daaa","Type":"ContainerStarted","Data":"56032c1f6c09ca08d67c12145edfdc5c851973d193e47728cf28929f2a20e433"} Jul 14 09:54:13.173500 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:13.173184 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae87aa32-5cb54d99b-89kfg" Jul 14 09:54:13.191533 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:13.191479 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae87aa32-5cb54d99b-89kfg" podStartSLOduration=1.7934006550000001 podStartE2EDuration="2.19146452s" totalImagesPullingTime="398.063865ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 09:54:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:54:13.18978418 +0000 UTC m=+1642.699015622" watchObservedRunningTime="2026-07-14 09:54:13.19146452 +0000 UTC m=+1642.700695967" Jul 14 09:54:16.708094 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:16.708047 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae87aa32-xtvb9"] Jul 14 09:54:16.728248 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:16.728219 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae87aa32-xtvb9" Jul 14 09:54:16.731424 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:16.731398 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8m76v\"" Jul 14 09:54:16.731591 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:16.731539 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae87aa32-xtvb9"] Jul 14 09:54:16.780620 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:16.780584 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6624bc71-519c-49a2-a18c-30ca7b435901-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ae87aa32-xtvb9\" (UID: \"6624bc71-519c-49a2-a18c-30ca7b435901\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae87aa32-xtvb9" Jul 14 09:54:16.881919 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:16.881882 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6624bc71-519c-49a2-a18c-30ca7b435901-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ae87aa32-xtvb9\" (UID: \"6624bc71-519c-49a2-a18c-30ca7b435901\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae87aa32-xtvb9" Jul 14 09:54:16.882520 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:16.882499 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6624bc71-519c-49a2-a18c-30ca7b435901-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ae87aa32-xtvb9\" (UID: \"6624bc71-519c-49a2-a18c-30ca7b435901\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae87aa32-xtvb9" Jul 14 09:54:17.040463 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:17.040373 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae87aa32-xtvb9" Jul 14 09:54:17.179073 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:17.179043 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae87aa32-xtvb9"] Jul 14 09:54:17.180136 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:54:17.180107 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6624bc71_519c_49a2_a18c_30ca7b435901.slice/crio-a6445bf819fa8ef5c620727e53b5435b6f183f0ae4bc148d393e828ba0659127 WatchSource:0}: Error finding container a6445bf819fa8ef5c620727e53b5435b6f183f0ae4bc148d393e828ba0659127: Status 404 returned error can't find the container with id a6445bf819fa8ef5c620727e53b5435b6f183f0ae4bc148d393e828ba0659127 Jul 14 09:54:17.197946 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:17.197917 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae87aa32-xtvb9" event={"ID":"6624bc71-519c-49a2-a18c-30ca7b435901","Type":"ContainerStarted","Data":"a6445bf819fa8ef5c620727e53b5435b6f183f0ae4bc148d393e828ba0659127"} Jul 14 09:54:19.185967 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:19.185938 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae87aa32-5cb54d99b-89kfg" Jul 14 09:54:21.230222 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:21.230182 2565 generic.go:356] "Generic (PLEG): container finished" podID="6624bc71-519c-49a2-a18c-30ca7b435901" containerID="f5617038b78a2c9170c765162b66daa19ed67b4cbc4e5ea360c7544e222a10f9" exitCode=0 Jul 14 09:54:21.230633 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:21.230257 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae87aa32-xtvb9" event={"ID":"6624bc71-519c-49a2-a18c-30ca7b435901","Type":"ContainerDied","Data":"f5617038b78a2c9170c765162b66daa19ed67b4cbc4e5ea360c7544e222a10f9"} Jul 14 09:54:22.371978 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:22.371949 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae87aa32-xtvb9" Jul 14 09:54:22.441030 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:22.440993 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6624bc71-519c-49a2-a18c-30ca7b435901-maas-api-service-ca\") pod \"6624bc71-519c-49a2-a18c-30ca7b435901\" (UID: \"6624bc71-519c-49a2-a18c-30ca7b435901\") " Jul 14 09:54:22.441363 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:22.441341 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6624bc71-519c-49a2-a18c-30ca7b435901-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6624bc71-519c-49a2-a18c-30ca7b435901" (UID: "6624bc71-519c-49a2-a18c-30ca7b435901"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 09:54:22.542909 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:22.542796 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6624bc71-519c-49a2-a18c-30ca7b435901-maas-api-service-ca\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:54:23.244336 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:23.244303 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae87aa32-xtvb9" Jul 14 09:54:23.244509 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:23.244339 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ae87aa32-xtvb9" event={"ID":"6624bc71-519c-49a2-a18c-30ca7b435901","Type":"ContainerDied","Data":"a6445bf819fa8ef5c620727e53b5435b6f183f0ae4bc148d393e828ba0659127"} Jul 14 09:54:23.244509 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:23.244384 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6445bf819fa8ef5c620727e53b5435b6f183f0ae4bc148d393e828ba0659127" Jul 14 09:54:27.099412 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:27.099380 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ae87aa32-5cb54d99b-89kfg"] Jul 14 09:54:27.099943 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:27.099631 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae87aa32-5cb54d99b-89kfg" podUID="b8b26100-7ea2-4641-b8be-fcc85cd8daaa" containerName="maas-api" containerID="cri-o://a4e60f530fe13eba7fadeebeca9a58b62fed4b3c852ef141b2c099461919300f" gracePeriod=30 Jul 14 09:54:27.269969 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:27.269930 2565 generic.go:356] "Generic (PLEG): container finished" podID="b8b26100-7ea2-4641-b8be-fcc85cd8daaa" containerID="a4e60f530fe13eba7fadeebeca9a58b62fed4b3c852ef141b2c099461919300f" exitCode=0 Jul 14 09:54:27.270120 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:27.270005 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae87aa32-5cb54d99b-89kfg" event={"ID":"b8b26100-7ea2-4641-b8be-fcc85cd8daaa","Type":"ContainerDied","Data":"a4e60f530fe13eba7fadeebeca9a58b62fed4b3c852ef141b2c099461919300f"} Jul 14 09:54:27.347236 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:27.347206 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae87aa32-5cb54d99b-89kfg" Jul 14 09:54:27.491341 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:27.491304 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8b26100-7ea2-4641-b8be-fcc85cd8daaa-maas-api-tls\") pod \"b8b26100-7ea2-4641-b8be-fcc85cd8daaa\" (UID: \"b8b26100-7ea2-4641-b8be-fcc85cd8daaa\") " Jul 14 09:54:27.491554 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:27.491368 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8b26100-7ea2-4641-b8be-fcc85cd8daaa-openshift-service-ca\") pod \"b8b26100-7ea2-4641-b8be-fcc85cd8daaa\" (UID: \"b8b26100-7ea2-4641-b8be-fcc85cd8daaa\") " Jul 14 09:54:27.491554 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:27.491398 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5gz96\" (UniqueName: \"kubernetes.io/projected/b8b26100-7ea2-4641-b8be-fcc85cd8daaa-kube-api-access-5gz96\") pod \"b8b26100-7ea2-4641-b8be-fcc85cd8daaa\" (UID: \"b8b26100-7ea2-4641-b8be-fcc85cd8daaa\") " Jul 14 09:54:27.491836 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:27.491776 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8b26100-7ea2-4641-b8be-fcc85cd8daaa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b8b26100-7ea2-4641-b8be-fcc85cd8daaa" (UID: "b8b26100-7ea2-4641-b8be-fcc85cd8daaa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 09:54:27.493566 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:27.493541 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8b26100-7ea2-4641-b8be-fcc85cd8daaa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b8b26100-7ea2-4641-b8be-fcc85cd8daaa" (UID: "b8b26100-7ea2-4641-b8be-fcc85cd8daaa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 09:54:27.493671 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:27.493577 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8b26100-7ea2-4641-b8be-fcc85cd8daaa-kube-api-access-5gz96" (OuterVolumeSpecName: "kube-api-access-5gz96") pod "b8b26100-7ea2-4641-b8be-fcc85cd8daaa" (UID: "b8b26100-7ea2-4641-b8be-fcc85cd8daaa"). InnerVolumeSpecName "kube-api-access-5gz96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 09:54:27.592736 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:27.592701 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8b26100-7ea2-4641-b8be-fcc85cd8daaa-maas-api-tls\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:54:27.592736 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:27.592734 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8b26100-7ea2-4641-b8be-fcc85cd8daaa-openshift-service-ca\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:54:27.592736 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:27.592745 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gz96\" (UniqueName: \"kubernetes.io/projected/b8b26100-7ea2-4641-b8be-fcc85cd8daaa-kube-api-access-5gz96\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:54:28.276700 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:28.276665 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae87aa32-5cb54d99b-89kfg" event={"ID":"b8b26100-7ea2-4641-b8be-fcc85cd8daaa","Type":"ContainerDied","Data":"56032c1f6c09ca08d67c12145edfdc5c851973d193e47728cf28929f2a20e433"} Jul 14 09:54:28.276700 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:28.276698 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae87aa32-5cb54d99b-89kfg" Jul 14 09:54:28.277238 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:28.276735 2565 scope.go:117] "RemoveContainer" containerID="a4e60f530fe13eba7fadeebeca9a58b62fed4b3c852ef141b2c099461919300f" Jul 14 09:54:28.298570 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:28.298540 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ae87aa32-5cb54d99b-89kfg"] Jul 14 09:54:28.303780 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:28.303755 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ae87aa32-5cb54d99b-89kfg"] Jul 14 09:54:28.969732 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:28.969694 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8b26100-7ea2-4641-b8be-fcc85cd8daaa" path="/var/lib/kubelet/pods/b8b26100-7ea2-4641-b8be-fcc85cd8daaa/volumes" Jul 14 09:54:44.351085 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:44.351050 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-376a5d8c-78f58c575d-swk67"] Jul 14 09:54:44.351497 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:44.351483 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6624bc71-519c-49a2-a18c-30ca7b435901" containerName="revoke-keys" Jul 14 09:54:44.351542 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:44.351499 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6624bc71-519c-49a2-a18c-30ca7b435901" containerName="revoke-keys" Jul 14 09:54:44.351542 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:44.351514 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8b26100-7ea2-4641-b8be-fcc85cd8daaa" containerName="maas-api" Jul 14 09:54:44.351542 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:44.351520 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8b26100-7ea2-4641-b8be-fcc85cd8daaa" containerName="maas-api" Jul 14 09:54:44.351638 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:44.351594 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8b26100-7ea2-4641-b8be-fcc85cd8daaa" containerName="maas-api" Jul 14 09:54:44.351638 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:44.351610 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6624bc71-519c-49a2-a18c-30ca7b435901" containerName="revoke-keys" Jul 14 09:54:44.356252 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:44.356231 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-376a5d8c-78f58c575d-swk67" Jul 14 09:54:44.358883 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:44.358859 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-376a5d8c-serving-cert\"" Jul 14 09:54:44.366203 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:44.366182 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-376a5d8c-78f58c575d-swk67"] Jul 14 09:54:44.469661 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:44.469626 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdfdb46d-1ba1-4dff-b9c0-ce919098625e-openshift-service-ca\") pod \"maas-api-e2e-derive-376a5d8c-78f58c575d-swk67\" (UID: \"fdfdb46d-1ba1-4dff-b9c0-ce919098625e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-376a5d8c-78f58c575d-swk67" Jul 14 09:54:44.469854 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:44.469668 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj7wk\" (UniqueName: \"kubernetes.io/projected/fdfdb46d-1ba1-4dff-b9c0-ce919098625e-kube-api-access-nj7wk\") pod \"maas-api-e2e-derive-376a5d8c-78f58c575d-swk67\" (UID: \"fdfdb46d-1ba1-4dff-b9c0-ce919098625e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-376a5d8c-78f58c575d-swk67" Jul 14 09:54:44.469854 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:44.469749 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdfdb46d-1ba1-4dff-b9c0-ce919098625e-maas-api-tls\") pod \"maas-api-e2e-derive-376a5d8c-78f58c575d-swk67\" (UID: \"fdfdb46d-1ba1-4dff-b9c0-ce919098625e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-376a5d8c-78f58c575d-swk67" Jul 14 09:54:44.570454 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:44.570414 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdfdb46d-1ba1-4dff-b9c0-ce919098625e-maas-api-tls\") pod \"maas-api-e2e-derive-376a5d8c-78f58c575d-swk67\" (UID: \"fdfdb46d-1ba1-4dff-b9c0-ce919098625e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-376a5d8c-78f58c575d-swk67" Jul 14 09:54:44.570622 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:44.570498 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdfdb46d-1ba1-4dff-b9c0-ce919098625e-openshift-service-ca\") pod \"maas-api-e2e-derive-376a5d8c-78f58c575d-swk67\" (UID: \"fdfdb46d-1ba1-4dff-b9c0-ce919098625e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-376a5d8c-78f58c575d-swk67" Jul 14 09:54:44.570622 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:44.570519 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj7wk\" (UniqueName: \"kubernetes.io/projected/fdfdb46d-1ba1-4dff-b9c0-ce919098625e-kube-api-access-nj7wk\") pod \"maas-api-e2e-derive-376a5d8c-78f58c575d-swk67\" (UID: \"fdfdb46d-1ba1-4dff-b9c0-ce919098625e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-376a5d8c-78f58c575d-swk67" Jul 14 09:54:44.571199 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:44.571179 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdfdb46d-1ba1-4dff-b9c0-ce919098625e-openshift-service-ca\") pod \"maas-api-e2e-derive-376a5d8c-78f58c575d-swk67\" (UID: \"fdfdb46d-1ba1-4dff-b9c0-ce919098625e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-376a5d8c-78f58c575d-swk67" Jul 14 09:54:44.572857 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:44.572826 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdfdb46d-1ba1-4dff-b9c0-ce919098625e-maas-api-tls\") pod \"maas-api-e2e-derive-376a5d8c-78f58c575d-swk67\" (UID: \"fdfdb46d-1ba1-4dff-b9c0-ce919098625e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-376a5d8c-78f58c575d-swk67" Jul 14 09:54:44.579715 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:44.579693 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj7wk\" (UniqueName: \"kubernetes.io/projected/fdfdb46d-1ba1-4dff-b9c0-ce919098625e-kube-api-access-nj7wk\") pod \"maas-api-e2e-derive-376a5d8c-78f58c575d-swk67\" (UID: \"fdfdb46d-1ba1-4dff-b9c0-ce919098625e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-376a5d8c-78f58c575d-swk67" Jul 14 09:54:44.669638 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:44.669606 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-376a5d8c-78f58c575d-swk67" Jul 14 09:54:44.807354 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:44.807326 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-376a5d8c-78f58c575d-swk67"] Jul 14 09:54:44.809456 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:54:44.809417 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdfdb46d_1ba1_4dff_b9c0_ce919098625e.slice/crio-f07e381fb7f942f440a7483c2f6bbe08a1571377b1da38475b672cfeee69450b WatchSource:0}: Error finding container f07e381fb7f942f440a7483c2f6bbe08a1571377b1da38475b672cfeee69450b: Status 404 returned error can't find the container with id f07e381fb7f942f440a7483c2f6bbe08a1571377b1da38475b672cfeee69450b Jul 14 09:54:45.374795 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:45.374749 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-376a5d8c-78f58c575d-swk67" event={"ID":"fdfdb46d-1ba1-4dff-b9c0-ce919098625e","Type":"ContainerStarted","Data":"f07e381fb7f942f440a7483c2f6bbe08a1571377b1da38475b672cfeee69450b"} Jul 14 09:54:46.382606 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:46.382503 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-376a5d8c-78f58c575d-swk67" event={"ID":"fdfdb46d-1ba1-4dff-b9c0-ce919098625e","Type":"ContainerStarted","Data":"6e9f0c2a13d9e6bdd3a3fe24845066c88d267c95eeca36ba59b53f06459d403d"} Jul 14 09:54:46.382606 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:46.382577 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-376a5d8c-78f58c575d-swk67" Jul 14 09:54:46.401444 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:46.401389 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-376a5d8c-78f58c575d-swk67" podStartSLOduration=1.124973461 podStartE2EDuration="2.401374124s" totalImagesPullingTime="1.276400663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 09:54:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:54:46.399172772 +0000 UTC m=+1675.908404232" watchObservedRunningTime="2026-07-14 09:54:46.401374124 +0000 UTC m=+1675.910605571" Jul 14 09:54:48.551899 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:48.551863 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-376a5d8c-ctdxg"] Jul 14 09:54:48.555519 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:48.555502 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-376a5d8c-ctdxg" Jul 14 09:54:48.558415 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:48.558393 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8m76v\"" Jul 14 09:54:48.563611 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:48.563587 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-376a5d8c-ctdxg"] Jul 14 09:54:48.610436 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:48.610398 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebcfcf38-04ba-47b4-8cab-fd3f4c7e2034-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-376a5d8c-ctdxg\" (UID: \"ebcfcf38-04ba-47b4-8cab-fd3f4c7e2034\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-376a5d8c-ctdxg" Jul 14 09:54:48.711354 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:48.711317 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebcfcf38-04ba-47b4-8cab-fd3f4c7e2034-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-376a5d8c-ctdxg\" (UID: \"ebcfcf38-04ba-47b4-8cab-fd3f4c7e2034\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-376a5d8c-ctdxg" Jul 14 09:54:48.712026 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:48.712008 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebcfcf38-04ba-47b4-8cab-fd3f4c7e2034-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-376a5d8c-ctdxg\" (UID: \"ebcfcf38-04ba-47b4-8cab-fd3f4c7e2034\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-376a5d8c-ctdxg" Jul 14 09:54:48.868377 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:48.868291 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-376a5d8c-ctdxg" Jul 14 09:54:49.008293 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:49.008265 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-376a5d8c-ctdxg"] Jul 14 09:54:49.010169 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:54:49.010141 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebcfcf38_04ba_47b4_8cab_fd3f4c7e2034.slice/crio-41345259f7da092746a89e100fa5d17278b334aabf450628ac514d523fc58e2a WatchSource:0}: Error finding container 41345259f7da092746a89e100fa5d17278b334aabf450628ac514d523fc58e2a: Status 404 returned error can't find the container with id 41345259f7da092746a89e100fa5d17278b334aabf450628ac514d523fc58e2a Jul 14 09:54:49.401471 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:49.401441 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-376a5d8c-ctdxg" event={"ID":"ebcfcf38-04ba-47b4-8cab-fd3f4c7e2034","Type":"ContainerStarted","Data":"190a95d4220f768118d799f48297ce481b07375a5d3ab2234bbe3f6f84fa1860"} Jul 14 09:54:49.401603 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:49.401490 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-376a5d8c-ctdxg" event={"ID":"ebcfcf38-04ba-47b4-8cab-fd3f4c7e2034","Type":"ContainerStarted","Data":"41345259f7da092746a89e100fa5d17278b334aabf450628ac514d523fc58e2a"} Jul 14 09:54:49.429611 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:49.429557 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-376a5d8c-ctdxg" podStartSLOduration=1.429540931 podStartE2EDuration="1.429540931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 09:54:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:54:49.428059098 +0000 UTC m=+1678.937290546" watchObservedRunningTime="2026-07-14 09:54:49.429540931 +0000 UTC m=+1678.938772379" Jul 14 09:54:50.414514 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:50.414480 2565 generic.go:356] "Generic (PLEG): container finished" podID="ebcfcf38-04ba-47b4-8cab-fd3f4c7e2034" containerID="190a95d4220f768118d799f48297ce481b07375a5d3ab2234bbe3f6f84fa1860" exitCode=7 Jul 14 09:54:50.414984 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:50.414556 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-376a5d8c-ctdxg" event={"ID":"ebcfcf38-04ba-47b4-8cab-fd3f4c7e2034","Type":"ContainerDied","Data":"190a95d4220f768118d799f48297ce481b07375a5d3ab2234bbe3f6f84fa1860"} Jul 14 09:54:50.414984 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:50.414703 2565 scope.go:117] "RemoveContainer" containerID="190a95d4220f768118d799f48297ce481b07375a5d3ab2234bbe3f6f84fa1860" Jul 14 09:54:51.422953 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:51.422920 2565 generic.go:356] "Generic (PLEG): container finished" podID="ebcfcf38-04ba-47b4-8cab-fd3f4c7e2034" containerID="d844049c6ded5ad4b651ff606903d3dadc80c9754b5b017167219c4f4fdc9e8a" exitCode=7 Jul 14 09:54:51.423392 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:51.422989 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-376a5d8c-ctdxg" event={"ID":"ebcfcf38-04ba-47b4-8cab-fd3f4c7e2034","Type":"ContainerDied","Data":"d844049c6ded5ad4b651ff606903d3dadc80c9754b5b017167219c4f4fdc9e8a"} Jul 14 09:54:51.423392 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:51.423053 2565 scope.go:117] "RemoveContainer" containerID="190a95d4220f768118d799f48297ce481b07375a5d3ab2234bbe3f6f84fa1860" Jul 14 09:54:51.423392 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:51.423229 2565 scope.go:117] "RemoveContainer" containerID="d844049c6ded5ad4b651ff606903d3dadc80c9754b5b017167219c4f4fdc9e8a" Jul 14 09:54:51.423587 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:54:51.423511 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-376a5d8c-ctdxg_odh-ai-gateway-infra(ebcfcf38-04ba-47b4-8cab-fd3f4c7e2034)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-376a5d8c-ctdxg" podUID="ebcfcf38-04ba-47b4-8cab-fd3f4c7e2034" Jul 14 09:54:52.394833 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:52.394791 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-376a5d8c-78f58c575d-swk67" Jul 14 09:54:52.432713 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:54:52.432674 2565 scope.go:117] "RemoveContainer" containerID="d844049c6ded5ad4b651ff606903d3dadc80c9754b5b017167219c4f4fdc9e8a" Jul 14 09:54:52.433130 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:54:52.432996 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-376a5d8c-ctdxg_odh-ai-gateway-infra(ebcfcf38-04ba-47b4-8cab-fd3f4c7e2034)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-376a5d8c-ctdxg" podUID="ebcfcf38-04ba-47b4-8cab-fd3f4c7e2034" Jul 14 09:55:05.963216 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:05.963184 2565 scope.go:117] "RemoveContainer" containerID="d844049c6ded5ad4b651ff606903d3dadc80c9754b5b017167219c4f4fdc9e8a" Jul 14 09:55:06.515596 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:06.515556 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-376a5d8c-ctdxg" event={"ID":"ebcfcf38-04ba-47b4-8cab-fd3f4c7e2034","Type":"ContainerStarted","Data":"696cbb247c0c39d0de6f28b90c7e04ee64dbd7f4292ed857e855142b698948e6"} Jul 14 09:55:07.523201 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:07.523160 2565 generic.go:356] "Generic (PLEG): container finished" podID="ebcfcf38-04ba-47b4-8cab-fd3f4c7e2034" containerID="696cbb247c0c39d0de6f28b90c7e04ee64dbd7f4292ed857e855142b698948e6" exitCode=0 Jul 14 09:55:07.523602 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:07.523232 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-376a5d8c-ctdxg" event={"ID":"ebcfcf38-04ba-47b4-8cab-fd3f4c7e2034","Type":"ContainerDied","Data":"696cbb247c0c39d0de6f28b90c7e04ee64dbd7f4292ed857e855142b698948e6"} Jul 14 09:55:07.523602 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:07.523294 2565 scope.go:117] "RemoveContainer" containerID="d844049c6ded5ad4b651ff606903d3dadc80c9754b5b017167219c4f4fdc9e8a" Jul 14 09:55:07.549455 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:07.549425 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-376a5d8c-ctdxg"] Jul 14 09:55:08.664456 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:08.664429 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-376a5d8c-ctdxg" Jul 14 09:55:08.813251 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:08.813166 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebcfcf38-04ba-47b4-8cab-fd3f4c7e2034-maas-api-service-ca\") pod \"ebcfcf38-04ba-47b4-8cab-fd3f4c7e2034\" (UID: \"ebcfcf38-04ba-47b4-8cab-fd3f4c7e2034\") " Jul 14 09:55:08.813533 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:08.813508 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebcfcf38-04ba-47b4-8cab-fd3f4c7e2034-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ebcfcf38-04ba-47b4-8cab-fd3f4c7e2034" (UID: "ebcfcf38-04ba-47b4-8cab-fd3f4c7e2034"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 09:55:08.914706 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:08.914668 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebcfcf38-04ba-47b4-8cab-fd3f4c7e2034-maas-api-service-ca\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:55:09.537247 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:09.537217 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-376a5d8c-ctdxg" Jul 14 09:55:09.537440 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:09.537210 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-376a5d8c-ctdxg" event={"ID":"ebcfcf38-04ba-47b4-8cab-fd3f4c7e2034","Type":"ContainerDied","Data":"41345259f7da092746a89e100fa5d17278b334aabf450628ac514d523fc58e2a"} Jul 14 09:55:09.537440 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:09.537356 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41345259f7da092746a89e100fa5d17278b334aabf450628ac514d523fc58e2a" Jul 14 09:55:09.569338 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:09.569305 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-376a5d8c-ctdxg"] Jul 14 09:55:09.572738 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:09.572709 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-376a5d8c-ctdxg"] Jul 14 09:55:10.970712 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:10.970677 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebcfcf38-04ba-47b4-8cab-fd3f4c7e2034" path="/var/lib/kubelet/pods/ebcfcf38-04ba-47b4-8cab-fd3f4c7e2034/volumes" Jul 14 09:55:48.598463 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:48.598429 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-376a5d8c-wdnb8"] Jul 14 09:55:48.598976 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:48.598849 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebcfcf38-04ba-47b4-8cab-fd3f4c7e2034" containerName="revoke-keys" Jul 14 09:55:48.598976 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:48.598861 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebcfcf38-04ba-47b4-8cab-fd3f4c7e2034" containerName="revoke-keys" Jul 14 09:55:48.598976 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:48.598884 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebcfcf38-04ba-47b4-8cab-fd3f4c7e2034" containerName="revoke-keys" Jul 14 09:55:48.598976 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:48.598892 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebcfcf38-04ba-47b4-8cab-fd3f4c7e2034" containerName="revoke-keys" Jul 14 09:55:48.598976 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:48.598950 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebcfcf38-04ba-47b4-8cab-fd3f4c7e2034" containerName="revoke-keys" Jul 14 09:55:48.598976 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:48.598961 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebcfcf38-04ba-47b4-8cab-fd3f4c7e2034" containerName="revoke-keys" Jul 14 09:55:48.602227 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:48.602211 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-376a5d8c-wdnb8" Jul 14 09:55:48.604640 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:48.604621 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8m76v\"" Jul 14 09:55:48.610117 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:48.609919 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-376a5d8c-wdnb8"] Jul 14 09:55:48.702748 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:48.702707 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dc86b02-1d67-4539-b3eb-df3ccdcda27c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-376a5d8c-wdnb8\" (UID: \"5dc86b02-1d67-4539-b3eb-df3ccdcda27c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-376a5d8c-wdnb8" Jul 14 09:55:48.804231 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:48.804196 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dc86b02-1d67-4539-b3eb-df3ccdcda27c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-376a5d8c-wdnb8\" (UID: \"5dc86b02-1d67-4539-b3eb-df3ccdcda27c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-376a5d8c-wdnb8" Jul 14 09:55:48.804907 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:48.804880 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dc86b02-1d67-4539-b3eb-df3ccdcda27c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-376a5d8c-wdnb8\" (UID: \"5dc86b02-1d67-4539-b3eb-df3ccdcda27c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-376a5d8c-wdnb8" Jul 14 09:55:48.914367 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:48.914332 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-376a5d8c-wdnb8" Jul 14 09:55:49.053847 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:49.051426 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-376a5d8c-wdnb8"] Jul 14 09:55:49.772602 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:49.772568 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-376a5d8c-wdnb8" event={"ID":"5dc86b02-1d67-4539-b3eb-df3ccdcda27c","Type":"ContainerStarted","Data":"843599eb33ee1f931eb78e8b2142a6a858bdc73fb450b1879a3bc4c12d74dd04"} Jul 14 09:55:49.773005 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:49.772617 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-376a5d8c-wdnb8" event={"ID":"5dc86b02-1d67-4539-b3eb-df3ccdcda27c","Type":"ContainerStarted","Data":"e2ce52bc59890aa5a88a9712db748f67d7fa82e98278608821279ee8b0e88d72"} Jul 14 09:55:49.790303 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:49.790258 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-376a5d8c-wdnb8" podStartSLOduration=1.7902436769999999 podStartE2EDuration="1.790243677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 09:55:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:55:49.788731995 +0000 UTC m=+1739.297963443" watchObservedRunningTime="2026-07-14 09:55:49.790243677 +0000 UTC m=+1739.299475128" Jul 14 09:55:50.780082 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:50.780046 2565 generic.go:356] "Generic (PLEG): container finished" podID="5dc86b02-1d67-4539-b3eb-df3ccdcda27c" containerID="843599eb33ee1f931eb78e8b2142a6a858bdc73fb450b1879a3bc4c12d74dd04" exitCode=0 Jul 14 09:55:50.780526 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:50.780105 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-376a5d8c-wdnb8" event={"ID":"5dc86b02-1d67-4539-b3eb-df3ccdcda27c","Type":"ContainerDied","Data":"843599eb33ee1f931eb78e8b2142a6a858bdc73fb450b1879a3bc4c12d74dd04"} Jul 14 09:55:51.920176 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:51.920152 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-376a5d8c-wdnb8" Jul 14 09:55:51.933023 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:51.932992 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dc86b02-1d67-4539-b3eb-df3ccdcda27c-maas-api-service-ca\") pod \"5dc86b02-1d67-4539-b3eb-df3ccdcda27c\" (UID: \"5dc86b02-1d67-4539-b3eb-df3ccdcda27c\") " Jul 14 09:55:51.933320 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:51.933299 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dc86b02-1d67-4539-b3eb-df3ccdcda27c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5dc86b02-1d67-4539-b3eb-df3ccdcda27c" (UID: "5dc86b02-1d67-4539-b3eb-df3ccdcda27c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 09:55:52.034303 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:52.034267 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dc86b02-1d67-4539-b3eb-df3ccdcda27c-maas-api-service-ca\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:55:52.792535 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:52.792507 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-376a5d8c-wdnb8" Jul 14 09:55:52.792535 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:52.792510 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-376a5d8c-wdnb8" event={"ID":"5dc86b02-1d67-4539-b3eb-df3ccdcda27c","Type":"ContainerDied","Data":"e2ce52bc59890aa5a88a9712db748f67d7fa82e98278608821279ee8b0e88d72"} Jul 14 09:55:52.792750 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:52.792558 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2ce52bc59890aa5a88a9712db748f67d7fa82e98278608821279ee8b0e88d72" Jul 14 09:55:59.168546 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:59.168514 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-376a5d8c-78f58c575d-swk67"] Jul 14 09:55:59.168986 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:59.168693 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-376a5d8c-78f58c575d-swk67" podUID="fdfdb46d-1ba1-4dff-b9c0-ce919098625e" containerName="maas-api" containerID="cri-o://6e9f0c2a13d9e6bdd3a3fe24845066c88d267c95eeca36ba59b53f06459d403d" gracePeriod=30 Jul 14 09:55:59.420453 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:59.420374 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-376a5d8c-78f58c575d-swk67" Jul 14 09:55:59.510151 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:59.510110 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdfdb46d-1ba1-4dff-b9c0-ce919098625e-maas-api-tls\") pod \"fdfdb46d-1ba1-4dff-b9c0-ce919098625e\" (UID: \"fdfdb46d-1ba1-4dff-b9c0-ce919098625e\") " Jul 14 09:55:59.510349 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:59.510192 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nj7wk\" (UniqueName: \"kubernetes.io/projected/fdfdb46d-1ba1-4dff-b9c0-ce919098625e-kube-api-access-nj7wk\") pod \"fdfdb46d-1ba1-4dff-b9c0-ce919098625e\" (UID: \"fdfdb46d-1ba1-4dff-b9c0-ce919098625e\") " Jul 14 09:55:59.510349 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:59.510289 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdfdb46d-1ba1-4dff-b9c0-ce919098625e-openshift-service-ca\") pod \"fdfdb46d-1ba1-4dff-b9c0-ce919098625e\" (UID: \"fdfdb46d-1ba1-4dff-b9c0-ce919098625e\") " Jul 14 09:55:59.510644 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:59.510620 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdfdb46d-1ba1-4dff-b9c0-ce919098625e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fdfdb46d-1ba1-4dff-b9c0-ce919098625e" (UID: "fdfdb46d-1ba1-4dff-b9c0-ce919098625e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 09:55:59.512236 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:59.512209 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdfdb46d-1ba1-4dff-b9c0-ce919098625e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fdfdb46d-1ba1-4dff-b9c0-ce919098625e" (UID: "fdfdb46d-1ba1-4dff-b9c0-ce919098625e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 09:55:59.512346 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:59.512296 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdfdb46d-1ba1-4dff-b9c0-ce919098625e-kube-api-access-nj7wk" (OuterVolumeSpecName: "kube-api-access-nj7wk") pod "fdfdb46d-1ba1-4dff-b9c0-ce919098625e" (UID: "fdfdb46d-1ba1-4dff-b9c0-ce919098625e"). InnerVolumeSpecName "kube-api-access-nj7wk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 09:55:59.611499 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:59.611453 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdfdb46d-1ba1-4dff-b9c0-ce919098625e-openshift-service-ca\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:55:59.611499 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:59.611494 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdfdb46d-1ba1-4dff-b9c0-ce919098625e-maas-api-tls\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:55:59.611499 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:59.611507 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj7wk\" (UniqueName: \"kubernetes.io/projected/fdfdb46d-1ba1-4dff-b9c0-ce919098625e-kube-api-access-nj7wk\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:55:59.833791 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:59.833653 2565 generic.go:356] "Generic (PLEG): container finished" podID="fdfdb46d-1ba1-4dff-b9c0-ce919098625e" containerID="6e9f0c2a13d9e6bdd3a3fe24845066c88d267c95eeca36ba59b53f06459d403d" exitCode=0 Jul 14 09:55:59.833791 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:59.833703 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-376a5d8c-78f58c575d-swk67" Jul 14 09:55:59.833791 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:59.833696 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-376a5d8c-78f58c575d-swk67" event={"ID":"fdfdb46d-1ba1-4dff-b9c0-ce919098625e","Type":"ContainerDied","Data":"6e9f0c2a13d9e6bdd3a3fe24845066c88d267c95eeca36ba59b53f06459d403d"} Jul 14 09:55:59.833791 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:59.833766 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-376a5d8c-78f58c575d-swk67" event={"ID":"fdfdb46d-1ba1-4dff-b9c0-ce919098625e","Type":"ContainerDied","Data":"f07e381fb7f942f440a7483c2f6bbe08a1571377b1da38475b672cfeee69450b"} Jul 14 09:55:59.834115 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:59.833798 2565 scope.go:117] "RemoveContainer" containerID="6e9f0c2a13d9e6bdd3a3fe24845066c88d267c95eeca36ba59b53f06459d403d" Jul 14 09:55:59.846409 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:59.846390 2565 scope.go:117] "RemoveContainer" containerID="6e9f0c2a13d9e6bdd3a3fe24845066c88d267c95eeca36ba59b53f06459d403d" Jul 14 09:55:59.846690 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:55:59.846671 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e9f0c2a13d9e6bdd3a3fe24845066c88d267c95eeca36ba59b53f06459d403d\": container with ID starting with 6e9f0c2a13d9e6bdd3a3fe24845066c88d267c95eeca36ba59b53f06459d403d not found: ID does not exist" containerID="6e9f0c2a13d9e6bdd3a3fe24845066c88d267c95eeca36ba59b53f06459d403d" Jul 14 09:55:59.846754 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:59.846707 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e9f0c2a13d9e6bdd3a3fe24845066c88d267c95eeca36ba59b53f06459d403d"} err="failed to get container status \"6e9f0c2a13d9e6bdd3a3fe24845066c88d267c95eeca36ba59b53f06459d403d\": rpc error: code = NotFound desc = could not find container \"6e9f0c2a13d9e6bdd3a3fe24845066c88d267c95eeca36ba59b53f06459d403d\": container with ID starting with 6e9f0c2a13d9e6bdd3a3fe24845066c88d267c95eeca36ba59b53f06459d403d not found: ID does not exist" Jul 14 09:55:59.863657 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:59.863608 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-376a5d8c-78f58c575d-swk67"] Jul 14 09:55:59.864235 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:55:59.864208 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-376a5d8c-78f58c575d-swk67"] Jul 14 09:56:00.970286 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:56:00.970250 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdfdb46d-1ba1-4dff-b9c0-ce919098625e" path="/var/lib/kubelet/pods/fdfdb46d-1ba1-4dff-b9c0-ce919098625e/volumes" Jul 14 09:57:54.148479 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:57:54.148446 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dc470fea-6c5f5b55bc-ntrxb"] Jul 14 09:57:54.149023 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:57:54.149005 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebcfcf38-04ba-47b4-8cab-fd3f4c7e2034" containerName="revoke-keys" Jul 14 09:57:54.149023 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:57:54.149024 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebcfcf38-04ba-47b4-8cab-fd3f4c7e2034" containerName="revoke-keys" Jul 14 09:57:54.149102 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:57:54.149036 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdfdb46d-1ba1-4dff-b9c0-ce919098625e" containerName="maas-api" Jul 14 09:57:54.149102 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:57:54.149042 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdfdb46d-1ba1-4dff-b9c0-ce919098625e" containerName="maas-api" Jul 14 09:57:54.149102 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:57:54.149065 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dc86b02-1d67-4539-b3eb-df3ccdcda27c" containerName="revoke-keys" Jul 14 09:57:54.149102 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:57:54.149071 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dc86b02-1d67-4539-b3eb-df3ccdcda27c" containerName="revoke-keys" Jul 14 09:57:54.149238 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:57:54.149134 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebcfcf38-04ba-47b4-8cab-fd3f4c7e2034" containerName="revoke-keys" Jul 14 09:57:54.149238 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:57:54.149144 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dc86b02-1d67-4539-b3eb-df3ccdcda27c" containerName="revoke-keys" Jul 14 09:57:54.149238 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:57:54.149150 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdfdb46d-1ba1-4dff-b9c0-ce919098625e" containerName="maas-api" Jul 14 09:57:54.152186 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:57:54.152168 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc470fea-6c5f5b55bc-ntrxb" Jul 14 09:57:54.154649 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:57:54.154628 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-dc470fea-serving-cert\"" Jul 14 09:57:54.166647 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:57:54.166621 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dc470fea-6c5f5b55bc-ntrxb"] Jul 14 09:57:54.212838 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:57:54.212773 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqcwc\" (UniqueName: \"kubernetes.io/projected/a4c1964c-245d-4195-84c5-6299e9308e86-kube-api-access-xqcwc\") pod \"maas-api-e2e-mt-dc470fea-6c5f5b55bc-ntrxb\" (UID: \"a4c1964c-245d-4195-84c5-6299e9308e86\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc470fea-6c5f5b55bc-ntrxb" Jul 14 09:57:54.213006 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:57:54.212895 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4c1964c-245d-4195-84c5-6299e9308e86-openshift-service-ca\") pod \"maas-api-e2e-mt-dc470fea-6c5f5b55bc-ntrxb\" (UID: \"a4c1964c-245d-4195-84c5-6299e9308e86\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc470fea-6c5f5b55bc-ntrxb" Jul 14 09:57:54.213006 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:57:54.212982 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4c1964c-245d-4195-84c5-6299e9308e86-maas-api-tls\") pod \"maas-api-e2e-mt-dc470fea-6c5f5b55bc-ntrxb\" (UID: \"a4c1964c-245d-4195-84c5-6299e9308e86\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc470fea-6c5f5b55bc-ntrxb" Jul 14 09:57:54.314140 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:57:54.314101 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqcwc\" (UniqueName: \"kubernetes.io/projected/a4c1964c-245d-4195-84c5-6299e9308e86-kube-api-access-xqcwc\") pod \"maas-api-e2e-mt-dc470fea-6c5f5b55bc-ntrxb\" (UID: \"a4c1964c-245d-4195-84c5-6299e9308e86\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc470fea-6c5f5b55bc-ntrxb" Jul 14 09:57:54.314343 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:57:54.314173 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4c1964c-245d-4195-84c5-6299e9308e86-openshift-service-ca\") pod \"maas-api-e2e-mt-dc470fea-6c5f5b55bc-ntrxb\" (UID: \"a4c1964c-245d-4195-84c5-6299e9308e86\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc470fea-6c5f5b55bc-ntrxb" Jul 14 09:57:54.314343 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:57:54.314227 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4c1964c-245d-4195-84c5-6299e9308e86-maas-api-tls\") pod \"maas-api-e2e-mt-dc470fea-6c5f5b55bc-ntrxb\" (UID: \"a4c1964c-245d-4195-84c5-6299e9308e86\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc470fea-6c5f5b55bc-ntrxb" Jul 14 09:57:54.314479 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:57:54.314351 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-dc470fea-serving-cert: secret "maas-api-e2e-mt-dc470fea-serving-cert" not found Jul 14 09:57:54.314479 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:57:54.314417 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4c1964c-245d-4195-84c5-6299e9308e86-maas-api-tls podName:a4c1964c-245d-4195-84c5-6299e9308e86 nodeName:}" failed. No retries permitted until 2026-07-14 09:57:54.814396896 +0000 UTC m=+1864.323628326 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a4c1964c-245d-4195-84c5-6299e9308e86-maas-api-tls") pod "maas-api-e2e-mt-dc470fea-6c5f5b55bc-ntrxb" (UID: "a4c1964c-245d-4195-84c5-6299e9308e86") : secret "maas-api-e2e-mt-dc470fea-serving-cert" not found Jul 14 09:57:54.314954 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:57:54.314928 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4c1964c-245d-4195-84c5-6299e9308e86-openshift-service-ca\") pod \"maas-api-e2e-mt-dc470fea-6c5f5b55bc-ntrxb\" (UID: \"a4c1964c-245d-4195-84c5-6299e9308e86\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc470fea-6c5f5b55bc-ntrxb" Jul 14 09:57:54.329890 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:57:54.329864 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqcwc\" (UniqueName: \"kubernetes.io/projected/a4c1964c-245d-4195-84c5-6299e9308e86-kube-api-access-xqcwc\") pod \"maas-api-e2e-mt-dc470fea-6c5f5b55bc-ntrxb\" (UID: \"a4c1964c-245d-4195-84c5-6299e9308e86\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc470fea-6c5f5b55bc-ntrxb" Jul 14 09:57:54.819757 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:57:54.819708 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4c1964c-245d-4195-84c5-6299e9308e86-maas-api-tls\") pod \"maas-api-e2e-mt-dc470fea-6c5f5b55bc-ntrxb\" (UID: \"a4c1964c-245d-4195-84c5-6299e9308e86\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc470fea-6c5f5b55bc-ntrxb" Jul 14 09:57:54.822108 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:57:54.822080 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4c1964c-245d-4195-84c5-6299e9308e86-maas-api-tls\") pod \"maas-api-e2e-mt-dc470fea-6c5f5b55bc-ntrxb\" (UID: \"a4c1964c-245d-4195-84c5-6299e9308e86\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc470fea-6c5f5b55bc-ntrxb" Jul 14 09:57:55.062907 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:57:55.062871 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc470fea-6c5f5b55bc-ntrxb" Jul 14 09:57:55.203474 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:57:55.203450 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dc470fea-6c5f5b55bc-ntrxb"] Jul 14 09:57:55.204784 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:57:55.204758 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4c1964c_245d_4195_84c5_6299e9308e86.slice/crio-10498e2075fe28d959933db2b0d55eb2cedefd781a1871ea77912ef7e4d85f1e WatchSource:0}: Error finding container 10498e2075fe28d959933db2b0d55eb2cedefd781a1871ea77912ef7e4d85f1e: Status 404 returned error can't find the container with id 10498e2075fe28d959933db2b0d55eb2cedefd781a1871ea77912ef7e4d85f1e Jul 14 09:57:55.206246 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:57:55.206229 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 09:57:55.491319 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:57:55.491283 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc470fea-6c5f5b55bc-ntrxb" event={"ID":"a4c1964c-245d-4195-84c5-6299e9308e86","Type":"ContainerStarted","Data":"10498e2075fe28d959933db2b0d55eb2cedefd781a1871ea77912ef7e4d85f1e"} Jul 14 09:57:56.499282 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:57:56.499245 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc470fea-6c5f5b55bc-ntrxb" event={"ID":"a4c1964c-245d-4195-84c5-6299e9308e86","Type":"ContainerStarted","Data":"9f8242e7e3c36636636d9694197e2fe043dfc3af305d46ef492c405b9e43985c"} Jul 14 09:57:56.499678 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:57:56.499344 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc470fea-6c5f5b55bc-ntrxb" Jul 14 09:57:56.521324 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:57:56.521266 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc470fea-6c5f5b55bc-ntrxb" podStartSLOduration=1.814627576 podStartE2EDuration="2.5212507s" totalImagesPullingTime="706.623124ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 09:57:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:57:56.518779085 +0000 UTC m=+1866.028010534" watchObservedRunningTime="2026-07-14 09:57:56.5212507 +0000 UTC m=+1866.030482147" Jul 14 09:58:02.511660 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:58:02.511628 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc470fea-6c5f5b55bc-ntrxb" Jul 14 09:58:16.307441 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:58:16.307350 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dc470fea-6c5f5b55bc-ntrxb"] Jul 14 09:58:16.307911 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:58:16.307646 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc470fea-6c5f5b55bc-ntrxb" podUID="a4c1964c-245d-4195-84c5-6299e9308e86" containerName="maas-api" containerID="cri-o://9f8242e7e3c36636636d9694197e2fe043dfc3af305d46ef492c405b9e43985c" gracePeriod=30 Jul 14 09:58:16.562137 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:58:16.562079 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc470fea-6c5f5b55bc-ntrxb" Jul 14 09:58:16.613501 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:58:16.613460 2565 generic.go:356] "Generic (PLEG): container finished" podID="a4c1964c-245d-4195-84c5-6299e9308e86" containerID="9f8242e7e3c36636636d9694197e2fe043dfc3af305d46ef492c405b9e43985c" exitCode=0 Jul 14 09:58:16.613669 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:58:16.613522 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc470fea-6c5f5b55bc-ntrxb" Jul 14 09:58:16.613669 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:58:16.613542 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc470fea-6c5f5b55bc-ntrxb" event={"ID":"a4c1964c-245d-4195-84c5-6299e9308e86","Type":"ContainerDied","Data":"9f8242e7e3c36636636d9694197e2fe043dfc3af305d46ef492c405b9e43985c"} Jul 14 09:58:16.613669 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:58:16.613631 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc470fea-6c5f5b55bc-ntrxb" event={"ID":"a4c1964c-245d-4195-84c5-6299e9308e86","Type":"ContainerDied","Data":"10498e2075fe28d959933db2b0d55eb2cedefd781a1871ea77912ef7e4d85f1e"} Jul 14 09:58:16.613669 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:58:16.613660 2565 scope.go:117] "RemoveContainer" containerID="9f8242e7e3c36636636d9694197e2fe043dfc3af305d46ef492c405b9e43985c" Jul 14 09:58:16.622777 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:58:16.622761 2565 scope.go:117] "RemoveContainer" containerID="9f8242e7e3c36636636d9694197e2fe043dfc3af305d46ef492c405b9e43985c" Jul 14 09:58:16.623066 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:58:16.623048 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f8242e7e3c36636636d9694197e2fe043dfc3af305d46ef492c405b9e43985c\": container with ID starting with 9f8242e7e3c36636636d9694197e2fe043dfc3af305d46ef492c405b9e43985c not found: ID does not exist" containerID="9f8242e7e3c36636636d9694197e2fe043dfc3af305d46ef492c405b9e43985c" Jul 14 09:58:16.623114 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:58:16.623079 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f8242e7e3c36636636d9694197e2fe043dfc3af305d46ef492c405b9e43985c"} err="failed to get container status \"9f8242e7e3c36636636d9694197e2fe043dfc3af305d46ef492c405b9e43985c\": rpc error: code = NotFound desc = could not find container \"9f8242e7e3c36636636d9694197e2fe043dfc3af305d46ef492c405b9e43985c\": container with ID starting with 9f8242e7e3c36636636d9694197e2fe043dfc3af305d46ef492c405b9e43985c not found: ID does not exist" Jul 14 09:58:16.741352 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:58:16.741304 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xqcwc\" (UniqueName: \"kubernetes.io/projected/a4c1964c-245d-4195-84c5-6299e9308e86-kube-api-access-xqcwc\") pod \"a4c1964c-245d-4195-84c5-6299e9308e86\" (UID: \"a4c1964c-245d-4195-84c5-6299e9308e86\") " Jul 14 09:58:16.741524 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:58:16.741426 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4c1964c-245d-4195-84c5-6299e9308e86-openshift-service-ca\") pod \"a4c1964c-245d-4195-84c5-6299e9308e86\" (UID: \"a4c1964c-245d-4195-84c5-6299e9308e86\") " Jul 14 09:58:16.741524 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:58:16.741448 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4c1964c-245d-4195-84c5-6299e9308e86-maas-api-tls\") pod \"a4c1964c-245d-4195-84c5-6299e9308e86\" (UID: \"a4c1964c-245d-4195-84c5-6299e9308e86\") " Jul 14 09:58:16.741826 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:58:16.741774 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4c1964c-245d-4195-84c5-6299e9308e86-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a4c1964c-245d-4195-84c5-6299e9308e86" (UID: "a4c1964c-245d-4195-84c5-6299e9308e86"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 09:58:16.743482 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:58:16.743463 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4c1964c-245d-4195-84c5-6299e9308e86-kube-api-access-xqcwc" (OuterVolumeSpecName: "kube-api-access-xqcwc") pod "a4c1964c-245d-4195-84c5-6299e9308e86" (UID: "a4c1964c-245d-4195-84c5-6299e9308e86"). InnerVolumeSpecName "kube-api-access-xqcwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 09:58:16.743586 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:58:16.743564 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4c1964c-245d-4195-84c5-6299e9308e86-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a4c1964c-245d-4195-84c5-6299e9308e86" (UID: "a4c1964c-245d-4195-84c5-6299e9308e86"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 09:58:16.843130 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:58:16.843028 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4c1964c-245d-4195-84c5-6299e9308e86-openshift-service-ca\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:58:16.843130 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:58:16.843071 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4c1964c-245d-4195-84c5-6299e9308e86-maas-api-tls\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:58:16.843130 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:58:16.843082 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqcwc\" (UniqueName: \"kubernetes.io/projected/a4c1964c-245d-4195-84c5-6299e9308e86-kube-api-access-xqcwc\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:58:16.937059 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:58:16.937029 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dc470fea-6c5f5b55bc-ntrxb"] Jul 14 09:58:16.942543 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:58:16.942515 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dc470fea-6c5f5b55bc-ntrxb"] Jul 14 09:58:16.970194 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:58:16.970157 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4c1964c-245d-4195-84c5-6299e9308e86" path="/var/lib/kubelet/pods/a4c1964c-245d-4195-84c5-6299e9308e86/volumes" Jul 14 09:58:54.192352 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:58:54.192311 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b2b30ce5-54d87597f-86xrq"] Jul 14 09:58:54.192988 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:58:54.192717 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4c1964c-245d-4195-84c5-6299e9308e86" containerName="maas-api" Jul 14 09:58:54.192988 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:58:54.192728 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4c1964c-245d-4195-84c5-6299e9308e86" containerName="maas-api" Jul 14 09:58:54.192988 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:58:54.192795 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4c1964c-245d-4195-84c5-6299e9308e86" containerName="maas-api" Jul 14 09:58:54.195929 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:58:54.195911 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2b30ce5-54d87597f-86xrq" Jul 14 09:58:54.198417 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:58:54.198393 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-b2b30ce5-serving-cert\"" Jul 14 09:58:54.209649 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:58:54.209619 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b2b30ce5-54d87597f-86xrq"] Jul 14 09:58:54.300707 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:58:54.300674 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a56cb06b-0df1-4821-907d-050d9a78511c-maas-api-tls\") pod \"maas-api-e2e-mt-b2b30ce5-54d87597f-86xrq\" (UID: \"a56cb06b-0df1-4821-907d-050d9a78511c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2b30ce5-54d87597f-86xrq" Jul 14 09:58:54.300944 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:58:54.300791 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a56cb06b-0df1-4821-907d-050d9a78511c-openshift-service-ca\") pod \"maas-api-e2e-mt-b2b30ce5-54d87597f-86xrq\" (UID: \"a56cb06b-0df1-4821-907d-050d9a78511c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2b30ce5-54d87597f-86xrq" Jul 14 09:58:54.301033 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:58:54.300968 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vwl2\" (UniqueName: \"kubernetes.io/projected/a56cb06b-0df1-4821-907d-050d9a78511c-kube-api-access-8vwl2\") pod \"maas-api-e2e-mt-b2b30ce5-54d87597f-86xrq\" (UID: \"a56cb06b-0df1-4821-907d-050d9a78511c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2b30ce5-54d87597f-86xrq" Jul 14 09:58:54.402410 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:58:54.402373 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vwl2\" (UniqueName: \"kubernetes.io/projected/a56cb06b-0df1-4821-907d-050d9a78511c-kube-api-access-8vwl2\") pod \"maas-api-e2e-mt-b2b30ce5-54d87597f-86xrq\" (UID: \"a56cb06b-0df1-4821-907d-050d9a78511c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2b30ce5-54d87597f-86xrq" Jul 14 09:58:54.402576 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:58:54.402434 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a56cb06b-0df1-4821-907d-050d9a78511c-maas-api-tls\") pod \"maas-api-e2e-mt-b2b30ce5-54d87597f-86xrq\" (UID: \"a56cb06b-0df1-4821-907d-050d9a78511c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2b30ce5-54d87597f-86xrq" Jul 14 09:58:54.402576 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:58:54.402479 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a56cb06b-0df1-4821-907d-050d9a78511c-openshift-service-ca\") pod \"maas-api-e2e-mt-b2b30ce5-54d87597f-86xrq\" (UID: \"a56cb06b-0df1-4821-907d-050d9a78511c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2b30ce5-54d87597f-86xrq" Jul 14 09:58:54.403134 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:58:54.403115 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a56cb06b-0df1-4821-907d-050d9a78511c-openshift-service-ca\") pod \"maas-api-e2e-mt-b2b30ce5-54d87597f-86xrq\" (UID: \"a56cb06b-0df1-4821-907d-050d9a78511c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2b30ce5-54d87597f-86xrq" Jul 14 09:58:54.404962 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:58:54.404940 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a56cb06b-0df1-4821-907d-050d9a78511c-maas-api-tls\") pod \"maas-api-e2e-mt-b2b30ce5-54d87597f-86xrq\" (UID: \"a56cb06b-0df1-4821-907d-050d9a78511c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2b30ce5-54d87597f-86xrq" Jul 14 09:58:54.411043 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:58:54.411021 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vwl2\" (UniqueName: \"kubernetes.io/projected/a56cb06b-0df1-4821-907d-050d9a78511c-kube-api-access-8vwl2\") pod \"maas-api-e2e-mt-b2b30ce5-54d87597f-86xrq\" (UID: \"a56cb06b-0df1-4821-907d-050d9a78511c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2b30ce5-54d87597f-86xrq" Jul 14 09:58:54.506412 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:58:54.506321 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2b30ce5-54d87597f-86xrq" Jul 14 09:58:54.658233 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:58:54.658204 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b2b30ce5-54d87597f-86xrq"] Jul 14 09:58:54.659649 ip-10-0-132-2 kubenswrapper[2565]: W0714 09:58:54.659614 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda56cb06b_0df1_4821_907d_050d9a78511c.slice/crio-26d0231d4f6a9eea8615ffd33aaed7e9c056b765b8b47f5137dc7c127366c47b WatchSource:0}: Error finding container 26d0231d4f6a9eea8615ffd33aaed7e9c056b765b8b47f5137dc7c127366c47b: Status 404 returned error can't find the container with id 26d0231d4f6a9eea8615ffd33aaed7e9c056b765b8b47f5137dc7c127366c47b Jul 14 09:58:54.851406 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:58:54.851321 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2b30ce5-54d87597f-86xrq" event={"ID":"a56cb06b-0df1-4821-907d-050d9a78511c","Type":"ContainerStarted","Data":"26d0231d4f6a9eea8615ffd33aaed7e9c056b765b8b47f5137dc7c127366c47b"} Jul 14 09:58:55.858364 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:58:55.858326 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2b30ce5-54d87597f-86xrq" event={"ID":"a56cb06b-0df1-4821-907d-050d9a78511c","Type":"ContainerStarted","Data":"d346f031aad9def09b70174fde3b83e30aea09e8cfe33e93bd73d9b26e318fb3"} Jul 14 09:58:55.858853 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:58:55.858435 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2b30ce5-54d87597f-86xrq" Jul 14 09:58:55.878731 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:58:55.878684 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2b30ce5-54d87597f-86xrq" podStartSLOduration=1.466229123 podStartE2EDuration="1.878669104s" totalImagesPullingTime="412.439981ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 09:58:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:58:55.875580552 +0000 UTC m=+1925.384812000" watchObservedRunningTime="2026-07-14 09:58:55.878669104 +0000 UTC m=+1925.387900552" Jul 14 09:59:01.869953 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:01.869926 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2b30ce5-54d87597f-86xrq" Jul 14 09:59:03.930872 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:03.930841 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-b2b30ce5/e2e-lifecycle-model-b2b30ce5-kserve-68c664867f-dlggr"] Jul 14 09:59:03.935633 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:03.935616 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b2b30ce5/e2e-lifecycle-model-b2b30ce5-kserve-68c664867f-dlggr" Jul 14 09:59:03.939292 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:03.939271 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-b2b30ce5\"/\"default-dockercfg-jz84r\"" Jul 14 09:59:03.939417 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:03.939310 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-b2b30ce5\"/\"openshift-service-ca.crt\"" Jul 14 09:59:03.939417 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:03.939310 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-b2b30ce5\"/\"e2e-lifecycle-model-b2b30ce5-kserve-self-signed-certs\"" Jul 14 09:59:03.939417 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:03.939269 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-b2b30ce5\"/\"kube-root-ca.crt\"" Jul 14 09:59:03.946572 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:03.946549 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-b2b30ce5/e2e-lifecycle-model-b2b30ce5-kserve-68c664867f-dlggr"] Jul 14 09:59:03.994264 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:03.994231 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff-tmp-dir\") pod \"e2e-lifecycle-model-b2b30ce5-kserve-68c664867f-dlggr\" (UID: \"63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff\") " pod="ai-tenant-e2e-mt-b2b30ce5/e2e-lifecycle-model-b2b30ce5-kserve-68c664867f-dlggr" Jul 14 09:59:03.994446 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:03.994285 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff-home\") pod \"e2e-lifecycle-model-b2b30ce5-kserve-68c664867f-dlggr\" (UID: \"63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff\") " pod="ai-tenant-e2e-mt-b2b30ce5/e2e-lifecycle-model-b2b30ce5-kserve-68c664867f-dlggr" Jul 14 09:59:03.994446 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:03.994312 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff-model-cache\") pod \"e2e-lifecycle-model-b2b30ce5-kserve-68c664867f-dlggr\" (UID: \"63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff\") " pod="ai-tenant-e2e-mt-b2b30ce5/e2e-lifecycle-model-b2b30ce5-kserve-68c664867f-dlggr" Jul 14 09:59:03.994446 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:03.994343 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff-tls-certs\") pod \"e2e-lifecycle-model-b2b30ce5-kserve-68c664867f-dlggr\" (UID: \"63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff\") " pod="ai-tenant-e2e-mt-b2b30ce5/e2e-lifecycle-model-b2b30ce5-kserve-68c664867f-dlggr" Jul 14 09:59:03.994446 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:03.994368 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff-dshm\") pod \"e2e-lifecycle-model-b2b30ce5-kserve-68c664867f-dlggr\" (UID: \"63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff\") " pod="ai-tenant-e2e-mt-b2b30ce5/e2e-lifecycle-model-b2b30ce5-kserve-68c664867f-dlggr" Jul 14 09:59:03.994446 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:03.994417 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf9xp\" (UniqueName: \"kubernetes.io/projected/63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff-kube-api-access-gf9xp\") pod \"e2e-lifecycle-model-b2b30ce5-kserve-68c664867f-dlggr\" (UID: \"63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff\") " pod="ai-tenant-e2e-mt-b2b30ce5/e2e-lifecycle-model-b2b30ce5-kserve-68c664867f-dlggr" Jul 14 09:59:04.095148 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:04.095108 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf9xp\" (UniqueName: \"kubernetes.io/projected/63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff-kube-api-access-gf9xp\") pod \"e2e-lifecycle-model-b2b30ce5-kserve-68c664867f-dlggr\" (UID: \"63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff\") " pod="ai-tenant-e2e-mt-b2b30ce5/e2e-lifecycle-model-b2b30ce5-kserve-68c664867f-dlggr" Jul 14 09:59:04.095329 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:04.095185 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff-tmp-dir\") pod \"e2e-lifecycle-model-b2b30ce5-kserve-68c664867f-dlggr\" (UID: \"63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff\") " pod="ai-tenant-e2e-mt-b2b30ce5/e2e-lifecycle-model-b2b30ce5-kserve-68c664867f-dlggr" Jul 14 09:59:04.095329 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:04.095227 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff-home\") pod \"e2e-lifecycle-model-b2b30ce5-kserve-68c664867f-dlggr\" (UID: \"63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff\") " pod="ai-tenant-e2e-mt-b2b30ce5/e2e-lifecycle-model-b2b30ce5-kserve-68c664867f-dlggr" Jul 14 09:59:04.095329 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:04.095247 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff-model-cache\") pod \"e2e-lifecycle-model-b2b30ce5-kserve-68c664867f-dlggr\" (UID: \"63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff\") " pod="ai-tenant-e2e-mt-b2b30ce5/e2e-lifecycle-model-b2b30ce5-kserve-68c664867f-dlggr" Jul 14 09:59:04.095329 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:04.095270 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff-tls-certs\") pod \"e2e-lifecycle-model-b2b30ce5-kserve-68c664867f-dlggr\" (UID: \"63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff\") " pod="ai-tenant-e2e-mt-b2b30ce5/e2e-lifecycle-model-b2b30ce5-kserve-68c664867f-dlggr" Jul 14 09:59:04.095329 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:04.095300 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff-dshm\") pod \"e2e-lifecycle-model-b2b30ce5-kserve-68c664867f-dlggr\" (UID: \"63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff\") " pod="ai-tenant-e2e-mt-b2b30ce5/e2e-lifecycle-model-b2b30ce5-kserve-68c664867f-dlggr" Jul 14 09:59:04.095658 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:04.095636 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff-model-cache\") pod \"e2e-lifecycle-model-b2b30ce5-kserve-68c664867f-dlggr\" (UID: \"63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff\") " pod="ai-tenant-e2e-mt-b2b30ce5/e2e-lifecycle-model-b2b30ce5-kserve-68c664867f-dlggr" Jul 14 09:59:04.095730 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:04.095703 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff-home\") pod \"e2e-lifecycle-model-b2b30ce5-kserve-68c664867f-dlggr\" (UID: \"63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff\") " pod="ai-tenant-e2e-mt-b2b30ce5/e2e-lifecycle-model-b2b30ce5-kserve-68c664867f-dlggr" Jul 14 09:59:04.095782 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:04.095726 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff-tmp-dir\") pod \"e2e-lifecycle-model-b2b30ce5-kserve-68c664867f-dlggr\" (UID: \"63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff\") " pod="ai-tenant-e2e-mt-b2b30ce5/e2e-lifecycle-model-b2b30ce5-kserve-68c664867f-dlggr" Jul 14 09:59:04.097684 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:04.097663 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff-dshm\") pod \"e2e-lifecycle-model-b2b30ce5-kserve-68c664867f-dlggr\" (UID: \"63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff\") " pod="ai-tenant-e2e-mt-b2b30ce5/e2e-lifecycle-model-b2b30ce5-kserve-68c664867f-dlggr" Jul 14 09:59:04.097896 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:04.097878 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff-tls-certs\") pod \"e2e-lifecycle-model-b2b30ce5-kserve-68c664867f-dlggr\" (UID: \"63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff\") " pod="ai-tenant-e2e-mt-b2b30ce5/e2e-lifecycle-model-b2b30ce5-kserve-68c664867f-dlggr" Jul 14 09:59:04.104361 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:04.104335 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf9xp\" (UniqueName: \"kubernetes.io/projected/63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff-kube-api-access-gf9xp\") pod \"e2e-lifecycle-model-b2b30ce5-kserve-68c664867f-dlggr\" (UID: \"63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff\") " pod="ai-tenant-e2e-mt-b2b30ce5/e2e-lifecycle-model-b2b30ce5-kserve-68c664867f-dlggr" Jul 14 09:59:04.246277 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:04.246176 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b2b30ce5/e2e-lifecycle-model-b2b30ce5-kserve-68c664867f-dlggr" Jul 14 09:59:04.385858 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:04.385833 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-b2b30ce5/e2e-lifecycle-model-b2b30ce5-kserve-68c664867f-dlggr"] Jul 14 09:59:04.912056 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:04.912018 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b2b30ce5/e2e-lifecycle-model-b2b30ce5-kserve-68c664867f-dlggr" event={"ID":"63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff","Type":"ContainerStarted","Data":"2333cbd8767b3d01ae5dad92180c82bbce24643718b17f5cd59e685bcd2f77d4"} Jul 14 09:59:04.912254 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:04.912065 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b2b30ce5/e2e-lifecycle-model-b2b30ce5-kserve-68c664867f-dlggr" event={"ID":"63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff","Type":"ContainerStarted","Data":"2d2db8fe3103c599307ca22b403e197e1d4d9db3ac845088480d19edc71152f3"} Jul 14 09:59:04.933976 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:04.933919 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-b2b30ce5/e2e-lifecycle-model-b2b30ce5-kserve-68c664867f-dlggr" podStartSLOduration=1.9339009539999998 podStartE2EDuration="1.933900954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 09:59:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 09:59:04.931073215 +0000 UTC m=+1934.440304680" watchObservedRunningTime="2026-07-14 09:59:04.933900954 +0000 UTC m=+1934.443132420" Jul 14 09:59:14.247181 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:14.247144 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-b2b30ce5/e2e-lifecycle-model-b2b30ce5-kserve-68c664867f-dlggr" Jul 14 09:59:14.247570 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:14.247198 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-b2b30ce5/e2e-lifecycle-model-b2b30ce5-kserve-68c664867f-dlggr" Jul 14 09:59:14.260028 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:14.260001 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-b2b30ce5/e2e-lifecycle-model-b2b30ce5-kserve-68c664867f-dlggr" Jul 14 09:59:14.981338 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:14.981308 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-b2b30ce5/e2e-lifecycle-model-b2b30ce5-kserve-68c664867f-dlggr" Jul 14 09:59:42.088426 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:42.088388 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b2b30ce5-54d87597f-86xrq"] Jul 14 09:59:42.088963 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:42.088644 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2b30ce5-54d87597f-86xrq" podUID="a56cb06b-0df1-4821-907d-050d9a78511c" containerName="maas-api" containerID="cri-o://d346f031aad9def09b70174fde3b83e30aea09e8cfe33e93bd73d9b26e318fb3" gracePeriod=30 Jul 14 09:59:42.336730 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:42.336707 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2b30ce5-54d87597f-86xrq" Jul 14 09:59:42.479934 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:42.479897 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vwl2\" (UniqueName: \"kubernetes.io/projected/a56cb06b-0df1-4821-907d-050d9a78511c-kube-api-access-8vwl2\") pod \"a56cb06b-0df1-4821-907d-050d9a78511c\" (UID: \"a56cb06b-0df1-4821-907d-050d9a78511c\") " Jul 14 09:59:42.480124 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:42.479958 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a56cb06b-0df1-4821-907d-050d9a78511c-openshift-service-ca\") pod \"a56cb06b-0df1-4821-907d-050d9a78511c\" (UID: \"a56cb06b-0df1-4821-907d-050d9a78511c\") " Jul 14 09:59:42.480124 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:42.480119 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a56cb06b-0df1-4821-907d-050d9a78511c-maas-api-tls\") pod \"a56cb06b-0df1-4821-907d-050d9a78511c\" (UID: \"a56cb06b-0df1-4821-907d-050d9a78511c\") " Jul 14 09:59:42.480366 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:42.480341 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a56cb06b-0df1-4821-907d-050d9a78511c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a56cb06b-0df1-4821-907d-050d9a78511c" (UID: "a56cb06b-0df1-4821-907d-050d9a78511c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 09:59:42.482157 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:42.482128 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a56cb06b-0df1-4821-907d-050d9a78511c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a56cb06b-0df1-4821-907d-050d9a78511c" (UID: "a56cb06b-0df1-4821-907d-050d9a78511c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 09:59:42.482246 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:42.482179 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a56cb06b-0df1-4821-907d-050d9a78511c-kube-api-access-8vwl2" (OuterVolumeSpecName: "kube-api-access-8vwl2") pod "a56cb06b-0df1-4821-907d-050d9a78511c" (UID: "a56cb06b-0df1-4821-907d-050d9a78511c"). InnerVolumeSpecName "kube-api-access-8vwl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 09:59:42.581277 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:42.581234 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a56cb06b-0df1-4821-907d-050d9a78511c-maas-api-tls\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:59:42.581277 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:42.581272 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vwl2\" (UniqueName: \"kubernetes.io/projected/a56cb06b-0df1-4821-907d-050d9a78511c-kube-api-access-8vwl2\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:59:42.581277 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:42.581286 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a56cb06b-0df1-4821-907d-050d9a78511c-openshift-service-ca\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 09:59:43.145957 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:43.145926 2565 generic.go:356] "Generic (PLEG): container finished" podID="a56cb06b-0df1-4821-907d-050d9a78511c" containerID="d346f031aad9def09b70174fde3b83e30aea09e8cfe33e93bd73d9b26e318fb3" exitCode=0 Jul 14 09:59:43.146385 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:43.145967 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2b30ce5-54d87597f-86xrq" Jul 14 09:59:43.146385 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:43.145972 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2b30ce5-54d87597f-86xrq" event={"ID":"a56cb06b-0df1-4821-907d-050d9a78511c","Type":"ContainerDied","Data":"d346f031aad9def09b70174fde3b83e30aea09e8cfe33e93bd73d9b26e318fb3"} Jul 14 09:59:43.146385 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:43.146073 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b2b30ce5-54d87597f-86xrq" event={"ID":"a56cb06b-0df1-4821-907d-050d9a78511c","Type":"ContainerDied","Data":"26d0231d4f6a9eea8615ffd33aaed7e9c056b765b8b47f5137dc7c127366c47b"} Jul 14 09:59:43.146385 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:43.146094 2565 scope.go:117] "RemoveContainer" containerID="d346f031aad9def09b70174fde3b83e30aea09e8cfe33e93bd73d9b26e318fb3" Jul 14 09:59:43.155426 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:43.155408 2565 scope.go:117] "RemoveContainer" containerID="d346f031aad9def09b70174fde3b83e30aea09e8cfe33e93bd73d9b26e318fb3" Jul 14 09:59:43.155661 ip-10-0-132-2 kubenswrapper[2565]: E0714 09:59:43.155643 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d346f031aad9def09b70174fde3b83e30aea09e8cfe33e93bd73d9b26e318fb3\": container with ID starting with d346f031aad9def09b70174fde3b83e30aea09e8cfe33e93bd73d9b26e318fb3 not found: ID does not exist" containerID="d346f031aad9def09b70174fde3b83e30aea09e8cfe33e93bd73d9b26e318fb3" Jul 14 09:59:43.155710 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:43.155671 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d346f031aad9def09b70174fde3b83e30aea09e8cfe33e93bd73d9b26e318fb3"} err="failed to get container status \"d346f031aad9def09b70174fde3b83e30aea09e8cfe33e93bd73d9b26e318fb3\": rpc error: code = NotFound desc = could not find container \"d346f031aad9def09b70174fde3b83e30aea09e8cfe33e93bd73d9b26e318fb3\": container with ID starting with d346f031aad9def09b70174fde3b83e30aea09e8cfe33e93bd73d9b26e318fb3 not found: ID does not exist" Jul 14 09:59:43.164156 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:43.164130 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b2b30ce5-54d87597f-86xrq"] Jul 14 09:59:43.169133 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:43.169112 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b2b30ce5-54d87597f-86xrq"] Jul 14 09:59:44.969042 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:44.969012 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a56cb06b-0df1-4821-907d-050d9a78511c" path="/var/lib/kubelet/pods/a56cb06b-0df1-4821-907d-050d9a78511c/volumes" Jul 14 09:59:51.717454 ip-10-0-132-2 kubenswrapper[2565]: I0714 09:59:51.717415 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-b2b30ce5/e2e-lifecycle-model-b2b30ce5-kserve-68c664867f-dlggr"] Jul 14 10:00:00.152749 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:00:00.152712 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29733720-zc2zt"] Jul 14 10:00:00.153190 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:00:00.153176 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a56cb06b-0df1-4821-907d-050d9a78511c" containerName="maas-api" Jul 14 10:00:00.153190 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:00:00.153191 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a56cb06b-0df1-4821-907d-050d9a78511c" containerName="maas-api" Jul 14 10:00:00.153272 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:00:00.153263 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a56cb06b-0df1-4821-907d-050d9a78511c" containerName="maas-api" Jul 14 10:00:00.156305 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:00:00.156283 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733720-zc2zt" Jul 14 10:00:00.159431 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:00:00.159411 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8m76v\"" Jul 14 10:00:00.166455 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:00:00.166432 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733720-zc2zt" Jul 14 10:00:00.169062 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:00:00.169037 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29733720-zc2zt"] Jul 14 10:00:00.314834 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:00:00.314782 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29733720-zc2zt"] Jul 14 10:00:00.318248 ip-10-0-132-2 kubenswrapper[2565]: W0714 10:00:00.318213 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e3bf8ce_3ebb_44b4_a5a7_bc780305f8b1.slice/crio-7ccd8b5d600137ed87dd0e5a9288d22a0ee05cac99f74b6877fd026fc11d0255 WatchSource:0}: Error finding container 7ccd8b5d600137ed87dd0e5a9288d22a0ee05cac99f74b6877fd026fc11d0255: Status 404 returned error can't find the container with id 7ccd8b5d600137ed87dd0e5a9288d22a0ee05cac99f74b6877fd026fc11d0255 Jul 14 10:00:01.256440 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:00:01.256338 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733720-zc2zt" event={"ID":"0e3bf8ce-3ebb-44b4-a5a7-bc780305f8b1","Type":"ContainerStarted","Data":"1e3a7b5db5a8f88dda86f9a69fc1c0f065a26ed6e725d0a8d2807f944c3784fc"} Jul 14 10:00:01.256440 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:00:01.256397 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733720-zc2zt" event={"ID":"0e3bf8ce-3ebb-44b4-a5a7-bc780305f8b1","Type":"ContainerStarted","Data":"7ccd8b5d600137ed87dd0e5a9288d22a0ee05cac99f74b6877fd026fc11d0255"} Jul 14 10:00:01.273320 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:00:01.273258 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733720-zc2zt" podStartSLOduration=1.273235823 podStartE2EDuration="1.273235823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 10:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 10:00:01.272748929 +0000 UTC m=+1990.781980396" watchObservedRunningTime="2026-07-14 10:00:01.273235823 +0000 UTC m=+1990.782467272" Jul 14 10:00:02.263377 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:00:02.263344 2565 generic.go:356] "Generic (PLEG): container finished" podID="0e3bf8ce-3ebb-44b4-a5a7-bc780305f8b1" containerID="1e3a7b5db5a8f88dda86f9a69fc1c0f065a26ed6e725d0a8d2807f944c3784fc" exitCode=0 Jul 14 10:00:02.263732 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:00:02.263438 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733720-zc2zt" event={"ID":"0e3bf8ce-3ebb-44b4-a5a7-bc780305f8b1","Type":"ContainerDied","Data":"1e3a7b5db5a8f88dda86f9a69fc1c0f065a26ed6e725d0a8d2807f944c3784fc"} Jul 14 10:00:03.409441 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:00:03.409411 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733720-zc2zt" Jul 14 10:00:04.276360 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:00:04.276333 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733720-zc2zt" Jul 14 10:00:04.276536 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:00:04.276359 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733720-zc2zt" event={"ID":"0e3bf8ce-3ebb-44b4-a5a7-bc780305f8b1","Type":"ContainerDied","Data":"7ccd8b5d600137ed87dd0e5a9288d22a0ee05cac99f74b6877fd026fc11d0255"} Jul 14 10:00:04.276536 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:00:04.276401 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ccd8b5d600137ed87dd0e5a9288d22a0ee05cac99f74b6877fd026fc11d0255" Jul 14 10:00:06.738470 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:00:06.738426 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-b2b30ce5/e2e-lifecycle-model-b2b30ce5-kserve-68c664867f-dlggr" podUID="63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff" containerName="main" containerID="cri-o://2333cbd8767b3d01ae5dad92180c82bbce24643718b17f5cd59e685bcd2f77d4" gracePeriod=45 Jul 14 10:00:07.008882 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:00:07.007722 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b2b30ce5/e2e-lifecycle-model-b2b30ce5-kserve-68c664867f-dlggr" Jul 14 10:00:07.116310 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:00:07.116276 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff-tmp-dir\") pod \"63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff\" (UID: \"63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff\") " Jul 14 10:00:07.116506 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:00:07.116329 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gf9xp\" (UniqueName: \"kubernetes.io/projected/63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff-kube-api-access-gf9xp\") pod \"63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff\" (UID: \"63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff\") " Jul 14 10:00:07.116506 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:00:07.116357 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff-dshm\") pod \"63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff\" (UID: \"63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff\") " Jul 14 10:00:07.116506 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:00:07.116388 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff-home\") pod \"63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff\" (UID: \"63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff\") " Jul 14 10:00:07.116506 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:00:07.116448 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff-model-cache\") pod \"63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff\" (UID: \"63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff\") " Jul 14 10:00:07.116506 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:00:07.116464 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff-tls-certs\") pod \"63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff\" (UID: \"63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff\") " Jul 14 10:00:07.116884 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:00:07.116638 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff" (UID: "63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 10:00:07.116884 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:00:07.116727 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff-model-cache" (OuterVolumeSpecName: "model-cache") pod "63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff" (UID: "63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 10:00:07.116884 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:00:07.116738 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff-home" (OuterVolumeSpecName: "home") pod "63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff" (UID: "63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 10:00:07.116884 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:00:07.116754 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff-tmp-dir\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 10:00:07.118639 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:00:07.118615 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff" (UID: "63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 10:00:07.118639 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:00:07.118621 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff-dshm" (OuterVolumeSpecName: "dshm") pod "63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff" (UID: "63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 10:00:07.118778 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:00:07.118639 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff-kube-api-access-gf9xp" (OuterVolumeSpecName: "kube-api-access-gf9xp") pod "63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff" (UID: "63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff"). InnerVolumeSpecName "kube-api-access-gf9xp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 10:00:07.218120 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:00:07.218083 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf9xp\" (UniqueName: \"kubernetes.io/projected/63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff-kube-api-access-gf9xp\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 10:00:07.218120 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:00:07.218114 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff-dshm\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 10:00:07.218120 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:00:07.218124 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff-home\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 10:00:07.218120 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:00:07.218131 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff-model-cache\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 10:00:07.218396 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:00:07.218141 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff-tls-certs\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 10:00:07.295152 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:00:07.295077 2565 generic.go:356] "Generic (PLEG): container finished" podID="63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff" containerID="2333cbd8767b3d01ae5dad92180c82bbce24643718b17f5cd59e685bcd2f77d4" exitCode=0 Jul 14 10:00:07.295152 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:00:07.295120 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b2b30ce5/e2e-lifecycle-model-b2b30ce5-kserve-68c664867f-dlggr" Jul 14 10:00:07.295320 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:00:07.295122 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b2b30ce5/e2e-lifecycle-model-b2b30ce5-kserve-68c664867f-dlggr" event={"ID":"63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff","Type":"ContainerDied","Data":"2333cbd8767b3d01ae5dad92180c82bbce24643718b17f5cd59e685bcd2f77d4"} Jul 14 10:00:07.295320 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:00:07.295246 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b2b30ce5/e2e-lifecycle-model-b2b30ce5-kserve-68c664867f-dlggr" event={"ID":"63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff","Type":"ContainerDied","Data":"2d2db8fe3103c599307ca22b403e197e1d4d9db3ac845088480d19edc71152f3"} Jul 14 10:00:07.295320 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:00:07.295273 2565 scope.go:117] "RemoveContainer" containerID="2333cbd8767b3d01ae5dad92180c82bbce24643718b17f5cd59e685bcd2f77d4" Jul 14 10:00:07.304583 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:00:07.304565 2565 scope.go:117] "RemoveContainer" containerID="2333cbd8767b3d01ae5dad92180c82bbce24643718b17f5cd59e685bcd2f77d4" Jul 14 10:00:07.304858 ip-10-0-132-2 kubenswrapper[2565]: E0714 10:00:07.304836 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2333cbd8767b3d01ae5dad92180c82bbce24643718b17f5cd59e685bcd2f77d4\": container with ID starting with 2333cbd8767b3d01ae5dad92180c82bbce24643718b17f5cd59e685bcd2f77d4 not found: ID does not exist" containerID="2333cbd8767b3d01ae5dad92180c82bbce24643718b17f5cd59e685bcd2f77d4" Jul 14 10:00:07.304910 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:00:07.304888 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2333cbd8767b3d01ae5dad92180c82bbce24643718b17f5cd59e685bcd2f77d4"} err="failed to get container status \"2333cbd8767b3d01ae5dad92180c82bbce24643718b17f5cd59e685bcd2f77d4\": rpc error: code = NotFound desc = could not find container \"2333cbd8767b3d01ae5dad92180c82bbce24643718b17f5cd59e685bcd2f77d4\": container with ID starting with 2333cbd8767b3d01ae5dad92180c82bbce24643718b17f5cd59e685bcd2f77d4 not found: ID does not exist" Jul 14 10:00:07.317458 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:00:07.317437 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-b2b30ce5/e2e-lifecycle-model-b2b30ce5-kserve-68c664867f-dlggr"] Jul 14 10:00:07.321895 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:00:07.321875 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-b2b30ce5/e2e-lifecycle-model-b2b30ce5-kserve-68c664867f-dlggr"] Jul 14 10:00:08.969011 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:00:08.968977 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff" path="/var/lib/kubelet/pods/63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff/volumes" Jul 14 10:01:37.197227 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:01:37.197191 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-06e8a8-9b988cd47-7xqsp"] Jul 14 10:01:37.197726 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:01:37.197602 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff" containerName="main" Jul 14 10:01:37.197726 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:01:37.197612 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff" containerName="main" Jul 14 10:01:37.197726 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:01:37.197626 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e3bf8ce-3ebb-44b4-a5a7-bc780305f8b1" containerName="cleanup" Jul 14 10:01:37.197726 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:01:37.197631 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e3bf8ce-3ebb-44b4-a5a7-bc780305f8b1" containerName="cleanup" Jul 14 10:01:37.197726 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:01:37.197709 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="63e5a8e5-7e3d-4bc8-add0-d5f01d08ebff" containerName="main" Jul 14 10:01:37.197726 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:01:37.197716 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e3bf8ce-3ebb-44b4-a5a7-bc780305f8b1" containerName="cleanup" Jul 14 10:01:37.200992 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:01:37.200974 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06e8a8-9b988cd47-7xqsp" Jul 14 10:01:37.203611 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:01:37.203592 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-06e8a8-serving-cert\"" Jul 14 10:01:37.212893 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:01:37.212868 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-06e8a8-9b988cd47-7xqsp"] Jul 14 10:01:37.307336 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:01:37.307298 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn86v\" (UniqueName: \"kubernetes.io/projected/77e597d9-6f5e-4cf3-bb63-72fddb89bfd2-kube-api-access-cn86v\") pod \"maas-api-e2e-inf-a-06e8a8-9b988cd47-7xqsp\" (UID: \"77e597d9-6f5e-4cf3-bb63-72fddb89bfd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06e8a8-9b988cd47-7xqsp" Jul 14 10:01:37.307517 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:01:37.307384 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77e597d9-6f5e-4cf3-bb63-72fddb89bfd2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-06e8a8-9b988cd47-7xqsp\" (UID: \"77e597d9-6f5e-4cf3-bb63-72fddb89bfd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06e8a8-9b988cd47-7xqsp" Jul 14 10:01:37.307517 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:01:37.307453 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77e597d9-6f5e-4cf3-bb63-72fddb89bfd2-maas-api-tls\") pod \"maas-api-e2e-inf-a-06e8a8-9b988cd47-7xqsp\" (UID: \"77e597d9-6f5e-4cf3-bb63-72fddb89bfd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06e8a8-9b988cd47-7xqsp" Jul 14 10:01:37.409001 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:01:37.408963 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77e597d9-6f5e-4cf3-bb63-72fddb89bfd2-maas-api-tls\") pod \"maas-api-e2e-inf-a-06e8a8-9b988cd47-7xqsp\" (UID: \"77e597d9-6f5e-4cf3-bb63-72fddb89bfd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06e8a8-9b988cd47-7xqsp" Jul 14 10:01:37.409169 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:01:37.409022 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn86v\" (UniqueName: \"kubernetes.io/projected/77e597d9-6f5e-4cf3-bb63-72fddb89bfd2-kube-api-access-cn86v\") pod \"maas-api-e2e-inf-a-06e8a8-9b988cd47-7xqsp\" (UID: \"77e597d9-6f5e-4cf3-bb63-72fddb89bfd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06e8a8-9b988cd47-7xqsp" Jul 14 10:01:37.409169 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:01:37.409068 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77e597d9-6f5e-4cf3-bb63-72fddb89bfd2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-06e8a8-9b988cd47-7xqsp\" (UID: \"77e597d9-6f5e-4cf3-bb63-72fddb89bfd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06e8a8-9b988cd47-7xqsp" Jul 14 10:01:37.409667 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:01:37.409643 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77e597d9-6f5e-4cf3-bb63-72fddb89bfd2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-06e8a8-9b988cd47-7xqsp\" (UID: \"77e597d9-6f5e-4cf3-bb63-72fddb89bfd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06e8a8-9b988cd47-7xqsp" Jul 14 10:01:37.411450 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:01:37.411432 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77e597d9-6f5e-4cf3-bb63-72fddb89bfd2-maas-api-tls\") pod \"maas-api-e2e-inf-a-06e8a8-9b988cd47-7xqsp\" (UID: \"77e597d9-6f5e-4cf3-bb63-72fddb89bfd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06e8a8-9b988cd47-7xqsp" Jul 14 10:01:37.417597 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:01:37.417572 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn86v\" (UniqueName: \"kubernetes.io/projected/77e597d9-6f5e-4cf3-bb63-72fddb89bfd2-kube-api-access-cn86v\") pod \"maas-api-e2e-inf-a-06e8a8-9b988cd47-7xqsp\" (UID: \"77e597d9-6f5e-4cf3-bb63-72fddb89bfd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06e8a8-9b988cd47-7xqsp" Jul 14 10:01:37.512723 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:01:37.512634 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06e8a8-9b988cd47-7xqsp" Jul 14 10:01:37.649861 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:01:37.649832 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-06e8a8-9b988cd47-7xqsp"] Jul 14 10:01:37.651556 ip-10-0-132-2 kubenswrapper[2565]: W0714 10:01:37.651533 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77e597d9_6f5e_4cf3_bb63_72fddb89bfd2.slice/crio-804bccacd168d64bf13abd81180092624e32cbbb3ea0d09188118217516ed288 WatchSource:0}: Error finding container 804bccacd168d64bf13abd81180092624e32cbbb3ea0d09188118217516ed288: Status 404 returned error can't find the container with id 804bccacd168d64bf13abd81180092624e32cbbb3ea0d09188118217516ed288 Jul 14 10:01:37.821373 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:01:37.821283 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06e8a8-9b988cd47-7xqsp" event={"ID":"77e597d9-6f5e-4cf3-bb63-72fddb89bfd2","Type":"ContainerStarted","Data":"804bccacd168d64bf13abd81180092624e32cbbb3ea0d09188118217516ed288"} Jul 14 10:01:38.828888 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:01:38.828846 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06e8a8-9b988cd47-7xqsp" event={"ID":"77e597d9-6f5e-4cf3-bb63-72fddb89bfd2","Type":"ContainerStarted","Data":"eeef9f6feafcfc16de601f7b5020f1b91d64f1708d4ac5b1c03006cb4ac7de10"} Jul 14 10:01:38.829297 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:01:38.829013 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06e8a8-9b988cd47-7xqsp" Jul 14 10:01:38.850280 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:01:38.850222 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06e8a8-9b988cd47-7xqsp" podStartSLOduration=1.361335075 podStartE2EDuration="1.850203094s" totalImagesPullingTime="488.868019ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 10:01:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 10:01:38.849998895 +0000 UTC m=+2088.359230375" watchObservedRunningTime="2026-07-14 10:01:38.850203094 +0000 UTC m=+2088.359434543" Jul 14 10:01:44.841751 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:01:44.841723 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06e8a8-9b988cd47-7xqsp" Jul 14 10:01:45.045891 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:01:45.045857 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d7ae40-dc7bddb6f-n6vwf"] Jul 14 10:01:45.050855 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:01:45.050829 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7ae40-dc7bddb6f-n6vwf" Jul 14 10:01:45.053901 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:01:45.053880 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-d7ae40-serving-cert\"" Jul 14 10:01:45.060335 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:01:45.060307 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d7ae40-dc7bddb6f-n6vwf"] Jul 14 10:01:45.080211 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:01:45.080175 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7frc\" (UniqueName: \"kubernetes.io/projected/9288af01-8c6a-4ae1-9908-0f5d05da0d7d-kube-api-access-r7frc\") pod \"maas-api-e2e-inf-b-d7ae40-dc7bddb6f-n6vwf\" (UID: \"9288af01-8c6a-4ae1-9908-0f5d05da0d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7ae40-dc7bddb6f-n6vwf" Jul 14 10:01:45.080211 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:01:45.080216 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9288af01-8c6a-4ae1-9908-0f5d05da0d7d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d7ae40-dc7bddb6f-n6vwf\" (UID: \"9288af01-8c6a-4ae1-9908-0f5d05da0d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7ae40-dc7bddb6f-n6vwf" Jul 14 10:01:45.080432 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:01:45.080249 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9288af01-8c6a-4ae1-9908-0f5d05da0d7d-maas-api-tls\") pod \"maas-api-e2e-inf-b-d7ae40-dc7bddb6f-n6vwf\" (UID: \"9288af01-8c6a-4ae1-9908-0f5d05da0d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7ae40-dc7bddb6f-n6vwf" Jul 14 10:01:45.181460 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:01:45.181419 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7frc\" (UniqueName: \"kubernetes.io/projected/9288af01-8c6a-4ae1-9908-0f5d05da0d7d-kube-api-access-r7frc\") pod \"maas-api-e2e-inf-b-d7ae40-dc7bddb6f-n6vwf\" (UID: \"9288af01-8c6a-4ae1-9908-0f5d05da0d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7ae40-dc7bddb6f-n6vwf" Jul 14 10:01:45.181460 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:01:45.181461 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9288af01-8c6a-4ae1-9908-0f5d05da0d7d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d7ae40-dc7bddb6f-n6vwf\" (UID: \"9288af01-8c6a-4ae1-9908-0f5d05da0d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7ae40-dc7bddb6f-n6vwf" Jul 14 10:01:45.181675 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:01:45.181485 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9288af01-8c6a-4ae1-9908-0f5d05da0d7d-maas-api-tls\") pod \"maas-api-e2e-inf-b-d7ae40-dc7bddb6f-n6vwf\" (UID: \"9288af01-8c6a-4ae1-9908-0f5d05da0d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7ae40-dc7bddb6f-n6vwf" Jul 14 10:01:45.181675 ip-10-0-132-2 kubenswrapper[2565]: E0714 10:01:45.181575 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-d7ae40-serving-cert: secret "maas-api-e2e-inf-b-d7ae40-serving-cert" not found Jul 14 10:01:45.181675 ip-10-0-132-2 kubenswrapper[2565]: E0714 10:01:45.181639 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9288af01-8c6a-4ae1-9908-0f5d05da0d7d-maas-api-tls podName:9288af01-8c6a-4ae1-9908-0f5d05da0d7d nodeName:}" failed. No retries permitted until 2026-07-14 10:01:45.681619979 +0000 UTC m=+2095.190851406 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9288af01-8c6a-4ae1-9908-0f5d05da0d7d-maas-api-tls") pod "maas-api-e2e-inf-b-d7ae40-dc7bddb6f-n6vwf" (UID: "9288af01-8c6a-4ae1-9908-0f5d05da0d7d") : secret "maas-api-e2e-inf-b-d7ae40-serving-cert" not found Jul 14 10:01:45.182190 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:01:45.182166 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9288af01-8c6a-4ae1-9908-0f5d05da0d7d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d7ae40-dc7bddb6f-n6vwf\" (UID: \"9288af01-8c6a-4ae1-9908-0f5d05da0d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7ae40-dc7bddb6f-n6vwf" Jul 14 10:01:45.192303 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:01:45.192270 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7frc\" (UniqueName: \"kubernetes.io/projected/9288af01-8c6a-4ae1-9908-0f5d05da0d7d-kube-api-access-r7frc\") pod \"maas-api-e2e-inf-b-d7ae40-dc7bddb6f-n6vwf\" (UID: \"9288af01-8c6a-4ae1-9908-0f5d05da0d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7ae40-dc7bddb6f-n6vwf" Jul 14 10:01:45.687921 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:01:45.687887 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9288af01-8c6a-4ae1-9908-0f5d05da0d7d-maas-api-tls\") pod \"maas-api-e2e-inf-b-d7ae40-dc7bddb6f-n6vwf\" (UID: \"9288af01-8c6a-4ae1-9908-0f5d05da0d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7ae40-dc7bddb6f-n6vwf" Jul 14 10:01:45.690417 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:01:45.690391 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9288af01-8c6a-4ae1-9908-0f5d05da0d7d-maas-api-tls\") pod \"maas-api-e2e-inf-b-d7ae40-dc7bddb6f-n6vwf\" (UID: \"9288af01-8c6a-4ae1-9908-0f5d05da0d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7ae40-dc7bddb6f-n6vwf" Jul 14 10:01:45.964755 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:01:45.964663 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7ae40-dc7bddb6f-n6vwf" Jul 14 10:01:46.110883 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:01:46.110854 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d7ae40-dc7bddb6f-n6vwf"] Jul 14 10:01:46.111705 ip-10-0-132-2 kubenswrapper[2565]: W0714 10:01:46.111681 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9288af01_8c6a_4ae1_9908_0f5d05da0d7d.slice/crio-e4bdfb08f5926e9b8dada20b39726b3b29e6282075b517c5a0018c1e708ad917 WatchSource:0}: Error finding container e4bdfb08f5926e9b8dada20b39726b3b29e6282075b517c5a0018c1e708ad917: Status 404 returned error can't find the container with id e4bdfb08f5926e9b8dada20b39726b3b29e6282075b517c5a0018c1e708ad917 Jul 14 10:01:46.878417 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:01:46.878382 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7ae40-dc7bddb6f-n6vwf" event={"ID":"9288af01-8c6a-4ae1-9908-0f5d05da0d7d","Type":"ContainerStarted","Data":"c398318d3613cc2826ecfb9bdd5c03289e3d845ed79e114d472d155cdb2f1a08"} Jul 14 10:01:46.878592 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:01:46.878446 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7ae40-dc7bddb6f-n6vwf" Jul 14 10:01:46.878592 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:01:46.878468 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7ae40-dc7bddb6f-n6vwf" event={"ID":"9288af01-8c6a-4ae1-9908-0f5d05da0d7d","Type":"ContainerStarted","Data":"e4bdfb08f5926e9b8dada20b39726b3b29e6282075b517c5a0018c1e708ad917"} Jul 14 10:01:46.898104 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:01:46.898055 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7ae40-dc7bddb6f-n6vwf" podStartSLOduration=1.275837723 podStartE2EDuration="1.898038105s" totalImagesPullingTime="622.200382ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 10:01:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 10:01:46.897060269 +0000 UTC m=+2096.406291717" watchObservedRunningTime="2026-07-14 10:01:46.898038105 +0000 UTC m=+2096.407269577" Jul 14 10:01:52.890171 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:01:52.890140 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7ae40-dc7bddb6f-n6vwf" Jul 14 10:02:01.756902 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:02:01.756798 2565 scope.go:117] "RemoveContainer" containerID="696cbb247c0c39d0de6f28b90c7e04ee64dbd7f4292ed857e855142b698948e6" Jul 14 10:02:41.726514 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:02:41.726483 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-d7ae40/test-model-d7ae40-kserve-9f4fdd555-k4h6t"] Jul 14 10:02:41.729267 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:02:41.729248 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d7ae40/test-model-d7ae40-kserve-9f4fdd555-k4h6t" Jul 14 10:02:41.734077 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:02:41.734056 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d7ae40\"/\"test-model-d7ae40-kserve-self-signed-certs\"" Jul 14 10:02:41.734213 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:02:41.734101 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d7ae40\"/\"default-dockercfg-q86hd\"" Jul 14 10:02:41.734213 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:02:41.734063 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d7ae40\"/\"openshift-service-ca.crt\"" Jul 14 10:02:41.734213 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:02:41.734150 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d7ae40\"/\"kube-root-ca.crt\"" Jul 14 10:02:41.742433 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:02:41.742414 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d7ae40/test-model-d7ae40-kserve-9f4fdd555-k4h6t"] Jul 14 10:02:41.829480 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:02:41.829444 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fad05fa2-730b-4467-a90b-77283f4657d4-tmp-dir\") pod \"test-model-d7ae40-kserve-9f4fdd555-k4h6t\" (UID: \"fad05fa2-730b-4467-a90b-77283f4657d4\") " pod="ai-tenant-e2e-inf-b-d7ae40/test-model-d7ae40-kserve-9f4fdd555-k4h6t" Jul 14 10:02:41.829480 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:02:41.829483 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fad05fa2-730b-4467-a90b-77283f4657d4-model-cache\") pod \"test-model-d7ae40-kserve-9f4fdd555-k4h6t\" (UID: \"fad05fa2-730b-4467-a90b-77283f4657d4\") " pod="ai-tenant-e2e-inf-b-d7ae40/test-model-d7ae40-kserve-9f4fdd555-k4h6t" Jul 14 10:02:41.829697 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:02:41.829600 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fad05fa2-730b-4467-a90b-77283f4657d4-tls-certs\") pod \"test-model-d7ae40-kserve-9f4fdd555-k4h6t\" (UID: \"fad05fa2-730b-4467-a90b-77283f4657d4\") " pod="ai-tenant-e2e-inf-b-d7ae40/test-model-d7ae40-kserve-9f4fdd555-k4h6t" Jul 14 10:02:41.829697 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:02:41.829635 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjk4f\" (UniqueName: \"kubernetes.io/projected/fad05fa2-730b-4467-a90b-77283f4657d4-kube-api-access-tjk4f\") pod \"test-model-d7ae40-kserve-9f4fdd555-k4h6t\" (UID: \"fad05fa2-730b-4467-a90b-77283f4657d4\") " pod="ai-tenant-e2e-inf-b-d7ae40/test-model-d7ae40-kserve-9f4fdd555-k4h6t" Jul 14 10:02:41.829697 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:02:41.829669 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fad05fa2-730b-4467-a90b-77283f4657d4-home\") pod \"test-model-d7ae40-kserve-9f4fdd555-k4h6t\" (UID: \"fad05fa2-730b-4467-a90b-77283f4657d4\") " pod="ai-tenant-e2e-inf-b-d7ae40/test-model-d7ae40-kserve-9f4fdd555-k4h6t" Jul 14 10:02:41.829796 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:02:41.829732 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fad05fa2-730b-4467-a90b-77283f4657d4-dshm\") pod \"test-model-d7ae40-kserve-9f4fdd555-k4h6t\" (UID: \"fad05fa2-730b-4467-a90b-77283f4657d4\") " pod="ai-tenant-e2e-inf-b-d7ae40/test-model-d7ae40-kserve-9f4fdd555-k4h6t" Jul 14 10:02:41.930430 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:02:41.930390 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fad05fa2-730b-4467-a90b-77283f4657d4-tmp-dir\") pod \"test-model-d7ae40-kserve-9f4fdd555-k4h6t\" (UID: \"fad05fa2-730b-4467-a90b-77283f4657d4\") " pod="ai-tenant-e2e-inf-b-d7ae40/test-model-d7ae40-kserve-9f4fdd555-k4h6t" Jul 14 10:02:41.930430 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:02:41.930432 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fad05fa2-730b-4467-a90b-77283f4657d4-model-cache\") pod \"test-model-d7ae40-kserve-9f4fdd555-k4h6t\" (UID: \"fad05fa2-730b-4467-a90b-77283f4657d4\") " pod="ai-tenant-e2e-inf-b-d7ae40/test-model-d7ae40-kserve-9f4fdd555-k4h6t" Jul 14 10:02:41.930648 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:02:41.930502 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fad05fa2-730b-4467-a90b-77283f4657d4-tls-certs\") pod \"test-model-d7ae40-kserve-9f4fdd555-k4h6t\" (UID: \"fad05fa2-730b-4467-a90b-77283f4657d4\") " pod="ai-tenant-e2e-inf-b-d7ae40/test-model-d7ae40-kserve-9f4fdd555-k4h6t" Jul 14 10:02:41.930648 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:02:41.930536 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjk4f\" (UniqueName: \"kubernetes.io/projected/fad05fa2-730b-4467-a90b-77283f4657d4-kube-api-access-tjk4f\") pod \"test-model-d7ae40-kserve-9f4fdd555-k4h6t\" (UID: \"fad05fa2-730b-4467-a90b-77283f4657d4\") " pod="ai-tenant-e2e-inf-b-d7ae40/test-model-d7ae40-kserve-9f4fdd555-k4h6t" Jul 14 10:02:41.930648 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:02:41.930571 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fad05fa2-730b-4467-a90b-77283f4657d4-home\") pod \"test-model-d7ae40-kserve-9f4fdd555-k4h6t\" (UID: \"fad05fa2-730b-4467-a90b-77283f4657d4\") " pod="ai-tenant-e2e-inf-b-d7ae40/test-model-d7ae40-kserve-9f4fdd555-k4h6t" Jul 14 10:02:41.930648 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:02:41.930622 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fad05fa2-730b-4467-a90b-77283f4657d4-dshm\") pod \"test-model-d7ae40-kserve-9f4fdd555-k4h6t\" (UID: \"fad05fa2-730b-4467-a90b-77283f4657d4\") " pod="ai-tenant-e2e-inf-b-d7ae40/test-model-d7ae40-kserve-9f4fdd555-k4h6t" Jul 14 10:02:41.930922 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:02:41.930899 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fad05fa2-730b-4467-a90b-77283f4657d4-tmp-dir\") pod \"test-model-d7ae40-kserve-9f4fdd555-k4h6t\" (UID: \"fad05fa2-730b-4467-a90b-77283f4657d4\") " pod="ai-tenant-e2e-inf-b-d7ae40/test-model-d7ae40-kserve-9f4fdd555-k4h6t" Jul 14 10:02:41.931014 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:02:41.930923 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fad05fa2-730b-4467-a90b-77283f4657d4-model-cache\") pod \"test-model-d7ae40-kserve-9f4fdd555-k4h6t\" (UID: \"fad05fa2-730b-4467-a90b-77283f4657d4\") " pod="ai-tenant-e2e-inf-b-d7ae40/test-model-d7ae40-kserve-9f4fdd555-k4h6t" Jul 14 10:02:41.931014 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:02:41.930984 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fad05fa2-730b-4467-a90b-77283f4657d4-home\") pod \"test-model-d7ae40-kserve-9f4fdd555-k4h6t\" (UID: \"fad05fa2-730b-4467-a90b-77283f4657d4\") " pod="ai-tenant-e2e-inf-b-d7ae40/test-model-d7ae40-kserve-9f4fdd555-k4h6t" Jul 14 10:02:41.932711 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:02:41.932689 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fad05fa2-730b-4467-a90b-77283f4657d4-dshm\") pod \"test-model-d7ae40-kserve-9f4fdd555-k4h6t\" (UID: \"fad05fa2-730b-4467-a90b-77283f4657d4\") " pod="ai-tenant-e2e-inf-b-d7ae40/test-model-d7ae40-kserve-9f4fdd555-k4h6t" Jul 14 10:02:41.933018 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:02:41.933001 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fad05fa2-730b-4467-a90b-77283f4657d4-tls-certs\") pod \"test-model-d7ae40-kserve-9f4fdd555-k4h6t\" (UID: \"fad05fa2-730b-4467-a90b-77283f4657d4\") " pod="ai-tenant-e2e-inf-b-d7ae40/test-model-d7ae40-kserve-9f4fdd555-k4h6t" Jul 14 10:02:41.940441 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:02:41.940415 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjk4f\" (UniqueName: \"kubernetes.io/projected/fad05fa2-730b-4467-a90b-77283f4657d4-kube-api-access-tjk4f\") pod \"test-model-d7ae40-kserve-9f4fdd555-k4h6t\" (UID: \"fad05fa2-730b-4467-a90b-77283f4657d4\") " pod="ai-tenant-e2e-inf-b-d7ae40/test-model-d7ae40-kserve-9f4fdd555-k4h6t" Jul 14 10:02:42.039674 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:02:42.039590 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d7ae40/test-model-d7ae40-kserve-9f4fdd555-k4h6t" Jul 14 10:02:42.179186 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:02:42.179159 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d7ae40/test-model-d7ae40-kserve-9f4fdd555-k4h6t"] Jul 14 10:02:42.180398 ip-10-0-132-2 kubenswrapper[2565]: W0714 10:02:42.180372 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfad05fa2_730b_4467_a90b_77283f4657d4.slice/crio-213913fbcb719601798173e10d5e4fcb68637e96d91874123ecc60cfc6ad2583 WatchSource:0}: Error finding container 213913fbcb719601798173e10d5e4fcb68637e96d91874123ecc60cfc6ad2583: Status 404 returned error can't find the container with id 213913fbcb719601798173e10d5e4fcb68637e96d91874123ecc60cfc6ad2583 Jul 14 10:02:42.199764 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:02:42.199728 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d7ae40/test-model-d7ae40-kserve-9f4fdd555-k4h6t" event={"ID":"fad05fa2-730b-4467-a90b-77283f4657d4","Type":"ContainerStarted","Data":"213913fbcb719601798173e10d5e4fcb68637e96d91874123ecc60cfc6ad2583"} Jul 14 10:02:43.207221 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:02:43.207187 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d7ae40/test-model-d7ae40-kserve-9f4fdd555-k4h6t" event={"ID":"fad05fa2-730b-4467-a90b-77283f4657d4","Type":"ContainerStarted","Data":"d9dcefcd21b728338ecf38ea07fed40abbbdb65288a2200a7b960f6e7944f96c"} Jul 14 10:02:43.226258 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:02:43.226204 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-d7ae40/test-model-d7ae40-kserve-9f4fdd555-k4h6t" podStartSLOduration=2.2261875939999998 podStartE2EDuration="2.226187594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 10:02:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 10:02:43.224993366 +0000 UTC m=+2152.734224817" watchObservedRunningTime="2026-07-14 10:02:43.226187594 +0000 UTC m=+2152.735419068" Jul 14 10:02:52.040415 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:02:52.040367 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-d7ae40/test-model-d7ae40-kserve-9f4fdd555-k4h6t" Jul 14 10:02:52.040955 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:02:52.040678 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-d7ae40/test-model-d7ae40-kserve-9f4fdd555-k4h6t" Jul 14 10:02:52.053033 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:02:52.053003 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-d7ae40/test-model-d7ae40-kserve-9f4fdd555-k4h6t" Jul 14 10:02:52.269362 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:02:52.269323 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-d7ae40/test-model-d7ae40-kserve-9f4fdd555-k4h6t" Jul 14 10:03:29.941333 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:29.941297 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d7ae40/test-model-d7ae40-kserve-9f4fdd555-k4h6t"] Jul 14 10:03:40.489093 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:40.489044 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-06e8a8-9b988cd47-7xqsp"] Jul 14 10:03:40.489524 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:40.489317 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06e8a8-9b988cd47-7xqsp" podUID="77e597d9-6f5e-4cf3-bb63-72fddb89bfd2" containerName="maas-api" containerID="cri-o://eeef9f6feafcfc16de601f7b5020f1b91d64f1708d4ac5b1c03006cb4ac7de10" gracePeriod=30 Jul 14 10:03:40.746393 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:40.746330 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06e8a8-9b988cd47-7xqsp" Jul 14 10:03:40.820915 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:40.820883 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cn86v\" (UniqueName: \"kubernetes.io/projected/77e597d9-6f5e-4cf3-bb63-72fddb89bfd2-kube-api-access-cn86v\") pod \"77e597d9-6f5e-4cf3-bb63-72fddb89bfd2\" (UID: \"77e597d9-6f5e-4cf3-bb63-72fddb89bfd2\") " Jul 14 10:03:40.821116 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:40.820934 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77e597d9-6f5e-4cf3-bb63-72fddb89bfd2-maas-api-tls\") pod \"77e597d9-6f5e-4cf3-bb63-72fddb89bfd2\" (UID: \"77e597d9-6f5e-4cf3-bb63-72fddb89bfd2\") " Jul 14 10:03:40.821116 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:40.820957 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77e597d9-6f5e-4cf3-bb63-72fddb89bfd2-openshift-service-ca\") pod \"77e597d9-6f5e-4cf3-bb63-72fddb89bfd2\" (UID: \"77e597d9-6f5e-4cf3-bb63-72fddb89bfd2\") " Jul 14 10:03:40.821357 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:40.821332 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77e597d9-6f5e-4cf3-bb63-72fddb89bfd2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "77e597d9-6f5e-4cf3-bb63-72fddb89bfd2" (UID: "77e597d9-6f5e-4cf3-bb63-72fddb89bfd2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 10:03:40.823122 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:40.823094 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77e597d9-6f5e-4cf3-bb63-72fddb89bfd2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "77e597d9-6f5e-4cf3-bb63-72fddb89bfd2" (UID: "77e597d9-6f5e-4cf3-bb63-72fddb89bfd2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 10:03:40.823222 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:40.823165 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77e597d9-6f5e-4cf3-bb63-72fddb89bfd2-kube-api-access-cn86v" (OuterVolumeSpecName: "kube-api-access-cn86v") pod "77e597d9-6f5e-4cf3-bb63-72fddb89bfd2" (UID: "77e597d9-6f5e-4cf3-bb63-72fddb89bfd2"). InnerVolumeSpecName "kube-api-access-cn86v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 10:03:40.922454 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:40.922420 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cn86v\" (UniqueName: \"kubernetes.io/projected/77e597d9-6f5e-4cf3-bb63-72fddb89bfd2-kube-api-access-cn86v\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 10:03:40.922454 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:40.922450 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77e597d9-6f5e-4cf3-bb63-72fddb89bfd2-maas-api-tls\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 10:03:40.922454 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:40.922463 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77e597d9-6f5e-4cf3-bb63-72fddb89bfd2-openshift-service-ca\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 10:03:41.547915 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:41.547883 2565 generic.go:356] "Generic (PLEG): container finished" podID="77e597d9-6f5e-4cf3-bb63-72fddb89bfd2" containerID="eeef9f6feafcfc16de601f7b5020f1b91d64f1708d4ac5b1c03006cb4ac7de10" exitCode=0 Jul 14 10:03:41.548336 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:41.547933 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06e8a8-9b988cd47-7xqsp" Jul 14 10:03:41.548336 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:41.547954 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06e8a8-9b988cd47-7xqsp" event={"ID":"77e597d9-6f5e-4cf3-bb63-72fddb89bfd2","Type":"ContainerDied","Data":"eeef9f6feafcfc16de601f7b5020f1b91d64f1708d4ac5b1c03006cb4ac7de10"} Jul 14 10:03:41.548336 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:41.548004 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-06e8a8-9b988cd47-7xqsp" event={"ID":"77e597d9-6f5e-4cf3-bb63-72fddb89bfd2","Type":"ContainerDied","Data":"804bccacd168d64bf13abd81180092624e32cbbb3ea0d09188118217516ed288"} Jul 14 10:03:41.548336 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:41.548021 2565 scope.go:117] "RemoveContainer" containerID="eeef9f6feafcfc16de601f7b5020f1b91d64f1708d4ac5b1c03006cb4ac7de10" Jul 14 10:03:41.557592 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:41.557567 2565 scope.go:117] "RemoveContainer" containerID="eeef9f6feafcfc16de601f7b5020f1b91d64f1708d4ac5b1c03006cb4ac7de10" Jul 14 10:03:41.557839 ip-10-0-132-2 kubenswrapper[2565]: E0714 10:03:41.557820 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eeef9f6feafcfc16de601f7b5020f1b91d64f1708d4ac5b1c03006cb4ac7de10\": container with ID starting with eeef9f6feafcfc16de601f7b5020f1b91d64f1708d4ac5b1c03006cb4ac7de10 not found: ID does not exist" containerID="eeef9f6feafcfc16de601f7b5020f1b91d64f1708d4ac5b1c03006cb4ac7de10" Jul 14 10:03:41.557897 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:41.557847 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eeef9f6feafcfc16de601f7b5020f1b91d64f1708d4ac5b1c03006cb4ac7de10"} err="failed to get container status \"eeef9f6feafcfc16de601f7b5020f1b91d64f1708d4ac5b1c03006cb4ac7de10\": rpc error: code = NotFound desc = could not find container \"eeef9f6feafcfc16de601f7b5020f1b91d64f1708d4ac5b1c03006cb4ac7de10\": container with ID starting with eeef9f6feafcfc16de601f7b5020f1b91d64f1708d4ac5b1c03006cb4ac7de10 not found: ID does not exist" Jul 14 10:03:41.565988 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:41.565965 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-06e8a8-9b988cd47-7xqsp"] Jul 14 10:03:41.568912 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:41.568886 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-06e8a8-9b988cd47-7xqsp"] Jul 14 10:03:42.969785 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:42.969750 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77e597d9-6f5e-4cf3-bb63-72fddb89bfd2" path="/var/lib/kubelet/pods/77e597d9-6f5e-4cf3-bb63-72fddb89bfd2/volumes" Jul 14 10:03:44.964121 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:44.964083 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-d7ae40/test-model-d7ae40-kserve-9f4fdd555-k4h6t" podUID="fad05fa2-730b-4467-a90b-77283f4657d4" containerName="main" containerID="cri-o://d9dcefcd21b728338ecf38ea07fed40abbbdb65288a2200a7b960f6e7944f96c" gracePeriod=45 Jul 14 10:03:45.220609 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:45.220548 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d7ae40/test-model-d7ae40-kserve-9f4fdd555-k4h6t" Jul 14 10:03:45.263343 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:45.263313 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fad05fa2-730b-4467-a90b-77283f4657d4-tls-certs\") pod \"fad05fa2-730b-4467-a90b-77283f4657d4\" (UID: \"fad05fa2-730b-4467-a90b-77283f4657d4\") " Jul 14 10:03:45.263507 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:45.263398 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fad05fa2-730b-4467-a90b-77283f4657d4-model-cache\") pod \"fad05fa2-730b-4467-a90b-77283f4657d4\" (UID: \"fad05fa2-730b-4467-a90b-77283f4657d4\") " Jul 14 10:03:45.263507 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:45.263469 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjk4f\" (UniqueName: \"kubernetes.io/projected/fad05fa2-730b-4467-a90b-77283f4657d4-kube-api-access-tjk4f\") pod \"fad05fa2-730b-4467-a90b-77283f4657d4\" (UID: \"fad05fa2-730b-4467-a90b-77283f4657d4\") " Jul 14 10:03:45.263507 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:45.263503 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fad05fa2-730b-4467-a90b-77283f4657d4-home\") pod \"fad05fa2-730b-4467-a90b-77283f4657d4\" (UID: \"fad05fa2-730b-4467-a90b-77283f4657d4\") " Jul 14 10:03:45.263688 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:45.263532 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fad05fa2-730b-4467-a90b-77283f4657d4-tmp-dir\") pod \"fad05fa2-730b-4467-a90b-77283f4657d4\" (UID: \"fad05fa2-730b-4467-a90b-77283f4657d4\") " Jul 14 10:03:45.263688 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:45.263593 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fad05fa2-730b-4467-a90b-77283f4657d4-dshm\") pod \"fad05fa2-730b-4467-a90b-77283f4657d4\" (UID: \"fad05fa2-730b-4467-a90b-77283f4657d4\") " Jul 14 10:03:45.263788 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:45.263718 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fad05fa2-730b-4467-a90b-77283f4657d4-model-cache" (OuterVolumeSpecName: "model-cache") pod "fad05fa2-730b-4467-a90b-77283f4657d4" (UID: "fad05fa2-730b-4467-a90b-77283f4657d4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 10:03:45.263788 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:45.263729 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fad05fa2-730b-4467-a90b-77283f4657d4-home" (OuterVolumeSpecName: "home") pod "fad05fa2-730b-4467-a90b-77283f4657d4" (UID: "fad05fa2-730b-4467-a90b-77283f4657d4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 10:03:45.264021 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:45.263973 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fad05fa2-730b-4467-a90b-77283f4657d4-model-cache\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 10:03:45.264021 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:45.263961 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fad05fa2-730b-4467-a90b-77283f4657d4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fad05fa2-730b-4467-a90b-77283f4657d4" (UID: "fad05fa2-730b-4467-a90b-77283f4657d4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 10:03:45.264021 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:45.264001 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fad05fa2-730b-4467-a90b-77283f4657d4-home\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 10:03:45.265690 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:45.265667 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fad05fa2-730b-4467-a90b-77283f4657d4-kube-api-access-tjk4f" (OuterVolumeSpecName: "kube-api-access-tjk4f") pod "fad05fa2-730b-4467-a90b-77283f4657d4" (UID: "fad05fa2-730b-4467-a90b-77283f4657d4"). InnerVolumeSpecName "kube-api-access-tjk4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 10:03:45.265883 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:45.265866 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fad05fa2-730b-4467-a90b-77283f4657d4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fad05fa2-730b-4467-a90b-77283f4657d4" (UID: "fad05fa2-730b-4467-a90b-77283f4657d4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 10:03:45.265974 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:45.265883 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fad05fa2-730b-4467-a90b-77283f4657d4-dshm" (OuterVolumeSpecName: "dshm") pod "fad05fa2-730b-4467-a90b-77283f4657d4" (UID: "fad05fa2-730b-4467-a90b-77283f4657d4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 10:03:45.365067 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:45.365033 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjk4f\" (UniqueName: \"kubernetes.io/projected/fad05fa2-730b-4467-a90b-77283f4657d4-kube-api-access-tjk4f\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 10:03:45.365067 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:45.365061 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fad05fa2-730b-4467-a90b-77283f4657d4-tmp-dir\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 10:03:45.365067 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:45.365070 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fad05fa2-730b-4467-a90b-77283f4657d4-dshm\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 10:03:45.365067 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:45.365077 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fad05fa2-730b-4467-a90b-77283f4657d4-tls-certs\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 10:03:45.573472 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:45.573376 2565 generic.go:356] "Generic (PLEG): container finished" podID="fad05fa2-730b-4467-a90b-77283f4657d4" containerID="d9dcefcd21b728338ecf38ea07fed40abbbdb65288a2200a7b960f6e7944f96c" exitCode=0 Jul 14 10:03:45.573472 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:45.573418 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d7ae40/test-model-d7ae40-kserve-9f4fdd555-k4h6t" Jul 14 10:03:45.573690 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:45.573465 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d7ae40/test-model-d7ae40-kserve-9f4fdd555-k4h6t" event={"ID":"fad05fa2-730b-4467-a90b-77283f4657d4","Type":"ContainerDied","Data":"d9dcefcd21b728338ecf38ea07fed40abbbdb65288a2200a7b960f6e7944f96c"} Jul 14 10:03:45.573690 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:45.573507 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d7ae40/test-model-d7ae40-kserve-9f4fdd555-k4h6t" event={"ID":"fad05fa2-730b-4467-a90b-77283f4657d4","Type":"ContainerDied","Data":"213913fbcb719601798173e10d5e4fcb68637e96d91874123ecc60cfc6ad2583"} Jul 14 10:03:45.573690 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:45.573525 2565 scope.go:117] "RemoveContainer" containerID="d9dcefcd21b728338ecf38ea07fed40abbbdb65288a2200a7b960f6e7944f96c" Jul 14 10:03:45.584018 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:45.583995 2565 scope.go:117] "RemoveContainer" containerID="d9dcefcd21b728338ecf38ea07fed40abbbdb65288a2200a7b960f6e7944f96c" Jul 14 10:03:45.584291 ip-10-0-132-2 kubenswrapper[2565]: E0714 10:03:45.584271 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9dcefcd21b728338ecf38ea07fed40abbbdb65288a2200a7b960f6e7944f96c\": container with ID starting with d9dcefcd21b728338ecf38ea07fed40abbbdb65288a2200a7b960f6e7944f96c not found: ID does not exist" containerID="d9dcefcd21b728338ecf38ea07fed40abbbdb65288a2200a7b960f6e7944f96c" Jul 14 10:03:45.584356 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:45.584298 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9dcefcd21b728338ecf38ea07fed40abbbdb65288a2200a7b960f6e7944f96c"} err="failed to get container status \"d9dcefcd21b728338ecf38ea07fed40abbbdb65288a2200a7b960f6e7944f96c\": rpc error: code = NotFound desc = could not find container \"d9dcefcd21b728338ecf38ea07fed40abbbdb65288a2200a7b960f6e7944f96c\": container with ID starting with d9dcefcd21b728338ecf38ea07fed40abbbdb65288a2200a7b960f6e7944f96c not found: ID does not exist" Jul 14 10:03:45.596985 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:45.596961 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d7ae40/test-model-d7ae40-kserve-9f4fdd555-k4h6t"] Jul 14 10:03:45.602668 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:45.602646 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-d7ae40/test-model-d7ae40-kserve-9f4fdd555-k4h6t"] Jul 14 10:03:46.969004 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:46.968970 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fad05fa2-730b-4467-a90b-77283f4657d4" path="/var/lib/kubelet/pods/fad05fa2-730b-4467-a90b-77283f4657d4/volumes" Jul 14 10:03:56.191731 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:56.191697 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d7ae40-pd6zm"] Jul 14 10:03:56.192357 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:56.192337 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fad05fa2-730b-4467-a90b-77283f4657d4" containerName="main" Jul 14 10:03:56.192461 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:56.192359 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fad05fa2-730b-4467-a90b-77283f4657d4" containerName="main" Jul 14 10:03:56.192461 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:56.192398 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77e597d9-6f5e-4cf3-bb63-72fddb89bfd2" containerName="maas-api" Jul 14 10:03:56.192461 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:56.192406 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="77e597d9-6f5e-4cf3-bb63-72fddb89bfd2" containerName="maas-api" Jul 14 10:03:56.192630 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:56.192505 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="77e597d9-6f5e-4cf3-bb63-72fddb89bfd2" containerName="maas-api" Jul 14 10:03:56.192630 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:56.192523 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fad05fa2-730b-4467-a90b-77283f4657d4" containerName="main" Jul 14 10:03:56.195598 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:56.195579 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d7ae40-pd6zm" Jul 14 10:03:56.199013 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:56.198994 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8m76v\"" Jul 14 10:03:56.212666 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:56.212634 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d7ae40-pd6zm"] Jul 14 10:03:56.270916 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:56.270877 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b819a49-91da-4f32-80be-3df03d321a8a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d7ae40-pd6zm\" (UID: \"9b819a49-91da-4f32-80be-3df03d321a8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d7ae40-pd6zm" Jul 14 10:03:56.371459 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:56.371424 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b819a49-91da-4f32-80be-3df03d321a8a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d7ae40-pd6zm\" (UID: \"9b819a49-91da-4f32-80be-3df03d321a8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d7ae40-pd6zm" Jul 14 10:03:56.372098 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:56.372078 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b819a49-91da-4f32-80be-3df03d321a8a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d7ae40-pd6zm\" (UID: \"9b819a49-91da-4f32-80be-3df03d321a8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d7ae40-pd6zm" Jul 14 10:03:56.506254 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:56.506172 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d7ae40-pd6zm" Jul 14 10:03:56.648084 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:56.648044 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d7ae40-pd6zm"] Jul 14 10:03:56.650293 ip-10-0-132-2 kubenswrapper[2565]: W0714 10:03:56.650265 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b819a49_91da_4f32_80be_3df03d321a8a.slice/crio-679c0ada5148713f3f5fe57be9aaa48e581b42da26de90150e6658927c6d2eaf WatchSource:0}: Error finding container 679c0ada5148713f3f5fe57be9aaa48e581b42da26de90150e6658927c6d2eaf: Status 404 returned error can't find the container with id 679c0ada5148713f3f5fe57be9aaa48e581b42da26de90150e6658927c6d2eaf Jul 14 10:03:57.644081 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:57.644038 2565 generic.go:356] "Generic (PLEG): container finished" podID="9b819a49-91da-4f32-80be-3df03d321a8a" containerID="be231eea200dd5f48b5cda51ec34141e7f79f4a6a2cdc6a5f816669cca4ca4bb" exitCode=0 Jul 14 10:03:57.644495 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:57.644118 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d7ae40-pd6zm" event={"ID":"9b819a49-91da-4f32-80be-3df03d321a8a","Type":"ContainerDied","Data":"be231eea200dd5f48b5cda51ec34141e7f79f4a6a2cdc6a5f816669cca4ca4bb"} Jul 14 10:03:57.644495 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:57.644167 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d7ae40-pd6zm" event={"ID":"9b819a49-91da-4f32-80be-3df03d321a8a","Type":"ContainerStarted","Data":"679c0ada5148713f3f5fe57be9aaa48e581b42da26de90150e6658927c6d2eaf"} Jul 14 10:03:58.784341 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:58.784314 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d7ae40-pd6zm" Jul 14 10:03:58.897085 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:58.897040 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b819a49-91da-4f32-80be-3df03d321a8a-maas-api-service-ca\") pod \"9b819a49-91da-4f32-80be-3df03d321a8a\" (UID: \"9b819a49-91da-4f32-80be-3df03d321a8a\") " Jul 14 10:03:58.897412 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:58.897390 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b819a49-91da-4f32-80be-3df03d321a8a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9b819a49-91da-4f32-80be-3df03d321a8a" (UID: "9b819a49-91da-4f32-80be-3df03d321a8a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 10:03:58.997774 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:58.997743 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b819a49-91da-4f32-80be-3df03d321a8a-maas-api-service-ca\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 10:03:59.656958 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:59.656924 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d7ae40-pd6zm" Jul 14 10:03:59.657138 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:59.656955 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d7ae40-pd6zm" event={"ID":"9b819a49-91da-4f32-80be-3df03d321a8a","Type":"ContainerDied","Data":"679c0ada5148713f3f5fe57be9aaa48e581b42da26de90150e6658927c6d2eaf"} Jul 14 10:03:59.657138 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:03:59.657006 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="679c0ada5148713f3f5fe57be9aaa48e581b42da26de90150e6658927c6d2eaf" Jul 14 10:04:06.654643 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:06.654611 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d7ae40-dc7bddb6f-n6vwf"] Jul 14 10:04:06.655477 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:06.654869 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7ae40-dc7bddb6f-n6vwf" podUID="9288af01-8c6a-4ae1-9908-0f5d05da0d7d" containerName="maas-api" containerID="cri-o://c398318d3613cc2826ecfb9bdd5c03289e3d845ed79e114d472d155cdb2f1a08" gracePeriod=30 Jul 14 10:04:06.671720 ip-10-0-132-2 kubenswrapper[2565]: E0714 10:04:06.671690 2565 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9288af01_8c6a_4ae1_9908_0f5d05da0d7d.slice/crio-c398318d3613cc2826ecfb9bdd5c03289e3d845ed79e114d472d155cdb2f1a08.scope\": RecentStats: unable to find data in memory cache]" Jul 14 10:04:06.672033 ip-10-0-132-2 kubenswrapper[2565]: E0714 10:04:06.672002 2565 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9288af01_8c6a_4ae1_9908_0f5d05da0d7d.slice/crio-c398318d3613cc2826ecfb9bdd5c03289e3d845ed79e114d472d155cdb2f1a08.scope\": RecentStats: unable to find data in memory cache]" Jul 14 10:04:06.913729 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:06.913663 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7ae40-dc7bddb6f-n6vwf" Jul 14 10:04:06.978084 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:06.978054 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9288af01-8c6a-4ae1-9908-0f5d05da0d7d-maas-api-tls\") pod \"9288af01-8c6a-4ae1-9908-0f5d05da0d7d\" (UID: \"9288af01-8c6a-4ae1-9908-0f5d05da0d7d\") " Jul 14 10:04:06.978229 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:06.978119 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7frc\" (UniqueName: \"kubernetes.io/projected/9288af01-8c6a-4ae1-9908-0f5d05da0d7d-kube-api-access-r7frc\") pod \"9288af01-8c6a-4ae1-9908-0f5d05da0d7d\" (UID: \"9288af01-8c6a-4ae1-9908-0f5d05da0d7d\") " Jul 14 10:04:06.978229 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:06.978167 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9288af01-8c6a-4ae1-9908-0f5d05da0d7d-openshift-service-ca\") pod \"9288af01-8c6a-4ae1-9908-0f5d05da0d7d\" (UID: \"9288af01-8c6a-4ae1-9908-0f5d05da0d7d\") " Jul 14 10:04:06.978533 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:06.978510 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9288af01-8c6a-4ae1-9908-0f5d05da0d7d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9288af01-8c6a-4ae1-9908-0f5d05da0d7d" (UID: "9288af01-8c6a-4ae1-9908-0f5d05da0d7d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 10:04:06.980240 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:06.980214 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9288af01-8c6a-4ae1-9908-0f5d05da0d7d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9288af01-8c6a-4ae1-9908-0f5d05da0d7d" (UID: "9288af01-8c6a-4ae1-9908-0f5d05da0d7d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 10:04:06.980395 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:06.980374 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9288af01-8c6a-4ae1-9908-0f5d05da0d7d-kube-api-access-r7frc" (OuterVolumeSpecName: "kube-api-access-r7frc") pod "9288af01-8c6a-4ae1-9908-0f5d05da0d7d" (UID: "9288af01-8c6a-4ae1-9908-0f5d05da0d7d"). InnerVolumeSpecName "kube-api-access-r7frc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 10:04:07.079661 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:07.079622 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9288af01-8c6a-4ae1-9908-0f5d05da0d7d-openshift-service-ca\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 10:04:07.079661 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:07.079653 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9288af01-8c6a-4ae1-9908-0f5d05da0d7d-maas-api-tls\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 10:04:07.079661 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:07.079664 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7frc\" (UniqueName: \"kubernetes.io/projected/9288af01-8c6a-4ae1-9908-0f5d05da0d7d-kube-api-access-r7frc\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 10:04:07.710744 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:07.710694 2565 generic.go:356] "Generic (PLEG): container finished" podID="9288af01-8c6a-4ae1-9908-0f5d05da0d7d" containerID="c398318d3613cc2826ecfb9bdd5c03289e3d845ed79e114d472d155cdb2f1a08" exitCode=0 Jul 14 10:04:07.710744 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:07.710742 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7ae40-dc7bddb6f-n6vwf" Jul 14 10:04:07.711307 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:07.710775 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7ae40-dc7bddb6f-n6vwf" event={"ID":"9288af01-8c6a-4ae1-9908-0f5d05da0d7d","Type":"ContainerDied","Data":"c398318d3613cc2826ecfb9bdd5c03289e3d845ed79e114d472d155cdb2f1a08"} Jul 14 10:04:07.711307 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:07.710866 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7ae40-dc7bddb6f-n6vwf" event={"ID":"9288af01-8c6a-4ae1-9908-0f5d05da0d7d","Type":"ContainerDied","Data":"e4bdfb08f5926e9b8dada20b39726b3b29e6282075b517c5a0018c1e708ad917"} Jul 14 10:04:07.711307 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:07.710902 2565 scope.go:117] "RemoveContainer" containerID="c398318d3613cc2826ecfb9bdd5c03289e3d845ed79e114d472d155cdb2f1a08" Jul 14 10:04:07.720755 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:07.720737 2565 scope.go:117] "RemoveContainer" containerID="c398318d3613cc2826ecfb9bdd5c03289e3d845ed79e114d472d155cdb2f1a08" Jul 14 10:04:07.721065 ip-10-0-132-2 kubenswrapper[2565]: E0714 10:04:07.721046 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c398318d3613cc2826ecfb9bdd5c03289e3d845ed79e114d472d155cdb2f1a08\": container with ID starting with c398318d3613cc2826ecfb9bdd5c03289e3d845ed79e114d472d155cdb2f1a08 not found: ID does not exist" containerID="c398318d3613cc2826ecfb9bdd5c03289e3d845ed79e114d472d155cdb2f1a08" Jul 14 10:04:07.721135 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:07.721075 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c398318d3613cc2826ecfb9bdd5c03289e3d845ed79e114d472d155cdb2f1a08"} err="failed to get container status \"c398318d3613cc2826ecfb9bdd5c03289e3d845ed79e114d472d155cdb2f1a08\": rpc error: code = NotFound desc = could not find container \"c398318d3613cc2826ecfb9bdd5c03289e3d845ed79e114d472d155cdb2f1a08\": container with ID starting with c398318d3613cc2826ecfb9bdd5c03289e3d845ed79e114d472d155cdb2f1a08 not found: ID does not exist" Jul 14 10:04:07.735169 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:07.735140 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d7ae40-dc7bddb6f-n6vwf"] Jul 14 10:04:07.746117 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:07.746091 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d7ae40-dc7bddb6f-n6vwf"] Jul 14 10:04:08.968543 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:08.968511 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9288af01-8c6a-4ae1-9908-0f5d05da0d7d" path="/var/lib/kubelet/pods/9288af01-8c6a-4ae1-9908-0f5d05da0d7d/volumes" Jul 14 10:04:26.112098 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:26.112055 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b26ed2-684c9569c9-ddwbj"] Jul 14 10:04:26.112505 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:26.112491 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9288af01-8c6a-4ae1-9908-0f5d05da0d7d" containerName="maas-api" Jul 14 10:04:26.112564 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:26.112506 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9288af01-8c6a-4ae1-9908-0f5d05da0d7d" containerName="maas-api" Jul 14 10:04:26.112564 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:26.112516 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b819a49-91da-4f32-80be-3df03d321a8a" containerName="revoke-keys" Jul 14 10:04:26.112564 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:26.112523 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b819a49-91da-4f32-80be-3df03d321a8a" containerName="revoke-keys" Jul 14 10:04:26.112670 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:26.112614 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b819a49-91da-4f32-80be-3df03d321a8a" containerName="revoke-keys" Jul 14 10:04:26.112670 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:26.112625 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9288af01-8c6a-4ae1-9908-0f5d05da0d7d" containerName="maas-api" Jul 14 10:04:26.117115 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:26.117095 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b26ed2-684c9569c9-ddwbj" Jul 14 10:04:26.120548 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:26.120525 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-b26ed2-serving-cert\"" Jul 14 10:04:26.166059 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:26.166028 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qfhd\" (UniqueName: \"kubernetes.io/projected/31a71677-52ff-4104-87cf-cb89575a64c7-kube-api-access-2qfhd\") pod \"maas-api-e2e-api-a-b26ed2-684c9569c9-ddwbj\" (UID: \"31a71677-52ff-4104-87cf-cb89575a64c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b26ed2-684c9569c9-ddwbj" Jul 14 10:04:26.166220 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:26.166130 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31a71677-52ff-4104-87cf-cb89575a64c7-maas-api-tls\") pod \"maas-api-e2e-api-a-b26ed2-684c9569c9-ddwbj\" (UID: \"31a71677-52ff-4104-87cf-cb89575a64c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b26ed2-684c9569c9-ddwbj" Jul 14 10:04:26.166220 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:26.166152 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31a71677-52ff-4104-87cf-cb89575a64c7-openshift-service-ca\") pod \"maas-api-e2e-api-a-b26ed2-684c9569c9-ddwbj\" (UID: \"31a71677-52ff-4104-87cf-cb89575a64c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b26ed2-684c9569c9-ddwbj" Jul 14 10:04:26.207466 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:26.207436 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b26ed2-684c9569c9-ddwbj"] Jul 14 10:04:26.267069 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:26.267032 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qfhd\" (UniqueName: \"kubernetes.io/projected/31a71677-52ff-4104-87cf-cb89575a64c7-kube-api-access-2qfhd\") pod \"maas-api-e2e-api-a-b26ed2-684c9569c9-ddwbj\" (UID: \"31a71677-52ff-4104-87cf-cb89575a64c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b26ed2-684c9569c9-ddwbj" Jul 14 10:04:26.267243 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:26.267159 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31a71677-52ff-4104-87cf-cb89575a64c7-maas-api-tls\") pod \"maas-api-e2e-api-a-b26ed2-684c9569c9-ddwbj\" (UID: \"31a71677-52ff-4104-87cf-cb89575a64c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b26ed2-684c9569c9-ddwbj" Jul 14 10:04:26.267243 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:26.267192 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31a71677-52ff-4104-87cf-cb89575a64c7-openshift-service-ca\") pod \"maas-api-e2e-api-a-b26ed2-684c9569c9-ddwbj\" (UID: \"31a71677-52ff-4104-87cf-cb89575a64c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b26ed2-684c9569c9-ddwbj" Jul 14 10:04:26.267318 ip-10-0-132-2 kubenswrapper[2565]: E0714 10:04:26.267295 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-b26ed2-serving-cert: secret "maas-api-e2e-api-a-b26ed2-serving-cert" not found Jul 14 10:04:26.267397 ip-10-0-132-2 kubenswrapper[2565]: E0714 10:04:26.267385 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31a71677-52ff-4104-87cf-cb89575a64c7-maas-api-tls podName:31a71677-52ff-4104-87cf-cb89575a64c7 nodeName:}" failed. No retries permitted until 2026-07-14 10:04:26.767364956 +0000 UTC m=+2256.276596398 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/31a71677-52ff-4104-87cf-cb89575a64c7-maas-api-tls") pod "maas-api-e2e-api-a-b26ed2-684c9569c9-ddwbj" (UID: "31a71677-52ff-4104-87cf-cb89575a64c7") : secret "maas-api-e2e-api-a-b26ed2-serving-cert" not found Jul 14 10:04:26.267905 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:26.267890 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31a71677-52ff-4104-87cf-cb89575a64c7-openshift-service-ca\") pod \"maas-api-e2e-api-a-b26ed2-684c9569c9-ddwbj\" (UID: \"31a71677-52ff-4104-87cf-cb89575a64c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b26ed2-684c9569c9-ddwbj" Jul 14 10:04:26.278966 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:26.278944 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qfhd\" (UniqueName: \"kubernetes.io/projected/31a71677-52ff-4104-87cf-cb89575a64c7-kube-api-access-2qfhd\") pod \"maas-api-e2e-api-a-b26ed2-684c9569c9-ddwbj\" (UID: \"31a71677-52ff-4104-87cf-cb89575a64c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b26ed2-684c9569c9-ddwbj" Jul 14 10:04:26.770532 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:26.770496 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31a71677-52ff-4104-87cf-cb89575a64c7-maas-api-tls\") pod \"maas-api-e2e-api-a-b26ed2-684c9569c9-ddwbj\" (UID: \"31a71677-52ff-4104-87cf-cb89575a64c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b26ed2-684c9569c9-ddwbj" Jul 14 10:04:26.772980 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:26.772956 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31a71677-52ff-4104-87cf-cb89575a64c7-maas-api-tls\") pod \"maas-api-e2e-api-a-b26ed2-684c9569c9-ddwbj\" (UID: \"31a71677-52ff-4104-87cf-cb89575a64c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b26ed2-684c9569c9-ddwbj" Jul 14 10:04:27.027697 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:27.027611 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b26ed2-684c9569c9-ddwbj" Jul 14 10:04:27.166235 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:27.166203 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b26ed2-684c9569c9-ddwbj"] Jul 14 10:04:27.167497 ip-10-0-132-2 kubenswrapper[2565]: W0714 10:04:27.167471 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31a71677_52ff_4104_87cf_cb89575a64c7.slice/crio-5a31692427c147b76361395e371d3613e26c9fa93f601b563c29c6c21c9ec7fb WatchSource:0}: Error finding container 5a31692427c147b76361395e371d3613e26c9fa93f601b563c29c6c21c9ec7fb: Status 404 returned error can't find the container with id 5a31692427c147b76361395e371d3613e26c9fa93f601b563c29c6c21c9ec7fb Jul 14 10:04:27.169210 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:27.169193 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 10:04:27.831554 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:27.831519 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b26ed2-684c9569c9-ddwbj" event={"ID":"31a71677-52ff-4104-87cf-cb89575a64c7","Type":"ContainerStarted","Data":"afe725bd058f748f1e8982e8bc0df06664ee7eeef80fe20413b3e48f0817bb6b"} Jul 14 10:04:27.831716 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:27.831582 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b26ed2-684c9569c9-ddwbj" event={"ID":"31a71677-52ff-4104-87cf-cb89575a64c7","Type":"ContainerStarted","Data":"5a31692427c147b76361395e371d3613e26c9fa93f601b563c29c6c21c9ec7fb"} Jul 14 10:04:27.831716 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:27.831606 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b26ed2-684c9569c9-ddwbj" Jul 14 10:04:27.851125 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:27.851073 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b26ed2-684c9569c9-ddwbj" podStartSLOduration=1.397188261 podStartE2EDuration="1.85105666s" totalImagesPullingTime="453.868399ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 10:04:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 10:04:27.850546114 +0000 UTC m=+2257.359777586" watchObservedRunningTime="2026-07-14 10:04:27.85105666 +0000 UTC m=+2257.360288107" Jul 14 10:04:33.851022 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:33.850993 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b26ed2-684c9569c9-ddwbj" Jul 14 10:04:34.096558 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:34.096524 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6cb46d-6bd44b4644-flzks"] Jul 14 10:04:34.100491 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:34.100468 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6cb46d-6bd44b4644-flzks" Jul 14 10:04:34.103587 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:34.103528 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-6cb46d-serving-cert\"" Jul 14 10:04:34.111490 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:34.111461 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6cb46d-6bd44b4644-flzks"] Jul 14 10:04:34.136347 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:34.136319 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38039008-3538-4120-91d2-ef47646afa98-openshift-service-ca\") pod \"maas-api-e2e-api-b-6cb46d-6bd44b4644-flzks\" (UID: \"38039008-3538-4120-91d2-ef47646afa98\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6cb46d-6bd44b4644-flzks" Jul 14 10:04:34.136509 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:34.136364 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktklc\" (UniqueName: \"kubernetes.io/projected/38039008-3538-4120-91d2-ef47646afa98-kube-api-access-ktklc\") pod \"maas-api-e2e-api-b-6cb46d-6bd44b4644-flzks\" (UID: \"38039008-3538-4120-91d2-ef47646afa98\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6cb46d-6bd44b4644-flzks" Jul 14 10:04:34.136509 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:34.136443 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38039008-3538-4120-91d2-ef47646afa98-maas-api-tls\") pod \"maas-api-e2e-api-b-6cb46d-6bd44b4644-flzks\" (UID: \"38039008-3538-4120-91d2-ef47646afa98\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6cb46d-6bd44b4644-flzks" Jul 14 10:04:34.237726 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:34.237693 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38039008-3538-4120-91d2-ef47646afa98-openshift-service-ca\") pod \"maas-api-e2e-api-b-6cb46d-6bd44b4644-flzks\" (UID: \"38039008-3538-4120-91d2-ef47646afa98\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6cb46d-6bd44b4644-flzks" Jul 14 10:04:34.237925 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:34.237738 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktklc\" (UniqueName: \"kubernetes.io/projected/38039008-3538-4120-91d2-ef47646afa98-kube-api-access-ktklc\") pod \"maas-api-e2e-api-b-6cb46d-6bd44b4644-flzks\" (UID: \"38039008-3538-4120-91d2-ef47646afa98\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6cb46d-6bd44b4644-flzks" Jul 14 10:04:34.237925 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:34.237784 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38039008-3538-4120-91d2-ef47646afa98-maas-api-tls\") pod \"maas-api-e2e-api-b-6cb46d-6bd44b4644-flzks\" (UID: \"38039008-3538-4120-91d2-ef47646afa98\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6cb46d-6bd44b4644-flzks" Jul 14 10:04:34.238420 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:34.238398 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38039008-3538-4120-91d2-ef47646afa98-openshift-service-ca\") pod \"maas-api-e2e-api-b-6cb46d-6bd44b4644-flzks\" (UID: \"38039008-3538-4120-91d2-ef47646afa98\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6cb46d-6bd44b4644-flzks" Jul 14 10:04:34.240255 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:34.240236 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38039008-3538-4120-91d2-ef47646afa98-maas-api-tls\") pod \"maas-api-e2e-api-b-6cb46d-6bd44b4644-flzks\" (UID: \"38039008-3538-4120-91d2-ef47646afa98\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6cb46d-6bd44b4644-flzks" Jul 14 10:04:34.249956 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:34.249934 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktklc\" (UniqueName: \"kubernetes.io/projected/38039008-3538-4120-91d2-ef47646afa98-kube-api-access-ktklc\") pod \"maas-api-e2e-api-b-6cb46d-6bd44b4644-flzks\" (UID: \"38039008-3538-4120-91d2-ef47646afa98\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6cb46d-6bd44b4644-flzks" Jul 14 10:04:34.412846 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:34.412787 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6cb46d-6bd44b4644-flzks" Jul 14 10:04:34.564071 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:34.564036 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6cb46d-6bd44b4644-flzks"] Jul 14 10:04:34.564880 ip-10-0-132-2 kubenswrapper[2565]: W0714 10:04:34.564856 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38039008_3538_4120_91d2_ef47646afa98.slice/crio-f8218212e1d2741bb262f7c5387b8e59be7ae92161adc547f7f9b4d1363c15a6 WatchSource:0}: Error finding container f8218212e1d2741bb262f7c5387b8e59be7ae92161adc547f7f9b4d1363c15a6: Status 404 returned error can't find the container with id f8218212e1d2741bb262f7c5387b8e59be7ae92161adc547f7f9b4d1363c15a6 Jul 14 10:04:34.882800 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:34.882767 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6cb46d-6bd44b4644-flzks" event={"ID":"38039008-3538-4120-91d2-ef47646afa98","Type":"ContainerStarted","Data":"f8218212e1d2741bb262f7c5387b8e59be7ae92161adc547f7f9b4d1363c15a6"} Jul 14 10:04:35.890767 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:35.890731 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6cb46d-6bd44b4644-flzks" event={"ID":"38039008-3538-4120-91d2-ef47646afa98","Type":"ContainerStarted","Data":"dabb7f3814cc1707f1a4b3bea1eacff4e759f83b46bce33929f961a79d6b288d"} Jul 14 10:04:35.891178 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:35.890956 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6cb46d-6bd44b4644-flzks" Jul 14 10:04:35.911101 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:35.911059 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6cb46d-6bd44b4644-flzks" podStartSLOduration=1.6366064580000002 podStartE2EDuration="1.91104281s" totalImagesPullingTime="274.436352ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 10:04:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 10:04:35.909993315 +0000 UTC m=+2265.419224764" watchObservedRunningTime="2026-07-14 10:04:35.91104281 +0000 UTC m=+2265.420274258" Jul 14 10:04:41.902650 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:41.902621 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6cb46d-6bd44b4644-flzks" Jul 14 10:04:46.456939 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:46.456902 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b26ed2-wrcng"] Jul 14 10:04:46.460745 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:46.460727 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b26ed2-wrcng" Jul 14 10:04:46.464915 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:46.464897 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8m76v\"" Jul 14 10:04:46.476318 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:46.476289 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b26ed2-wrcng"] Jul 14 10:04:46.558711 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:46.558670 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41540f64-cbe0-45c5-9087-856a4c63bdfc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b26ed2-wrcng\" (UID: \"41540f64-cbe0-45c5-9087-856a4c63bdfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b26ed2-wrcng" Jul 14 10:04:46.659576 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:46.659543 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41540f64-cbe0-45c5-9087-856a4c63bdfc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b26ed2-wrcng\" (UID: \"41540f64-cbe0-45c5-9087-856a4c63bdfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b26ed2-wrcng" Jul 14 10:04:46.660915 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:46.660883 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41540f64-cbe0-45c5-9087-856a4c63bdfc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b26ed2-wrcng\" (UID: \"41540f64-cbe0-45c5-9087-856a4c63bdfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b26ed2-wrcng" Jul 14 10:04:46.771661 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:46.771572 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b26ed2-wrcng" Jul 14 10:04:46.927658 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:46.927627 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b26ed2-wrcng"] Jul 14 10:04:46.928492 ip-10-0-132-2 kubenswrapper[2565]: W0714 10:04:46.928465 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41540f64_cbe0_45c5_9087_856a4c63bdfc.slice/crio-027ddb0415db3b3f3d7f2fff57e55633c8a85fb6422d70ebda7a247d89abcc8e WatchSource:0}: Error finding container 027ddb0415db3b3f3d7f2fff57e55633c8a85fb6422d70ebda7a247d89abcc8e: Status 404 returned error can't find the container with id 027ddb0415db3b3f3d7f2fff57e55633c8a85fb6422d70ebda7a247d89abcc8e Jul 14 10:04:46.957071 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:46.957037 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b26ed2-wrcng" event={"ID":"41540f64-cbe0-45c5-9087-856a4c63bdfc","Type":"ContainerStarted","Data":"027ddb0415db3b3f3d7f2fff57e55633c8a85fb6422d70ebda7a247d89abcc8e"} Jul 14 10:04:47.964984 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:47.964956 2565 generic.go:356] "Generic (PLEG): container finished" podID="41540f64-cbe0-45c5-9087-856a4c63bdfc" containerID="fdbf30e59de2f3cae79e82c0cbf796fcba7bd787ba3a2626d21de074f046a2fa" exitCode=0 Jul 14 10:04:47.965378 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:47.965016 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b26ed2-wrcng" event={"ID":"41540f64-cbe0-45c5-9087-856a4c63bdfc","Type":"ContainerDied","Data":"fdbf30e59de2f3cae79e82c0cbf796fcba7bd787ba3a2626d21de074f046a2fa"} Jul 14 10:04:49.108690 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:49.108666 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b26ed2-wrcng" Jul 14 10:04:49.188767 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:49.188739 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41540f64-cbe0-45c5-9087-856a4c63bdfc-maas-api-service-ca\") pod \"41540f64-cbe0-45c5-9087-856a4c63bdfc\" (UID: \"41540f64-cbe0-45c5-9087-856a4c63bdfc\") " Jul 14 10:04:49.189181 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:49.189154 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41540f64-cbe0-45c5-9087-856a4c63bdfc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "41540f64-cbe0-45c5-9087-856a4c63bdfc" (UID: "41540f64-cbe0-45c5-9087-856a4c63bdfc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 10:04:49.289654 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:49.289574 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41540f64-cbe0-45c5-9087-856a4c63bdfc-maas-api-service-ca\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 10:04:49.977337 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:49.977308 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b26ed2-wrcng" Jul 14 10:04:49.977337 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:49.977324 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b26ed2-wrcng" event={"ID":"41540f64-cbe0-45c5-9087-856a4c63bdfc","Type":"ContainerDied","Data":"027ddb0415db3b3f3d7f2fff57e55633c8a85fb6422d70ebda7a247d89abcc8e"} Jul 14 10:04:49.977548 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:49.977365 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="027ddb0415db3b3f3d7f2fff57e55633c8a85fb6422d70ebda7a247d89abcc8e" Jul 14 10:04:57.342142 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:57.342109 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b26ed2-684c9569c9-ddwbj"] Jul 14 10:04:57.342552 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:57.342303 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b26ed2-684c9569c9-ddwbj" podUID="31a71677-52ff-4104-87cf-cb89575a64c7" containerName="maas-api" containerID="cri-o://afe725bd058f748f1e8982e8bc0df06664ee7eeef80fe20413b3e48f0817bb6b" gracePeriod=30 Jul 14 10:04:57.612793 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:57.612765 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b26ed2-684c9569c9-ddwbj" Jul 14 10:04:57.771760 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:57.771726 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31a71677-52ff-4104-87cf-cb89575a64c7-openshift-service-ca\") pod \"31a71677-52ff-4104-87cf-cb89575a64c7\" (UID: \"31a71677-52ff-4104-87cf-cb89575a64c7\") " Jul 14 10:04:57.771967 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:57.771785 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qfhd\" (UniqueName: \"kubernetes.io/projected/31a71677-52ff-4104-87cf-cb89575a64c7-kube-api-access-2qfhd\") pod \"31a71677-52ff-4104-87cf-cb89575a64c7\" (UID: \"31a71677-52ff-4104-87cf-cb89575a64c7\") " Jul 14 10:04:57.771967 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:57.771876 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31a71677-52ff-4104-87cf-cb89575a64c7-maas-api-tls\") pod \"31a71677-52ff-4104-87cf-cb89575a64c7\" (UID: \"31a71677-52ff-4104-87cf-cb89575a64c7\") " Jul 14 10:04:57.772121 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:57.772096 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31a71677-52ff-4104-87cf-cb89575a64c7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "31a71677-52ff-4104-87cf-cb89575a64c7" (UID: "31a71677-52ff-4104-87cf-cb89575a64c7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 10:04:57.772209 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:57.772193 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31a71677-52ff-4104-87cf-cb89575a64c7-openshift-service-ca\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 10:04:57.773881 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:57.773855 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31a71677-52ff-4104-87cf-cb89575a64c7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "31a71677-52ff-4104-87cf-cb89575a64c7" (UID: "31a71677-52ff-4104-87cf-cb89575a64c7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 10:04:57.774002 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:57.773921 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31a71677-52ff-4104-87cf-cb89575a64c7-kube-api-access-2qfhd" (OuterVolumeSpecName: "kube-api-access-2qfhd") pod "31a71677-52ff-4104-87cf-cb89575a64c7" (UID: "31a71677-52ff-4104-87cf-cb89575a64c7"). InnerVolumeSpecName "kube-api-access-2qfhd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 10:04:57.873504 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:57.873423 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qfhd\" (UniqueName: \"kubernetes.io/projected/31a71677-52ff-4104-87cf-cb89575a64c7-kube-api-access-2qfhd\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 10:04:57.873504 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:57.873450 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31a71677-52ff-4104-87cf-cb89575a64c7-maas-api-tls\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 10:04:58.028374 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:58.028344 2565 generic.go:356] "Generic (PLEG): container finished" podID="31a71677-52ff-4104-87cf-cb89575a64c7" containerID="afe725bd058f748f1e8982e8bc0df06664ee7eeef80fe20413b3e48f0817bb6b" exitCode=0 Jul 14 10:04:58.028583 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:58.028399 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b26ed2-684c9569c9-ddwbj" Jul 14 10:04:58.028583 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:58.028443 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b26ed2-684c9569c9-ddwbj" event={"ID":"31a71677-52ff-4104-87cf-cb89575a64c7","Type":"ContainerDied","Data":"afe725bd058f748f1e8982e8bc0df06664ee7eeef80fe20413b3e48f0817bb6b"} Jul 14 10:04:58.028583 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:58.028496 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b26ed2-684c9569c9-ddwbj" event={"ID":"31a71677-52ff-4104-87cf-cb89575a64c7","Type":"ContainerDied","Data":"5a31692427c147b76361395e371d3613e26c9fa93f601b563c29c6c21c9ec7fb"} Jul 14 10:04:58.028583 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:58.028520 2565 scope.go:117] "RemoveContainer" containerID="afe725bd058f748f1e8982e8bc0df06664ee7eeef80fe20413b3e48f0817bb6b" Jul 14 10:04:58.038978 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:58.038960 2565 scope.go:117] "RemoveContainer" containerID="afe725bd058f748f1e8982e8bc0df06664ee7eeef80fe20413b3e48f0817bb6b" Jul 14 10:04:58.039256 ip-10-0-132-2 kubenswrapper[2565]: E0714 10:04:58.039234 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afe725bd058f748f1e8982e8bc0df06664ee7eeef80fe20413b3e48f0817bb6b\": container with ID starting with afe725bd058f748f1e8982e8bc0df06664ee7eeef80fe20413b3e48f0817bb6b not found: ID does not exist" containerID="afe725bd058f748f1e8982e8bc0df06664ee7eeef80fe20413b3e48f0817bb6b" Jul 14 10:04:58.039310 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:58.039265 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afe725bd058f748f1e8982e8bc0df06664ee7eeef80fe20413b3e48f0817bb6b"} err="failed to get container status \"afe725bd058f748f1e8982e8bc0df06664ee7eeef80fe20413b3e48f0817bb6b\": rpc error: code = NotFound desc = could not find container \"afe725bd058f748f1e8982e8bc0df06664ee7eeef80fe20413b3e48f0817bb6b\": container with ID starting with afe725bd058f748f1e8982e8bc0df06664ee7eeef80fe20413b3e48f0817bb6b not found: ID does not exist" Jul 14 10:04:58.053055 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:58.053026 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b26ed2-684c9569c9-ddwbj"] Jul 14 10:04:58.057671 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:58.057647 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b26ed2-684c9569c9-ddwbj"] Jul 14 10:04:58.970172 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:04:58.970135 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31a71677-52ff-4104-87cf-cb89575a64c7" path="/var/lib/kubelet/pods/31a71677-52ff-4104-87cf-cb89575a64c7/volumes" Jul 14 10:05:12.605853 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:12.605813 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6cb46d-ghvcm"] Jul 14 10:05:12.606344 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:12.606308 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31a71677-52ff-4104-87cf-cb89575a64c7" containerName="maas-api" Jul 14 10:05:12.606344 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:12.606320 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="31a71677-52ff-4104-87cf-cb89575a64c7" containerName="maas-api" Jul 14 10:05:12.606344 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:12.606333 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41540f64-cbe0-45c5-9087-856a4c63bdfc" containerName="revoke-keys" Jul 14 10:05:12.606344 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:12.606338 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="41540f64-cbe0-45c5-9087-856a4c63bdfc" containerName="revoke-keys" Jul 14 10:05:12.606488 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:12.606398 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="41540f64-cbe0-45c5-9087-856a4c63bdfc" containerName="revoke-keys" Jul 14 10:05:12.606488 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:12.606410 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="31a71677-52ff-4104-87cf-cb89575a64c7" containerName="maas-api" Jul 14 10:05:12.609624 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:12.609607 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6cb46d-ghvcm" Jul 14 10:05:12.612274 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:12.612250 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8m76v\"" Jul 14 10:05:12.618353 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:12.618324 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6cb46d-ghvcm"] Jul 14 10:05:12.719692 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:12.719654 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5c5a23c-6742-4da3-92a2-131898d57a64-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6cb46d-ghvcm\" (UID: \"d5c5a23c-6742-4da3-92a2-131898d57a64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6cb46d-ghvcm" Jul 14 10:05:12.821285 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:12.821241 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5c5a23c-6742-4da3-92a2-131898d57a64-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6cb46d-ghvcm\" (UID: \"d5c5a23c-6742-4da3-92a2-131898d57a64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6cb46d-ghvcm" Jul 14 10:05:12.821976 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:12.821952 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5c5a23c-6742-4da3-92a2-131898d57a64-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6cb46d-ghvcm\" (UID: \"d5c5a23c-6742-4da3-92a2-131898d57a64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6cb46d-ghvcm" Jul 14 10:05:12.921700 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:12.921663 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6cb46d-ghvcm" Jul 14 10:05:13.065594 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:13.065567 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6cb46d-ghvcm"] Jul 14 10:05:13.066459 ip-10-0-132-2 kubenswrapper[2565]: W0714 10:05:13.066438 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5c5a23c_6742_4da3_92a2_131898d57a64.slice/crio-01d5aa1fbd3fe04f5e8fc3ecc73aed4594ab7adf2b038fa1c1f92bd8fd2b3d1e WatchSource:0}: Error finding container 01d5aa1fbd3fe04f5e8fc3ecc73aed4594ab7adf2b038fa1c1f92bd8fd2b3d1e: Status 404 returned error can't find the container with id 01d5aa1fbd3fe04f5e8fc3ecc73aed4594ab7adf2b038fa1c1f92bd8fd2b3d1e Jul 14 10:05:13.122475 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:13.122438 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6cb46d-ghvcm" event={"ID":"d5c5a23c-6742-4da3-92a2-131898d57a64","Type":"ContainerStarted","Data":"01d5aa1fbd3fe04f5e8fc3ecc73aed4594ab7adf2b038fa1c1f92bd8fd2b3d1e"} Jul 14 10:05:14.130934 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:14.130842 2565 generic.go:356] "Generic (PLEG): container finished" podID="d5c5a23c-6742-4da3-92a2-131898d57a64" containerID="e8d925e7a4daec53f4fd656f326f1f29e52b0549aa370a628eef9ec8d349ce00" exitCode=0 Jul 14 10:05:14.130934 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:14.130890 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6cb46d-ghvcm" event={"ID":"d5c5a23c-6742-4da3-92a2-131898d57a64","Type":"ContainerDied","Data":"e8d925e7a4daec53f4fd656f326f1f29e52b0549aa370a628eef9ec8d349ce00"} Jul 14 10:05:15.273050 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:15.273019 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6cb46d-ghvcm" Jul 14 10:05:15.349116 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:15.349084 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5c5a23c-6742-4da3-92a2-131898d57a64-maas-api-service-ca\") pod \"d5c5a23c-6742-4da3-92a2-131898d57a64\" (UID: \"d5c5a23c-6742-4da3-92a2-131898d57a64\") " Jul 14 10:05:15.349418 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:15.349396 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5c5a23c-6742-4da3-92a2-131898d57a64-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d5c5a23c-6742-4da3-92a2-131898d57a64" (UID: "d5c5a23c-6742-4da3-92a2-131898d57a64"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 10:05:15.450218 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:15.450138 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5c5a23c-6742-4da3-92a2-131898d57a64-maas-api-service-ca\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 10:05:16.143796 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:16.143757 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6cb46d-ghvcm" event={"ID":"d5c5a23c-6742-4da3-92a2-131898d57a64","Type":"ContainerDied","Data":"01d5aa1fbd3fe04f5e8fc3ecc73aed4594ab7adf2b038fa1c1f92bd8fd2b3d1e"} Jul 14 10:05:16.144036 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:16.143826 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01d5aa1fbd3fe04f5e8fc3ecc73aed4594ab7adf2b038fa1c1f92bd8fd2b3d1e" Jul 14 10:05:16.144036 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:16.143769 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6cb46d-ghvcm" Jul 14 10:05:23.040399 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:23.040367 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6cb46d-6bd44b4644-flzks"] Jul 14 10:05:23.040896 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:23.040617 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6cb46d-6bd44b4644-flzks" podUID="38039008-3538-4120-91d2-ef47646afa98" containerName="maas-api" containerID="cri-o://dabb7f3814cc1707f1a4b3bea1eacff4e759f83b46bce33929f961a79d6b288d" gracePeriod=30 Jul 14 10:05:23.187888 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:23.187851 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6cb46d-6bd44b4644-flzks" event={"ID":"38039008-3538-4120-91d2-ef47646afa98","Type":"ContainerDied","Data":"dabb7f3814cc1707f1a4b3bea1eacff4e759f83b46bce33929f961a79d6b288d"} Jul 14 10:05:23.188086 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:23.187799 2565 generic.go:356] "Generic (PLEG): container finished" podID="38039008-3538-4120-91d2-ef47646afa98" containerID="dabb7f3814cc1707f1a4b3bea1eacff4e759f83b46bce33929f961a79d6b288d" exitCode=0 Jul 14 10:05:23.312750 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:23.312725 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6cb46d-6bd44b4644-flzks" Jul 14 10:05:23.427677 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:23.427644 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38039008-3538-4120-91d2-ef47646afa98-openshift-service-ca\") pod \"38039008-3538-4120-91d2-ef47646afa98\" (UID: \"38039008-3538-4120-91d2-ef47646afa98\") " Jul 14 10:05:23.427874 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:23.427699 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ktklc\" (UniqueName: \"kubernetes.io/projected/38039008-3538-4120-91d2-ef47646afa98-kube-api-access-ktklc\") pod \"38039008-3538-4120-91d2-ef47646afa98\" (UID: \"38039008-3538-4120-91d2-ef47646afa98\") " Jul 14 10:05:23.427874 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:23.427846 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38039008-3538-4120-91d2-ef47646afa98-maas-api-tls\") pod \"38039008-3538-4120-91d2-ef47646afa98\" (UID: \"38039008-3538-4120-91d2-ef47646afa98\") " Jul 14 10:05:23.428083 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:23.428061 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38039008-3538-4120-91d2-ef47646afa98-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "38039008-3538-4120-91d2-ef47646afa98" (UID: "38039008-3538-4120-91d2-ef47646afa98"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 10:05:23.428175 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:23.428153 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38039008-3538-4120-91d2-ef47646afa98-openshift-service-ca\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 10:05:23.429852 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:23.429798 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38039008-3538-4120-91d2-ef47646afa98-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "38039008-3538-4120-91d2-ef47646afa98" (UID: "38039008-3538-4120-91d2-ef47646afa98"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 10:05:23.429852 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:23.429836 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38039008-3538-4120-91d2-ef47646afa98-kube-api-access-ktklc" (OuterVolumeSpecName: "kube-api-access-ktklc") pod "38039008-3538-4120-91d2-ef47646afa98" (UID: "38039008-3538-4120-91d2-ef47646afa98"). InnerVolumeSpecName "kube-api-access-ktklc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 10:05:23.529363 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:23.529329 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38039008-3538-4120-91d2-ef47646afa98-maas-api-tls\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 10:05:23.529363 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:23.529358 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktklc\" (UniqueName: \"kubernetes.io/projected/38039008-3538-4120-91d2-ef47646afa98-kube-api-access-ktklc\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 10:05:24.195308 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:24.195278 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6cb46d-6bd44b4644-flzks" Jul 14 10:05:24.195308 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:24.195296 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6cb46d-6bd44b4644-flzks" event={"ID":"38039008-3538-4120-91d2-ef47646afa98","Type":"ContainerDied","Data":"f8218212e1d2741bb262f7c5387b8e59be7ae92161adc547f7f9b4d1363c15a6"} Jul 14 10:05:24.195758 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:24.195363 2565 scope.go:117] "RemoveContainer" containerID="dabb7f3814cc1707f1a4b3bea1eacff4e759f83b46bce33929f961a79d6b288d" Jul 14 10:05:24.226686 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:24.226659 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6cb46d-6bd44b4644-flzks"] Jul 14 10:05:24.229356 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:24.229335 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6cb46d-6bd44b4644-flzks"] Jul 14 10:05:24.969378 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:24.969343 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38039008-3538-4120-91d2-ef47646afa98" path="/var/lib/kubelet/pods/38039008-3538-4120-91d2-ef47646afa98/volumes" Jul 14 10:05:42.487005 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:42.486925 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-5d32fd-5f7b88fc59-2272f"] Jul 14 10:05:42.487370 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:42.487356 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38039008-3538-4120-91d2-ef47646afa98" containerName="maas-api" Jul 14 10:05:42.487370 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:42.487367 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="38039008-3538-4120-91d2-ef47646afa98" containerName="maas-api" Jul 14 10:05:42.487691 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:42.487390 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5c5a23c-6742-4da3-92a2-131898d57a64" containerName="revoke-keys" Jul 14 10:05:42.487691 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:42.487396 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5c5a23c-6742-4da3-92a2-131898d57a64" containerName="revoke-keys" Jul 14 10:05:42.487691 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:42.487464 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="38039008-3538-4120-91d2-ef47646afa98" containerName="maas-api" Jul 14 10:05:42.487691 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:42.487477 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5c5a23c-6742-4da3-92a2-131898d57a64" containerName="revoke-keys" Jul 14 10:05:42.490443 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:42.490425 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5d32fd-5f7b88fc59-2272f" Jul 14 10:05:42.493057 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:42.493029 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-5d32fd-serving-cert\"" Jul 14 10:05:42.504099 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:42.504064 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-5d32fd-5f7b88fc59-2272f"] Jul 14 10:05:42.521127 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:42.521097 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/462dd0d8-b00b-407d-84a1-2bf66cd4e58f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-5d32fd-5f7b88fc59-2272f\" (UID: \"462dd0d8-b00b-407d-84a1-2bf66cd4e58f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5d32fd-5f7b88fc59-2272f" Jul 14 10:05:42.521267 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:42.521159 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/462dd0d8-b00b-407d-84a1-2bf66cd4e58f-maas-api-tls\") pod \"maas-api-e2e-shared-a-5d32fd-5f7b88fc59-2272f\" (UID: \"462dd0d8-b00b-407d-84a1-2bf66cd4e58f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5d32fd-5f7b88fc59-2272f" Jul 14 10:05:42.521267 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:42.521207 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zf8n\" (UniqueName: \"kubernetes.io/projected/462dd0d8-b00b-407d-84a1-2bf66cd4e58f-kube-api-access-6zf8n\") pod \"maas-api-e2e-shared-a-5d32fd-5f7b88fc59-2272f\" (UID: \"462dd0d8-b00b-407d-84a1-2bf66cd4e58f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5d32fd-5f7b88fc59-2272f" Jul 14 10:05:42.629076 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:42.629044 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/462dd0d8-b00b-407d-84a1-2bf66cd4e58f-maas-api-tls\") pod \"maas-api-e2e-shared-a-5d32fd-5f7b88fc59-2272f\" (UID: \"462dd0d8-b00b-407d-84a1-2bf66cd4e58f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5d32fd-5f7b88fc59-2272f" Jul 14 10:05:42.629315 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:42.629095 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zf8n\" (UniqueName: \"kubernetes.io/projected/462dd0d8-b00b-407d-84a1-2bf66cd4e58f-kube-api-access-6zf8n\") pod \"maas-api-e2e-shared-a-5d32fd-5f7b88fc59-2272f\" (UID: \"462dd0d8-b00b-407d-84a1-2bf66cd4e58f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5d32fd-5f7b88fc59-2272f" Jul 14 10:05:42.629315 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:42.629176 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/462dd0d8-b00b-407d-84a1-2bf66cd4e58f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-5d32fd-5f7b88fc59-2272f\" (UID: \"462dd0d8-b00b-407d-84a1-2bf66cd4e58f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5d32fd-5f7b88fc59-2272f" Jul 14 10:05:42.629315 ip-10-0-132-2 kubenswrapper[2565]: E0714 10:05:42.629213 2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-5d32fd-serving-cert: secret "maas-api-e2e-shared-a-5d32fd-serving-cert" not found Jul 14 10:05:42.629315 ip-10-0-132-2 kubenswrapper[2565]: E0714 10:05:42.629296 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/462dd0d8-b00b-407d-84a1-2bf66cd4e58f-maas-api-tls podName:462dd0d8-b00b-407d-84a1-2bf66cd4e58f nodeName:}" failed. No retries permitted until 2026-07-14 10:05:43.129275758 +0000 UTC m=+2332.638507184 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/462dd0d8-b00b-407d-84a1-2bf66cd4e58f-maas-api-tls") pod "maas-api-e2e-shared-a-5d32fd-5f7b88fc59-2272f" (UID: "462dd0d8-b00b-407d-84a1-2bf66cd4e58f") : secret "maas-api-e2e-shared-a-5d32fd-serving-cert" not found Jul 14 10:05:42.629739 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:42.629723 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/462dd0d8-b00b-407d-84a1-2bf66cd4e58f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-5d32fd-5f7b88fc59-2272f\" (UID: \"462dd0d8-b00b-407d-84a1-2bf66cd4e58f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5d32fd-5f7b88fc59-2272f" Jul 14 10:05:42.642797 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:42.642768 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zf8n\" (UniqueName: \"kubernetes.io/projected/462dd0d8-b00b-407d-84a1-2bf66cd4e58f-kube-api-access-6zf8n\") pod \"maas-api-e2e-shared-a-5d32fd-5f7b88fc59-2272f\" (UID: \"462dd0d8-b00b-407d-84a1-2bf66cd4e58f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5d32fd-5f7b88fc59-2272f" Jul 14 10:05:43.133711 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:43.133677 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/462dd0d8-b00b-407d-84a1-2bf66cd4e58f-maas-api-tls\") pod \"maas-api-e2e-shared-a-5d32fd-5f7b88fc59-2272f\" (UID: \"462dd0d8-b00b-407d-84a1-2bf66cd4e58f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5d32fd-5f7b88fc59-2272f" Jul 14 10:05:43.136414 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:43.136378 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/462dd0d8-b00b-407d-84a1-2bf66cd4e58f-maas-api-tls\") pod \"maas-api-e2e-shared-a-5d32fd-5f7b88fc59-2272f\" (UID: \"462dd0d8-b00b-407d-84a1-2bf66cd4e58f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5d32fd-5f7b88fc59-2272f" Jul 14 10:05:43.401216 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:43.401183 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5d32fd-5f7b88fc59-2272f" Jul 14 10:05:43.560209 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:43.560182 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-5d32fd-5f7b88fc59-2272f"] Jul 14 10:05:43.561953 ip-10-0-132-2 kubenswrapper[2565]: W0714 10:05:43.561926 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod462dd0d8_b00b_407d_84a1_2bf66cd4e58f.slice/crio-454d4d8b64150e8a510b2517d92d60d3f6d37b7a8b7be211c46426f02ce75ff3 WatchSource:0}: Error finding container 454d4d8b64150e8a510b2517d92d60d3f6d37b7a8b7be211c46426f02ce75ff3: Status 404 returned error can't find the container with id 454d4d8b64150e8a510b2517d92d60d3f6d37b7a8b7be211c46426f02ce75ff3 Jul 14 10:05:44.317975 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:44.317940 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5d32fd-5f7b88fc59-2272f" event={"ID":"462dd0d8-b00b-407d-84a1-2bf66cd4e58f","Type":"ContainerStarted","Data":"454d4d8b64150e8a510b2517d92d60d3f6d37b7a8b7be211c46426f02ce75ff3"} Jul 14 10:05:45.324951 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:45.324911 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5d32fd-5f7b88fc59-2272f" event={"ID":"462dd0d8-b00b-407d-84a1-2bf66cd4e58f","Type":"ContainerStarted","Data":"57c6d998c45cd5e0bffcc1a74dfed609d52d340ace67d4bcd7705424a3c4ea01"} Jul 14 10:05:45.325448 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:45.325029 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5d32fd-5f7b88fc59-2272f" Jul 14 10:05:45.344954 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:45.344907 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5d32fd-5f7b88fc59-2272f" podStartSLOduration=2.267801299 podStartE2EDuration="3.344892924s" totalImagesPullingTime="1.077091625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 10:05:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 10:05:45.341616296 +0000 UTC m=+2334.850847760" watchObservedRunningTime="2026-07-14 10:05:45.344892924 +0000 UTC m=+2334.854124373" Jul 14 10:05:50.461060 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:50.461018 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ad5ead-7bb88b7bb8-wkqks"] Jul 14 10:05:50.464987 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:50.464966 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ad5ead-7bb88b7bb8-wkqks" Jul 14 10:05:50.469340 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:50.469318 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-ad5ead-serving-cert\"" Jul 14 10:05:50.477734 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:50.477710 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ad5ead-7bb88b7bb8-wkqks"] Jul 14 10:05:50.610900 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:50.610864 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7167fa9f-5153-4d52-9e75-f6a4b43a78f1-maas-api-tls\") pod \"maas-api-e2e-shared-b-ad5ead-7bb88b7bb8-wkqks\" (UID: \"7167fa9f-5153-4d52-9e75-f6a4b43a78f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ad5ead-7bb88b7bb8-wkqks" Jul 14 10:05:50.611089 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:50.610940 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7167fa9f-5153-4d52-9e75-f6a4b43a78f1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-ad5ead-7bb88b7bb8-wkqks\" (UID: \"7167fa9f-5153-4d52-9e75-f6a4b43a78f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ad5ead-7bb88b7bb8-wkqks" Jul 14 10:05:50.611089 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:50.611006 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnzj2\" (UniqueName: \"kubernetes.io/projected/7167fa9f-5153-4d52-9e75-f6a4b43a78f1-kube-api-access-mnzj2\") pod \"maas-api-e2e-shared-b-ad5ead-7bb88b7bb8-wkqks\" (UID: \"7167fa9f-5153-4d52-9e75-f6a4b43a78f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ad5ead-7bb88b7bb8-wkqks" Jul 14 10:05:50.712470 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:50.712385 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnzj2\" (UniqueName: \"kubernetes.io/projected/7167fa9f-5153-4d52-9e75-f6a4b43a78f1-kube-api-access-mnzj2\") pod \"maas-api-e2e-shared-b-ad5ead-7bb88b7bb8-wkqks\" (UID: \"7167fa9f-5153-4d52-9e75-f6a4b43a78f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ad5ead-7bb88b7bb8-wkqks" Jul 14 10:05:50.712470 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:50.712427 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7167fa9f-5153-4d52-9e75-f6a4b43a78f1-maas-api-tls\") pod \"maas-api-e2e-shared-b-ad5ead-7bb88b7bb8-wkqks\" (UID: \"7167fa9f-5153-4d52-9e75-f6a4b43a78f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ad5ead-7bb88b7bb8-wkqks" Jul 14 10:05:50.712661 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:50.712492 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7167fa9f-5153-4d52-9e75-f6a4b43a78f1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-ad5ead-7bb88b7bb8-wkqks\" (UID: \"7167fa9f-5153-4d52-9e75-f6a4b43a78f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ad5ead-7bb88b7bb8-wkqks" Jul 14 10:05:50.713220 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:50.713187 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7167fa9f-5153-4d52-9e75-f6a4b43a78f1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-ad5ead-7bb88b7bb8-wkqks\" (UID: \"7167fa9f-5153-4d52-9e75-f6a4b43a78f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ad5ead-7bb88b7bb8-wkqks" Jul 14 10:05:50.714945 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:50.714925 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7167fa9f-5153-4d52-9e75-f6a4b43a78f1-maas-api-tls\") pod \"maas-api-e2e-shared-b-ad5ead-7bb88b7bb8-wkqks\" (UID: \"7167fa9f-5153-4d52-9e75-f6a4b43a78f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ad5ead-7bb88b7bb8-wkqks" Jul 14 10:05:50.723303 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:50.723285 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnzj2\" (UniqueName: \"kubernetes.io/projected/7167fa9f-5153-4d52-9e75-f6a4b43a78f1-kube-api-access-mnzj2\") pod \"maas-api-e2e-shared-b-ad5ead-7bb88b7bb8-wkqks\" (UID: \"7167fa9f-5153-4d52-9e75-f6a4b43a78f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ad5ead-7bb88b7bb8-wkqks" Jul 14 10:05:50.776571 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:50.776533 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ad5ead-7bb88b7bb8-wkqks" Jul 14 10:05:50.917173 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:50.917141 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ad5ead-7bb88b7bb8-wkqks"] Jul 14 10:05:50.918145 ip-10-0-132-2 kubenswrapper[2565]: W0714 10:05:50.918119 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7167fa9f_5153_4d52_9e75_f6a4b43a78f1.slice/crio-2d52e6b41a95ba847f4b564e0d889f752c4942f541e1162f8e812feb0ae5c00b WatchSource:0}: Error finding container 2d52e6b41a95ba847f4b564e0d889f752c4942f541e1162f8e812feb0ae5c00b: Status 404 returned error can't find the container with id 2d52e6b41a95ba847f4b564e0d889f752c4942f541e1162f8e812feb0ae5c00b Jul 14 10:05:51.336980 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:51.336952 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5d32fd-5f7b88fc59-2272f" Jul 14 10:05:51.362685 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:51.362651 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ad5ead-7bb88b7bb8-wkqks" event={"ID":"7167fa9f-5153-4d52-9e75-f6a4b43a78f1","Type":"ContainerStarted","Data":"2d52e6b41a95ba847f4b564e0d889f752c4942f541e1162f8e812feb0ae5c00b"} Jul 14 10:05:52.370103 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:52.370062 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ad5ead-7bb88b7bb8-wkqks" event={"ID":"7167fa9f-5153-4d52-9e75-f6a4b43a78f1","Type":"ContainerStarted","Data":"4cc4381dd7f15ca2f152456967a08a8be1c5fadc9435c6ccb19ac61ff980d129"} Jul 14 10:05:52.370505 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:52.370131 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ad5ead-7bb88b7bb8-wkqks" Jul 14 10:05:52.397489 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:52.397433 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ad5ead-7bb88b7bb8-wkqks" podStartSLOduration=1.496956162 podStartE2EDuration="2.397415942s" totalImagesPullingTime="900.45978ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 10:05:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 10:05:52.395408024 +0000 UTC m=+2341.904639475" watchObservedRunningTime="2026-07-14 10:05:52.397415942 +0000 UTC m=+2341.906647390" Jul 14 10:05:58.381411 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:05:58.381380 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ad5ead-7bb88b7bb8-wkqks" Jul 14 10:06:40.337629 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:06:40.337596 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-ad5ead/auth-test-model-ad5ead-kserve-5f7b684f9-l5cgp"] Jul 14 10:06:40.341279 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:06:40.341263 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ad5ead/auth-test-model-ad5ead-kserve-5f7b684f9-l5cgp" Jul 14 10:06:40.344516 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:06:40.344487 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ad5ead\"/\"auth-test-model-ad5ead-kserve-self-signed-certs\"" Jul 14 10:06:40.344698 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:06:40.344534 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ad5ead\"/\"kube-root-ca.crt\"" Jul 14 10:06:40.344698 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:06:40.344547 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ad5ead\"/\"default-dockercfg-gz8nm\"" Jul 14 10:06:40.344854 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:06:40.344840 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ad5ead\"/\"openshift-service-ca.crt\"" Jul 14 10:06:40.351083 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:06:40.351006 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ad5ead/auth-test-model-ad5ead-kserve-5f7b684f9-l5cgp"] Jul 14 10:06:40.516633 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:06:40.516595 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97ee5208-dac9-47ee-a880-58b65264b0b0-tmp-dir\") pod \"auth-test-model-ad5ead-kserve-5f7b684f9-l5cgp\" (UID: \"97ee5208-dac9-47ee-a880-58b65264b0b0\") " pod="ai-tenant-e2e-shared-b-ad5ead/auth-test-model-ad5ead-kserve-5f7b684f9-l5cgp" Jul 14 10:06:40.516633 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:06:40.516638 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97ee5208-dac9-47ee-a880-58b65264b0b0-model-cache\") pod \"auth-test-model-ad5ead-kserve-5f7b684f9-l5cgp\" (UID: \"97ee5208-dac9-47ee-a880-58b65264b0b0\") " pod="ai-tenant-e2e-shared-b-ad5ead/auth-test-model-ad5ead-kserve-5f7b684f9-l5cgp" Jul 14 10:06:40.516949 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:06:40.516662 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97ee5208-dac9-47ee-a880-58b65264b0b0-tls-certs\") pod \"auth-test-model-ad5ead-kserve-5f7b684f9-l5cgp\" (UID: \"97ee5208-dac9-47ee-a880-58b65264b0b0\") " pod="ai-tenant-e2e-shared-b-ad5ead/auth-test-model-ad5ead-kserve-5f7b684f9-l5cgp" Jul 14 10:06:40.516949 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:06:40.516725 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwxc8\" (UniqueName: \"kubernetes.io/projected/97ee5208-dac9-47ee-a880-58b65264b0b0-kube-api-access-wwxc8\") pod \"auth-test-model-ad5ead-kserve-5f7b684f9-l5cgp\" (UID: \"97ee5208-dac9-47ee-a880-58b65264b0b0\") " pod="ai-tenant-e2e-shared-b-ad5ead/auth-test-model-ad5ead-kserve-5f7b684f9-l5cgp" Jul 14 10:06:40.516949 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:06:40.516835 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97ee5208-dac9-47ee-a880-58b65264b0b0-home\") pod \"auth-test-model-ad5ead-kserve-5f7b684f9-l5cgp\" (UID: \"97ee5208-dac9-47ee-a880-58b65264b0b0\") " pod="ai-tenant-e2e-shared-b-ad5ead/auth-test-model-ad5ead-kserve-5f7b684f9-l5cgp" Jul 14 10:06:40.516949 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:06:40.516901 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97ee5208-dac9-47ee-a880-58b65264b0b0-dshm\") pod \"auth-test-model-ad5ead-kserve-5f7b684f9-l5cgp\" (UID: \"97ee5208-dac9-47ee-a880-58b65264b0b0\") " pod="ai-tenant-e2e-shared-b-ad5ead/auth-test-model-ad5ead-kserve-5f7b684f9-l5cgp" Jul 14 10:06:40.618221 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:06:40.618125 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97ee5208-dac9-47ee-a880-58b65264b0b0-model-cache\") pod \"auth-test-model-ad5ead-kserve-5f7b684f9-l5cgp\" (UID: \"97ee5208-dac9-47ee-a880-58b65264b0b0\") " pod="ai-tenant-e2e-shared-b-ad5ead/auth-test-model-ad5ead-kserve-5f7b684f9-l5cgp" Jul 14 10:06:40.618221 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:06:40.618173 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97ee5208-dac9-47ee-a880-58b65264b0b0-tls-certs\") pod \"auth-test-model-ad5ead-kserve-5f7b684f9-l5cgp\" (UID: \"97ee5208-dac9-47ee-a880-58b65264b0b0\") " pod="ai-tenant-e2e-shared-b-ad5ead/auth-test-model-ad5ead-kserve-5f7b684f9-l5cgp" Jul 14 10:06:40.618221 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:06:40.618200 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwxc8\" (UniqueName: \"kubernetes.io/projected/97ee5208-dac9-47ee-a880-58b65264b0b0-kube-api-access-wwxc8\") pod \"auth-test-model-ad5ead-kserve-5f7b684f9-l5cgp\" (UID: \"97ee5208-dac9-47ee-a880-58b65264b0b0\") " pod="ai-tenant-e2e-shared-b-ad5ead/auth-test-model-ad5ead-kserve-5f7b684f9-l5cgp" Jul 14 10:06:40.618522 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:06:40.618238 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97ee5208-dac9-47ee-a880-58b65264b0b0-home\") pod \"auth-test-model-ad5ead-kserve-5f7b684f9-l5cgp\" (UID: \"97ee5208-dac9-47ee-a880-58b65264b0b0\") " pod="ai-tenant-e2e-shared-b-ad5ead/auth-test-model-ad5ead-kserve-5f7b684f9-l5cgp" Jul 14 10:06:40.618522 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:06:40.618435 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97ee5208-dac9-47ee-a880-58b65264b0b0-dshm\") pod \"auth-test-model-ad5ead-kserve-5f7b684f9-l5cgp\" (UID: \"97ee5208-dac9-47ee-a880-58b65264b0b0\") " pod="ai-tenant-e2e-shared-b-ad5ead/auth-test-model-ad5ead-kserve-5f7b684f9-l5cgp" Jul 14 10:06:40.618632 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:06:40.618598 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97ee5208-dac9-47ee-a880-58b65264b0b0-model-cache\") pod \"auth-test-model-ad5ead-kserve-5f7b684f9-l5cgp\" (UID: \"97ee5208-dac9-47ee-a880-58b65264b0b0\") " pod="ai-tenant-e2e-shared-b-ad5ead/auth-test-model-ad5ead-kserve-5f7b684f9-l5cgp" Jul 14 10:06:40.618632 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:06:40.618618 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97ee5208-dac9-47ee-a880-58b65264b0b0-tmp-dir\") pod \"auth-test-model-ad5ead-kserve-5f7b684f9-l5cgp\" (UID: \"97ee5208-dac9-47ee-a880-58b65264b0b0\") " pod="ai-tenant-e2e-shared-b-ad5ead/auth-test-model-ad5ead-kserve-5f7b684f9-l5cgp" Jul 14 10:06:40.618632 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:06:40.618628 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97ee5208-dac9-47ee-a880-58b65264b0b0-home\") pod \"auth-test-model-ad5ead-kserve-5f7b684f9-l5cgp\" (UID: \"97ee5208-dac9-47ee-a880-58b65264b0b0\") " pod="ai-tenant-e2e-shared-b-ad5ead/auth-test-model-ad5ead-kserve-5f7b684f9-l5cgp" Jul 14 10:06:40.618935 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:06:40.618909 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97ee5208-dac9-47ee-a880-58b65264b0b0-tmp-dir\") pod \"auth-test-model-ad5ead-kserve-5f7b684f9-l5cgp\" (UID: \"97ee5208-dac9-47ee-a880-58b65264b0b0\") " pod="ai-tenant-e2e-shared-b-ad5ead/auth-test-model-ad5ead-kserve-5f7b684f9-l5cgp" Jul 14 10:06:40.620640 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:06:40.620611 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97ee5208-dac9-47ee-a880-58b65264b0b0-dshm\") pod \"auth-test-model-ad5ead-kserve-5f7b684f9-l5cgp\" (UID: \"97ee5208-dac9-47ee-a880-58b65264b0b0\") " pod="ai-tenant-e2e-shared-b-ad5ead/auth-test-model-ad5ead-kserve-5f7b684f9-l5cgp" Jul 14 10:06:40.620758 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:06:40.620717 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97ee5208-dac9-47ee-a880-58b65264b0b0-tls-certs\") pod \"auth-test-model-ad5ead-kserve-5f7b684f9-l5cgp\" (UID: \"97ee5208-dac9-47ee-a880-58b65264b0b0\") " pod="ai-tenant-e2e-shared-b-ad5ead/auth-test-model-ad5ead-kserve-5f7b684f9-l5cgp" Jul 14 10:06:40.627107 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:06:40.627084 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwxc8\" (UniqueName: \"kubernetes.io/projected/97ee5208-dac9-47ee-a880-58b65264b0b0-kube-api-access-wwxc8\") pod \"auth-test-model-ad5ead-kserve-5f7b684f9-l5cgp\" (UID: \"97ee5208-dac9-47ee-a880-58b65264b0b0\") " pod="ai-tenant-e2e-shared-b-ad5ead/auth-test-model-ad5ead-kserve-5f7b684f9-l5cgp" Jul 14 10:06:40.651855 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:06:40.651795 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ad5ead/auth-test-model-ad5ead-kserve-5f7b684f9-l5cgp" Jul 14 10:06:40.795476 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:06:40.795443 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ad5ead/auth-test-model-ad5ead-kserve-5f7b684f9-l5cgp"] Jul 14 10:06:40.797315 ip-10-0-132-2 kubenswrapper[2565]: W0714 10:06:40.797286 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97ee5208_dac9_47ee_a880_58b65264b0b0.slice/crio-75152790b03ce5a6a71c1cff67d30b37c3b77f0db8f339b88ab48823f931a168 WatchSource:0}: Error finding container 75152790b03ce5a6a71c1cff67d30b37c3b77f0db8f339b88ab48823f931a168: Status 404 returned error can't find the container with id 75152790b03ce5a6a71c1cff67d30b37c3b77f0db8f339b88ab48823f931a168 Jul 14 10:06:41.675058 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:06:41.675026 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ad5ead/auth-test-model-ad5ead-kserve-5f7b684f9-l5cgp" event={"ID":"97ee5208-dac9-47ee-a880-58b65264b0b0","Type":"ContainerStarted","Data":"746d0c96f6239f175f8ea84f7abd98892f0675c409e6e30adab7c30b77d05d21"} Jul 14 10:06:41.675058 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:06:41.675071 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ad5ead/auth-test-model-ad5ead-kserve-5f7b684f9-l5cgp" event={"ID":"97ee5208-dac9-47ee-a880-58b65264b0b0","Type":"ContainerStarted","Data":"75152790b03ce5a6a71c1cff67d30b37c3b77f0db8f339b88ab48823f931a168"} Jul 14 10:06:41.696822 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:06:41.696755 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-ad5ead/auth-test-model-ad5ead-kserve-5f7b684f9-l5cgp" podStartSLOduration=1.696740826 podStartE2EDuration="1.696740826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 10:06:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 10:06:41.69488706 +0000 UTC m=+2391.204118510" watchObservedRunningTime="2026-07-14 10:06:41.696740826 +0000 UTC m=+2391.205972273" Jul 14 10:06:50.652831 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:06:50.652770 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-ad5ead/auth-test-model-ad5ead-kserve-5f7b684f9-l5cgp" Jul 14 10:06:50.653259 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:06:50.652851 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-ad5ead/auth-test-model-ad5ead-kserve-5f7b684f9-l5cgp" Jul 14 10:06:50.665441 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:06:50.665413 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-ad5ead/auth-test-model-ad5ead-kserve-5f7b684f9-l5cgp" Jul 14 10:06:50.740340 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:06:50.740314 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-ad5ead/auth-test-model-ad5ead-kserve-5f7b684f9-l5cgp" Jul 14 10:08:45.535389 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:08:45.535311 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ad5ead/auth-test-model-ad5ead-kserve-5f7b684f9-l5cgp"] Jul 14 10:09:00.556669 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:00.556620 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-ad5ead/auth-test-model-ad5ead-kserve-5f7b684f9-l5cgp" podUID="97ee5208-dac9-47ee-a880-58b65264b0b0" containerName="main" containerID="cri-o://746d0c96f6239f175f8ea84f7abd98892f0675c409e6e30adab7c30b77d05d21" gracePeriod=45 Jul 14 10:09:00.730148 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:00.730109 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-ad5ead/auth-test-model-ad5ead-kserve-5f7b684f9-l5cgp" podUID="97ee5208-dac9-47ee-a880-58b65264b0b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": dial tcp 10.134.0.80:8000: connect: connection refused" Jul 14 10:09:00.817481 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:00.817416 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ad5ead/auth-test-model-ad5ead-kserve-5f7b684f9-l5cgp" Jul 14 10:09:00.886134 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:00.886103 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97ee5208-dac9-47ee-a880-58b65264b0b0-dshm\") pod \"97ee5208-dac9-47ee-a880-58b65264b0b0\" (UID: \"97ee5208-dac9-47ee-a880-58b65264b0b0\") " Jul 14 10:09:00.886300 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:00.886171 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97ee5208-dac9-47ee-a880-58b65264b0b0-model-cache\") pod \"97ee5208-dac9-47ee-a880-58b65264b0b0\" (UID: \"97ee5208-dac9-47ee-a880-58b65264b0b0\") " Jul 14 10:09:00.886300 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:00.886276 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97ee5208-dac9-47ee-a880-58b65264b0b0-tmp-dir\") pod \"97ee5208-dac9-47ee-a880-58b65264b0b0\" (UID: \"97ee5208-dac9-47ee-a880-58b65264b0b0\") " Jul 14 10:09:00.886374 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:00.886312 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97ee5208-dac9-47ee-a880-58b65264b0b0-tls-certs\") pod \"97ee5208-dac9-47ee-a880-58b65264b0b0\" (UID: \"97ee5208-dac9-47ee-a880-58b65264b0b0\") " Jul 14 10:09:00.886374 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:00.886340 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwxc8\" (UniqueName: \"kubernetes.io/projected/97ee5208-dac9-47ee-a880-58b65264b0b0-kube-api-access-wwxc8\") pod \"97ee5208-dac9-47ee-a880-58b65264b0b0\" (UID: \"97ee5208-dac9-47ee-a880-58b65264b0b0\") " Jul 14 10:09:00.886374 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:00.886362 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97ee5208-dac9-47ee-a880-58b65264b0b0-home\") pod \"97ee5208-dac9-47ee-a880-58b65264b0b0\" (UID: \"97ee5208-dac9-47ee-a880-58b65264b0b0\") " Jul 14 10:09:00.886520 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:00.886420 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97ee5208-dac9-47ee-a880-58b65264b0b0-model-cache" (OuterVolumeSpecName: "model-cache") pod "97ee5208-dac9-47ee-a880-58b65264b0b0" (UID: "97ee5208-dac9-47ee-a880-58b65264b0b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 10:09:00.886587 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:00.886541 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97ee5208-dac9-47ee-a880-58b65264b0b0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "97ee5208-dac9-47ee-a880-58b65264b0b0" (UID: "97ee5208-dac9-47ee-a880-58b65264b0b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 10:09:00.886700 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:00.886677 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97ee5208-dac9-47ee-a880-58b65264b0b0-home" (OuterVolumeSpecName: "home") pod "97ee5208-dac9-47ee-a880-58b65264b0b0" (UID: "97ee5208-dac9-47ee-a880-58b65264b0b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 10:09:00.886792 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:00.886774 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97ee5208-dac9-47ee-a880-58b65264b0b0-model-cache\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 10:09:00.886792 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:00.886792 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97ee5208-dac9-47ee-a880-58b65264b0b0-tmp-dir\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 10:09:00.886942 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:00.886800 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97ee5208-dac9-47ee-a880-58b65264b0b0-home\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 10:09:00.888507 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:00.888479 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97ee5208-dac9-47ee-a880-58b65264b0b0-dshm" (OuterVolumeSpecName: "dshm") pod "97ee5208-dac9-47ee-a880-58b65264b0b0" (UID: "97ee5208-dac9-47ee-a880-58b65264b0b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 10:09:00.888612 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:00.888503 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97ee5208-dac9-47ee-a880-58b65264b0b0-kube-api-access-wwxc8" (OuterVolumeSpecName: "kube-api-access-wwxc8") pod "97ee5208-dac9-47ee-a880-58b65264b0b0" (UID: "97ee5208-dac9-47ee-a880-58b65264b0b0"). InnerVolumeSpecName "kube-api-access-wwxc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 10:09:00.888662 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:00.888643 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97ee5208-dac9-47ee-a880-58b65264b0b0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "97ee5208-dac9-47ee-a880-58b65264b0b0" (UID: "97ee5208-dac9-47ee-a880-58b65264b0b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 10:09:00.988108 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:00.988072 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97ee5208-dac9-47ee-a880-58b65264b0b0-dshm\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 10:09:00.988108 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:00.988104 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97ee5208-dac9-47ee-a880-58b65264b0b0-tls-certs\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 10:09:00.988285 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:00.988118 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwxc8\" (UniqueName: \"kubernetes.io/projected/97ee5208-dac9-47ee-a880-58b65264b0b0-kube-api-access-wwxc8\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 10:09:01.512944 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:01.512902 2565 generic.go:356] "Generic (PLEG): container finished" podID="97ee5208-dac9-47ee-a880-58b65264b0b0" containerID="746d0c96f6239f175f8ea84f7abd98892f0675c409e6e30adab7c30b77d05d21" exitCode=0 Jul 14 10:09:01.512944 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:01.512938 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ad5ead/auth-test-model-ad5ead-kserve-5f7b684f9-l5cgp" Jul 14 10:09:01.513214 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:01.512980 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ad5ead/auth-test-model-ad5ead-kserve-5f7b684f9-l5cgp" event={"ID":"97ee5208-dac9-47ee-a880-58b65264b0b0","Type":"ContainerDied","Data":"746d0c96f6239f175f8ea84f7abd98892f0675c409e6e30adab7c30b77d05d21"} Jul 14 10:09:01.513214 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:01.513020 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ad5ead/auth-test-model-ad5ead-kserve-5f7b684f9-l5cgp" event={"ID":"97ee5208-dac9-47ee-a880-58b65264b0b0","Type":"ContainerDied","Data":"75152790b03ce5a6a71c1cff67d30b37c3b77f0db8f339b88ab48823f931a168"} Jul 14 10:09:01.513214 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:01.513037 2565 scope.go:117] "RemoveContainer" containerID="746d0c96f6239f175f8ea84f7abd98892f0675c409e6e30adab7c30b77d05d21" Jul 14 10:09:01.524577 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:01.524553 2565 scope.go:117] "RemoveContainer" containerID="746d0c96f6239f175f8ea84f7abd98892f0675c409e6e30adab7c30b77d05d21" Jul 14 10:09:01.525027 ip-10-0-132-2 kubenswrapper[2565]: E0714 10:09:01.525001 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"746d0c96f6239f175f8ea84f7abd98892f0675c409e6e30adab7c30b77d05d21\": container with ID starting with 746d0c96f6239f175f8ea84f7abd98892f0675c409e6e30adab7c30b77d05d21 not found: ID does not exist" containerID="746d0c96f6239f175f8ea84f7abd98892f0675c409e6e30adab7c30b77d05d21" Jul 14 10:09:01.525103 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:01.525040 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"746d0c96f6239f175f8ea84f7abd98892f0675c409e6e30adab7c30b77d05d21"} err="failed to get container status \"746d0c96f6239f175f8ea84f7abd98892f0675c409e6e30adab7c30b77d05d21\": rpc error: code = NotFound desc = could not find container \"746d0c96f6239f175f8ea84f7abd98892f0675c409e6e30adab7c30b77d05d21\": container with ID starting with 746d0c96f6239f175f8ea84f7abd98892f0675c409e6e30adab7c30b77d05d21 not found: ID does not exist" Jul 14 10:09:01.537107 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:01.537072 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ad5ead/auth-test-model-ad5ead-kserve-5f7b684f9-l5cgp"] Jul 14 10:09:01.542227 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:01.542202 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-ad5ead/auth-test-model-ad5ead-kserve-5f7b684f9-l5cgp"] Jul 14 10:09:02.969392 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:02.969356 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97ee5208-dac9-47ee-a880-58b65264b0b0" path="/var/lib/kubelet/pods/97ee5208-dac9-47ee-a880-58b65264b0b0/volumes" Jul 14 10:09:29.742255 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:29.742218 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-ad5ead/sub-test-model-ad5ead-kserve-7dcb57b467-px55w"] Jul 14 10:09:29.742941 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:29.742795 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97ee5208-dac9-47ee-a880-58b65264b0b0" containerName="main" Jul 14 10:09:29.742941 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:29.742829 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="97ee5208-dac9-47ee-a880-58b65264b0b0" containerName="main" Jul 14 10:09:29.742941 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:29.742943 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="97ee5208-dac9-47ee-a880-58b65264b0b0" containerName="main" Jul 14 10:09:29.746380 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:29.746358 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ad5ead/sub-test-model-ad5ead-kserve-7dcb57b467-px55w" Jul 14 10:09:29.750107 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:29.750081 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ad5ead\"/\"openshift-service-ca.crt\"" Jul 14 10:09:29.750256 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:29.750149 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ad5ead\"/\"kube-root-ca.crt\"" Jul 14 10:09:29.750256 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:29.750119 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ad5ead\"/\"sub-test-model-ad5ead-kserve-self-signed-certs\"" Jul 14 10:09:29.750397 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:29.750081 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ad5ead\"/\"default-dockercfg-gz8nm\"" Jul 14 10:09:29.758189 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:29.758165 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ad5ead/sub-test-model-ad5ead-kserve-7dcb57b467-px55w"] Jul 14 10:09:29.775158 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:29.775127 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e521bc8-cd01-4986-bb59-4358e185ae31-home\") pod \"sub-test-model-ad5ead-kserve-7dcb57b467-px55w\" (UID: \"9e521bc8-cd01-4986-bb59-4358e185ae31\") " pod="ai-tenant-e2e-shared-b-ad5ead/sub-test-model-ad5ead-kserve-7dcb57b467-px55w" Jul 14 10:09:29.775314 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:29.775167 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e521bc8-cd01-4986-bb59-4358e185ae31-model-cache\") pod \"sub-test-model-ad5ead-kserve-7dcb57b467-px55w\" (UID: \"9e521bc8-cd01-4986-bb59-4358e185ae31\") " pod="ai-tenant-e2e-shared-b-ad5ead/sub-test-model-ad5ead-kserve-7dcb57b467-px55w" Jul 14 10:09:29.775314 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:29.775214 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e521bc8-cd01-4986-bb59-4358e185ae31-tmp-dir\") pod \"sub-test-model-ad5ead-kserve-7dcb57b467-px55w\" (UID: \"9e521bc8-cd01-4986-bb59-4358e185ae31\") " pod="ai-tenant-e2e-shared-b-ad5ead/sub-test-model-ad5ead-kserve-7dcb57b467-px55w" Jul 14 10:09:29.775314 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:29.775231 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e521bc8-cd01-4986-bb59-4358e185ae31-tls-certs\") pod \"sub-test-model-ad5ead-kserve-7dcb57b467-px55w\" (UID: \"9e521bc8-cd01-4986-bb59-4358e185ae31\") " pod="ai-tenant-e2e-shared-b-ad5ead/sub-test-model-ad5ead-kserve-7dcb57b467-px55w" Jul 14 10:09:29.775314 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:29.775285 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmmcs\" (UniqueName: \"kubernetes.io/projected/9e521bc8-cd01-4986-bb59-4358e185ae31-kube-api-access-gmmcs\") pod \"sub-test-model-ad5ead-kserve-7dcb57b467-px55w\" (UID: \"9e521bc8-cd01-4986-bb59-4358e185ae31\") " pod="ai-tenant-e2e-shared-b-ad5ead/sub-test-model-ad5ead-kserve-7dcb57b467-px55w" Jul 14 10:09:29.775485 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:29.775347 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e521bc8-cd01-4986-bb59-4358e185ae31-dshm\") pod \"sub-test-model-ad5ead-kserve-7dcb57b467-px55w\" (UID: \"9e521bc8-cd01-4986-bb59-4358e185ae31\") " pod="ai-tenant-e2e-shared-b-ad5ead/sub-test-model-ad5ead-kserve-7dcb57b467-px55w" Jul 14 10:09:29.876335 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:29.876297 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e521bc8-cd01-4986-bb59-4358e185ae31-tmp-dir\") pod \"sub-test-model-ad5ead-kserve-7dcb57b467-px55w\" (UID: \"9e521bc8-cd01-4986-bb59-4358e185ae31\") " pod="ai-tenant-e2e-shared-b-ad5ead/sub-test-model-ad5ead-kserve-7dcb57b467-px55w" Jul 14 10:09:29.876335 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:29.876334 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e521bc8-cd01-4986-bb59-4358e185ae31-tls-certs\") pod \"sub-test-model-ad5ead-kserve-7dcb57b467-px55w\" (UID: \"9e521bc8-cd01-4986-bb59-4358e185ae31\") " pod="ai-tenant-e2e-shared-b-ad5ead/sub-test-model-ad5ead-kserve-7dcb57b467-px55w" Jul 14 10:09:29.876596 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:29.876358 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmmcs\" (UniqueName: \"kubernetes.io/projected/9e521bc8-cd01-4986-bb59-4358e185ae31-kube-api-access-gmmcs\") pod \"sub-test-model-ad5ead-kserve-7dcb57b467-px55w\" (UID: \"9e521bc8-cd01-4986-bb59-4358e185ae31\") " pod="ai-tenant-e2e-shared-b-ad5ead/sub-test-model-ad5ead-kserve-7dcb57b467-px55w" Jul 14 10:09:29.876596 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:29.876406 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e521bc8-cd01-4986-bb59-4358e185ae31-dshm\") pod \"sub-test-model-ad5ead-kserve-7dcb57b467-px55w\" (UID: \"9e521bc8-cd01-4986-bb59-4358e185ae31\") " pod="ai-tenant-e2e-shared-b-ad5ead/sub-test-model-ad5ead-kserve-7dcb57b467-px55w" Jul 14 10:09:29.876596 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:29.876494 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e521bc8-cd01-4986-bb59-4358e185ae31-home\") pod \"sub-test-model-ad5ead-kserve-7dcb57b467-px55w\" (UID: \"9e521bc8-cd01-4986-bb59-4358e185ae31\") " pod="ai-tenant-e2e-shared-b-ad5ead/sub-test-model-ad5ead-kserve-7dcb57b467-px55w" Jul 14 10:09:29.876596 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:29.876537 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e521bc8-cd01-4986-bb59-4358e185ae31-model-cache\") pod \"sub-test-model-ad5ead-kserve-7dcb57b467-px55w\" (UID: \"9e521bc8-cd01-4986-bb59-4358e185ae31\") " pod="ai-tenant-e2e-shared-b-ad5ead/sub-test-model-ad5ead-kserve-7dcb57b467-px55w" Jul 14 10:09:29.876845 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:29.876777 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e521bc8-cd01-4986-bb59-4358e185ae31-tmp-dir\") pod \"sub-test-model-ad5ead-kserve-7dcb57b467-px55w\" (UID: \"9e521bc8-cd01-4986-bb59-4358e185ae31\") " pod="ai-tenant-e2e-shared-b-ad5ead/sub-test-model-ad5ead-kserve-7dcb57b467-px55w" Jul 14 10:09:29.876904 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:29.876849 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e521bc8-cd01-4986-bb59-4358e185ae31-home\") pod \"sub-test-model-ad5ead-kserve-7dcb57b467-px55w\" (UID: \"9e521bc8-cd01-4986-bb59-4358e185ae31\") " pod="ai-tenant-e2e-shared-b-ad5ead/sub-test-model-ad5ead-kserve-7dcb57b467-px55w" Jul 14 10:09:29.876943 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:29.876934 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e521bc8-cd01-4986-bb59-4358e185ae31-model-cache\") pod \"sub-test-model-ad5ead-kserve-7dcb57b467-px55w\" (UID: \"9e521bc8-cd01-4986-bb59-4358e185ae31\") " pod="ai-tenant-e2e-shared-b-ad5ead/sub-test-model-ad5ead-kserve-7dcb57b467-px55w" Jul 14 10:09:29.878870 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:29.878840 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e521bc8-cd01-4986-bb59-4358e185ae31-dshm\") pod \"sub-test-model-ad5ead-kserve-7dcb57b467-px55w\" (UID: \"9e521bc8-cd01-4986-bb59-4358e185ae31\") " pod="ai-tenant-e2e-shared-b-ad5ead/sub-test-model-ad5ead-kserve-7dcb57b467-px55w" Jul 14 10:09:29.879222 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:29.879203 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e521bc8-cd01-4986-bb59-4358e185ae31-tls-certs\") pod \"sub-test-model-ad5ead-kserve-7dcb57b467-px55w\" (UID: \"9e521bc8-cd01-4986-bb59-4358e185ae31\") " pod="ai-tenant-e2e-shared-b-ad5ead/sub-test-model-ad5ead-kserve-7dcb57b467-px55w" Jul 14 10:09:29.885143 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:29.885122 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmmcs\" (UniqueName: \"kubernetes.io/projected/9e521bc8-cd01-4986-bb59-4358e185ae31-kube-api-access-gmmcs\") pod \"sub-test-model-ad5ead-kserve-7dcb57b467-px55w\" (UID: \"9e521bc8-cd01-4986-bb59-4358e185ae31\") " pod="ai-tenant-e2e-shared-b-ad5ead/sub-test-model-ad5ead-kserve-7dcb57b467-px55w" Jul 14 10:09:30.057576 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:30.057487 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ad5ead/sub-test-model-ad5ead-kserve-7dcb57b467-px55w" Jul 14 10:09:30.195615 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:30.195583 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ad5ead/sub-test-model-ad5ead-kserve-7dcb57b467-px55w"] Jul 14 10:09:30.196698 ip-10-0-132-2 kubenswrapper[2565]: W0714 10:09:30.196670 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e521bc8_cd01_4986_bb59_4358e185ae31.slice/crio-cacf7dac0a42b791807fedfc99ca0a8115615311304bbdf2f14ccccfc9c4eeff WatchSource:0}: Error finding container cacf7dac0a42b791807fedfc99ca0a8115615311304bbdf2f14ccccfc9c4eeff: Status 404 returned error can't find the container with id cacf7dac0a42b791807fedfc99ca0a8115615311304bbdf2f14ccccfc9c4eeff Jul 14 10:09:30.686792 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:30.686757 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ad5ead/sub-test-model-ad5ead-kserve-7dcb57b467-px55w" event={"ID":"9e521bc8-cd01-4986-bb59-4358e185ae31","Type":"ContainerStarted","Data":"293ed2229c50d2198fd4f0010f50d02c648eea9e055d8d0e8674f170e790df35"} Jul 14 10:09:30.686996 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:30.686801 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ad5ead/sub-test-model-ad5ead-kserve-7dcb57b467-px55w" event={"ID":"9e521bc8-cd01-4986-bb59-4358e185ae31","Type":"ContainerStarted","Data":"cacf7dac0a42b791807fedfc99ca0a8115615311304bbdf2f14ccccfc9c4eeff"} Jul 14 10:09:30.706678 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:30.706599 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-ad5ead/sub-test-model-ad5ead-kserve-7dcb57b467-px55w" podStartSLOduration=1.706581127 podStartE2EDuration="1.706581127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 10:09:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 10:09:30.704238476 +0000 UTC m=+2560.213469944" watchObservedRunningTime="2026-07-14 10:09:30.706581127 +0000 UTC m=+2560.215812576" Jul 14 10:09:40.058523 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:40.058476 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-ad5ead/sub-test-model-ad5ead-kserve-7dcb57b467-px55w" Jul 14 10:09:40.058523 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:40.058533 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-ad5ead/sub-test-model-ad5ead-kserve-7dcb57b467-px55w" Jul 14 10:09:40.071325 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:40.071300 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-ad5ead/sub-test-model-ad5ead-kserve-7dcb57b467-px55w" Jul 14 10:09:40.763402 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:09:40.763373 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-ad5ead/sub-test-model-ad5ead-kserve-7dcb57b467-px55w" Jul 14 10:10:19.222732 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:19.222651 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ad5ead/sub-test-model-ad5ead-kserve-7dcb57b467-px55w"] Jul 14 10:10:26.037010 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:26.036977 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-5d32fd/rate-test-model-5d32fd-kserve-869cd567bc-b77ws"] Jul 14 10:10:26.040708 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:26.040690 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5d32fd/rate-test-model-5d32fd-kserve-869cd567bc-b77ws" Jul 14 10:10:26.045470 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:26.045450 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-5d32fd\"/\"kube-root-ca.crt\"" Jul 14 10:10:26.046736 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:26.046684 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-5d32fd\"/\"openshift-service-ca.crt\"" Jul 14 10:10:26.046736 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:26.046702 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-5d32fd\"/\"rate-test-model-5d32fd-kserve-self-signed-certs\"" Jul 14 10:10:26.046736 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:26.046727 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-5d32fd\"/\"default-dockercfg-pmmcl\"" Jul 14 10:10:26.053472 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:26.053449 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-5d32fd/rate-test-model-5d32fd-kserve-869cd567bc-b77ws"] Jul 14 10:10:26.152211 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:26.152171 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f0b468a-0f45-43ff-87b7-7d9e95086be3-model-cache\") pod \"rate-test-model-5d32fd-kserve-869cd567bc-b77ws\" (UID: \"5f0b468a-0f45-43ff-87b7-7d9e95086be3\") " pod="ai-tenant-e2e-shared-a-5d32fd/rate-test-model-5d32fd-kserve-869cd567bc-b77ws" Jul 14 10:10:26.152211 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:26.152211 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f0b468a-0f45-43ff-87b7-7d9e95086be3-tmp-dir\") pod \"rate-test-model-5d32fd-kserve-869cd567bc-b77ws\" (UID: \"5f0b468a-0f45-43ff-87b7-7d9e95086be3\") " pod="ai-tenant-e2e-shared-a-5d32fd/rate-test-model-5d32fd-kserve-869cd567bc-b77ws" Jul 14 10:10:26.152458 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:26.152254 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfc2x\" (UniqueName: \"kubernetes.io/projected/5f0b468a-0f45-43ff-87b7-7d9e95086be3-kube-api-access-tfc2x\") pod \"rate-test-model-5d32fd-kserve-869cd567bc-b77ws\" (UID: \"5f0b468a-0f45-43ff-87b7-7d9e95086be3\") " pod="ai-tenant-e2e-shared-a-5d32fd/rate-test-model-5d32fd-kserve-869cd567bc-b77ws" Jul 14 10:10:26.152458 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:26.152335 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f0b468a-0f45-43ff-87b7-7d9e95086be3-dshm\") pod \"rate-test-model-5d32fd-kserve-869cd567bc-b77ws\" (UID: \"5f0b468a-0f45-43ff-87b7-7d9e95086be3\") " pod="ai-tenant-e2e-shared-a-5d32fd/rate-test-model-5d32fd-kserve-869cd567bc-b77ws" Jul 14 10:10:26.152458 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:26.152360 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f0b468a-0f45-43ff-87b7-7d9e95086be3-home\") pod \"rate-test-model-5d32fd-kserve-869cd567bc-b77ws\" (UID: \"5f0b468a-0f45-43ff-87b7-7d9e95086be3\") " pod="ai-tenant-e2e-shared-a-5d32fd/rate-test-model-5d32fd-kserve-869cd567bc-b77ws" Jul 14 10:10:26.152611 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:26.152465 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f0b468a-0f45-43ff-87b7-7d9e95086be3-tls-certs\") pod \"rate-test-model-5d32fd-kserve-869cd567bc-b77ws\" (UID: \"5f0b468a-0f45-43ff-87b7-7d9e95086be3\") " pod="ai-tenant-e2e-shared-a-5d32fd/rate-test-model-5d32fd-kserve-869cd567bc-b77ws" Jul 14 10:10:26.253469 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:26.253428 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f0b468a-0f45-43ff-87b7-7d9e95086be3-model-cache\") pod \"rate-test-model-5d32fd-kserve-869cd567bc-b77ws\" (UID: \"5f0b468a-0f45-43ff-87b7-7d9e95086be3\") " pod="ai-tenant-e2e-shared-a-5d32fd/rate-test-model-5d32fd-kserve-869cd567bc-b77ws" Jul 14 10:10:26.253469 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:26.253469 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f0b468a-0f45-43ff-87b7-7d9e95086be3-tmp-dir\") pod \"rate-test-model-5d32fd-kserve-869cd567bc-b77ws\" (UID: \"5f0b468a-0f45-43ff-87b7-7d9e95086be3\") " pod="ai-tenant-e2e-shared-a-5d32fd/rate-test-model-5d32fd-kserve-869cd567bc-b77ws" Jul 14 10:10:26.253729 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:26.253511 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfc2x\" (UniqueName: \"kubernetes.io/projected/5f0b468a-0f45-43ff-87b7-7d9e95086be3-kube-api-access-tfc2x\") pod \"rate-test-model-5d32fd-kserve-869cd567bc-b77ws\" (UID: \"5f0b468a-0f45-43ff-87b7-7d9e95086be3\") " pod="ai-tenant-e2e-shared-a-5d32fd/rate-test-model-5d32fd-kserve-869cd567bc-b77ws" Jul 14 10:10:26.253729 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:26.253592 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f0b468a-0f45-43ff-87b7-7d9e95086be3-dshm\") pod \"rate-test-model-5d32fd-kserve-869cd567bc-b77ws\" (UID: \"5f0b468a-0f45-43ff-87b7-7d9e95086be3\") " pod="ai-tenant-e2e-shared-a-5d32fd/rate-test-model-5d32fd-kserve-869cd567bc-b77ws" Jul 14 10:10:26.253729 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:26.253616 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f0b468a-0f45-43ff-87b7-7d9e95086be3-home\") pod \"rate-test-model-5d32fd-kserve-869cd567bc-b77ws\" (UID: \"5f0b468a-0f45-43ff-87b7-7d9e95086be3\") " pod="ai-tenant-e2e-shared-a-5d32fd/rate-test-model-5d32fd-kserve-869cd567bc-b77ws" Jul 14 10:10:26.253729 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:26.253698 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f0b468a-0f45-43ff-87b7-7d9e95086be3-tls-certs\") pod \"rate-test-model-5d32fd-kserve-869cd567bc-b77ws\" (UID: \"5f0b468a-0f45-43ff-87b7-7d9e95086be3\") " pod="ai-tenant-e2e-shared-a-5d32fd/rate-test-model-5d32fd-kserve-869cd567bc-b77ws" Jul 14 10:10:26.254013 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:26.253891 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f0b468a-0f45-43ff-87b7-7d9e95086be3-model-cache\") pod \"rate-test-model-5d32fd-kserve-869cd567bc-b77ws\" (UID: \"5f0b468a-0f45-43ff-87b7-7d9e95086be3\") " pod="ai-tenant-e2e-shared-a-5d32fd/rate-test-model-5d32fd-kserve-869cd567bc-b77ws" Jul 14 10:10:26.254013 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:26.253932 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f0b468a-0f45-43ff-87b7-7d9e95086be3-tmp-dir\") pod \"rate-test-model-5d32fd-kserve-869cd567bc-b77ws\" (UID: \"5f0b468a-0f45-43ff-87b7-7d9e95086be3\") " pod="ai-tenant-e2e-shared-a-5d32fd/rate-test-model-5d32fd-kserve-869cd567bc-b77ws" Jul 14 10:10:26.254144 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:26.254037 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f0b468a-0f45-43ff-87b7-7d9e95086be3-home\") pod \"rate-test-model-5d32fd-kserve-869cd567bc-b77ws\" (UID: \"5f0b468a-0f45-43ff-87b7-7d9e95086be3\") " pod="ai-tenant-e2e-shared-a-5d32fd/rate-test-model-5d32fd-kserve-869cd567bc-b77ws" Jul 14 10:10:26.255820 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:26.255783 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f0b468a-0f45-43ff-87b7-7d9e95086be3-dshm\") pod \"rate-test-model-5d32fd-kserve-869cd567bc-b77ws\" (UID: \"5f0b468a-0f45-43ff-87b7-7d9e95086be3\") " pod="ai-tenant-e2e-shared-a-5d32fd/rate-test-model-5d32fd-kserve-869cd567bc-b77ws" Jul 14 10:10:26.256118 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:26.256100 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f0b468a-0f45-43ff-87b7-7d9e95086be3-tls-certs\") pod \"rate-test-model-5d32fd-kserve-869cd567bc-b77ws\" (UID: \"5f0b468a-0f45-43ff-87b7-7d9e95086be3\") " pod="ai-tenant-e2e-shared-a-5d32fd/rate-test-model-5d32fd-kserve-869cd567bc-b77ws" Jul 14 10:10:26.262645 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:26.262615 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfc2x\" (UniqueName: \"kubernetes.io/projected/5f0b468a-0f45-43ff-87b7-7d9e95086be3-kube-api-access-tfc2x\") pod \"rate-test-model-5d32fd-kserve-869cd567bc-b77ws\" (UID: \"5f0b468a-0f45-43ff-87b7-7d9e95086be3\") " pod="ai-tenant-e2e-shared-a-5d32fd/rate-test-model-5d32fd-kserve-869cd567bc-b77ws" Jul 14 10:10:26.350971 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:26.350876 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5d32fd/rate-test-model-5d32fd-kserve-869cd567bc-b77ws" Jul 14 10:10:26.494860 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:26.494824 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-5d32fd/rate-test-model-5d32fd-kserve-869cd567bc-b77ws"] Jul 14 10:10:26.495990 ip-10-0-132-2 kubenswrapper[2565]: W0714 10:10:26.495961 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f0b468a_0f45_43ff_87b7_7d9e95086be3.slice/crio-c9929b59e293f0aa7db22d4312cc4b253dfd05e5d3a297a0909a9ab9344b39d6 WatchSource:0}: Error finding container c9929b59e293f0aa7db22d4312cc4b253dfd05e5d3a297a0909a9ab9344b39d6: Status 404 returned error can't find the container with id c9929b59e293f0aa7db22d4312cc4b253dfd05e5d3a297a0909a9ab9344b39d6 Jul 14 10:10:27.034884 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:27.034846 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5d32fd/rate-test-model-5d32fd-kserve-869cd567bc-b77ws" event={"ID":"5f0b468a-0f45-43ff-87b7-7d9e95086be3","Type":"ContainerStarted","Data":"65809e3c9a425399157a6f3af73d2e64198770c0d8776c4eb436b6f0e7a8d172"} Jul 14 10:10:27.035069 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:27.034896 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5d32fd/rate-test-model-5d32fd-kserve-869cd567bc-b77ws" event={"ID":"5f0b468a-0f45-43ff-87b7-7d9e95086be3","Type":"ContainerStarted","Data":"c9929b59e293f0aa7db22d4312cc4b253dfd05e5d3a297a0909a9ab9344b39d6"} Jul 14 10:10:27.053639 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:27.053585 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-5d32fd/rate-test-model-5d32fd-kserve-869cd567bc-b77ws" podStartSLOduration=1.0535683279999999 podStartE2EDuration="1.053568328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 10:10:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 10:10:27.050939574 +0000 UTC m=+2616.560171058" watchObservedRunningTime="2026-07-14 10:10:27.053568328 +0000 UTC m=+2616.562799775" Jul 14 10:10:34.243378 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:34.243339 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-ad5ead/sub-test-model-ad5ead-kserve-7dcb57b467-px55w" podUID="9e521bc8-cd01-4986-bb59-4358e185ae31" containerName="main" containerID="cri-o://293ed2229c50d2198fd4f0010f50d02c648eea9e055d8d0e8674f170e790df35" gracePeriod=45 Jul 14 10:10:34.501423 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:34.501350 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ad5ead/sub-test-model-ad5ead-kserve-7dcb57b467-px55w" Jul 14 10:10:34.537526 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:34.537488 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e521bc8-cd01-4986-bb59-4358e185ae31-tmp-dir\") pod \"9e521bc8-cd01-4986-bb59-4358e185ae31\" (UID: \"9e521bc8-cd01-4986-bb59-4358e185ae31\") " Jul 14 10:10:34.537526 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:34.537523 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmmcs\" (UniqueName: \"kubernetes.io/projected/9e521bc8-cd01-4986-bb59-4358e185ae31-kube-api-access-gmmcs\") pod \"9e521bc8-cd01-4986-bb59-4358e185ae31\" (UID: \"9e521bc8-cd01-4986-bb59-4358e185ae31\") " Jul 14 10:10:34.537787 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:34.537559 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e521bc8-cd01-4986-bb59-4358e185ae31-model-cache\") pod \"9e521bc8-cd01-4986-bb59-4358e185ae31\" (UID: \"9e521bc8-cd01-4986-bb59-4358e185ae31\") " Jul 14 10:10:34.537787 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:34.537612 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e521bc8-cd01-4986-bb59-4358e185ae31-dshm\") pod \"9e521bc8-cd01-4986-bb59-4358e185ae31\" (UID: \"9e521bc8-cd01-4986-bb59-4358e185ae31\") " Jul 14 10:10:34.537787 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:34.537682 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e521bc8-cd01-4986-bb59-4358e185ae31-home\") pod \"9e521bc8-cd01-4986-bb59-4358e185ae31\" (UID: \"9e521bc8-cd01-4986-bb59-4358e185ae31\") " Jul 14 10:10:34.537787 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:34.537721 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e521bc8-cd01-4986-bb59-4358e185ae31-tls-certs\") pod \"9e521bc8-cd01-4986-bb59-4358e185ae31\" (UID: \"9e521bc8-cd01-4986-bb59-4358e185ae31\") " Jul 14 10:10:34.538045 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:34.537799 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e521bc8-cd01-4986-bb59-4358e185ae31-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9e521bc8-cd01-4986-bb59-4358e185ae31" (UID: "9e521bc8-cd01-4986-bb59-4358e185ae31"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 10:10:34.538045 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:34.537872 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e521bc8-cd01-4986-bb59-4358e185ae31-model-cache" (OuterVolumeSpecName: "model-cache") pod "9e521bc8-cd01-4986-bb59-4358e185ae31" (UID: "9e521bc8-cd01-4986-bb59-4358e185ae31"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 10:10:34.538045 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:34.538004 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e521bc8-cd01-4986-bb59-4358e185ae31-home" (OuterVolumeSpecName: "home") pod "9e521bc8-cd01-4986-bb59-4358e185ae31" (UID: "9e521bc8-cd01-4986-bb59-4358e185ae31"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 10:10:34.538221 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:34.538162 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e521bc8-cd01-4986-bb59-4358e185ae31-model-cache\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 10:10:34.538221 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:34.538182 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e521bc8-cd01-4986-bb59-4358e185ae31-home\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 10:10:34.538221 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:34.538194 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e521bc8-cd01-4986-bb59-4358e185ae31-tmp-dir\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 10:10:34.539833 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:34.539775 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e521bc8-cd01-4986-bb59-4358e185ae31-dshm" (OuterVolumeSpecName: "dshm") pod "9e521bc8-cd01-4986-bb59-4358e185ae31" (UID: "9e521bc8-cd01-4986-bb59-4358e185ae31"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 10:10:34.539833 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:34.539793 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e521bc8-cd01-4986-bb59-4358e185ae31-kube-api-access-gmmcs" (OuterVolumeSpecName: "kube-api-access-gmmcs") pod "9e521bc8-cd01-4986-bb59-4358e185ae31" (UID: "9e521bc8-cd01-4986-bb59-4358e185ae31"). InnerVolumeSpecName "kube-api-access-gmmcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 10:10:34.539999 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:34.539941 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e521bc8-cd01-4986-bb59-4358e185ae31-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9e521bc8-cd01-4986-bb59-4358e185ae31" (UID: "9e521bc8-cd01-4986-bb59-4358e185ae31"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 10:10:34.639223 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:34.639187 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e521bc8-cd01-4986-bb59-4358e185ae31-tls-certs\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 10:10:34.639223 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:34.639218 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmmcs\" (UniqueName: \"kubernetes.io/projected/9e521bc8-cd01-4986-bb59-4358e185ae31-kube-api-access-gmmcs\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 10:10:34.639223 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:34.639229 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e521bc8-cd01-4986-bb59-4358e185ae31-dshm\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 10:10:35.084264 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:35.084225 2565 generic.go:356] "Generic (PLEG): container finished" podID="9e521bc8-cd01-4986-bb59-4358e185ae31" containerID="293ed2229c50d2198fd4f0010f50d02c648eea9e055d8d0e8674f170e790df35" exitCode=0 Jul 14 10:10:35.084449 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:35.084283 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ad5ead/sub-test-model-ad5ead-kserve-7dcb57b467-px55w" Jul 14 10:10:35.084449 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:35.084300 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ad5ead/sub-test-model-ad5ead-kserve-7dcb57b467-px55w" event={"ID":"9e521bc8-cd01-4986-bb59-4358e185ae31","Type":"ContainerDied","Data":"293ed2229c50d2198fd4f0010f50d02c648eea9e055d8d0e8674f170e790df35"} Jul 14 10:10:35.084449 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:35.084336 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ad5ead/sub-test-model-ad5ead-kserve-7dcb57b467-px55w" event={"ID":"9e521bc8-cd01-4986-bb59-4358e185ae31","Type":"ContainerDied","Data":"cacf7dac0a42b791807fedfc99ca0a8115615311304bbdf2f14ccccfc9c4eeff"} Jul 14 10:10:35.084449 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:35.084353 2565 scope.go:117] "RemoveContainer" containerID="293ed2229c50d2198fd4f0010f50d02c648eea9e055d8d0e8674f170e790df35" Jul 14 10:10:35.093792 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:35.093772 2565 scope.go:117] "RemoveContainer" containerID="293ed2229c50d2198fd4f0010f50d02c648eea9e055d8d0e8674f170e790df35" Jul 14 10:10:35.094115 ip-10-0-132-2 kubenswrapper[2565]: E0714 10:10:35.094096 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"293ed2229c50d2198fd4f0010f50d02c648eea9e055d8d0e8674f170e790df35\": container with ID starting with 293ed2229c50d2198fd4f0010f50d02c648eea9e055d8d0e8674f170e790df35 not found: ID does not exist" containerID="293ed2229c50d2198fd4f0010f50d02c648eea9e055d8d0e8674f170e790df35" Jul 14 10:10:35.094181 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:35.094124 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"293ed2229c50d2198fd4f0010f50d02c648eea9e055d8d0e8674f170e790df35"} err="failed to get container status \"293ed2229c50d2198fd4f0010f50d02c648eea9e055d8d0e8674f170e790df35\": rpc error: code = NotFound desc = could not find container \"293ed2229c50d2198fd4f0010f50d02c648eea9e055d8d0e8674f170e790df35\": container with ID starting with 293ed2229c50d2198fd4f0010f50d02c648eea9e055d8d0e8674f170e790df35 not found: ID does not exist" Jul 14 10:10:35.102162 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:35.102136 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ad5ead/sub-test-model-ad5ead-kserve-7dcb57b467-px55w"] Jul 14 10:10:35.106072 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:35.106048 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-ad5ead/sub-test-model-ad5ead-kserve-7dcb57b467-px55w"] Jul 14 10:10:36.351821 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:36.351781 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-5d32fd/rate-test-model-5d32fd-kserve-869cd567bc-b77ws" Jul 14 10:10:36.352306 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:36.351860 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-5d32fd/rate-test-model-5d32fd-kserve-869cd567bc-b77ws" Jul 14 10:10:36.364581 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:36.364549 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-5d32fd/rate-test-model-5d32fd-kserve-869cd567bc-b77ws" Jul 14 10:10:36.969924 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:36.969891 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e521bc8-cd01-4986-bb59-4358e185ae31" path="/var/lib/kubelet/pods/9e521bc8-cd01-4986-bb59-4358e185ae31/volumes" Jul 14 10:10:37.109404 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:10:37.109374 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-5d32fd/rate-test-model-5d32fd-kserve-869cd567bc-b77ws" Jul 14 10:11:32.611397 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:32.611363 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-5d32fd/rate-test-model-5d32fd-kserve-869cd567bc-b77ws"] Jul 14 10:11:37.823594 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:37.823553 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7151-139766691427248"] Jul 14 10:11:37.824230 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:37.824209 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e521bc8-cd01-4986-bb59-4358e185ae31" containerName="main" Jul 14 10:11:37.824324 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:37.824232 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e521bc8-cd01-4986-bb59-4358e185ae31" containerName="main" Jul 14 10:11:37.824405 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:37.824392 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e521bc8-cd01-4986-bb59-4358e185ae31" containerName="main" Jul 14 10:11:37.827498 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:37.827477 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7151-139766691427248" Jul 14 10:11:37.830509 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:37.830477 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-zhv22\"" Jul 14 10:11:37.834233 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:37.834209 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7151-139766691427248"] Jul 14 10:11:37.955583 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:37.955546 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gvhb\" (UniqueName: \"kubernetes.io/projected/eb88cb51-5863-49ba-884c-73d01bc20290-kube-api-access-8gvhb\") pod \"test-curl-7151-139766691427248\" (UID: \"eb88cb51-5863-49ba-884c-73d01bc20290\") " pod="openshift-ingress/test-curl-7151-139766691427248" Jul 14 10:11:38.056272 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:38.056238 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gvhb\" (UniqueName: \"kubernetes.io/projected/eb88cb51-5863-49ba-884c-73d01bc20290-kube-api-access-8gvhb\") pod \"test-curl-7151-139766691427248\" (UID: \"eb88cb51-5863-49ba-884c-73d01bc20290\") " pod="openshift-ingress/test-curl-7151-139766691427248" Jul 14 10:11:38.066377 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:38.066350 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gvhb\" (UniqueName: \"kubernetes.io/projected/eb88cb51-5863-49ba-884c-73d01bc20290-kube-api-access-8gvhb\") pod \"test-curl-7151-139766691427248\" (UID: \"eb88cb51-5863-49ba-884c-73d01bc20290\") " pod="openshift-ingress/test-curl-7151-139766691427248" Jul 14 10:11:38.137403 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:38.137375 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7151-139766691427248" Jul 14 10:11:38.279544 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:38.279519 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7151-139766691427248"] Jul 14 10:11:38.281989 ip-10-0-132-2 kubenswrapper[2565]: W0714 10:11:38.281962 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb88cb51_5863_49ba_884c_73d01bc20290.slice/crio-3a22a59a4571fcf604e314f5f150f5a48535eece2172815103c01c622f5375ec WatchSource:0}: Error finding container 3a22a59a4571fcf604e314f5f150f5a48535eece2172815103c01c622f5375ec: Status 404 returned error can't find the container with id 3a22a59a4571fcf604e314f5f150f5a48535eece2172815103c01c622f5375ec Jul 14 10:11:38.283756 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:38.283739 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 10:11:38.464592 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:38.464504 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7151-139766691427248" event={"ID":"eb88cb51-5863-49ba-884c-73d01bc20290","Type":"ContainerStarted","Data":"3a22a59a4571fcf604e314f5f150f5a48535eece2172815103c01c622f5375ec"} Jul 14 10:11:39.471390 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:39.471356 2565 generic.go:356] "Generic (PLEG): container finished" podID="eb88cb51-5863-49ba-884c-73d01bc20290" containerID="af639e5b6ee2adca870c4542194d5d86834d4bea7bb4e6ecea461916fa0b4eff" exitCode=0 Jul 14 10:11:39.471790 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:39.471438 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7151-139766691427248" event={"ID":"eb88cb51-5863-49ba-884c-73d01bc20290","Type":"ContainerDied","Data":"af639e5b6ee2adca870c4542194d5d86834d4bea7bb4e6ecea461916fa0b4eff"} Jul 14 10:11:40.622706 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:40.622679 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7151-139766691427248" Jul 14 10:11:40.662316 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:40.662283 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7151-139766691427248_eb88cb51-5863-49ba-884c-73d01bc20290/test-curl-7151-139766691427248/0.log" Jul 14 10:11:40.688619 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:40.688589 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7151-139766691427248"] Jul 14 10:11:40.692506 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:40.692481 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7151-139766691427248"] Jul 14 10:11:40.784306 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:40.784218 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8gvhb\" (UniqueName: \"kubernetes.io/projected/eb88cb51-5863-49ba-884c-73d01bc20290-kube-api-access-8gvhb\") pod \"eb88cb51-5863-49ba-884c-73d01bc20290\" (UID: \"eb88cb51-5863-49ba-884c-73d01bc20290\") " Jul 14 10:11:40.786296 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:40.786268 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb88cb51-5863-49ba-884c-73d01bc20290-kube-api-access-8gvhb" (OuterVolumeSpecName: "kube-api-access-8gvhb") pod "eb88cb51-5863-49ba-884c-73d01bc20290" (UID: "eb88cb51-5863-49ba-884c-73d01bc20290"). InnerVolumeSpecName "kube-api-access-8gvhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 10:11:40.885190 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:40.885153 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gvhb\" (UniqueName: \"kubernetes.io/projected/eb88cb51-5863-49ba-884c-73d01bc20290-kube-api-access-8gvhb\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 10:11:40.969977 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:40.969936 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb88cb51-5863-49ba-884c-73d01bc20290" path="/var/lib/kubelet/pods/eb88cb51-5863-49ba-884c-73d01bc20290/volumes" Jul 14 10:11:41.484333 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:41.484306 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7151-139766691427248" Jul 14 10:11:41.484513 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:41.484366 2565 scope.go:117] "RemoveContainer" containerID="af639e5b6ee2adca870c4542194d5d86834d4bea7bb4e6ecea461916fa0b4eff" Jul 14 10:11:46.839348 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:46.839252 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7151-139766691505184"] Jul 14 10:11:46.839973 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:46.839954 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb88cb51-5863-49ba-884c-73d01bc20290" containerName="test-curl-7151-139766691427248" Jul 14 10:11:46.840072 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:46.839975 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb88cb51-5863-49ba-884c-73d01bc20290" containerName="test-curl-7151-139766691427248" Jul 14 10:11:46.840138 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:46.840083 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb88cb51-5863-49ba-884c-73d01bc20290" containerName="test-curl-7151-139766691427248" Jul 14 10:11:46.844603 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:46.844582 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7151-139766691505184" Jul 14 10:11:46.847557 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:46.847536 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7151-139766691505184"] Jul 14 10:11:46.847700 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:46.847683 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-zhv22\"" Jul 14 10:11:46.941168 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:46.941133 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph5pv\" (UniqueName: \"kubernetes.io/projected/3d35c9ab-419c-4562-a62a-08697f1769f3-kube-api-access-ph5pv\") pod \"test-curl-7151-139766691505184\" (UID: \"3d35c9ab-419c-4562-a62a-08697f1769f3\") " pod="openshift-ingress/test-curl-7151-139766691505184" Jul 14 10:11:47.041882 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:47.041838 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph5pv\" (UniqueName: \"kubernetes.io/projected/3d35c9ab-419c-4562-a62a-08697f1769f3-kube-api-access-ph5pv\") pod \"test-curl-7151-139766691505184\" (UID: \"3d35c9ab-419c-4562-a62a-08697f1769f3\") " pod="openshift-ingress/test-curl-7151-139766691505184" Jul 14 10:11:47.051022 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:47.050993 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph5pv\" (UniqueName: \"kubernetes.io/projected/3d35c9ab-419c-4562-a62a-08697f1769f3-kube-api-access-ph5pv\") pod \"test-curl-7151-139766691505184\" (UID: \"3d35c9ab-419c-4562-a62a-08697f1769f3\") " pod="openshift-ingress/test-curl-7151-139766691505184" Jul 14 10:11:47.155403 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:47.155369 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7151-139766691505184" Jul 14 10:11:47.295127 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:47.295094 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7151-139766691505184"] Jul 14 10:11:47.296119 ip-10-0-132-2 kubenswrapper[2565]: W0714 10:11:47.296088 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d35c9ab_419c_4562_a62a_08697f1769f3.slice/crio-7480c4abd31af57e6f0fb8b56c18f4cb11689c96adb74078a5c6203672283884 WatchSource:0}: Error finding container 7480c4abd31af57e6f0fb8b56c18f4cb11689c96adb74078a5c6203672283884: Status 404 returned error can't find the container with id 7480c4abd31af57e6f0fb8b56c18f4cb11689c96adb74078a5c6203672283884 Jul 14 10:11:47.522340 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:47.522251 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7151-139766691505184" event={"ID":"3d35c9ab-419c-4562-a62a-08697f1769f3","Type":"ContainerStarted","Data":"7480c4abd31af57e6f0fb8b56c18f4cb11689c96adb74078a5c6203672283884"} Jul 14 10:11:47.631405 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:47.631368 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-5d32fd/rate-test-model-5d32fd-kserve-869cd567bc-b77ws" podUID="5f0b468a-0f45-43ff-87b7-7d9e95086be3" containerName="main" containerID="cri-o://65809e3c9a425399157a6f3af73d2e64198770c0d8776c4eb436b6f0e7a8d172" gracePeriod=45 Jul 14 10:11:47.872609 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:47.872584 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5d32fd/rate-test-model-5d32fd-kserve-869cd567bc-b77ws" Jul 14 10:11:48.052484 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:48.052393 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f0b468a-0f45-43ff-87b7-7d9e95086be3-dshm\") pod \"5f0b468a-0f45-43ff-87b7-7d9e95086be3\" (UID: \"5f0b468a-0f45-43ff-87b7-7d9e95086be3\") " Jul 14 10:11:48.052484 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:48.052468 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f0b468a-0f45-43ff-87b7-7d9e95086be3-tmp-dir\") pod \"5f0b468a-0f45-43ff-87b7-7d9e95086be3\" (UID: \"5f0b468a-0f45-43ff-87b7-7d9e95086be3\") " Jul 14 10:11:48.052694 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:48.052520 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tfc2x\" (UniqueName: \"kubernetes.io/projected/5f0b468a-0f45-43ff-87b7-7d9e95086be3-kube-api-access-tfc2x\") pod \"5f0b468a-0f45-43ff-87b7-7d9e95086be3\" (UID: \"5f0b468a-0f45-43ff-87b7-7d9e95086be3\") " Jul 14 10:11:48.052694 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:48.052542 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f0b468a-0f45-43ff-87b7-7d9e95086be3-model-cache\") pod \"5f0b468a-0f45-43ff-87b7-7d9e95086be3\" (UID: \"5f0b468a-0f45-43ff-87b7-7d9e95086be3\") " Jul 14 10:11:48.052694 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:48.052577 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f0b468a-0f45-43ff-87b7-7d9e95086be3-tls-certs\") pod \"5f0b468a-0f45-43ff-87b7-7d9e95086be3\" (UID: \"5f0b468a-0f45-43ff-87b7-7d9e95086be3\") " Jul 14 10:11:48.052951 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:48.052748 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f0b468a-0f45-43ff-87b7-7d9e95086be3-home\") pod \"5f0b468a-0f45-43ff-87b7-7d9e95086be3\" (UID: \"5f0b468a-0f45-43ff-87b7-7d9e95086be3\") " Jul 14 10:11:48.052951 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:48.052766 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f0b468a-0f45-43ff-87b7-7d9e95086be3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5f0b468a-0f45-43ff-87b7-7d9e95086be3" (UID: "5f0b468a-0f45-43ff-87b7-7d9e95086be3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 10:11:48.052951 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:48.052781 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f0b468a-0f45-43ff-87b7-7d9e95086be3-model-cache" (OuterVolumeSpecName: "model-cache") pod "5f0b468a-0f45-43ff-87b7-7d9e95086be3" (UID: "5f0b468a-0f45-43ff-87b7-7d9e95086be3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 10:11:48.053091 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:48.052984 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f0b468a-0f45-43ff-87b7-7d9e95086be3-home" (OuterVolumeSpecName: "home") pod "5f0b468a-0f45-43ff-87b7-7d9e95086be3" (UID: "5f0b468a-0f45-43ff-87b7-7d9e95086be3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 10:11:48.053266 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:48.053248 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f0b468a-0f45-43ff-87b7-7d9e95086be3-tmp-dir\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 10:11:48.053331 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:48.053267 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f0b468a-0f45-43ff-87b7-7d9e95086be3-model-cache\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 10:11:48.053331 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:48.053276 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f0b468a-0f45-43ff-87b7-7d9e95086be3-home\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 10:11:48.055387 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:48.055368 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f0b468a-0f45-43ff-87b7-7d9e95086be3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5f0b468a-0f45-43ff-87b7-7d9e95086be3" (UID: "5f0b468a-0f45-43ff-87b7-7d9e95086be3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 10:11:48.055447 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:48.055387 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f0b468a-0f45-43ff-87b7-7d9e95086be3-kube-api-access-tfc2x" (OuterVolumeSpecName: "kube-api-access-tfc2x") pod "5f0b468a-0f45-43ff-87b7-7d9e95086be3" (UID: "5f0b468a-0f45-43ff-87b7-7d9e95086be3"). InnerVolumeSpecName "kube-api-access-tfc2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 10:11:48.055447 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:48.055402 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f0b468a-0f45-43ff-87b7-7d9e95086be3-dshm" (OuterVolumeSpecName: "dshm") pod "5f0b468a-0f45-43ff-87b7-7d9e95086be3" (UID: "5f0b468a-0f45-43ff-87b7-7d9e95086be3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 10:11:48.154174 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:48.154120 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfc2x\" (UniqueName: \"kubernetes.io/projected/5f0b468a-0f45-43ff-87b7-7d9e95086be3-kube-api-access-tfc2x\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 10:11:48.154174 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:48.154167 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f0b468a-0f45-43ff-87b7-7d9e95086be3-tls-certs\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 10:11:48.154174 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:48.154178 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f0b468a-0f45-43ff-87b7-7d9e95086be3-dshm\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 10:11:48.529239 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:48.529206 2565 generic.go:356] "Generic (PLEG): container finished" podID="3d35c9ab-419c-4562-a62a-08697f1769f3" containerID="f86a09a588ea5048dc728f15a2d5b5fcdf44750e6f68fa9e7944618ab2bc871c" exitCode=0 Jul 14 10:11:48.529437 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:48.529289 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7151-139766691505184" event={"ID":"3d35c9ab-419c-4562-a62a-08697f1769f3","Type":"ContainerDied","Data":"f86a09a588ea5048dc728f15a2d5b5fcdf44750e6f68fa9e7944618ab2bc871c"} Jul 14 10:11:48.530465 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:48.530434 2565 generic.go:356] "Generic (PLEG): container finished" podID="5f0b468a-0f45-43ff-87b7-7d9e95086be3" containerID="65809e3c9a425399157a6f3af73d2e64198770c0d8776c4eb436b6f0e7a8d172" exitCode=0 Jul 14 10:11:48.530591 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:48.530467 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5d32fd/rate-test-model-5d32fd-kserve-869cd567bc-b77ws" Jul 14 10:11:48.530591 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:48.530463 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5d32fd/rate-test-model-5d32fd-kserve-869cd567bc-b77ws" event={"ID":"5f0b468a-0f45-43ff-87b7-7d9e95086be3","Type":"ContainerDied","Data":"65809e3c9a425399157a6f3af73d2e64198770c0d8776c4eb436b6f0e7a8d172"} Jul 14 10:11:48.530591 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:48.530584 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5d32fd/rate-test-model-5d32fd-kserve-869cd567bc-b77ws" event={"ID":"5f0b468a-0f45-43ff-87b7-7d9e95086be3","Type":"ContainerDied","Data":"c9929b59e293f0aa7db22d4312cc4b253dfd05e5d3a297a0909a9ab9344b39d6"} Jul 14 10:11:48.530701 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:48.530604 2565 scope.go:117] "RemoveContainer" containerID="65809e3c9a425399157a6f3af73d2e64198770c0d8776c4eb436b6f0e7a8d172" Jul 14 10:11:48.540376 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:48.540356 2565 scope.go:117] "RemoveContainer" containerID="65809e3c9a425399157a6f3af73d2e64198770c0d8776c4eb436b6f0e7a8d172" Jul 14 10:11:48.540630 ip-10-0-132-2 kubenswrapper[2565]: E0714 10:11:48.540610 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65809e3c9a425399157a6f3af73d2e64198770c0d8776c4eb436b6f0e7a8d172\": container with ID starting with 65809e3c9a425399157a6f3af73d2e64198770c0d8776c4eb436b6f0e7a8d172 not found: ID does not exist" containerID="65809e3c9a425399157a6f3af73d2e64198770c0d8776c4eb436b6f0e7a8d172" Jul 14 10:11:48.540693 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:48.540639 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65809e3c9a425399157a6f3af73d2e64198770c0d8776c4eb436b6f0e7a8d172"} err="failed to get container status \"65809e3c9a425399157a6f3af73d2e64198770c0d8776c4eb436b6f0e7a8d172\": rpc error: code = NotFound desc = could not find container \"65809e3c9a425399157a6f3af73d2e64198770c0d8776c4eb436b6f0e7a8d172\": container with ID starting with 65809e3c9a425399157a6f3af73d2e64198770c0d8776c4eb436b6f0e7a8d172 not found: ID does not exist" Jul 14 10:11:48.557558 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:48.557534 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-5d32fd/rate-test-model-5d32fd-kserve-869cd567bc-b77ws"] Jul 14 10:11:48.563874 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:48.563850 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-5d32fd/rate-test-model-5d32fd-kserve-869cd567bc-b77ws"] Jul 14 10:11:48.969277 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:48.969245 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f0b468a-0f45-43ff-87b7-7d9e95086be3" path="/var/lib/kubelet/pods/5f0b468a-0f45-43ff-87b7-7d9e95086be3/volumes" Jul 14 10:11:49.681621 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:49.681599 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7151-139766691505184" Jul 14 10:11:49.701290 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:49.701265 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7151-139766691505184_3d35c9ab-419c-4562-a62a-08697f1769f3/test-curl-7151-139766691505184/0.log" Jul 14 10:11:49.728510 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:49.728477 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7151-139766691505184"] Jul 14 10:11:49.731283 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:49.731264 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7151-139766691505184"] Jul 14 10:11:49.863416 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:49.863335 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7151-139766691505184"] Jul 14 10:11:49.863779 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:49.863766 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f0b468a-0f45-43ff-87b7-7d9e95086be3" containerName="main" Jul 14 10:11:49.863844 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:49.863780 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f0b468a-0f45-43ff-87b7-7d9e95086be3" containerName="main" Jul 14 10:11:49.863844 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:49.863788 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d35c9ab-419c-4562-a62a-08697f1769f3" containerName="test-curl-7151-139766691505184" Jul 14 10:11:49.863844 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:49.863795 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d35c9ab-419c-4562-a62a-08697f1769f3" containerName="test-curl-7151-139766691505184" Jul 14 10:11:49.863958 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:49.863878 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f0b468a-0f45-43ff-87b7-7d9e95086be3" containerName="main" Jul 14 10:11:49.863958 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:49.863887 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d35c9ab-419c-4562-a62a-08697f1769f3" containerName="test-curl-7151-139766691505184" Jul 14 10:11:49.866985 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:49.866969 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7151-139766691505184" Jul 14 10:11:49.872593 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:49.872566 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ph5pv\" (UniqueName: \"kubernetes.io/projected/3d35c9ab-419c-4562-a62a-08697f1769f3-kube-api-access-ph5pv\") pod \"3d35c9ab-419c-4562-a62a-08697f1769f3\" (UID: \"3d35c9ab-419c-4562-a62a-08697f1769f3\") " Jul 14 10:11:49.873863 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:49.873839 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7151-139766691505184"] Jul 14 10:11:49.875118 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:49.875093 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d35c9ab-419c-4562-a62a-08697f1769f3-kube-api-access-ph5pv" (OuterVolumeSpecName: "kube-api-access-ph5pv") pod "3d35c9ab-419c-4562-a62a-08697f1769f3" (UID: "3d35c9ab-419c-4562-a62a-08697f1769f3"). InnerVolumeSpecName "kube-api-access-ph5pv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 10:11:49.973845 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:49.973786 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsmgl\" (UniqueName: \"kubernetes.io/projected/690cdf57-b378-466b-a3a6-2ffb71c426a5-kube-api-access-vsmgl\") pod \"test-curl-7151-139766691505184\" (UID: \"690cdf57-b378-466b-a3a6-2ffb71c426a5\") " pod="openshift-ingress/test-curl-7151-139766691505184" Jul 14 10:11:49.974229 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:49.973905 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph5pv\" (UniqueName: \"kubernetes.io/projected/3d35c9ab-419c-4562-a62a-08697f1769f3-kube-api-access-ph5pv\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 10:11:50.075098 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:50.075059 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsmgl\" (UniqueName: \"kubernetes.io/projected/690cdf57-b378-466b-a3a6-2ffb71c426a5-kube-api-access-vsmgl\") pod \"test-curl-7151-139766691505184\" (UID: \"690cdf57-b378-466b-a3a6-2ffb71c426a5\") " pod="openshift-ingress/test-curl-7151-139766691505184" Jul 14 10:11:50.083895 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:50.083869 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsmgl\" (UniqueName: \"kubernetes.io/projected/690cdf57-b378-466b-a3a6-2ffb71c426a5-kube-api-access-vsmgl\") pod \"test-curl-7151-139766691505184\" (UID: \"690cdf57-b378-466b-a3a6-2ffb71c426a5\") " pod="openshift-ingress/test-curl-7151-139766691505184" Jul 14 10:11:50.184699 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:50.184661 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7151-139766691505184" Jul 14 10:11:50.325072 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:50.325047 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7151-139766691505184"] Jul 14 10:11:50.326497 ip-10-0-132-2 kubenswrapper[2565]: W0714 10:11:50.326469 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod690cdf57_b378_466b_a3a6_2ffb71c426a5.slice/crio-07eec63342dabc23d8dd55bfe989226753f5e8e595ac7a5d3af45438321df1fc WatchSource:0}: Error finding container 07eec63342dabc23d8dd55bfe989226753f5e8e595ac7a5d3af45438321df1fc: Status 404 returned error can't find the container with id 07eec63342dabc23d8dd55bfe989226753f5e8e595ac7a5d3af45438321df1fc Jul 14 10:11:50.544997 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:50.544915 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7151-139766691505184" Jul 14 10:11:50.545152 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:50.544918 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7480c4abd31af57e6f0fb8b56c18f4cb11689c96adb74078a5c6203672283884" Jul 14 10:11:50.546110 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:50.546084 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7151-139766691505184" event={"ID":"690cdf57-b378-466b-a3a6-2ffb71c426a5","Type":"ContainerStarted","Data":"07eec63342dabc23d8dd55bfe989226753f5e8e595ac7a5d3af45438321df1fc"} Jul 14 10:11:50.556316 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:50.556286 2565 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="openshift-ingress/test-curl-7151-139766691505184" oldPodUID="3d35c9ab-419c-4562-a62a-08697f1769f3" podUID="690cdf57-b378-466b-a3a6-2ffb71c426a5" Jul 14 10:11:50.970586 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:50.970554 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d35c9ab-419c-4562-a62a-08697f1769f3" path="/var/lib/kubelet/pods/3d35c9ab-419c-4562-a62a-08697f1769f3/volumes" Jul 14 10:11:51.554769 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:51.554732 2565 generic.go:356] "Generic (PLEG): container finished" podID="690cdf57-b378-466b-a3a6-2ffb71c426a5" containerID="51568fa0e58bb57705dffcd5467bd91ef914b99ef3ffb58cec2c8583abec6eec" exitCode=0 Jul 14 10:11:51.555199 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:51.554830 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7151-139766691505184" event={"ID":"690cdf57-b378-466b-a3a6-2ffb71c426a5","Type":"ContainerDied","Data":"51568fa0e58bb57705dffcd5467bd91ef914b99ef3ffb58cec2c8583abec6eec"} Jul 14 10:11:52.701847 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:52.701820 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7151-139766691505184" Jul 14 10:11:52.723687 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:52.723655 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7151-139766691505184_690cdf57-b378-466b-a3a6-2ffb71c426a5/test-curl-7151-139766691505184/0.log" Jul 14 10:11:52.748715 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:52.748683 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7151-139766691505184"] Jul 14 10:11:52.753024 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:52.752999 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7151-139766691505184"] Jul 14 10:11:52.798583 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:52.798556 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vsmgl\" (UniqueName: \"kubernetes.io/projected/690cdf57-b378-466b-a3a6-2ffb71c426a5-kube-api-access-vsmgl\") pod \"690cdf57-b378-466b-a3a6-2ffb71c426a5\" (UID: \"690cdf57-b378-466b-a3a6-2ffb71c426a5\") " Jul 14 10:11:52.800692 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:52.800671 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/690cdf57-b378-466b-a3a6-2ffb71c426a5-kube-api-access-vsmgl" (OuterVolumeSpecName: "kube-api-access-vsmgl") pod "690cdf57-b378-466b-a3a6-2ffb71c426a5" (UID: "690cdf57-b378-466b-a3a6-2ffb71c426a5"). InnerVolumeSpecName "kube-api-access-vsmgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 10:11:52.899349 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:52.899316 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsmgl\" (UniqueName: \"kubernetes.io/projected/690cdf57-b378-466b-a3a6-2ffb71c426a5-kube-api-access-vsmgl\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 10:11:52.969344 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:52.969312 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="690cdf57-b378-466b-a3a6-2ffb71c426a5" path="/var/lib/kubelet/pods/690cdf57-b378-466b-a3a6-2ffb71c426a5/volumes" Jul 14 10:11:53.569256 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:53.569222 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7151-139766691505184" Jul 14 10:11:53.569458 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:11:53.569271 2565 scope.go:117] "RemoveContainer" containerID="51568fa0e58bb57705dffcd5467bd91ef914b99ef3ffb58cec2c8583abec6eec" Jul 14 10:12:03.413084 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:03.413053 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-5d32fd-5f7b88fc59-2272f"] Jul 14 10:12:03.413560 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:03.413261 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5d32fd-5f7b88fc59-2272f" podUID="462dd0d8-b00b-407d-84a1-2bf66cd4e58f" containerName="maas-api" containerID="cri-o://57c6d998c45cd5e0bffcc1a74dfed609d52d340ace67d4bcd7705424a3c4ea01" gracePeriod=30 Jul 14 10:12:03.633537 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:03.633505 2565 generic.go:356] "Generic (PLEG): container finished" podID="462dd0d8-b00b-407d-84a1-2bf66cd4e58f" containerID="57c6d998c45cd5e0bffcc1a74dfed609d52d340ace67d4bcd7705424a3c4ea01" exitCode=0 Jul 14 10:12:03.633714 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:03.633590 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5d32fd-5f7b88fc59-2272f" event={"ID":"462dd0d8-b00b-407d-84a1-2bf66cd4e58f","Type":"ContainerDied","Data":"57c6d998c45cd5e0bffcc1a74dfed609d52d340ace67d4bcd7705424a3c4ea01"} Jul 14 10:12:03.664168 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:03.664100 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5d32fd-5f7b88fc59-2272f" Jul 14 10:12:03.709817 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:03.709771 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6zf8n\" (UniqueName: \"kubernetes.io/projected/462dd0d8-b00b-407d-84a1-2bf66cd4e58f-kube-api-access-6zf8n\") pod \"462dd0d8-b00b-407d-84a1-2bf66cd4e58f\" (UID: \"462dd0d8-b00b-407d-84a1-2bf66cd4e58f\") " Jul 14 10:12:03.710005 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:03.709896 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/462dd0d8-b00b-407d-84a1-2bf66cd4e58f-openshift-service-ca\") pod \"462dd0d8-b00b-407d-84a1-2bf66cd4e58f\" (UID: \"462dd0d8-b00b-407d-84a1-2bf66cd4e58f\") " Jul 14 10:12:03.710005 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:03.709956 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/462dd0d8-b00b-407d-84a1-2bf66cd4e58f-maas-api-tls\") pod \"462dd0d8-b00b-407d-84a1-2bf66cd4e58f\" (UID: \"462dd0d8-b00b-407d-84a1-2bf66cd4e58f\") " Jul 14 10:12:03.710335 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:03.710303 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/462dd0d8-b00b-407d-84a1-2bf66cd4e58f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "462dd0d8-b00b-407d-84a1-2bf66cd4e58f" (UID: "462dd0d8-b00b-407d-84a1-2bf66cd4e58f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 10:12:03.711987 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:03.711960 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/462dd0d8-b00b-407d-84a1-2bf66cd4e58f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "462dd0d8-b00b-407d-84a1-2bf66cd4e58f" (UID: "462dd0d8-b00b-407d-84a1-2bf66cd4e58f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 10:12:03.712110 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:03.711992 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/462dd0d8-b00b-407d-84a1-2bf66cd4e58f-kube-api-access-6zf8n" (OuterVolumeSpecName: "kube-api-access-6zf8n") pod "462dd0d8-b00b-407d-84a1-2bf66cd4e58f" (UID: "462dd0d8-b00b-407d-84a1-2bf66cd4e58f"). InnerVolumeSpecName "kube-api-access-6zf8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 10:12:03.811022 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:03.810985 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/462dd0d8-b00b-407d-84a1-2bf66cd4e58f-maas-api-tls\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 10:12:03.811022 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:03.811016 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zf8n\" (UniqueName: \"kubernetes.io/projected/462dd0d8-b00b-407d-84a1-2bf66cd4e58f-kube-api-access-6zf8n\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 10:12:03.811022 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:03.811025 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/462dd0d8-b00b-407d-84a1-2bf66cd4e58f-openshift-service-ca\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 10:12:04.640503 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:04.640466 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5d32fd-5f7b88fc59-2272f" event={"ID":"462dd0d8-b00b-407d-84a1-2bf66cd4e58f","Type":"ContainerDied","Data":"454d4d8b64150e8a510b2517d92d60d3f6d37b7a8b7be211c46426f02ce75ff3"} Jul 14 10:12:04.640503 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:04.640490 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5d32fd-5f7b88fc59-2272f" Jul 14 10:12:04.641047 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:04.640537 2565 scope.go:117] "RemoveContainer" containerID="57c6d998c45cd5e0bffcc1a74dfed609d52d340ace67d4bcd7705424a3c4ea01" Jul 14 10:12:04.663000 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:04.662975 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-5d32fd-5f7b88fc59-2272f"] Jul 14 10:12:04.669408 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:04.669375 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-5d32fd-5f7b88fc59-2272f"] Jul 14 10:12:04.969732 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:04.969649 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="462dd0d8-b00b-407d-84a1-2bf66cd4e58f" path="/var/lib/kubelet/pods/462dd0d8-b00b-407d-84a1-2bf66cd4e58f/volumes" Jul 14 10:12:19.173204 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:19.173169 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ad5ead-fvn5k"] Jul 14 10:12:19.173676 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:19.173598 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="690cdf57-b378-466b-a3a6-2ffb71c426a5" containerName="test-curl-7151-139766691505184" Jul 14 10:12:19.173676 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:19.173610 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="690cdf57-b378-466b-a3a6-2ffb71c426a5" containerName="test-curl-7151-139766691505184" Jul 14 10:12:19.173676 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:19.173624 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="462dd0d8-b00b-407d-84a1-2bf66cd4e58f" containerName="maas-api" Jul 14 10:12:19.173676 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:19.173629 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="462dd0d8-b00b-407d-84a1-2bf66cd4e58f" containerName="maas-api" Jul 14 10:12:19.173860 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:19.173697 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="690cdf57-b378-466b-a3a6-2ffb71c426a5" containerName="test-curl-7151-139766691505184" Jul 14 10:12:19.173860 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:19.173707 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="462dd0d8-b00b-407d-84a1-2bf66cd4e58f" containerName="maas-api" Jul 14 10:12:19.184895 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:19.184865 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ad5ead-fvn5k" Jul 14 10:12:19.186081 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:19.186046 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ad5ead-fvn5k"] Jul 14 10:12:19.187754 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:19.187731 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8m76v\"" Jul 14 10:12:19.271676 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:19.271637 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e96d402a-359a-496e-9089-c8a4539091d6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-ad5ead-fvn5k\" (UID: \"e96d402a-359a-496e-9089-c8a4539091d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ad5ead-fvn5k" Jul 14 10:12:19.373238 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:19.373198 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e96d402a-359a-496e-9089-c8a4539091d6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-ad5ead-fvn5k\" (UID: \"e96d402a-359a-496e-9089-c8a4539091d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ad5ead-fvn5k" Jul 14 10:12:19.373893 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:19.373872 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e96d402a-359a-496e-9089-c8a4539091d6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-ad5ead-fvn5k\" (UID: \"e96d402a-359a-496e-9089-c8a4539091d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ad5ead-fvn5k" Jul 14 10:12:19.496861 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:19.496747 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ad5ead-fvn5k" Jul 14 10:12:19.636734 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:19.636707 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ad5ead-fvn5k"] Jul 14 10:12:19.638203 ip-10-0-132-2 kubenswrapper[2565]: W0714 10:12:19.638180 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode96d402a_359a_496e_9089_c8a4539091d6.slice/crio-b5137c7e3f2970f36d2f82b96f6461bc7a9b1916e2968483986f1105254a70d2 WatchSource:0}: Error finding container b5137c7e3f2970f36d2f82b96f6461bc7a9b1916e2968483986f1105254a70d2: Status 404 returned error can't find the container with id b5137c7e3f2970f36d2f82b96f6461bc7a9b1916e2968483986f1105254a70d2 Jul 14 10:12:19.734403 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:19.734367 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ad5ead-fvn5k" event={"ID":"e96d402a-359a-496e-9089-c8a4539091d6","Type":"ContainerStarted","Data":"b5137c7e3f2970f36d2f82b96f6461bc7a9b1916e2968483986f1105254a70d2"} Jul 14 10:12:20.742690 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:20.742661 2565 generic.go:356] "Generic (PLEG): container finished" podID="e96d402a-359a-496e-9089-c8a4539091d6" containerID="3fe3ea1c27f0749fae8fb7fbf0d9812a2b75e2f223e0711a0f2affb7f63aaa38" exitCode=0 Jul 14 10:12:20.743106 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:20.742750 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ad5ead-fvn5k" event={"ID":"e96d402a-359a-496e-9089-c8a4539091d6","Type":"ContainerDied","Data":"3fe3ea1c27f0749fae8fb7fbf0d9812a2b75e2f223e0711a0f2affb7f63aaa38"} Jul 14 10:12:21.886014 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:21.885992 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ad5ead-fvn5k" Jul 14 10:12:22.001849 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:22.001729 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e96d402a-359a-496e-9089-c8a4539091d6-maas-api-service-ca\") pod \"e96d402a-359a-496e-9089-c8a4539091d6\" (UID: \"e96d402a-359a-496e-9089-c8a4539091d6\") " Jul 14 10:12:22.002086 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:22.002061 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e96d402a-359a-496e-9089-c8a4539091d6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e96d402a-359a-496e-9089-c8a4539091d6" (UID: "e96d402a-359a-496e-9089-c8a4539091d6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 10:12:22.002162 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:22.002143 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e96d402a-359a-496e-9089-c8a4539091d6-maas-api-service-ca\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 10:12:22.754937 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:22.754913 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ad5ead-fvn5k" Jul 14 10:12:22.755111 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:22.754954 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ad5ead-fvn5k" event={"ID":"e96d402a-359a-496e-9089-c8a4539091d6","Type":"ContainerDied","Data":"b5137c7e3f2970f36d2f82b96f6461bc7a9b1916e2968483986f1105254a70d2"} Jul 14 10:12:22.755111 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:22.754999 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5137c7e3f2970f36d2f82b96f6461bc7a9b1916e2968483986f1105254a70d2" Jul 14 10:12:29.653663 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:29.653629 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ad5ead-7bb88b7bb8-wkqks"] Jul 14 10:12:29.654250 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:29.653986 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ad5ead-7bb88b7bb8-wkqks" podUID="7167fa9f-5153-4d52-9e75-f6a4b43a78f1" containerName="maas-api" containerID="cri-o://4cc4381dd7f15ca2f152456967a08a8be1c5fadc9435c6ccb19ac61ff980d129" gracePeriod=30 Jul 14 10:12:29.800515 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:29.800481 2565 generic.go:356] "Generic (PLEG): container finished" podID="7167fa9f-5153-4d52-9e75-f6a4b43a78f1" containerID="4cc4381dd7f15ca2f152456967a08a8be1c5fadc9435c6ccb19ac61ff980d129" exitCode=0 Jul 14 10:12:29.800687 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:29.800561 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ad5ead-7bb88b7bb8-wkqks" event={"ID":"7167fa9f-5153-4d52-9e75-f6a4b43a78f1","Type":"ContainerDied","Data":"4cc4381dd7f15ca2f152456967a08a8be1c5fadc9435c6ccb19ac61ff980d129"} Jul 14 10:12:29.907694 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:29.907629 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ad5ead-7bb88b7bb8-wkqks" Jul 14 10:12:29.971636 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:29.971606 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mnzj2\" (UniqueName: \"kubernetes.io/projected/7167fa9f-5153-4d52-9e75-f6a4b43a78f1-kube-api-access-mnzj2\") pod \"7167fa9f-5153-4d52-9e75-f6a4b43a78f1\" (UID: \"7167fa9f-5153-4d52-9e75-f6a4b43a78f1\") " Jul 14 10:12:29.971801 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:29.971675 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7167fa9f-5153-4d52-9e75-f6a4b43a78f1-openshift-service-ca\") pod \"7167fa9f-5153-4d52-9e75-f6a4b43a78f1\" (UID: \"7167fa9f-5153-4d52-9e75-f6a4b43a78f1\") " Jul 14 10:12:29.971801 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:29.971721 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7167fa9f-5153-4d52-9e75-f6a4b43a78f1-maas-api-tls\") pod \"7167fa9f-5153-4d52-9e75-f6a4b43a78f1\" (UID: \"7167fa9f-5153-4d52-9e75-f6a4b43a78f1\") " Jul 14 10:12:29.972096 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:29.972069 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7167fa9f-5153-4d52-9e75-f6a4b43a78f1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7167fa9f-5153-4d52-9e75-f6a4b43a78f1" (UID: "7167fa9f-5153-4d52-9e75-f6a4b43a78f1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 10:12:29.973773 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:29.973746 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7167fa9f-5153-4d52-9e75-f6a4b43a78f1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7167fa9f-5153-4d52-9e75-f6a4b43a78f1" (UID: "7167fa9f-5153-4d52-9e75-f6a4b43a78f1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 10:12:29.973886 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:29.973828 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7167fa9f-5153-4d52-9e75-f6a4b43a78f1-kube-api-access-mnzj2" (OuterVolumeSpecName: "kube-api-access-mnzj2") pod "7167fa9f-5153-4d52-9e75-f6a4b43a78f1" (UID: "7167fa9f-5153-4d52-9e75-f6a4b43a78f1"). InnerVolumeSpecName "kube-api-access-mnzj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 10:12:30.072784 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:30.072743 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnzj2\" (UniqueName: \"kubernetes.io/projected/7167fa9f-5153-4d52-9e75-f6a4b43a78f1-kube-api-access-mnzj2\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 10:12:30.072784 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:30.072777 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7167fa9f-5153-4d52-9e75-f6a4b43a78f1-openshift-service-ca\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 10:12:30.072784 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:30.072790 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7167fa9f-5153-4d52-9e75-f6a4b43a78f1-maas-api-tls\") on node \"ip-10-0-132-2.ec2.internal\" DevicePath \"\"" Jul 14 10:12:30.808603 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:30.808574 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ad5ead-7bb88b7bb8-wkqks" Jul 14 10:12:30.809070 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:30.808607 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ad5ead-7bb88b7bb8-wkqks" event={"ID":"7167fa9f-5153-4d52-9e75-f6a4b43a78f1","Type":"ContainerDied","Data":"2d52e6b41a95ba847f4b564e0d889f752c4942f541e1162f8e812feb0ae5c00b"} Jul 14 10:12:30.809070 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:30.808661 2565 scope.go:117] "RemoveContainer" containerID="4cc4381dd7f15ca2f152456967a08a8be1c5fadc9435c6ccb19ac61ff980d129" Jul 14 10:12:30.831393 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:30.831369 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ad5ead-7bb88b7bb8-wkqks"] Jul 14 10:12:30.835609 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:30.835583 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ad5ead-7bb88b7bb8-wkqks"] Jul 14 10:12:30.969795 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:30.969749 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7167fa9f-5153-4d52-9e75-f6a4b43a78f1" path="/var/lib/kubelet/pods/7167fa9f-5153-4d52-9e75-f6a4b43a78f1/volumes" Jul 14 10:12:50.979966 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:50.979938 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-644f4b6bc9-9r548_59b87710-b331-4ad8-bc72-bc9f35c2d4f5/manager/0.log" Jul 14 10:12:50.986398 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:50.986379 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-644f4b6bc9-9r548_59b87710-b331-4ad8-bc72-bc9f35c2d4f5/copy-manifests/0.log" Jul 14 10:12:51.570937 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:51.570902 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-85d65f5d9f-xjxbs_4036b4fc-90ec-43de-a369-ce999774cc3d/maas-api/0.log" Jul 14 10:12:51.787817 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:51.787786 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29733720-zc2zt_0e3bf8ce-3ebb-44b4-a5a7-bc780305f8b1/cleanup/0.log" Jul 14 10:12:51.891770 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:51.891746 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-ae87aa32-xtvb9_6624bc71-519c-49a2-a18c-30ca7b435901/revoke-keys/0.log" Jul 14 10:12:52.144955 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:52.144861 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-a-b26ed2-wrcng_41540f64-cbe0-45c5-9087-856a4c63bdfc/revoke-keys/0.log" Jul 14 10:12:52.259474 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:52.259449 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-6cb46d-ghvcm_d5c5a23c-6742-4da3-92a2-131898d57a64/revoke-keys/0.log" Jul 14 10:12:52.401833 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:52.401719 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-derive-376a5d8c-wdnb8_5dc86b02-1d67-4539-b3eb-df3ccdcda27c/revoke-keys/0.log" Jul 14 10:12:52.634881 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:52.634851 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-b-d7ae40-pd6zm_9b819a49-91da-4f32-80be-3df03d321a8a/revoke-keys/0.log" Jul 14 10:12:53.188426 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:53.188400 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-b-ad5ead-fvn5k_e96d402a-359a-496e-9089-c8a4539091d6/revoke-keys/0.log" Jul 14 10:12:53.295987 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:53.295952 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-wtrvz_f681359f-dde2-4f16-ba74-7b13cd6b6a26/postgres/0.log" Jul 14 10:12:54.082839 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:54.082791 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7597xwn5_d1f66f44-3ca4-4d76-bb20-4c7e830ac11e/util/0.log" Jul 14 10:12:54.090218 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:54.090187 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7597xwn5_d1f66f44-3ca4-4d76-bb20-4c7e830ac11e/pull/0.log" Jul 14 10:12:54.104614 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:54.104578 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7597xwn5_d1f66f44-3ca4-4d76-bb20-4c7e830ac11e/extract/0.log" Jul 14 10:12:54.218018 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:54.217986 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf59x_9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e/extract/0.log" Jul 14 10:12:54.223559 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:54.223537 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf59x_9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e/util/0.log" Jul 14 10:12:54.229265 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:54.229224 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf59x_9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e/pull/0.log" Jul 14 10:12:54.340767 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:54.340683 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739h42f_b51c5501-6d06-4b37-b989-08b6016bf4e5/util/0.log" Jul 14 10:12:54.347824 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:54.347785 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739h42f_b51c5501-6d06-4b37-b989-08b6016bf4e5/pull/0.log" Jul 14 10:12:54.354459 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:54.354428 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739h42f_b51c5501-6d06-4b37-b989-08b6016bf4e5/extract/0.log" Jul 14 10:12:54.459490 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:54.459464 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14cnhz_c1cfca97-329a-4ce6-bca8-d711c05cfe2c/util/0.log" Jul 14 10:12:54.466334 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:54.466298 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14cnhz_c1cfca97-329a-4ce6-bca8-d711c05cfe2c/pull/0.log" Jul 14 10:12:54.472312 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:54.472287 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14cnhz_c1cfca97-329a-4ce6-bca8-d711c05cfe2c/extract/0.log" Jul 14 10:12:54.699551 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:54.699517 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-829mm_4efb1cb8-0291-4753-a575-dfbb9aebf533/manager/0.log" Jul 14 10:12:54.919525 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:54.919495 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-stf5s_851849d6-2fb4-4aa7-ac07-bbd0c3daaedd/kuadrant-console-plugin/0.log" Jul 14 10:12:55.041625 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:55.041541 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-wlmn2_16802750-6790-487f-95ac-cc5acd349214/registry-server/0.log" Jul 14 10:12:56.193402 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:56.193370 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-6c9dg_5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a/istio-proxy/1.log" Jul 14 10:12:56.314455 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:56.314429 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-24r2v_56e18c92-abdf-47ad-98da-0c68f6c38e81/payload-pre-processing/0.log" Jul 14 10:12:56.884541 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:56.884512 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-4jhsg_d181d90d-3be0-4c6e-aeb0-7f458819d582/main/0.log" Jul 14 10:12:57.101451 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:57.101421 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-hz9cw_62a6b10d-5e6a-46f1-9eb9-3174ea1a264d/main/0.log" Jul 14 10:12:57.204638 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:12:57.204606 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbl86wm_ff85a849-d87e-481d-a8fd-2dcff42e4a71/main/0.log" Jul 14 10:13:01.427666 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:01.427630 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-s29td/must-gather-sz8wt"] Jul 14 10:13:01.428130 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:01.428107 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e96d402a-359a-496e-9089-c8a4539091d6" containerName="revoke-keys" Jul 14 10:13:01.428130 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:01.428120 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e96d402a-359a-496e-9089-c8a4539091d6" containerName="revoke-keys" Jul 14 10:13:01.428130 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:01.428129 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7167fa9f-5153-4d52-9e75-f6a4b43a78f1" containerName="maas-api" Jul 14 10:13:01.428258 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:01.428134 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7167fa9f-5153-4d52-9e75-f6a4b43a78f1" containerName="maas-api" Jul 14 10:13:01.428258 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:01.428206 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7167fa9f-5153-4d52-9e75-f6a4b43a78f1" containerName="maas-api" Jul 14 10:13:01.428258 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:01.428219 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e96d402a-359a-496e-9089-c8a4539091d6" containerName="revoke-keys" Jul 14 10:13:01.431486 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:01.431470 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s29td/must-gather-sz8wt" Jul 14 10:13:01.434717 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:01.434695 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-s29td\"/\"default-dockercfg-zq65n\"" Jul 14 10:13:01.434950 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:01.434931 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s29td\"/\"openshift-service-ca.crt\"" Jul 14 10:13:01.436048 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:01.436029 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s29td\"/\"kube-root-ca.crt\"" Jul 14 10:13:01.453545 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:01.453522 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s29td/must-gather-sz8wt"] Jul 14 10:13:01.580982 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:01.580947 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhlwg\" (UniqueName: \"kubernetes.io/projected/ca549d8a-dd0b-4ee9-a28e-16eac3d9c669-kube-api-access-fhlwg\") pod \"must-gather-sz8wt\" (UID: \"ca549d8a-dd0b-4ee9-a28e-16eac3d9c669\") " pod="openshift-must-gather-s29td/must-gather-sz8wt" Jul 14 10:13:01.581160 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:01.581009 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ca549d8a-dd0b-4ee9-a28e-16eac3d9c669-must-gather-output\") pod \"must-gather-sz8wt\" (UID: \"ca549d8a-dd0b-4ee9-a28e-16eac3d9c669\") " pod="openshift-must-gather-s29td/must-gather-sz8wt" Jul 14 10:13:01.682066 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:01.681980 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ca549d8a-dd0b-4ee9-a28e-16eac3d9c669-must-gather-output\") pod \"must-gather-sz8wt\" (UID: \"ca549d8a-dd0b-4ee9-a28e-16eac3d9c669\") " pod="openshift-must-gather-s29td/must-gather-sz8wt" Jul 14 10:13:01.682232 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:01.682083 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhlwg\" (UniqueName: \"kubernetes.io/projected/ca549d8a-dd0b-4ee9-a28e-16eac3d9c669-kube-api-access-fhlwg\") pod \"must-gather-sz8wt\" (UID: \"ca549d8a-dd0b-4ee9-a28e-16eac3d9c669\") " pod="openshift-must-gather-s29td/must-gather-sz8wt" Jul 14 10:13:01.682348 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:01.682327 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ca549d8a-dd0b-4ee9-a28e-16eac3d9c669-must-gather-output\") pod \"must-gather-sz8wt\" (UID: \"ca549d8a-dd0b-4ee9-a28e-16eac3d9c669\") " pod="openshift-must-gather-s29td/must-gather-sz8wt" Jul 14 10:13:01.691957 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:01.691921 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhlwg\" (UniqueName: \"kubernetes.io/projected/ca549d8a-dd0b-4ee9-a28e-16eac3d9c669-kube-api-access-fhlwg\") pod \"must-gather-sz8wt\" (UID: \"ca549d8a-dd0b-4ee9-a28e-16eac3d9c669\") " pod="openshift-must-gather-s29td/must-gather-sz8wt" Jul 14 10:13:01.741187 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:01.741151 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s29td/must-gather-sz8wt" Jul 14 10:13:01.891105 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:01.891067 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s29td/must-gather-sz8wt"] Jul 14 10:13:01.892381 ip-10-0-132-2 kubenswrapper[2565]: W0714 10:13:01.892354 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca549d8a_dd0b_4ee9_a28e_16eac3d9c669.slice/crio-3836041a95f4ac455e97400fe9fa0ab3ddf485751530f2a59906a03b9213ab84 WatchSource:0}: Error finding container 3836041a95f4ac455e97400fe9fa0ab3ddf485751530f2a59906a03b9213ab84: Status 404 returned error can't find the container with id 3836041a95f4ac455e97400fe9fa0ab3ddf485751530f2a59906a03b9213ab84 Jul 14 10:13:01.995838 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:01.995790 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s29td/must-gather-sz8wt" event={"ID":"ca549d8a-dd0b-4ee9-a28e-16eac3d9c669","Type":"ContainerStarted","Data":"3836041a95f4ac455e97400fe9fa0ab3ddf485751530f2a59906a03b9213ab84"} Jul 14 10:13:04.009874 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:04.009838 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s29td/must-gather-sz8wt" event={"ID":"ca549d8a-dd0b-4ee9-a28e-16eac3d9c669","Type":"ContainerStarted","Data":"ce428bfece2c0fc2550a6c76ff64e03c568b7aaf5a39280d33f2acd3aa820d00"} Jul 14 10:13:04.010297 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:04.009884 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s29td/must-gather-sz8wt" event={"ID":"ca549d8a-dd0b-4ee9-a28e-16eac3d9c669","Type":"ContainerStarted","Data":"7bd0a059499e3fd347bf8199c2ba114558ae3dfbff0671456a16fd286d4f3b14"} Jul 14 10:13:04.028881 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:04.028799 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-s29td/must-gather-sz8wt" podStartSLOduration=1.865340249 podStartE2EDuration="3.028777277s" totalImagesPullingTime="1.163437028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 10:13:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 10:13:04.027866799 +0000 UTC m=+2773.537098249" watchObservedRunningTime="2026-07-14 10:13:04.028777277 +0000 UTC m=+2773.538008726" Jul 14 10:13:04.664742 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:04.664702 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2tcgq_a5c62b6a-f781-4066-9773-a5f4e8921276/global-pull-secret-syncer/0.log" Jul 14 10:13:04.852883 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:04.852848 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jjwzx_dc4d1b63-99f8-4738-9dee-76fe0e4e4e4d/konnectivity-agent/0.log" Jul 14 10:13:04.937388 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:04.937291 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-2.ec2.internal_9a1079819acdf201f17fa12e60d64a01/haproxy/0.log" Jul 14 10:13:09.250068 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:09.250038 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7597xwn5_d1f66f44-3ca4-4d76-bb20-4c7e830ac11e/extract/0.log" Jul 14 10:13:09.280751 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:09.280722 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7597xwn5_d1f66f44-3ca4-4d76-bb20-4c7e830ac11e/util/0.log" Jul 14 10:13:09.306638 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:09.306605 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7597xwn5_d1f66f44-3ca4-4d76-bb20-4c7e830ac11e/pull/0.log" Jul 14 10:13:09.334703 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:09.334674 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf59x_9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e/extract/0.log" Jul 14 10:13:09.360959 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:09.360929 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf59x_9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e/util/0.log" Jul 14 10:13:09.386418 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:09.386385 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0sf59x_9c77c9bc-4bbb-4d23-8ebf-1b552a5cdf4e/pull/0.log" Jul 14 10:13:09.415013 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:09.414981 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739h42f_b51c5501-6d06-4b37-b989-08b6016bf4e5/extract/0.log" Jul 14 10:13:09.440057 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:09.440022 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739h42f_b51c5501-6d06-4b37-b989-08b6016bf4e5/util/0.log" Jul 14 10:13:09.463758 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:09.463724 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739h42f_b51c5501-6d06-4b37-b989-08b6016bf4e5/pull/0.log" Jul 14 10:13:09.493341 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:09.493309 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14cnhz_c1cfca97-329a-4ce6-bca8-d711c05cfe2c/extract/0.log" Jul 14 10:13:09.522594 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:09.522247 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14cnhz_c1cfca97-329a-4ce6-bca8-d711c05cfe2c/util/0.log" Jul 14 10:13:09.550424 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:09.550367 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14cnhz_c1cfca97-329a-4ce6-bca8-d711c05cfe2c/pull/0.log" Jul 14 10:13:09.899825 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:09.899555 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-829mm_4efb1cb8-0291-4753-a575-dfbb9aebf533/manager/0.log" Jul 14 10:13:09.956845 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:09.956711 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-stf5s_851849d6-2fb4-4aa7-ac07-bbd0c3daaedd/kuadrant-console-plugin/0.log" Jul 14 10:13:10.029360 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:10.028720 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-wlmn2_16802750-6790-487f-95ac-cc5acd349214/registry-server/0.log" Jul 14 10:13:11.833490 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:11.833453 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc/alertmanager/0.log" Jul 14 10:13:11.864556 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:11.864524 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc/config-reloader/0.log" Jul 14 10:13:11.891587 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:11.891562 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc/kube-rbac-proxy-web/0.log" Jul 14 10:13:11.918134 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:11.918099 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc/kube-rbac-proxy/0.log" Jul 14 10:13:11.948135 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:11.948075 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc/kube-rbac-proxy-metric/0.log" Jul 14 10:13:11.973928 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:11.973902 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc/prom-label-proxy/0.log" Jul 14 10:13:12.001528 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:12.001500 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5c3172dc-8d4c-41cb-b7dd-4b9b29ba7adc/init-config-reloader/0.log" Jul 14 10:13:12.105845 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:12.105795 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-sw226_493cb380-3b2f-4f83-b251-c40f63b0b655/kube-state-metrics/0.log" Jul 14 10:13:12.134194 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:12.134162 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-sw226_493cb380-3b2f-4f83-b251-c40f63b0b655/kube-rbac-proxy-main/0.log" Jul 14 10:13:12.163153 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:12.163119 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-sw226_493cb380-3b2f-4f83-b251-c40f63b0b655/kube-rbac-proxy-self/0.log" Jul 14 10:13:12.197370 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:12.197337 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6cdc69fb5d-nqq6h_0f13c0a0-4bcc-4dde-b3ea-a738d8d129e0/metrics-server/0.log" Jul 14 10:13:12.224117 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:12.224081 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-48hk5_3834c168-54af-47d8-9787-8a74fe157c53/monitoring-plugin/0.log" Jul 14 10:13:12.334699 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:12.334625 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mh6sh_be19da8e-9e05-4c57-87b8-3da6b97e9514/node-exporter/0.log" Jul 14 10:13:12.354566 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:12.354493 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mh6sh_be19da8e-9e05-4c57-87b8-3da6b97e9514/kube-rbac-proxy/0.log" Jul 14 10:13:12.378181 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:12.378152 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mh6sh_be19da8e-9e05-4c57-87b8-3da6b97e9514/init-textfile/0.log" Jul 14 10:13:12.498942 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:12.498915 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-5x5kh_6eab5a46-4afa-4e95-afa5-2a235e5e463c/kube-rbac-proxy-main/0.log" Jul 14 10:13:12.521978 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:12.521948 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-5x5kh_6eab5a46-4afa-4e95-afa5-2a235e5e463c/kube-rbac-proxy-self/0.log" Jul 14 10:13:12.545701 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:12.545672 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-5x5kh_6eab5a46-4afa-4e95-afa5-2a235e5e463c/openshift-state-metrics/0.log" Jul 14 10:13:12.766755 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:12.766717 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-qszj9_48183b82-41a3-4075-b553-36bfd3742176/prometheus-operator/0.log" Jul 14 10:13:12.787878 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:12.787849 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-qszj9_48183b82-41a3-4075-b553-36bfd3742176/kube-rbac-proxy/0.log" Jul 14 10:13:12.810164 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:12.810131 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-vtqjr_a2766a64-d5c3-4685-b76a-b61e0ffb0104/prometheus-operator-admission-webhook/0.log" Jul 14 10:13:12.839697 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:12.839659 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-69cf5d869b-qwtcz_7f3c3a84-58b7-4794-b743-4ea53077b8ce/telemeter-client/0.log" Jul 14 10:13:12.863378 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:12.863344 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-69cf5d869b-qwtcz_7f3c3a84-58b7-4794-b743-4ea53077b8ce/reload/0.log" Jul 14 10:13:12.896840 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:12.896793 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-69cf5d869b-qwtcz_7f3c3a84-58b7-4794-b743-4ea53077b8ce/kube-rbac-proxy/0.log" Jul 14 10:13:12.932827 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:12.932782 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9c5fd5b4b-gbbk7_85e26c8a-faa2-43bb-90e9-1d8175b8f8a3/thanos-query/0.log" Jul 14 10:13:12.954191 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:12.954158 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9c5fd5b4b-gbbk7_85e26c8a-faa2-43bb-90e9-1d8175b8f8a3/kube-rbac-proxy-web/0.log" Jul 14 10:13:12.981500 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:12.981459 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9c5fd5b4b-gbbk7_85e26c8a-faa2-43bb-90e9-1d8175b8f8a3/kube-rbac-proxy/0.log" Jul 14 10:13:13.005492 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:13.005378 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9c5fd5b4b-gbbk7_85e26c8a-faa2-43bb-90e9-1d8175b8f8a3/prom-label-proxy/0.log" Jul 14 10:13:13.028475 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:13.028386 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9c5fd5b4b-gbbk7_85e26c8a-faa2-43bb-90e9-1d8175b8f8a3/kube-rbac-proxy-rules/0.log" Jul 14 10:13:13.050510 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:13.050478 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9c5fd5b4b-gbbk7_85e26c8a-faa2-43bb-90e9-1d8175b8f8a3/kube-rbac-proxy-metrics/0.log" Jul 14 10:13:14.995305 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:14.995257 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-s29td/perf-node-gather-daemonset-gm4hz"] Jul 14 10:13:15.003853 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:15.003826 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s29td/perf-node-gather-daemonset-gm4hz" Jul 14 10:13:15.013955 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:15.013923 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s29td/perf-node-gather-daemonset-gm4hz"] Jul 14 10:13:15.055189 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:15.055142 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x4rd\" (UniqueName: \"kubernetes.io/projected/c4050a38-0a06-4a50-9484-0f3aaea8f5ab-kube-api-access-4x4rd\") pod \"perf-node-gather-daemonset-gm4hz\" (UID: \"c4050a38-0a06-4a50-9484-0f3aaea8f5ab\") " pod="openshift-must-gather-s29td/perf-node-gather-daemonset-gm4hz" Jul 14 10:13:15.055395 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:15.055268 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c4050a38-0a06-4a50-9484-0f3aaea8f5ab-proc\") pod \"perf-node-gather-daemonset-gm4hz\" (UID: \"c4050a38-0a06-4a50-9484-0f3aaea8f5ab\") " pod="openshift-must-gather-s29td/perf-node-gather-daemonset-gm4hz" Jul 14 10:13:15.055395 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:15.055318 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c4050a38-0a06-4a50-9484-0f3aaea8f5ab-lib-modules\") pod \"perf-node-gather-daemonset-gm4hz\" (UID: \"c4050a38-0a06-4a50-9484-0f3aaea8f5ab\") " pod="openshift-must-gather-s29td/perf-node-gather-daemonset-gm4hz" Jul 14 10:13:15.055395 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:15.055381 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c4050a38-0a06-4a50-9484-0f3aaea8f5ab-sys\") pod \"perf-node-gather-daemonset-gm4hz\" (UID: \"c4050a38-0a06-4a50-9484-0f3aaea8f5ab\") " pod="openshift-must-gather-s29td/perf-node-gather-daemonset-gm4hz" Jul 14 10:13:15.055583 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:15.055424 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c4050a38-0a06-4a50-9484-0f3aaea8f5ab-podres\") pod \"perf-node-gather-daemonset-gm4hz\" (UID: \"c4050a38-0a06-4a50-9484-0f3aaea8f5ab\") " pod="openshift-must-gather-s29td/perf-node-gather-daemonset-gm4hz" Jul 14 10:13:15.156994 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:15.156954 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c4050a38-0a06-4a50-9484-0f3aaea8f5ab-podres\") pod \"perf-node-gather-daemonset-gm4hz\" (UID: \"c4050a38-0a06-4a50-9484-0f3aaea8f5ab\") " pod="openshift-must-gather-s29td/perf-node-gather-daemonset-gm4hz" Jul 14 10:13:15.157189 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:15.157055 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x4rd\" (UniqueName: \"kubernetes.io/projected/c4050a38-0a06-4a50-9484-0f3aaea8f5ab-kube-api-access-4x4rd\") pod \"perf-node-gather-daemonset-gm4hz\" (UID: \"c4050a38-0a06-4a50-9484-0f3aaea8f5ab\") " pod="openshift-must-gather-s29td/perf-node-gather-daemonset-gm4hz" Jul 14 10:13:15.157189 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:15.157184 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c4050a38-0a06-4a50-9484-0f3aaea8f5ab-proc\") pod \"perf-node-gather-daemonset-gm4hz\" (UID: \"c4050a38-0a06-4a50-9484-0f3aaea8f5ab\") " pod="openshift-must-gather-s29td/perf-node-gather-daemonset-gm4hz" Jul 14 10:13:15.157330 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:15.157224 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c4050a38-0a06-4a50-9484-0f3aaea8f5ab-podres\") pod \"perf-node-gather-daemonset-gm4hz\" (UID: \"c4050a38-0a06-4a50-9484-0f3aaea8f5ab\") " pod="openshift-must-gather-s29td/perf-node-gather-daemonset-gm4hz" Jul 14 10:13:15.157330 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:15.157308 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c4050a38-0a06-4a50-9484-0f3aaea8f5ab-lib-modules\") pod \"perf-node-gather-daemonset-gm4hz\" (UID: \"c4050a38-0a06-4a50-9484-0f3aaea8f5ab\") " pod="openshift-must-gather-s29td/perf-node-gather-daemonset-gm4hz" Jul 14 10:13:15.157448 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:15.157393 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c4050a38-0a06-4a50-9484-0f3aaea8f5ab-sys\") pod \"perf-node-gather-daemonset-gm4hz\" (UID: \"c4050a38-0a06-4a50-9484-0f3aaea8f5ab\") " pod="openshift-must-gather-s29td/perf-node-gather-daemonset-gm4hz" Jul 14 10:13:15.157558 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:15.157540 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c4050a38-0a06-4a50-9484-0f3aaea8f5ab-sys\") pod \"perf-node-gather-daemonset-gm4hz\" (UID: \"c4050a38-0a06-4a50-9484-0f3aaea8f5ab\") " pod="openshift-must-gather-s29td/perf-node-gather-daemonset-gm4hz" Jul 14 10:13:15.157632 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:15.157619 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c4050a38-0a06-4a50-9484-0f3aaea8f5ab-proc\") pod \"perf-node-gather-daemonset-gm4hz\" (UID: \"c4050a38-0a06-4a50-9484-0f3aaea8f5ab\") " pod="openshift-must-gather-s29td/perf-node-gather-daemonset-gm4hz" Jul 14 10:13:15.157690 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:15.157661 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c4050a38-0a06-4a50-9484-0f3aaea8f5ab-lib-modules\") pod \"perf-node-gather-daemonset-gm4hz\" (UID: \"c4050a38-0a06-4a50-9484-0f3aaea8f5ab\") " pod="openshift-must-gather-s29td/perf-node-gather-daemonset-gm4hz" Jul 14 10:13:15.166321 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:15.166236 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x4rd\" (UniqueName: \"kubernetes.io/projected/c4050a38-0a06-4a50-9484-0f3aaea8f5ab-kube-api-access-4x4rd\") pod \"perf-node-gather-daemonset-gm4hz\" (UID: \"c4050a38-0a06-4a50-9484-0f3aaea8f5ab\") " pod="openshift-must-gather-s29td/perf-node-gather-daemonset-gm4hz" Jul 14 10:13:15.325637 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:15.325540 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s29td/perf-node-gather-daemonset-gm4hz" Jul 14 10:13:15.597847 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:15.597799 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s29td/perf-node-gather-daemonset-gm4hz"] Jul 14 10:13:15.599018 ip-10-0-132-2 kubenswrapper[2565]: W0714 10:13:15.598987 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc4050a38_0a06_4a50_9484_0f3aaea8f5ab.slice/crio-9d9300562f9660b5df35d5de57be57aeb6a4057c07d7525b4be7b91ce814096f WatchSource:0}: Error finding container 9d9300562f9660b5df35d5de57be57aeb6a4057c07d7525b4be7b91ce814096f: Status 404 returned error can't find the container with id 9d9300562f9660b5df35d5de57be57aeb6a4057c07d7525b4be7b91ce814096f Jul 14 10:13:15.604150 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:15.604127 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-56894c8c45-vc4p9_45a87a74-8b57-4bf1-87ec-891cc41b76ef/console/0.log" Jul 14 10:13:16.139363 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:16.139320 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s29td/perf-node-gather-daemonset-gm4hz" event={"ID":"c4050a38-0a06-4a50-9484-0f3aaea8f5ab","Type":"ContainerStarted","Data":"f4b35baca0f0fb84ce89428eabf71a2853471764263d6db28234008904c0606f"} Jul 14 10:13:16.139875 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:16.139441 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-s29td/perf-node-gather-daemonset-gm4hz" Jul 14 10:13:16.139875 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:16.139470 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s29td/perf-node-gather-daemonset-gm4hz" event={"ID":"c4050a38-0a06-4a50-9484-0f3aaea8f5ab","Type":"ContainerStarted","Data":"9d9300562f9660b5df35d5de57be57aeb6a4057c07d7525b4be7b91ce814096f"} Jul 14 10:13:16.157469 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:16.157414 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-s29td/perf-node-gather-daemonset-gm4hz" podStartSLOduration=2.1573947159999998 podStartE2EDuration="2.157394716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 10:13:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 10:13:16.15468906 +0000 UTC m=+2785.663920534" watchObservedRunningTime="2026-07-14 10:13:16.157394716 +0000 UTC m=+2785.666626163" Jul 14 10:13:17.374890 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:17.374847 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-m7mrj_537a71a2-5db0-421b-8f77-76c91b07ac77/dns/0.log" Jul 14 10:13:17.398219 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:17.398185 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-m7mrj_537a71a2-5db0-421b-8f77-76c91b07ac77/kube-rbac-proxy/0.log" Jul 14 10:13:17.471696 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:17.471664 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5r769_4a7e6b7e-3c6a-4def-bc24-b9a00ccad67b/dns-node-resolver/0.log" Jul 14 10:13:18.098727 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:18.098698 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4kb48_3cfb37de-4170-43d8-aa35-d2f1c3c6cc90/node-ca/0.log" Jul 14 10:13:19.328770 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:19.328728 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-6c9dg_5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a/istio-proxy/0.log" Jul 14 10:13:19.385364 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:19.385322 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-6c9dg_5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a/istio-proxy/1.log" Jul 14 10:13:19.420844 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:19.420794 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-24r2v_56e18c92-abdf-47ad-98da-0c68f6c38e81/payload-pre-processing/0.log" Jul 14 10:13:20.222359 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:20.222330 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zlzfk_8f2133f7-7ee2-48c5-bc8f-1f17049b6c46/serve-healthcheck-canary/0.log" Jul 14 10:13:20.702277 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:20.702244 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hr4j6_5cffbd14-98ff-4790-aaaa-3b0ca91450fa/kube-rbac-proxy/0.log" Jul 14 10:13:20.725873 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:20.725844 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hr4j6_5cffbd14-98ff-4790-aaaa-3b0ca91450fa/exporter/0.log" Jul 14 10:13:20.749411 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:20.749354 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hr4j6_5cffbd14-98ff-4790-aaaa-3b0ca91450fa/extractor/0.log" Jul 14 10:13:22.158291 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:22.158260 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-s29td/perf-node-gather-daemonset-gm4hz" Jul 14 10:13:23.314070 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:23.314040 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-644f4b6bc9-9r548_59b87710-b331-4ad8-bc72-bc9f35c2d4f5/manager/0.log" Jul 14 10:13:23.341108 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:23.341083 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-644f4b6bc9-9r548_59b87710-b331-4ad8-bc72-bc9f35c2d4f5/copy-manifests/0.log" Jul 14 10:13:24.952279 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:24.952244 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-56954f6f7d-5zgsf_5cf780ae-cba9-467b-a3b0-32928bbb4c57/manager/0.log" Jul 14 10:13:24.984877 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:24.984784 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-xcrns_80022556-10a2-464a-8b8a-a1798cb32ae9/openshift-lws-operator/0.log" Jul 14 10:13:25.648496 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:25.648463 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-6c9dg_5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a/istio-proxy/0.log" Jul 14 10:13:25.715910 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:25.715878 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-6c9dg_5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a/istio-proxy/1.log" Jul 14 10:13:25.747941 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:25.747913 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-24r2v_56e18c92-abdf-47ad-98da-0c68f6c38e81/payload-pre-processing/0.log" Jul 14 10:13:27.247916 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:27.247888 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-6c9dg_5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a/istio-proxy/0.log" Jul 14 10:13:27.307636 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:27.307554 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-6c9dg_5506cd2a-20fa-48d2-a5bb-4e2b81d2c35a/istio-proxy/1.log" Jul 14 10:13:27.338083 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:27.338041 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-24r2v_56e18c92-abdf-47ad-98da-0c68f6c38e81/payload-pre-processing/0.log" Jul 14 10:13:31.607640 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:31.607607 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-9xsfz_0274da0d-7c52-4932-9ac3-732af03b0c85/kube-multus/0.log" Jul 14 10:13:31.812691 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:31.812665 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9z4f8_82f2aa67-8042-4bbb-930d-09ad08fd1696/kube-multus-additional-cni-plugins/0.log" Jul 14 10:13:31.841016 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:31.840989 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9z4f8_82f2aa67-8042-4bbb-930d-09ad08fd1696/egress-router-binary-copy/0.log" Jul 14 10:13:31.872640 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:31.872567 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9z4f8_82f2aa67-8042-4bbb-930d-09ad08fd1696/cni-plugins/0.log" Jul 14 10:13:31.900513 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:31.900488 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9z4f8_82f2aa67-8042-4bbb-930d-09ad08fd1696/bond-cni-plugin/0.log" Jul 14 10:13:31.928730 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:31.928696 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9z4f8_82f2aa67-8042-4bbb-930d-09ad08fd1696/routeoverride-cni/0.log" Jul 14 10:13:31.958471 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:31.958446 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9z4f8_82f2aa67-8042-4bbb-930d-09ad08fd1696/whereabouts-cni-bincopy/0.log" Jul 14 10:13:31.993260 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:31.993233 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9z4f8_82f2aa67-8042-4bbb-930d-09ad08fd1696/whereabouts-cni/0.log" Jul 14 10:13:32.262259 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:32.262227 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-p5547_c2190520-5806-41dd-b25f-b7d1076cfcf3/network-metrics-daemon/0.log" Jul 14 10:13:32.282115 ip-10-0-132-2 kubenswrapper[2565]: I0714 10:13:32.282066 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-p5547_c2190520-5806-41dd-b25f-b7d1076cfcf3/kube-rbac-proxy/0.log"