Jul 17 18:53:01.555980 ip-10-0-134-137 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 17 18:53:01.555997 ip-10-0-134-137 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 17 18:53:01.556006 ip-10-0-134-137 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 17 18:53:01.556357 ip-10-0-134-137 systemd[1]: Failed to start Kubernetes Kubelet. Jul 17 18:53:11.600335 ip-10-0-134-137 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found. Jul 17 18:53:11.600353 ip-10-0-134-137 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 82f120ee06a74edf910b3d48021afae3 -- Jul 17 18:55:01.572827 ip-10-0-134-137 systemd[1]: Starting Kubernetes Kubelet... Jul 17 18:55:01.883691 ip-10-0-134-137 kubenswrapper[2561]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 17 18:55:01.883691 ip-10-0-134-137 kubenswrapper[2561]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 17 18:55:01.883691 ip-10-0-134-137 kubenswrapper[2561]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 17 18:55:01.883691 ip-10-0-134-137 kubenswrapper[2561]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 17 18:55:01.883691 ip-10-0-134-137 kubenswrapper[2561]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 17 18:55:01.884405 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.884316 2561 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 17 18:55:01.889100 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889078 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 18:55:01.889100 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889099 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 18:55:01.889100 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889103 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 18:55:01.889207 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889107 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 18:55:01.889207 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889110 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 18:55:01.889207 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889113 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 18:55:01.889207 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889115 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 18:55:01.889207 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889123 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 18:55:01.889207 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889127 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 18:55:01.889207 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889130 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 18:55:01.889207 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889133 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 18:55:01.889207 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889135 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 18:55:01.889207 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889138 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 18:55:01.889207 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889140 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 18:55:01.889207 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889143 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 18:55:01.889207 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889146 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 18:55:01.889207 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889148 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 18:55:01.889207 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889151 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 18:55:01.889207 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889153 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 18:55:01.889207 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889156 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 18:55:01.889207 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889158 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 18:55:01.889207 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889160 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 18:55:01.889207 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889163 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 18:55:01.889697 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889165 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 18:55:01.889697 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889168 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 18:55:01.889697 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889170 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 18:55:01.889697 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889173 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 18:55:01.889697 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889175 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 18:55:01.889697 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889177 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 18:55:01.889697 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889180 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 18:55:01.889697 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889182 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 18:55:01.889697 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889184 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 18:55:01.889697 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889187 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 18:55:01.889697 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889189 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 18:55:01.889697 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889192 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 18:55:01.889697 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889194 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 18:55:01.889697 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889198 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 18:55:01.889697 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889201 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 18:55:01.889697 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889204 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 18:55:01.889697 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889206 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 18:55:01.889697 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889209 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 18:55:01.889697 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889211 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 18:55:01.889697 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889214 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 18:55:01.890198 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889218 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 18:55:01.890198 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889222 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 18:55:01.890198 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889224 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 18:55:01.890198 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889227 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 18:55:01.890198 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889229 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 18:55:01.890198 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889232 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 18:55:01.890198 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889234 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 18:55:01.890198 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889238 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 18:55:01.890198 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889242 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 18:55:01.890198 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889245 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 18:55:01.890198 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889248 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 18:55:01.890198 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889250 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 18:55:01.890198 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889253 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 18:55:01.890198 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889257 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 18:55:01.890198 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889260 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 18:55:01.890198 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889264 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 18:55:01.890198 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889266 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 18:55:01.890198 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889268 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 18:55:01.890198 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889271 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 18:55:01.890712 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889274 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 18:55:01.890712 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889276 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 18:55:01.890712 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889280 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 18:55:01.890712 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889283 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 18:55:01.890712 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889286 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 18:55:01.890712 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889289 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 18:55:01.890712 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889292 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 18:55:01.890712 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889297 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 18:55:01.890712 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889299 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 18:55:01.890712 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889303 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 18:55:01.890712 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889305 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 18:55:01.890712 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889308 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 18:55:01.890712 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889310 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 18:55:01.890712 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889313 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 18:55:01.890712 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889315 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 18:55:01.890712 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889318 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 18:55:01.890712 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889320 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 18:55:01.890712 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889322 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 18:55:01.890712 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889325 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 18:55:01.890712 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889327 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 18:55:01.891303 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889330 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 18:55:01.891303 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889332 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 18:55:01.891303 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889335 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 18:55:01.891303 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889337 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 18:55:01.891303 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889339 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 18:55:01.891303 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889342 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 18:55:01.891303 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889344 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 18:55:01.891303 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889347 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 18:55:01.891303 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889349 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 18:55:01.891303 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889352 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 18:55:01.891303 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889355 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 18:55:01.891303 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889357 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 18:55:01.891303 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889360 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 18:55:01.891303 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889362 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 18:55:01.891303 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889364 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 17 18:55:01.891303 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889367 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 18:55:01.891303 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.889369 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 18:55:01.891303 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890093 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 18:55:01.891303 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890100 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 18:55:01.891303 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890103 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 18:55:01.891801 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890106 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 18:55:01.891801 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890109 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 18:55:01.891801 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890112 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 18:55:01.891801 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890115 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 18:55:01.891801 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890118 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 18:55:01.891801 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890121 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 18:55:01.891801 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890124 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 18:55:01.891801 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890128 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 18:55:01.891801 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890132 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 18:55:01.891801 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890135 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 18:55:01.891801 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890138 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 18:55:01.891801 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890141 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 18:55:01.891801 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890143 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 18:55:01.891801 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890146 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 18:55:01.891801 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890148 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 18:55:01.891801 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890152 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 18:55:01.891801 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890155 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 18:55:01.891801 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890158 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 18:55:01.891801 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890160 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 18:55:01.891801 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890163 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 18:55:01.892294 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890166 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 18:55:01.892294 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890168 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 18:55:01.892294 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890171 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 18:55:01.892294 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890173 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 18:55:01.892294 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890175 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 18:55:01.892294 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890178 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 18:55:01.892294 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890181 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 18:55:01.892294 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890183 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 18:55:01.892294 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890186 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 18:55:01.892294 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890188 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 18:55:01.892294 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890191 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 18:55:01.892294 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890194 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 18:55:01.892294 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890196 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 18:55:01.892294 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890199 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 18:55:01.892294 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890201 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 18:55:01.892294 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890204 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 18:55:01.892294 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890207 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 18:55:01.892294 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890210 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 18:55:01.892294 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890212 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 18:55:01.892294 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890215 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 18:55:01.892789 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890218 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 18:55:01.892789 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890220 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 18:55:01.892789 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890223 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 18:55:01.892789 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890225 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 18:55:01.892789 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890228 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 18:55:01.892789 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890230 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 17 18:55:01.892789 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890233 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 18:55:01.892789 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890236 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 18:55:01.892789 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890238 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 18:55:01.892789 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890241 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 18:55:01.892789 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890243 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 18:55:01.892789 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890246 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 18:55:01.892789 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890249 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 18:55:01.892789 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890252 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 18:55:01.892789 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890254 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 18:55:01.892789 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890256 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 18:55:01.892789 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890259 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 18:55:01.892789 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890261 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 18:55:01.892789 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890264 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 18:55:01.892789 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890266 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 18:55:01.893274 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890268 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 18:55:01.893274 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890271 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 18:55:01.893274 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890274 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 18:55:01.893274 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890277 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 18:55:01.893274 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890279 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 18:55:01.893274 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890282 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 18:55:01.893274 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890284 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 18:55:01.893274 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890286 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 18:55:01.893274 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890290 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 18:55:01.893274 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890292 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 18:55:01.893274 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890294 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 18:55:01.893274 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890297 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 18:55:01.893274 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890300 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 18:55:01.893274 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890302 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 18:55:01.893274 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890306 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 18:55:01.893274 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890310 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 18:55:01.893274 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890313 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 18:55:01.893274 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890315 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 18:55:01.893274 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890318 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 18:55:01.893759 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890322 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 18:55:01.893759 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890326 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 18:55:01.893759 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890328 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 18:55:01.893759 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890331 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 18:55:01.893759 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890333 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 18:55:01.893759 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890336 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 18:55:01.893759 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890338 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 18:55:01.893759 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890341 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 18:55:01.893759 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890343 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 18:55:01.893759 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890345 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 18:55:01.893759 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890348 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 18:55:01.893759 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890351 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 18:55:01.893759 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890353 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 18:55:01.893759 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890355 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 18:55:01.893759 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890358 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 18:55:01.893759 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890361 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 18:55:01.893759 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890363 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 18:55:01.893759 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890774 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 18:55:01.893759 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890779 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 18:55:01.894223 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890781 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 18:55:01.894223 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890784 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 18:55:01.894223 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890787 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 18:55:01.894223 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890790 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 18:55:01.894223 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890793 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 18:55:01.894223 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890795 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 18:55:01.894223 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890798 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 18:55:01.894223 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890800 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 18:55:01.894223 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890803 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 18:55:01.894223 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890805 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 18:55:01.894223 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890808 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 18:55:01.894223 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890810 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 18:55:01.894223 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890813 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 18:55:01.894223 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890816 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 18:55:01.894223 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890819 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 18:55:01.894223 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890821 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 18:55:01.894223 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890824 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 18:55:01.894223 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890826 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 18:55:01.894223 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890829 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 18:55:01.894223 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890832 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 18:55:01.894716 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890834 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 18:55:01.894716 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890837 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 18:55:01.894716 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890839 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 18:55:01.894716 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890842 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 18:55:01.894716 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890844 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 18:55:01.894716 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890847 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 18:55:01.894716 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890850 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 18:55:01.894716 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890853 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 18:55:01.894716 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890856 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 18:55:01.894716 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890859 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 18:55:01.894716 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890861 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 18:55:01.894716 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890864 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 18:55:01.894716 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890866 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 18:55:01.894716 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890869 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 18:55:01.894716 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890871 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 18:55:01.894716 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890874 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 18:55:01.894716 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890876 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 17 18:55:01.894716 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890879 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 18:55:01.894716 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890881 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 18:55:01.894716 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890884 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 18:55:01.895263 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890886 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 18:55:01.895263 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890889 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 18:55:01.895263 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890891 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 18:55:01.895263 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890894 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 18:55:01.895263 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890897 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 18:55:01.895263 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890900 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 18:55:01.895263 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890902 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 18:55:01.895263 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890905 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 18:55:01.895263 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890907 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 18:55:01.895263 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890910 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 18:55:01.895263 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890912 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 18:55:01.895263 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890914 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 18:55:01.895263 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890917 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 18:55:01.895263 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890919 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 18:55:01.895263 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890922 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 18:55:01.895263 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890924 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 18:55:01.895263 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890926 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 18:55:01.895263 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890929 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 18:55:01.895263 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890931 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 18:55:01.895263 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890934 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 18:55:01.895781 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890936 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 18:55:01.895781 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890939 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 18:55:01.895781 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890944 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 18:55:01.895781 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890947 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 18:55:01.895781 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890950 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 18:55:01.895781 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890953 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 18:55:01.895781 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890956 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 18:55:01.895781 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890960 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 18:55:01.895781 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890963 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 18:55:01.895781 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890966 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 18:55:01.895781 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890969 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 18:55:01.895781 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890971 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 18:55:01.895781 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890974 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 18:55:01.895781 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890977 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 18:55:01.895781 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890979 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 18:55:01.895781 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890983 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 18:55:01.895781 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890986 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 18:55:01.895781 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890988 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 18:55:01.895781 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890991 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 18:55:01.896236 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890993 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 18:55:01.896236 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890996 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 18:55:01.896236 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.890998 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 18:55:01.896236 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891002 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 18:55:01.896236 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891005 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 18:55:01.896236 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891007 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 18:55:01.896236 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891010 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 18:55:01.896236 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891012 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 18:55:01.896236 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891015 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 18:55:01.896236 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891017 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 18:55:01.896236 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891019 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 18:55:01.896236 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891022 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 18:55:01.896236 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891024 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 18:55:01.896236 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891027 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 18:55:01.896236 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891029 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 18:55:01.896236 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891032 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 18:55:01.896236 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891034 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 18:55:01.896236 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891037 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 18:55:01.896236 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891114 2561 flags.go:64] FLAG: --address="0.0.0.0" Jul 17 18:55:01.896236 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891122 2561 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 17 18:55:01.896727 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891128 2561 flags.go:64] FLAG: --anonymous-auth="true" Jul 17 18:55:01.896727 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891133 2561 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 17 18:55:01.896727 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891138 2561 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 17 18:55:01.896727 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891141 2561 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 17 18:55:01.896727 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891145 2561 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 17 18:55:01.896727 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891150 2561 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 17 18:55:01.896727 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891153 2561 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 17 18:55:01.896727 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891156 2561 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 17 18:55:01.896727 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891159 2561 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 17 18:55:01.896727 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891162 2561 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 17 18:55:01.896727 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891165 2561 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 17 18:55:01.896727 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891169 2561 flags.go:64] FLAG: --cgroup-root="" Jul 17 18:55:01.896727 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891172 2561 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 17 18:55:01.896727 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891175 2561 flags.go:64] FLAG: --client-ca-file="" Jul 17 18:55:01.896727 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891177 2561 flags.go:64] FLAG: --cloud-provider="external" Jul 17 18:55:01.896727 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891181 2561 flags.go:64] FLAG: --cluster-dns="[]" Jul 17 18:55:01.896727 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891185 2561 flags.go:64] FLAG: --cluster-domain="" Jul 17 18:55:01.896727 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891188 2561 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 17 18:55:01.896727 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891191 2561 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 17 18:55:01.896727 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891195 2561 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 17 18:55:01.896727 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891198 2561 flags.go:64] FLAG: --container-log-max-files="5" Jul 17 18:55:01.896727 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891202 2561 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 17 18:55:01.896727 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891205 2561 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 17 18:55:01.897279 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891208 2561 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 17 18:55:01.897279 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891211 2561 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 17 18:55:01.897279 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891214 2561 flags.go:64] FLAG: --contention-profiling="false" Jul 17 18:55:01.897279 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891217 2561 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 17 18:55:01.897279 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891220 2561 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 17 18:55:01.897279 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891223 2561 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 17 18:55:01.897279 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891226 2561 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 17 18:55:01.897279 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891230 2561 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 17 18:55:01.897279 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891233 2561 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 17 18:55:01.897279 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891236 2561 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 17 18:55:01.897279 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891238 2561 flags.go:64] FLAG: --enable-load-reader="false" Jul 17 18:55:01.897279 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891241 2561 flags.go:64] FLAG: --enable-server="true" Jul 17 18:55:01.897279 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891245 2561 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 17 18:55:01.897279 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891249 2561 flags.go:64] FLAG: --event-burst="100" Jul 17 18:55:01.897279 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891252 2561 flags.go:64] FLAG: --event-qps="50" Jul 17 18:55:01.897279 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891254 2561 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 17 18:55:01.897279 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891257 2561 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 17 18:55:01.897279 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891260 2561 flags.go:64] FLAG: --eviction-hard="" Jul 17 18:55:01.897279 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891264 2561 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 17 18:55:01.897279 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891267 2561 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 17 18:55:01.897279 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891270 2561 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 17 18:55:01.897279 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891273 2561 flags.go:64] FLAG: --eviction-soft="" Jul 17 18:55:01.897279 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891276 2561 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 17 18:55:01.897279 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891279 2561 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 17 18:55:01.897279 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891281 2561 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 17 18:55:01.897949 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891284 2561 flags.go:64] FLAG: --experimental-mounter-path="" Jul 17 18:55:01.897949 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891287 2561 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 17 18:55:01.897949 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891290 2561 flags.go:64] FLAG: --fail-swap-on="true" Jul 17 18:55:01.897949 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891292 2561 flags.go:64] FLAG: --feature-gates="" Jul 17 18:55:01.897949 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891297 2561 flags.go:64] FLAG: --file-check-frequency="20s" Jul 17 18:55:01.897949 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891300 2561 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 17 18:55:01.897949 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891302 2561 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 17 18:55:01.897949 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891308 2561 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 17 18:55:01.897949 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891311 2561 flags.go:64] FLAG: --healthz-port="10248" Jul 17 18:55:01.897949 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891314 2561 flags.go:64] FLAG: --help="false" Jul 17 18:55:01.897949 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891317 2561 flags.go:64] FLAG: --hostname-override="ip-10-0-134-137.ec2.internal" Jul 17 18:55:01.897949 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891320 2561 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 17 18:55:01.897949 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891323 2561 flags.go:64] FLAG: --http-check-frequency="20s" Jul 17 18:55:01.897949 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891326 2561 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 17 18:55:01.897949 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891329 2561 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 17 18:55:01.897949 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891332 2561 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 17 18:55:01.897949 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891335 2561 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 17 18:55:01.897949 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891338 2561 flags.go:64] FLAG: --image-service-endpoint="" Jul 17 18:55:01.897949 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891341 2561 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 17 18:55:01.897949 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891343 2561 flags.go:64] FLAG: --kube-api-burst="100" Jul 17 18:55:01.897949 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891347 2561 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 17 18:55:01.897949 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891350 2561 flags.go:64] FLAG: --kube-api-qps="50" Jul 17 18:55:01.897949 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891352 2561 flags.go:64] FLAG: --kube-reserved="" Jul 17 18:55:01.897949 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891355 2561 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 17 18:55:01.898540 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891358 2561 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 17 18:55:01.898540 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891363 2561 flags.go:64] FLAG: --kubelet-cgroups="" Jul 17 18:55:01.898540 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891365 2561 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 17 18:55:01.898540 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891368 2561 flags.go:64] FLAG: --lock-file="" Jul 17 18:55:01.898540 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891371 2561 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 17 18:55:01.898540 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891373 2561 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 17 18:55:01.898540 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891377 2561 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 17 18:55:01.898540 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891382 2561 flags.go:64] FLAG: --log-json-split-stream="false" Jul 17 18:55:01.898540 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891385 2561 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 17 18:55:01.898540 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891388 2561 flags.go:64] FLAG: --log-text-split-stream="false" Jul 17 18:55:01.898540 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891390 2561 flags.go:64] FLAG: --logging-format="text" Jul 17 18:55:01.898540 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891393 2561 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 17 18:55:01.898540 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891397 2561 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 17 18:55:01.898540 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891400 2561 flags.go:64] FLAG: --manifest-url="" Jul 17 18:55:01.898540 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891403 2561 flags.go:64] FLAG: --manifest-url-header="" Jul 17 18:55:01.898540 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891408 2561 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 17 18:55:01.898540 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891411 2561 flags.go:64] FLAG: --max-open-files="1000000" Jul 17 18:55:01.898540 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891416 2561 flags.go:64] FLAG: --max-pods="110" Jul 17 18:55:01.898540 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891419 2561 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 17 18:55:01.898540 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891422 2561 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 17 18:55:01.898540 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891424 2561 flags.go:64] FLAG: --memory-manager-policy="None" Jul 17 18:55:01.898540 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891427 2561 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 17 18:55:01.898540 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891430 2561 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 17 18:55:01.898540 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891433 2561 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 17 18:55:01.898540 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891436 2561 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 17 18:55:01.899176 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891447 2561 flags.go:64] FLAG: --node-status-max-images="50" Jul 17 18:55:01.899176 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891451 2561 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 17 18:55:01.899176 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891453 2561 flags.go:64] FLAG: --oom-score-adj="-999" Jul 17 18:55:01.899176 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891456 2561 flags.go:64] FLAG: --pod-cidr="" Jul 17 18:55:01.899176 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891460 2561 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 17 18:55:01.899176 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891465 2561 flags.go:64] FLAG: --pod-manifest-path="" Jul 17 18:55:01.899176 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891468 2561 flags.go:64] FLAG: --pod-max-pids="-1" Jul 17 18:55:01.899176 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891471 2561 flags.go:64] FLAG: --pods-per-core="0" Jul 17 18:55:01.899176 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891476 2561 flags.go:64] FLAG: --port="10250" Jul 17 18:55:01.899176 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891479 2561 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 17 18:55:01.899176 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891482 2561 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e174d2bb91b88c9b" Jul 17 18:55:01.899176 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891484 2561 flags.go:64] FLAG: --qos-reserved="" Jul 17 18:55:01.899176 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891487 2561 flags.go:64] FLAG: --read-only-port="10255" Jul 17 18:55:01.899176 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891490 2561 flags.go:64] FLAG: --register-node="true" Jul 17 18:55:01.899176 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891493 2561 flags.go:64] FLAG: --register-with-taints="" Jul 17 18:55:01.899176 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891497 2561 flags.go:64] FLAG: --registry-burst="10" Jul 17 18:55:01.899176 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891499 2561 flags.go:64] FLAG: --registry-qps="5" Jul 17 18:55:01.899176 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891502 2561 flags.go:64] FLAG: --reserved-cpus="" Jul 17 18:55:01.899176 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891505 2561 flags.go:64] FLAG: --reserved-memory="" Jul 17 18:55:01.899176 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891509 2561 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 17 18:55:01.899176 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891511 2561 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 17 18:55:01.899176 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891515 2561 flags.go:64] FLAG: --rotate-certificates="false" Jul 17 18:55:01.899176 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891517 2561 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 17 18:55:01.899176 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891535 2561 flags.go:64] FLAG: --runonce="false" Jul 17 18:55:01.899176 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891538 2561 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 17 18:55:01.899802 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891541 2561 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 17 18:55:01.899802 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891544 2561 flags.go:64] FLAG: --seccomp-default="false" Jul 17 18:55:01.899802 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891547 2561 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 17 18:55:01.899802 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891550 2561 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 17 18:55:01.899802 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891553 2561 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 17 18:55:01.899802 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891556 2561 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 17 18:55:01.899802 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891559 2561 flags.go:64] FLAG: --storage-driver-password="root" Jul 17 18:55:01.899802 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891561 2561 flags.go:64] FLAG: --storage-driver-secure="false" Jul 17 18:55:01.899802 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891565 2561 flags.go:64] FLAG: --storage-driver-table="stats" Jul 17 18:55:01.899802 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891568 2561 flags.go:64] FLAG: --storage-driver-user="root" Jul 17 18:55:01.899802 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891570 2561 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 17 18:55:01.899802 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891573 2561 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 17 18:55:01.899802 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891577 2561 flags.go:64] FLAG: --system-cgroups="" Jul 17 18:55:01.899802 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891579 2561 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 17 18:55:01.899802 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891584 2561 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 17 18:55:01.899802 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891591 2561 flags.go:64] FLAG: --tls-cert-file="" Jul 17 18:55:01.899802 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891594 2561 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 17 18:55:01.899802 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891598 2561 flags.go:64] FLAG: --tls-min-version="" Jul 17 18:55:01.899802 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891601 2561 flags.go:64] FLAG: --tls-private-key-file="" Jul 17 18:55:01.899802 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891603 2561 flags.go:64] FLAG: --topology-manager-policy="none" Jul 17 18:55:01.899802 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891606 2561 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 17 18:55:01.899802 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891609 2561 flags.go:64] FLAG: --topology-manager-scope="container" Jul 17 18:55:01.899802 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891612 2561 flags.go:64] FLAG: --v="2" Jul 17 18:55:01.899802 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891615 2561 flags.go:64] FLAG: --version="false" Jul 17 18:55:01.899802 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891619 2561 flags.go:64] FLAG: --vmodule="" Jul 17 18:55:01.900415 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891623 2561 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 17 18:55:01.900415 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.891627 2561 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 17 18:55:01.900415 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891751 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 18:55:01.900415 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891754 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 18:55:01.900415 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891757 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 18:55:01.900415 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891761 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 18:55:01.900415 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891764 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 18:55:01.900415 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891766 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 18:55:01.900415 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891769 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 18:55:01.900415 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891771 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 18:55:01.900415 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891774 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 18:55:01.900415 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891776 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 18:55:01.900415 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891778 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 18:55:01.900415 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891781 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 18:55:01.900415 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891784 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 18:55:01.900415 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891786 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 18:55:01.900415 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891789 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 18:55:01.900415 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891791 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 18:55:01.900415 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891795 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 18:55:01.900415 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891799 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 18:55:01.900919 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891802 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 18:55:01.900919 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891805 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 18:55:01.900919 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891809 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 18:55:01.900919 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891811 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 18:55:01.900919 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891814 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 18:55:01.900919 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891817 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 18:55:01.900919 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891820 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 18:55:01.900919 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891822 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 18:55:01.900919 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891825 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 18:55:01.900919 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891827 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 18:55:01.900919 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891830 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 18:55:01.900919 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891832 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 18:55:01.900919 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891835 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 18:55:01.900919 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891837 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 18:55:01.900919 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891840 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 18:55:01.900919 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891842 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 18:55:01.900919 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891844 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 18:55:01.900919 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891849 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 18:55:01.900919 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891852 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 18:55:01.900919 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891855 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 18:55:01.901396 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891857 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 18:55:01.901396 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891860 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 18:55:01.901396 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891862 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 18:55:01.901396 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891865 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 18:55:01.901396 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891867 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 18:55:01.901396 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891870 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 18:55:01.901396 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891872 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 18:55:01.901396 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891874 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 18:55:01.901396 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891877 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 18:55:01.901396 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891880 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 18:55:01.901396 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891882 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 18:55:01.901396 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891885 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 18:55:01.901396 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891889 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 18:55:01.901396 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891892 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 18:55:01.901396 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891896 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 18:55:01.901396 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891899 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 18:55:01.901396 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891903 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 18:55:01.901396 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891906 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 18:55:01.901396 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891909 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 18:55:01.901876 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891912 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 18:55:01.901876 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891915 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 18:55:01.901876 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891917 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 18:55:01.901876 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891920 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 18:55:01.901876 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891922 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 18:55:01.901876 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891924 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 18:55:01.901876 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891927 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 18:55:01.901876 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891929 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 18:55:01.901876 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891932 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 18:55:01.901876 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891934 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 18:55:01.901876 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891938 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 18:55:01.901876 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891941 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 18:55:01.901876 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891943 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 18:55:01.901876 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891946 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 18:55:01.901876 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891948 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 18:55:01.901876 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891951 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 18:55:01.901876 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891953 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 18:55:01.901876 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891956 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 18:55:01.901876 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891958 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 18:55:01.901876 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891961 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 17 18:55:01.902357 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891963 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 18:55:01.902357 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891966 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 18:55:01.902357 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891968 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 18:55:01.902357 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891971 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 18:55:01.902357 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891973 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 18:55:01.902357 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891976 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 18:55:01.902357 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891978 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 18:55:01.902357 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891985 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 18:55:01.902357 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891988 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 18:55:01.902357 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891990 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 18:55:01.902357 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891993 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 18:55:01.902357 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891996 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 18:55:01.902357 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.891998 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 18:55:01.902357 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.892001 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 18:55:01.902357 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.892003 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 18:55:01.902357 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.892006 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 18:55:01.902357 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.892008 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 18:55:01.902357 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.892011 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 18:55:01.902357 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.892013 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 18:55:01.902357 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.892016 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 18:55:01.902867 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.892018 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 18:55:01.902867 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.892021 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 18:55:01.902867 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.892028 2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 17 18:55:01.902867 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.898920 2561 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 17 18:55:01.902867 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.898938 2561 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 17 18:55:01.902867 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.899546 2561 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 17 18:55:01.902867 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.899553 2561 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 17 18:55:01.902867 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899613 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 18:55:01.902867 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899618 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 18:55:01.902867 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899622 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 18:55:01.902867 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899625 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 18:55:01.902867 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899628 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 18:55:01.902867 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899631 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 18:55:01.902867 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899634 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 18:55:01.902867 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899637 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 18:55:01.903240 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899640 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 18:55:01.903240 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899644 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 18:55:01.903240 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899646 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 18:55:01.903240 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899648 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 18:55:01.903240 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899651 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 18:55:01.903240 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899654 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 17 18:55:01.903240 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899656 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 18:55:01.903240 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899659 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 18:55:01.903240 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899661 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 18:55:01.903240 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899672 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 18:55:01.903240 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899676 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 18:55:01.903240 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899679 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 18:55:01.903240 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899681 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 18:55:01.903240 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899684 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 18:55:01.903240 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899686 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 18:55:01.903240 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899689 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 18:55:01.903240 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899692 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 18:55:01.903240 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899694 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 18:55:01.903240 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899696 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 18:55:01.903240 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899699 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 18:55:01.903760 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899701 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 18:55:01.903760 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899706 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 18:55:01.903760 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899709 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 18:55:01.903760 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899712 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 18:55:01.903760 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899714 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 18:55:01.903760 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899717 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 18:55:01.903760 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899719 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 18:55:01.903760 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899721 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 18:55:01.903760 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899724 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 18:55:01.903760 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899726 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 18:55:01.903760 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899729 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 18:55:01.903760 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899731 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 18:55:01.903760 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899734 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 18:55:01.903760 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899736 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 18:55:01.903760 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899739 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 18:55:01.903760 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899741 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 18:55:01.903760 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899743 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 18:55:01.903760 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899746 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 18:55:01.903760 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899749 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 18:55:01.903760 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899752 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 18:55:01.904237 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899754 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 18:55:01.904237 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899756 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 18:55:01.904237 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899759 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 18:55:01.904237 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899761 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 18:55:01.904237 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899763 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 18:55:01.904237 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899766 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 18:55:01.904237 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899768 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 18:55:01.904237 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899770 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 18:55:01.904237 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899772 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 18:55:01.904237 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899775 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 18:55:01.904237 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899777 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 18:55:01.904237 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899780 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 18:55:01.904237 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899782 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 18:55:01.904237 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899787 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 18:55:01.904237 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899791 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 18:55:01.904237 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899794 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 18:55:01.904237 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899797 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 18:55:01.904237 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899800 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 18:55:01.904237 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899803 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 18:55:01.904237 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899806 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 18:55:01.904723 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899809 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 18:55:01.904723 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899812 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 18:55:01.904723 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899814 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 18:55:01.904723 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899817 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 18:55:01.904723 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899819 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 18:55:01.904723 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899822 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 18:55:01.904723 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899824 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 18:55:01.904723 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899827 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 18:55:01.904723 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899829 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 18:55:01.904723 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899832 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 18:55:01.904723 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899834 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 18:55:01.904723 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899837 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 18:55:01.904723 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899839 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 18:55:01.904723 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899843 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 18:55:01.904723 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899847 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 18:55:01.904723 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899850 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 18:55:01.904723 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899852 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 18:55:01.904723 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899856 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 18:55:01.904723 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899859 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 18:55:01.905191 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899862 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 18:55:01.905191 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899865 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 18:55:01.905191 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899868 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 18:55:01.905191 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899870 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 18:55:01.905191 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899873 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 18:55:01.905191 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899875 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 18:55:01.905191 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899878 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 18:55:01.905191 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899881 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 18:55:01.905191 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899884 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 18:55:01.905191 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899887 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 18:55:01.905191 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899889 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 18:55:01.905191 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.899892 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 18:55:01.905191 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.899897 2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 17 18:55:01.905191 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900016 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 18:55:01.905191 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900021 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 18:55:01.905582 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900024 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 18:55:01.905582 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900027 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 18:55:01.905582 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900029 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 18:55:01.905582 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900032 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 18:55:01.905582 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900035 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 18:55:01.905582 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900037 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 18:55:01.905582 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900039 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 18:55:01.905582 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900042 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 18:55:01.905582 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900044 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 18:55:01.905582 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900047 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 18:55:01.905582 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900049 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 18:55:01.905582 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900051 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 18:55:01.905582 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900054 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 18:55:01.905582 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900056 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 18:55:01.905582 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900058 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 18:55:01.905582 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900062 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 18:55:01.905582 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900064 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 18:55:01.905582 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900066 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 18:55:01.905582 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900069 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 18:55:01.905582 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900071 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 18:55:01.906061 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900074 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 18:55:01.906061 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900076 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 18:55:01.906061 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900078 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 18:55:01.906061 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900081 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 18:55:01.906061 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900085 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 18:55:01.906061 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900087 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 18:55:01.906061 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900090 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 18:55:01.906061 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900093 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 18:55:01.906061 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900095 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 18:55:01.906061 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900097 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 18:55:01.906061 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900100 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 18:55:01.906061 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900102 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 18:55:01.906061 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900104 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 18:55:01.906061 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900106 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 18:55:01.906061 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900109 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 18:55:01.906061 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900111 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 18:55:01.906061 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900113 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 18:55:01.906061 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900116 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 18:55:01.906061 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900118 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 18:55:01.906650 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900120 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 18:55:01.906650 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900123 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 18:55:01.906650 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900125 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 18:55:01.906650 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900127 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 18:55:01.906650 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900130 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 18:55:01.906650 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900132 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 18:55:01.906650 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900134 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 18:55:01.906650 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900137 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 18:55:01.906650 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900139 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 18:55:01.906650 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900142 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 18:55:01.906650 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900144 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 17 18:55:01.906650 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900146 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 18:55:01.906650 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900149 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 18:55:01.906650 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900152 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 18:55:01.906650 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900154 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 18:55:01.906650 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900156 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 18:55:01.906650 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900160 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 18:55:01.906650 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900164 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 18:55:01.906650 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900168 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 18:55:01.906650 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900171 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 18:55:01.907147 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900175 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 18:55:01.907147 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900178 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 18:55:01.907147 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900181 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 18:55:01.907147 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900183 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 18:55:01.907147 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900186 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 18:55:01.907147 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900189 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 18:55:01.907147 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900191 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 18:55:01.907147 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900194 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 18:55:01.907147 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900196 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 18:55:01.907147 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900198 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 18:55:01.907147 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900201 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 18:55:01.907147 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900203 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 18:55:01.907147 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900205 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 18:55:01.907147 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900208 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 18:55:01.907147 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900210 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 18:55:01.907147 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900213 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 18:55:01.907147 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900217 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 18:55:01.907147 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900219 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 18:55:01.907147 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900222 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 18:55:01.907625 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900224 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 18:55:01.907625 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900227 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 18:55:01.907625 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900230 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 18:55:01.907625 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900233 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 18:55:01.907625 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900237 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 18:55:01.907625 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900240 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 18:55:01.907625 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900242 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 18:55:01.907625 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900245 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 18:55:01.907625 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900248 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 18:55:01.907625 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900250 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 18:55:01.907625 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900253 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 18:55:01.907625 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900255 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 18:55:01.907625 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900258 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 18:55:01.907625 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900260 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 18:55:01.907625 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900263 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 18:55:01.907625 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900265 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 18:55:01.907625 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900268 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 18:55:01.907625 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900270 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 18:55:01.907625 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:01.900273 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 18:55:01.908114 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.900278 2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 17 18:55:01.908114 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.900411 2561 server.go:962] "Client rotation is on, will bootstrap in background" Jul 17 18:55:01.908114 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.902355 2561 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 17 18:55:01.908114 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.903461 2561 server.go:1019] "Starting client certificate rotation" Jul 17 18:55:01.908114 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.903564 2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 17 18:55:01.908114 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.903617 2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 17 18:55:01.908114 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.905714 2561 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 17 18:55:01.910964 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.910938 2561 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 17 18:55:01.919550 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.919514 2561 log.go:25] "Validated CRI v1 runtime API" Jul 17 18:55:01.921464 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.921444 2561 log.go:25] "Validated CRI v1 image API" Jul 17 18:55:01.922334 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.922318 2561 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 17 18:55:01.924323 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.924293 2561 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 869a1d14-d22e-453b-8284-7bedffcec106:/dev/nvme0n1p4 ca85aa5b-27cf-4cf0-b92a-164bfefc4996:/dev/nvme0n1p3] Jul 17 18:55:01.924376 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.924323 2561 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 17 18:55:01.929296 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.929183 2561 manager.go:216] Machine: {Timestamp:2026-07-17 18:55:01.928006653 +0000 UTC m=+0.257822852 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3106560 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a8e921f27f906b697ebf21d728f26 SystemUUID:ec2a8e92-1f27-f906-b697-ebf21d728f26 BootID:82f120ee-06a7-4edf-910b-3d48021afae3 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:bb:f5:29:f3:71 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:bb:f5:29:f3:71 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9a:63:e7:e4:34:40 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 17 18:55:01.929296 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.929286 2561 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 17 18:55:01.929426 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.929379 2561 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 17 18:55:01.931693 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.931654 2561 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 17 18:55:01.931838 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.931698 2561 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-137.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 17 18:55:01.932305 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.932288 2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 17 18:55:01.932823 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.932810 2561 topology_manager.go:138] "Creating topology manager with none policy" Jul 17 18:55:01.932881 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.932826 2561 container_manager_linux.go:309] "Creating device plugin manager" Jul 17 18:55:01.932881 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.932834 2561 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 17 18:55:01.933365 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.933354 2561 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 17 18:55:01.934562 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.934551 2561 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 17 18:55:01.938387 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.938375 2561 state_mem.go:36] "Initialized new in-memory state store" Jul 17 18:55:01.938507 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.938497 2561 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 17 18:55:01.938563 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.938540 2561 kubelet.go:486] "Attempting to sync node with API server" Jul 17 18:55:01.938563 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.938551 2561 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 17 18:55:01.938651 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.938566 2561 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 17 18:55:01.938651 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.938577 2561 kubelet.go:398] "Adding apiserver pod source" Jul 17 18:55:01.938651 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.938585 2561 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 17 18:55:01.938998 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.938987 2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 17 18:55:01.939035 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.939011 2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 17 18:55:01.940840 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.940806 2561 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 17 18:55:01.942754 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.942740 2561 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 17 18:55:01.942820 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.942772 2561 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 17 18:55:01.944300 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.944285 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 17 18:55:01.944356 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.944308 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 17 18:55:01.944356 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.944318 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 17 18:55:01.944356 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.944325 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 17 18:55:01.944356 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.944338 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 17 18:55:01.944356 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.944345 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 17 18:55:01.944356 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.944356 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 17 18:55:01.944505 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.944364 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 17 18:55:01.944505 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.944373 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 17 18:55:01.944505 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.944381 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 17 18:55:01.944505 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.944395 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 17 18:55:01.944505 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.944404 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 17 18:55:01.945002 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.944988 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 17 18:55:01.945031 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.945006 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 17 18:55:01.948162 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.948138 2561 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-137.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 17 18:55:01.948414 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.948401 2561 server.go:1268] "Started kubelet" Jul 17 18:55:01.951353 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:01.951318 2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-137.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 17 18:55:01.951450 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:01.951333 2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 17 18:55:01.952034 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.948476 2561 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 17 18:55:01.952165 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.952087 2561 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 17 18:55:01.952309 ip-10-0-134-137 systemd[1]: Started Kubernetes Kubelet. Jul 17 18:55:01.952433 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.952337 2561 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 17 18:55:01.954086 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.954070 2561 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 17 18:55:01.956320 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.956239 2561 server.go:310] "Adding debug handlers to kubelet server" Jul 17 18:55:01.958077 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:01.958056 2561 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 17 18:55:01.958346 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:01.957642 2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-137.ec2.internal.18c32895f835f235 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-137.ec2.internal,UID:ip-10-0-134-137.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-137.ec2.internal,},FirstTimestamp:2026-07-17 18:55:01.948371509 +0000 UTC m=+0.278187708,LastTimestamp:2026-07-17 18:55:01.948371509 +0000 UTC m=+0.278187708,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-137.ec2.internal,}" Jul 17 18:55:01.960606 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.960589 2561 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 17 18:55:01.960684 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.960634 2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 17 18:55:01.960754 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:01.960740 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-137.ec2.internal\" not found" Jul 17 18:55:01.961274 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.961219 2561 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 17 18:55:01.961274 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.961237 2561 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 17 18:55:01.961403 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.961379 2561 reconstruct.go:97] "Volume reconstruction finished" Jul 17 18:55:01.961403 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.961387 2561 reconciler.go:29] "Reconciler: start to sync state" Jul 17 18:55:01.961541 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.961501 2561 factory.go:55] Registering systemd factory Jul 17 18:55:01.961541 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.961516 2561 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 17 18:55:01.961640 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.961519 2561 factory.go:223] Registration of the systemd container factory successfully Jul 17 18:55:01.961985 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.961970 2561 factory.go:153] Registering CRI-O factory Jul 17 18:55:01.962062 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.961989 2561 factory.go:223] Registration of the crio container factory successfully Jul 17 18:55:01.962062 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.962037 2561 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 17 18:55:01.962167 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.962077 2561 factory.go:103] Registering Raw factory Jul 17 18:55:01.962167 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.962093 2561 manager.go:1196] Started watching for new ooms in manager Jul 17 18:55:01.963269 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.963252 2561 manager.go:349] Starting recovery of all containers Jul 17 18:55:01.964130 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:01.964102 2561 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-137.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 17 18:55:01.964228 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:01.964206 2561 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 17 18:55:01.968864 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.968843 2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-g7s9m" Jul 17 18:55:01.972332 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.972168 2561 manager.go:354] Recovery completed Jul 17 18:55:01.976606 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.976586 2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-g7s9m" Jul 17 18:55:01.976693 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.976652 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 18:55:01.978563 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.978543 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-137.ec2.internal" event="NodeHasSufficientMemory" Jul 17 18:55:01.978651 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.978577 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-137.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 18:55:01.978651 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.978590 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-137.ec2.internal" event="NodeHasSufficientPID" Jul 17 18:55:01.979108 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.979095 2561 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 17 18:55:01.979176 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.979108 2561 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 17 18:55:01.979176 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.979127 2561 state_mem.go:36] "Initialized new in-memory state store" Jul 17 18:55:01.979176 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.979135 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 18:55:01.979906 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.979891 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-137.ec2.internal" event="NodeHasSufficientMemory" Jul 17 18:55:01.979976 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.979920 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-137.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 18:55:01.979976 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.979930 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-137.ec2.internal" event="NodeHasSufficientPID" Jul 17 18:55:01.980321 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:01.980241 2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-137.ec2.internal.18c32895fa029d7d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-137.ec2.internal,UID:ip-10-0-134-137.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-137.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-137.ec2.internal,},FirstTimestamp:2026-07-17 18:55:01.978561917 +0000 UTC m=+0.308378120,LastTimestamp:2026-07-17 18:55:01.978561917 +0000 UTC m=+0.308378120,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-137.ec2.internal,}" Jul 17 18:55:01.981638 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.981624 2561 policy_none.go:49] "None policy: Start" Jul 17 18:55:01.981699 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.981645 2561 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 17 18:55:01.981699 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.981659 2561 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 17 18:55:01.983795 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:01.983783 2561 policy_none.go:47] "Start" Jul 17 18:55:02.022124 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.022106 2561 manager.go:337] "Starting Device Plugin manager" Jul 17 18:55:02.022228 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:02.022144 2561 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 17 18:55:02.022228 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.022155 2561 server.go:87] "Starting device plugin registration server" Jul 17 18:55:02.022408 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.022397 2561 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 17 18:55:02.022450 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.022409 2561 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 17 18:55:02.022515 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.022498 2561 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 17 18:55:02.022609 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.022596 2561 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 17 18:55:02.022609 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.022608 2561 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 17 18:55:02.023149 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:02.023130 2561 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 17 18:55:02.023207 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:02.023178 2561 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-137.ec2.internal\" not found" Jul 17 18:55:02.091254 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.091207 2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 17 18:55:02.093559 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.092598 2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 17 18:55:02.093559 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.092647 2561 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 17 18:55:02.093559 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.092679 2561 kubelet.go:2448] "Starting kubelet main sync loop" Jul 17 18:55:02.093559 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:02.092721 2561 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 17 18:55:02.095313 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.095291 2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 18:55:02.122669 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.122627 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 18:55:02.123584 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.123569 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-137.ec2.internal" event="NodeHasSufficientMemory" Jul 17 18:55:02.123707 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.123597 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-137.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 18:55:02.123707 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.123607 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-137.ec2.internal" event="NodeHasSufficientPID" Jul 17 18:55:02.123707 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.123630 2561 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-137.ec2.internal" Jul 17 18:55:02.132027 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.132008 2561 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-137.ec2.internal" Jul 17 18:55:02.132128 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:02.132033 2561 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-137.ec2.internal\": node \"ip-10-0-134-137.ec2.internal\" not found" Jul 17 18:55:02.147117 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:02.147046 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-137.ec2.internal\" not found" Jul 17 18:55:02.193411 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.193368 2561 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-134-137.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-137.ec2.internal"] Jul 17 18:55:02.193626 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.193456 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 18:55:02.195016 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.195000 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-137.ec2.internal" event="NodeHasSufficientMemory" Jul 17 18:55:02.195073 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.195030 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-137.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 18:55:02.195073 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.195039 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-137.ec2.internal" event="NodeHasSufficientPID" Jul 17 18:55:02.197412 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.197396 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 18:55:02.197511 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.197494 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-137.ec2.internal" Jul 17 18:55:02.197601 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.197545 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 18:55:02.198212 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.198198 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-137.ec2.internal" event="NodeHasSufficientMemory" Jul 17 18:55:02.198286 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.198221 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-137.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 18:55:02.198286 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.198231 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-137.ec2.internal" event="NodeHasSufficientPID" Jul 17 18:55:02.198286 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.198240 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-137.ec2.internal" event="NodeHasSufficientMemory" Jul 17 18:55:02.198286 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.198258 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-137.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 18:55:02.198286 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.198267 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-137.ec2.internal" event="NodeHasSufficientPID" Jul 17 18:55:02.201976 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.201957 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-137.ec2.internal" Jul 17 18:55:02.202085 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.201984 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 18:55:02.202765 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.202748 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-137.ec2.internal" event="NodeHasSufficientMemory" Jul 17 18:55:02.202844 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.202779 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-137.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 18:55:02.202844 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.202790 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-137.ec2.internal" event="NodeHasSufficientPID" Jul 17 18:55:02.220246 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:02.220214 2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-137.ec2.internal\" not found" node="ip-10-0-134-137.ec2.internal" Jul 17 18:55:02.226157 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:02.226135 2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-137.ec2.internal\" not found" node="ip-10-0-134-137.ec2.internal" Jul 17 18:55:02.247913 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:02.247881 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-137.ec2.internal\" not found" Jul 17 18:55:02.348767 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:02.348738 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-137.ec2.internal\" not found" Jul 17 18:55:02.362118 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.362096 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4055332a3b86c15989d82b71c24d29a9-config\") pod \"kube-apiserver-proxy-ip-10-0-134-137.ec2.internal\" (UID: \"4055332a3b86c15989d82b71c24d29a9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-137.ec2.internal" Jul 17 18:55:02.362211 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.362128 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f4f6391d232b2f3f6af57e5daac336ae-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-137.ec2.internal\" (UID: \"f4f6391d232b2f3f6af57e5daac336ae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-137.ec2.internal" Jul 17 18:55:02.362211 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.362147 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4f6391d232b2f3f6af57e5daac336ae-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-137.ec2.internal\" (UID: \"f4f6391d232b2f3f6af57e5daac336ae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-137.ec2.internal" Jul 17 18:55:02.449600 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:02.449516 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-137.ec2.internal\" not found" Jul 17 18:55:02.462843 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.462819 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4055332a3b86c15989d82b71c24d29a9-config\") pod \"kube-apiserver-proxy-ip-10-0-134-137.ec2.internal\" (UID: \"4055332a3b86c15989d82b71c24d29a9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-137.ec2.internal" Jul 17 18:55:02.462957 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.462853 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f4f6391d232b2f3f6af57e5daac336ae-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-137.ec2.internal\" (UID: \"f4f6391d232b2f3f6af57e5daac336ae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-137.ec2.internal" Jul 17 18:55:02.462957 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.462878 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4f6391d232b2f3f6af57e5daac336ae-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-137.ec2.internal\" (UID: \"f4f6391d232b2f3f6af57e5daac336ae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-137.ec2.internal" Jul 17 18:55:02.462957 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.462917 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4055332a3b86c15989d82b71c24d29a9-config\") pod \"kube-apiserver-proxy-ip-10-0-134-137.ec2.internal\" (UID: \"4055332a3b86c15989d82b71c24d29a9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-137.ec2.internal" Jul 17 18:55:02.462957 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.462924 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f4f6391d232b2f3f6af57e5daac336ae-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-137.ec2.internal\" (UID: \"f4f6391d232b2f3f6af57e5daac336ae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-137.ec2.internal" Jul 17 18:55:02.462957 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.462925 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4f6391d232b2f3f6af57e5daac336ae-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-137.ec2.internal\" (UID: \"f4f6391d232b2f3f6af57e5daac336ae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-137.ec2.internal" Jul 17 18:55:02.524035 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.524004 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-137.ec2.internal" Jul 17 18:55:02.529090 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.528968 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-137.ec2.internal" Jul 17 18:55:02.551063 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:02.551019 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-137.ec2.internal\" not found" Jul 17 18:55:02.651595 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:02.651557 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-137.ec2.internal\" not found" Jul 17 18:55:02.752142 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:02.752044 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-137.ec2.internal\" not found" Jul 17 18:55:02.792360 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.792336 2561 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 18:55:02.861718 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.861685 2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-137.ec2.internal" Jul 17 18:55:02.874908 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.874879 2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 17 18:55:02.876399 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.876379 2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-137.ec2.internal" Jul 17 18:55:02.889905 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.889879 2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 17 18:55:02.903049 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.903021 2561 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 17 18:55:02.903181 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.903164 2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 17 18:55:02.903228 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.903207 2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 17 18:55:02.939406 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.939373 2561 apiserver.go:52] "Watching apiserver" Jul 17 18:55:02.956291 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.956256 2561 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 17 18:55:02.956666 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.956645 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-n2wtb","openshift-multus/network-metrics-daemon-7l6k9","openshift-ovn-kubernetes/ovnkube-node-hpjzq","kube-system/global-pull-secret-syncer-m9wzh","kube-system/konnectivity-agent-2fgtz","openshift-dns/node-resolver-pvpbg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-137.ec2.internal","openshift-multus/multus-additional-cni-plugins-nb8rb","openshift-multus/multus-nrss7","openshift-network-diagnostics/network-check-target-c54gz","openshift-network-operator/iptables-alerter-7g4zx","kube-system/kube-apiserver-proxy-ip-10-0-134-137.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92cjd","openshift-cluster-node-tuning-operator/tuned-kj2t7"] Jul 17 18:55:02.959167 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.959147 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2fgtz" Jul 17 18:55:02.961135 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.961115 2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 17 18:55:02.961469 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.961448 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 17 18:55:02.961595 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.961453 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-cn88j\"" Jul 17 18:55:02.961595 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.961498 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 17 18:55:02.961777 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.961758 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7g4zx" Jul 17 18:55:02.965023 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.964202 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 17 18:55:02.965023 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.964322 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 17 18:55:02.965023 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.964407 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-n2wtb" Jul 17 18:55:02.965023 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.964426 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-p2m2d\"" Jul 17 18:55:02.965023 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.964455 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 17 18:55:02.965023 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.964859 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0135e7e9-6294-4524-ad81-17f251822127-agent-certs\") pod \"konnectivity-agent-2fgtz\" (UID: \"0135e7e9-6294-4524-ad81-17f251822127\") " pod="kube-system/konnectivity-agent-2fgtz" Jul 17 18:55:02.965221 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.965124 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0135e7e9-6294-4524-ad81-17f251822127-konnectivity-ca\") pod \"konnectivity-agent-2fgtz\" (UID: \"0135e7e9-6294-4524-ad81-17f251822127\") " pod="kube-system/konnectivity-agent-2fgtz" Jul 17 18:55:02.965221 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.965152 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c6b17b15-c9dc-4e3e-9fb3-8477f0a734ed-iptables-alerter-script\") pod \"iptables-alerter-7g4zx\" (UID: \"c6b17b15-c9dc-4e3e-9fb3-8477f0a734ed\") " pod="openshift-network-operator/iptables-alerter-7g4zx" Jul 17 18:55:02.965221 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.965172 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c6b17b15-c9dc-4e3e-9fb3-8477f0a734ed-host-slash\") pod \"iptables-alerter-7g4zx\" (UID: \"c6b17b15-c9dc-4e3e-9fb3-8477f0a734ed\") " pod="openshift-network-operator/iptables-alerter-7g4zx" Jul 17 18:55:02.965221 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.965201 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl87p\" (UniqueName: \"kubernetes.io/projected/c6b17b15-c9dc-4e3e-9fb3-8477f0a734ed-kube-api-access-jl87p\") pod \"iptables-alerter-7g4zx\" (UID: \"c6b17b15-c9dc-4e3e-9fb3-8477f0a734ed\") " pod="openshift-network-operator/iptables-alerter-7g4zx" Jul 17 18:55:02.966616 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.966597 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m9wzh" Jul 17 18:55:02.966695 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.966651 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 17 18:55:02.966695 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:02.966657 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m9wzh" podUID="3591b9f8-a2ff-4adb-93ec-449225ca6390" Jul 17 18:55:02.966791 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.966695 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 17 18:55:02.966791 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.966748 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-nqqts\"" Jul 17 18:55:02.966862 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.966805 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 17 18:55:02.968609 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.968588 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pvpbg" Jul 17 18:55:02.970645 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.970629 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:02.970918 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.970793 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-282dl\"" Jul 17 18:55:02.970918 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.970811 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 17 18:55:02.970918 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.970842 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 17 18:55:02.971097 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.970980 2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 17 18:55:02.972370 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.972354 2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 18:55:02.972633 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.972616 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 17 18:55:02.973008 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.972991 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nb8rb" Jul 17 18:55:02.973008 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.973002 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 17 18:55:02.973136 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.973098 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 17 18:55:02.973187 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.973142 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 17 18:55:02.973418 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.973405 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 17 18:55:02.973492 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.973425 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 17 18:55:02.973492 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.973465 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-dfwq7\"" Jul 17 18:55:02.974949 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.974933 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7994l\"" Jul 17 18:55:02.975068 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.975055 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 17 18:55:02.975163 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.975150 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nrss7" Jul 17 18:55:02.975420 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.975402 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 17 18:55:02.975420 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.975403 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 17 18:55:02.975567 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.975407 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 17 18:55:02.975567 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.975423 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 17 18:55:02.977336 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.977323 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c54gz" Jul 17 18:55:02.977405 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:02.977381 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c54gz" podUID="2c91e513-4f09-4551-8f50-75a4ba131d12" Jul 17 18:55:02.977466 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.977455 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-sq5d2\"" Jul 17 18:55:02.977660 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.977647 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 17 18:55:02.979464 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.979447 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l6k9" Jul 17 18:55:02.979568 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:02.979510 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7l6k9" podUID="16db4114-eec4-4eb4-80be-1e6238e5c67b" Jul 17 18:55:02.981503 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.981463 2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-16 18:50:01 +0000 UTC" deadline="2028-03-24 00:18:57.942689233 +0000 UTC" Jul 17 18:55:02.981589 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.981503 2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14765h23m54.961188402s" Jul 17 18:55:02.981639 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.981586 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92cjd" Jul 17 18:55:02.983597 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.983582 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kj2t7" Jul 17 18:55:02.983940 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.983921 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 17 18:55:02.983940 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.983934 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 17 18:55:02.984073 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.983975 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ckbw4\"" Jul 17 18:55:02.984264 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.984247 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 17 18:55:02.985657 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.985641 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-wjcjh\"" Jul 17 18:55:02.985973 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.985955 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 17 18:55:02.986053 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.985956 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 17 18:55:02.990917 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.990900 2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xj5hf" Jul 17 18:55:02.997630 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:02.997613 2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xj5hf" Jul 17 18:55:03.063260 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.063232 2561 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 17 18:55:03.065738 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.065720 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c649f1a2-c525-4818-bde3-f5c394101099-etc-sysconfig\") pod \"tuned-kj2t7\" (UID: \"c649f1a2-c525-4818-bde3-f5c394101099\") " pod="openshift-cluster-node-tuning-operator/tuned-kj2t7" Jul 17 18:55:03.065819 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.065745 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4b162535-066a-471a-aee4-99c07c6e1bb4-log-socket\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.065819 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.065761 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3591b9f8-a2ff-4adb-93ec-449225ca6390-original-pull-secret\") pod \"global-pull-secret-syncer-m9wzh\" (UID: \"3591b9f8-a2ff-4adb-93ec-449225ca6390\") " pod="kube-system/global-pull-secret-syncer-m9wzh" Jul 17 18:55:03.065819 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.065776 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08c31757-7c13-4c31-a1ad-570aad3cde70-system-cni-dir\") pod \"multus-additional-cni-plugins-nb8rb\" (UID: \"08c31757-7c13-4c31-a1ad-570aad3cde70\") " pod="openshift-multus/multus-additional-cni-plugins-nb8rb" Jul 17 18:55:03.065819 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.065789 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/08c31757-7c13-4c31-a1ad-570aad3cde70-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nb8rb\" (UID: \"08c31757-7c13-4c31-a1ad-570aad3cde70\") " pod="openshift-multus/multus-additional-cni-plugins-nb8rb" Jul 17 18:55:03.065952 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.065829 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c649f1a2-c525-4818-bde3-f5c394101099-tmp\") pod \"tuned-kj2t7\" (UID: \"c649f1a2-c525-4818-bde3-f5c394101099\") " pod="openshift-cluster-node-tuning-operator/tuned-kj2t7" Jul 17 18:55:03.065952 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.065856 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp8d8\" (UniqueName: \"kubernetes.io/projected/c649f1a2-c525-4818-bde3-f5c394101099-kube-api-access-sp8d8\") pod \"tuned-kj2t7\" (UID: \"c649f1a2-c525-4818-bde3-f5c394101099\") " pod="openshift-cluster-node-tuning-operator/tuned-kj2t7" Jul 17 18:55:03.065952 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.065881 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4b162535-066a-471a-aee4-99c07c6e1bb4-run-systemd\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.065952 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.065903 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjtmm\" (UniqueName: \"kubernetes.io/projected/16db4114-eec4-4eb4-80be-1e6238e5c67b-kube-api-access-kjtmm\") pod \"network-metrics-daemon-7l6k9\" (UID: \"16db4114-eec4-4eb4-80be-1e6238e5c67b\") " pod="openshift-multus/network-metrics-daemon-7l6k9" Jul 17 18:55:03.065952 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.065920 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/64f94c18-0eab-41bb-88e0-824db8dfa7e4-etc-selinux\") pod \"aws-ebs-csi-driver-node-92cjd\" (UID: \"64f94c18-0eab-41bb-88e0-824db8dfa7e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92cjd" Jul 17 18:55:03.065952 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.065943 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/64f94c18-0eab-41bb-88e0-824db8dfa7e4-sys-fs\") pod \"aws-ebs-csi-driver-node-92cjd\" (UID: \"64f94c18-0eab-41bb-88e0-824db8dfa7e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92cjd" Jul 17 18:55:03.066145 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.065960 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/08c31757-7c13-4c31-a1ad-570aad3cde70-os-release\") pod \"multus-additional-cni-plugins-nb8rb\" (UID: \"08c31757-7c13-4c31-a1ad-570aad3cde70\") " pod="openshift-multus/multus-additional-cni-plugins-nb8rb" Jul 17 18:55:03.066145 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.065975 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/08c31757-7c13-4c31-a1ad-570aad3cde70-cni-binary-copy\") pod \"multus-additional-cni-plugins-nb8rb\" (UID: \"08c31757-7c13-4c31-a1ad-570aad3cde70\") " pod="openshift-multus/multus-additional-cni-plugins-nb8rb" Jul 17 18:55:03.066145 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.065989 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1edd2f99-7d44-4eed-8850-50ca38aa5153-multus-cni-dir\") pod \"multus-nrss7\" (UID: \"1edd2f99-7d44-4eed-8850-50ca38aa5153\") " pod="openshift-multus/multus-nrss7" Jul 17 18:55:03.066145 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.066002 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c649f1a2-c525-4818-bde3-f5c394101099-host\") pod \"tuned-kj2t7\" (UID: \"c649f1a2-c525-4818-bde3-f5c394101099\") " pod="openshift-cluster-node-tuning-operator/tuned-kj2t7" Jul 17 18:55:03.066145 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.066019 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl87p\" (UniqueName: \"kubernetes.io/projected/c6b17b15-c9dc-4e3e-9fb3-8477f0a734ed-kube-api-access-jl87p\") pod \"iptables-alerter-7g4zx\" (UID: \"c6b17b15-c9dc-4e3e-9fb3-8477f0a734ed\") " pod="openshift-network-operator/iptables-alerter-7g4zx" Jul 17 18:55:03.066145 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.066034 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4b162535-066a-471a-aee4-99c07c6e1bb4-ovn-node-metrics-cert\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.066145 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.066047 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/728c6dad-4aae-44c2-ac11-60be3b680cca-tmp-dir\") pod \"node-resolver-pvpbg\" (UID: \"728c6dad-4aae-44c2-ac11-60be3b680cca\") " pod="openshift-dns/node-resolver-pvpbg" Jul 17 18:55:03.066145 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.066060 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1edd2f99-7d44-4eed-8850-50ca38aa5153-host-var-lib-cni-bin\") pod \"multus-nrss7\" (UID: \"1edd2f99-7d44-4eed-8850-50ca38aa5153\") " pod="openshift-multus/multus-nrss7" Jul 17 18:55:03.066145 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.066072 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c649f1a2-c525-4818-bde3-f5c394101099-etc-sysctl-d\") pod \"tuned-kj2t7\" (UID: \"c649f1a2-c525-4818-bde3-f5c394101099\") " pod="openshift-cluster-node-tuning-operator/tuned-kj2t7" Jul 17 18:55:03.066145 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.066083 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c649f1a2-c525-4818-bde3-f5c394101099-etc-systemd\") pod \"tuned-kj2t7\" (UID: \"c649f1a2-c525-4818-bde3-f5c394101099\") " pod="openshift-cluster-node-tuning-operator/tuned-kj2t7" Jul 17 18:55:03.066145 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.066096 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4b162535-066a-471a-aee4-99c07c6e1bb4-systemd-units\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.066145 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.066109 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7f5ebd27-cd62-4b2b-9c30-4863ca02f422-host\") pod \"node-ca-n2wtb\" (UID: \"7f5ebd27-cd62-4b2b-9c30-4863ca02f422\") " pod="openshift-image-registry/node-ca-n2wtb" Jul 17 18:55:03.066719 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.066200 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1edd2f99-7d44-4eed-8850-50ca38aa5153-cni-binary-copy\") pod \"multus-nrss7\" (UID: \"1edd2f99-7d44-4eed-8850-50ca38aa5153\") " pod="openshift-multus/multus-nrss7" Jul 17 18:55:03.066719 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.066234 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf4zc\" (UniqueName: \"kubernetes.io/projected/1edd2f99-7d44-4eed-8850-50ca38aa5153-kube-api-access-sf4zc\") pod \"multus-nrss7\" (UID: \"1edd2f99-7d44-4eed-8850-50ca38aa5153\") " pod="openshift-multus/multus-nrss7" Jul 17 18:55:03.066719 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.066256 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c649f1a2-c525-4818-bde3-f5c394101099-etc-kubernetes\") pod \"tuned-kj2t7\" (UID: \"c649f1a2-c525-4818-bde3-f5c394101099\") " pod="openshift-cluster-node-tuning-operator/tuned-kj2t7" Jul 17 18:55:03.066719 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.066277 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c649f1a2-c525-4818-bde3-f5c394101099-lib-modules\") pod \"tuned-kj2t7\" (UID: \"c649f1a2-c525-4818-bde3-f5c394101099\") " pod="openshift-cluster-node-tuning-operator/tuned-kj2t7" Jul 17 18:55:03.066719 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.066308 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b162535-066a-471a-aee4-99c07c6e1bb4-host-run-ovn-kubernetes\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.066719 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.066330 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16db4114-eec4-4eb4-80be-1e6238e5c67b-metrics-certs\") pod \"network-metrics-daemon-7l6k9\" (UID: \"16db4114-eec4-4eb4-80be-1e6238e5c67b\") " pod="openshift-multus/network-metrics-daemon-7l6k9" Jul 17 18:55:03.066719 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.066351 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/64f94c18-0eab-41bb-88e0-824db8dfa7e4-registration-dir\") pod \"aws-ebs-csi-driver-node-92cjd\" (UID: \"64f94c18-0eab-41bb-88e0-824db8dfa7e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92cjd" Jul 17 18:55:03.066719 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.066372 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/08c31757-7c13-4c31-a1ad-570aad3cde70-cnibin\") pod \"multus-additional-cni-plugins-nb8rb\" (UID: \"08c31757-7c13-4c31-a1ad-570aad3cde70\") " pod="openshift-multus/multus-additional-cni-plugins-nb8rb" Jul 17 18:55:03.066719 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.066396 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4b162535-066a-471a-aee4-99c07c6e1bb4-var-lib-openvswitch\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.066719 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.066418 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq57r\" (UniqueName: \"kubernetes.io/projected/7f5ebd27-cd62-4b2b-9c30-4863ca02f422-kube-api-access-gq57r\") pod \"node-ca-n2wtb\" (UID: \"7f5ebd27-cd62-4b2b-9c30-4863ca02f422\") " pod="openshift-image-registry/node-ca-n2wtb" Jul 17 18:55:03.066719 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.066440 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s827\" (UniqueName: \"kubernetes.io/projected/728c6dad-4aae-44c2-ac11-60be3b680cca-kube-api-access-5s827\") pod \"node-resolver-pvpbg\" (UID: \"728c6dad-4aae-44c2-ac11-60be3b680cca\") " pod="openshift-dns/node-resolver-pvpbg" Jul 17 18:55:03.066719 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.066463 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/08c31757-7c13-4c31-a1ad-570aad3cde70-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nb8rb\" (UID: \"08c31757-7c13-4c31-a1ad-570aad3cde70\") " pod="openshift-multus/multus-additional-cni-plugins-nb8rb" Jul 17 18:55:03.066719 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.066484 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1edd2f99-7d44-4eed-8850-50ca38aa5153-hostroot\") pod \"multus-nrss7\" (UID: \"1edd2f99-7d44-4eed-8850-50ca38aa5153\") " pod="openshift-multus/multus-nrss7" Jul 17 18:55:03.066719 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.066512 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkrll\" (UniqueName: \"kubernetes.io/projected/08c31757-7c13-4c31-a1ad-570aad3cde70-kube-api-access-zkrll\") pod \"multus-additional-cni-plugins-nb8rb\" (UID: \"08c31757-7c13-4c31-a1ad-570aad3cde70\") " pod="openshift-multus/multus-additional-cni-plugins-nb8rb" Jul 17 18:55:03.066719 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.066558 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1edd2f99-7d44-4eed-8850-50ca38aa5153-multus-socket-dir-parent\") pod \"multus-nrss7\" (UID: \"1edd2f99-7d44-4eed-8850-50ca38aa5153\") " pod="openshift-multus/multus-nrss7" Jul 17 18:55:03.066719 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.066593 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c649f1a2-c525-4818-bde3-f5c394101099-run\") pod \"tuned-kj2t7\" (UID: \"c649f1a2-c525-4818-bde3-f5c394101099\") " pod="openshift-cluster-node-tuning-operator/tuned-kj2t7" Jul 17 18:55:03.067229 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.066626 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c6b17b15-c9dc-4e3e-9fb3-8477f0a734ed-host-slash\") pod \"iptables-alerter-7g4zx\" (UID: \"c6b17b15-c9dc-4e3e-9fb3-8477f0a734ed\") " pod="openshift-network-operator/iptables-alerter-7g4zx" Jul 17 18:55:03.067229 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.066664 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c6b17b15-c9dc-4e3e-9fb3-8477f0a734ed-host-slash\") pod \"iptables-alerter-7g4zx\" (UID: \"c6b17b15-c9dc-4e3e-9fb3-8477f0a734ed\") " pod="openshift-network-operator/iptables-alerter-7g4zx" Jul 17 18:55:03.067229 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.066673 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4b162535-066a-471a-aee4-99c07c6e1bb4-ovnkube-script-lib\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.067229 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.066708 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7f5ebd27-cd62-4b2b-9c30-4863ca02f422-serviceca\") pod \"node-ca-n2wtb\" (UID: \"7f5ebd27-cd62-4b2b-9c30-4863ca02f422\") " pod="openshift-image-registry/node-ca-n2wtb" Jul 17 18:55:03.067229 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.066737 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c649f1a2-c525-4818-bde3-f5c394101099-sys\") pod \"tuned-kj2t7\" (UID: \"c649f1a2-c525-4818-bde3-f5c394101099\") " pod="openshift-cluster-node-tuning-operator/tuned-kj2t7" Jul 17 18:55:03.067229 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.066762 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0135e7e9-6294-4524-ad81-17f251822127-agent-certs\") pod \"konnectivity-agent-2fgtz\" (UID: \"0135e7e9-6294-4524-ad81-17f251822127\") " pod="kube-system/konnectivity-agent-2fgtz" Jul 17 18:55:03.067229 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.066783 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/728c6dad-4aae-44c2-ac11-60be3b680cca-hosts-file\") pod \"node-resolver-pvpbg\" (UID: \"728c6dad-4aae-44c2-ac11-60be3b680cca\") " pod="openshift-dns/node-resolver-pvpbg" Jul 17 18:55:03.067229 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.066799 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1edd2f99-7d44-4eed-8850-50ca38aa5153-host-run-k8s-cni-cncf-io\") pod \"multus-nrss7\" (UID: \"1edd2f99-7d44-4eed-8850-50ca38aa5153\") " pod="openshift-multus/multus-nrss7" Jul 17 18:55:03.067229 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.066827 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1edd2f99-7d44-4eed-8850-50ca38aa5153-multus-conf-dir\") pod \"multus-nrss7\" (UID: \"1edd2f99-7d44-4eed-8850-50ca38aa5153\") " pod="openshift-multus/multus-nrss7" Jul 17 18:55:03.067229 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.066852 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1edd2f99-7d44-4eed-8850-50ca38aa5153-multus-daemon-config\") pod \"multus-nrss7\" (UID: \"1edd2f99-7d44-4eed-8850-50ca38aa5153\") " pod="openshift-multus/multus-nrss7" Jul 17 18:55:03.067229 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.066866 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4b162535-066a-471a-aee4-99c07c6e1bb4-host-slash\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.067229 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.066887 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4b162535-066a-471a-aee4-99c07c6e1bb4-run-ovn\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.067229 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.066902 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4b162535-066a-471a-aee4-99c07c6e1bb4-env-overrides\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.067229 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.066915 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1edd2f99-7d44-4eed-8850-50ca38aa5153-system-cni-dir\") pod \"multus-nrss7\" (UID: \"1edd2f99-7d44-4eed-8850-50ca38aa5153\") " pod="openshift-multus/multus-nrss7" Jul 17 18:55:03.067229 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.066928 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1edd2f99-7d44-4eed-8850-50ca38aa5153-os-release\") pod \"multus-nrss7\" (UID: \"1edd2f99-7d44-4eed-8850-50ca38aa5153\") " pod="openshift-multus/multus-nrss7" Jul 17 18:55:03.067229 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.066940 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c649f1a2-c525-4818-bde3-f5c394101099-var-lib-kubelet\") pod \"tuned-kj2t7\" (UID: \"c649f1a2-c525-4818-bde3-f5c394101099\") " pod="openshift-cluster-node-tuning-operator/tuned-kj2t7" Jul 17 18:55:03.067229 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.066964 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4b162535-066a-471a-aee4-99c07c6e1bb4-etc-openvswitch\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.067878 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.066980 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4b162535-066a-471a-aee4-99c07c6e1bb4-run-openvswitch\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.067878 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.067000 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b162535-066a-471a-aee4-99c07c6e1bb4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.067878 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.067019 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3591b9f8-a2ff-4adb-93ec-449225ca6390-kubelet-config\") pod \"global-pull-secret-syncer-m9wzh\" (UID: \"3591b9f8-a2ff-4adb-93ec-449225ca6390\") " pod="kube-system/global-pull-secret-syncer-m9wzh" Jul 17 18:55:03.067878 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.067031 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3591b9f8-a2ff-4adb-93ec-449225ca6390-dbus\") pod \"global-pull-secret-syncer-m9wzh\" (UID: \"3591b9f8-a2ff-4adb-93ec-449225ca6390\") " pod="kube-system/global-pull-secret-syncer-m9wzh" Jul 17 18:55:03.067878 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.067045 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1edd2f99-7d44-4eed-8850-50ca38aa5153-host-var-lib-cni-multus\") pod \"multus-nrss7\" (UID: \"1edd2f99-7d44-4eed-8850-50ca38aa5153\") " pod="openshift-multus/multus-nrss7" Jul 17 18:55:03.067878 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.067080 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1edd2f99-7d44-4eed-8850-50ca38aa5153-host-run-multus-certs\") pod \"multus-nrss7\" (UID: \"1edd2f99-7d44-4eed-8850-50ca38aa5153\") " pod="openshift-multus/multus-nrss7" Jul 17 18:55:03.067878 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.067108 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c649f1a2-c525-4818-bde3-f5c394101099-etc-sysctl-conf\") pod \"tuned-kj2t7\" (UID: \"c649f1a2-c525-4818-bde3-f5c394101099\") " pod="openshift-cluster-node-tuning-operator/tuned-kj2t7" Jul 17 18:55:03.067878 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.067121 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b162535-066a-471a-aee4-99c07c6e1bb4-host-kubelet\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.067878 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.067150 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4b162535-066a-471a-aee4-99c07c6e1bb4-host-run-netns\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.067878 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.067168 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/08c31757-7c13-4c31-a1ad-570aad3cde70-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nb8rb\" (UID: \"08c31757-7c13-4c31-a1ad-570aad3cde70\") " pod="openshift-multus/multus-additional-cni-plugins-nb8rb" Jul 17 18:55:03.067878 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.067182 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4b162535-066a-471a-aee4-99c07c6e1bb4-ovnkube-config\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.067878 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.067198 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/64f94c18-0eab-41bb-88e0-824db8dfa7e4-socket-dir\") pod \"aws-ebs-csi-driver-node-92cjd\" (UID: \"64f94c18-0eab-41bb-88e0-824db8dfa7e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92cjd" Jul 17 18:55:03.067878 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.067231 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1edd2f99-7d44-4eed-8850-50ca38aa5153-host-var-lib-kubelet\") pod \"multus-nrss7\" (UID: \"1edd2f99-7d44-4eed-8850-50ca38aa5153\") " pod="openshift-multus/multus-nrss7" Jul 17 18:55:03.067878 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.067264 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1edd2f99-7d44-4eed-8850-50ca38aa5153-etc-kubernetes\") pod \"multus-nrss7\" (UID: \"1edd2f99-7d44-4eed-8850-50ca38aa5153\") " pod="openshift-multus/multus-nrss7" Jul 17 18:55:03.067878 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.067284 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c649f1a2-c525-4818-bde3-f5c394101099-etc-tuned\") pod \"tuned-kj2t7\" (UID: \"c649f1a2-c525-4818-bde3-f5c394101099\") " pod="openshift-cluster-node-tuning-operator/tuned-kj2t7" Jul 17 18:55:03.067878 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.067297 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4b162535-066a-471a-aee4-99c07c6e1bb4-node-log\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.068407 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.067310 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4b162535-066a-471a-aee4-99c07c6e1bb4-host-cni-netd\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.068407 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.067338 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1edd2f99-7d44-4eed-8850-50ca38aa5153-cnibin\") pod \"multus-nrss7\" (UID: \"1edd2f99-7d44-4eed-8850-50ca38aa5153\") " pod="openshift-multus/multus-nrss7" Jul 17 18:55:03.068407 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.067369 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1edd2f99-7d44-4eed-8850-50ca38aa5153-host-run-netns\") pod \"multus-nrss7\" (UID: \"1edd2f99-7d44-4eed-8850-50ca38aa5153\") " pod="openshift-multus/multus-nrss7" Jul 17 18:55:03.068407 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.067388 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4b162535-066a-471a-aee4-99c07c6e1bb4-host-cni-bin\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.068407 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.067410 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlmhc\" (UniqueName: \"kubernetes.io/projected/2c91e513-4f09-4551-8f50-75a4ba131d12-kube-api-access-mlmhc\") pod \"network-check-target-c54gz\" (UID: \"2c91e513-4f09-4551-8f50-75a4ba131d12\") " pod="openshift-network-diagnostics/network-check-target-c54gz" Jul 17 18:55:03.068407 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.067430 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c649f1a2-c525-4818-bde3-f5c394101099-etc-modprobe-d\") pod \"tuned-kj2t7\" (UID: \"c649f1a2-c525-4818-bde3-f5c394101099\") " pod="openshift-cluster-node-tuning-operator/tuned-kj2t7" Jul 17 18:55:03.068407 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.067455 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0135e7e9-6294-4524-ad81-17f251822127-konnectivity-ca\") pod \"konnectivity-agent-2fgtz\" (UID: \"0135e7e9-6294-4524-ad81-17f251822127\") " pod="kube-system/konnectivity-agent-2fgtz" Jul 17 18:55:03.068407 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.067472 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c6b17b15-c9dc-4e3e-9fb3-8477f0a734ed-iptables-alerter-script\") pod \"iptables-alerter-7g4zx\" (UID: \"c6b17b15-c9dc-4e3e-9fb3-8477f0a734ed\") " pod="openshift-network-operator/iptables-alerter-7g4zx" Jul 17 18:55:03.068407 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.067487 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn2gs\" (UniqueName: \"kubernetes.io/projected/4b162535-066a-471a-aee4-99c07c6e1bb4-kube-api-access-fn2gs\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.068407 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.067503 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/64f94c18-0eab-41bb-88e0-824db8dfa7e4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-92cjd\" (UID: \"64f94c18-0eab-41bb-88e0-824db8dfa7e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92cjd" Jul 17 18:55:03.068407 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.067540 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/64f94c18-0eab-41bb-88e0-824db8dfa7e4-device-dir\") pod \"aws-ebs-csi-driver-node-92cjd\" (UID: \"64f94c18-0eab-41bb-88e0-824db8dfa7e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92cjd" Jul 17 18:55:03.068407 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.067554 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsm5p\" (UniqueName: \"kubernetes.io/projected/64f94c18-0eab-41bb-88e0-824db8dfa7e4-kube-api-access-rsm5p\") pod \"aws-ebs-csi-driver-node-92cjd\" (UID: \"64f94c18-0eab-41bb-88e0-824db8dfa7e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92cjd" Jul 17 18:55:03.068407 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.067765 2561 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 17 18:55:03.068407 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.068073 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0135e7e9-6294-4524-ad81-17f251822127-konnectivity-ca\") pod \"konnectivity-agent-2fgtz\" (UID: \"0135e7e9-6294-4524-ad81-17f251822127\") " pod="kube-system/konnectivity-agent-2fgtz" Jul 17 18:55:03.068808 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.068756 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c6b17b15-c9dc-4e3e-9fb3-8477f0a734ed-iptables-alerter-script\") pod \"iptables-alerter-7g4zx\" (UID: \"c6b17b15-c9dc-4e3e-9fb3-8477f0a734ed\") " pod="openshift-network-operator/iptables-alerter-7g4zx" Jul 17 18:55:03.070680 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.070664 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0135e7e9-6294-4524-ad81-17f251822127-agent-certs\") pod \"konnectivity-agent-2fgtz\" (UID: \"0135e7e9-6294-4524-ad81-17f251822127\") " pod="kube-system/konnectivity-agent-2fgtz" Jul 17 18:55:03.075756 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.075739 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl87p\" (UniqueName: \"kubernetes.io/projected/c6b17b15-c9dc-4e3e-9fb3-8477f0a734ed-kube-api-access-jl87p\") pod \"iptables-alerter-7g4zx\" (UID: \"c6b17b15-c9dc-4e3e-9fb3-8477f0a734ed\") " pod="openshift-network-operator/iptables-alerter-7g4zx" Jul 17 18:55:03.168325 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.168296 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3591b9f8-a2ff-4adb-93ec-449225ca6390-kubelet-config\") pod \"global-pull-secret-syncer-m9wzh\" (UID: \"3591b9f8-a2ff-4adb-93ec-449225ca6390\") " pod="kube-system/global-pull-secret-syncer-m9wzh" Jul 17 18:55:03.168325 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.168327 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3591b9f8-a2ff-4adb-93ec-449225ca6390-dbus\") pod \"global-pull-secret-syncer-m9wzh\" (UID: \"3591b9f8-a2ff-4adb-93ec-449225ca6390\") " pod="kube-system/global-pull-secret-syncer-m9wzh" Jul 17 18:55:03.168586 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.168344 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1edd2f99-7d44-4eed-8850-50ca38aa5153-host-var-lib-cni-multus\") pod \"multus-nrss7\" (UID: \"1edd2f99-7d44-4eed-8850-50ca38aa5153\") " pod="openshift-multus/multus-nrss7" Jul 17 18:55:03.168586 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.168357 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1edd2f99-7d44-4eed-8850-50ca38aa5153-host-run-multus-certs\") pod \"multus-nrss7\" (UID: \"1edd2f99-7d44-4eed-8850-50ca38aa5153\") " pod="openshift-multus/multus-nrss7" Jul 17 18:55:03.168586 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.168371 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c649f1a2-c525-4818-bde3-f5c394101099-etc-sysctl-conf\") pod \"tuned-kj2t7\" (UID: \"c649f1a2-c525-4818-bde3-f5c394101099\") " pod="openshift-cluster-node-tuning-operator/tuned-kj2t7" Jul 17 18:55:03.168586 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.168424 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1edd2f99-7d44-4eed-8850-50ca38aa5153-host-var-lib-cni-multus\") pod \"multus-nrss7\" (UID: \"1edd2f99-7d44-4eed-8850-50ca38aa5153\") " pod="openshift-multus/multus-nrss7" Jul 17 18:55:03.168586 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.168449 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b162535-066a-471a-aee4-99c07c6e1bb4-host-kubelet\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.168586 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.168433 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1edd2f99-7d44-4eed-8850-50ca38aa5153-host-run-multus-certs\") pod \"multus-nrss7\" (UID: \"1edd2f99-7d44-4eed-8850-50ca38aa5153\") " pod="openshift-multus/multus-nrss7" Jul 17 18:55:03.168586 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.168424 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3591b9f8-a2ff-4adb-93ec-449225ca6390-kubelet-config\") pod \"global-pull-secret-syncer-m9wzh\" (UID: \"3591b9f8-a2ff-4adb-93ec-449225ca6390\") " pod="kube-system/global-pull-secret-syncer-m9wzh" Jul 17 18:55:03.168586 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.168483 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c649f1a2-c525-4818-bde3-f5c394101099-etc-sysctl-conf\") pod \"tuned-kj2t7\" (UID: \"c649f1a2-c525-4818-bde3-f5c394101099\") " pod="openshift-cluster-node-tuning-operator/tuned-kj2t7" Jul 17 18:55:03.168586 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.168499 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3591b9f8-a2ff-4adb-93ec-449225ca6390-dbus\") pod \"global-pull-secret-syncer-m9wzh\" (UID: \"3591b9f8-a2ff-4adb-93ec-449225ca6390\") " pod="kube-system/global-pull-secret-syncer-m9wzh" Jul 17 18:55:03.168586 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.168517 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4b162535-066a-471a-aee4-99c07c6e1bb4-host-run-netns\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.168586 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.168536 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b162535-066a-471a-aee4-99c07c6e1bb4-host-kubelet\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.168586 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.168479 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4b162535-066a-471a-aee4-99c07c6e1bb4-host-run-netns\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.168586 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.168565 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/08c31757-7c13-4c31-a1ad-570aad3cde70-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nb8rb\" (UID: \"08c31757-7c13-4c31-a1ad-570aad3cde70\") " pod="openshift-multus/multus-additional-cni-plugins-nb8rb" Jul 17 18:55:03.168586 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.168584 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4b162535-066a-471a-aee4-99c07c6e1bb4-ovnkube-config\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.169147 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.168599 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/64f94c18-0eab-41bb-88e0-824db8dfa7e4-socket-dir\") pod \"aws-ebs-csi-driver-node-92cjd\" (UID: \"64f94c18-0eab-41bb-88e0-824db8dfa7e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92cjd" Jul 17 18:55:03.169147 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.168620 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1edd2f99-7d44-4eed-8850-50ca38aa5153-host-var-lib-kubelet\") pod \"multus-nrss7\" (UID: \"1edd2f99-7d44-4eed-8850-50ca38aa5153\") " pod="openshift-multus/multus-nrss7" Jul 17 18:55:03.169147 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.168635 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1edd2f99-7d44-4eed-8850-50ca38aa5153-etc-kubernetes\") pod \"multus-nrss7\" (UID: \"1edd2f99-7d44-4eed-8850-50ca38aa5153\") " pod="openshift-multus/multus-nrss7" Jul 17 18:55:03.169147 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.168647 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c649f1a2-c525-4818-bde3-f5c394101099-etc-tuned\") pod \"tuned-kj2t7\" (UID: \"c649f1a2-c525-4818-bde3-f5c394101099\") " pod="openshift-cluster-node-tuning-operator/tuned-kj2t7" Jul 17 18:55:03.169147 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.168661 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4b162535-066a-471a-aee4-99c07c6e1bb4-node-log\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.169147 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.168673 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4b162535-066a-471a-aee4-99c07c6e1bb4-host-cni-netd\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.169147 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.168687 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1edd2f99-7d44-4eed-8850-50ca38aa5153-host-var-lib-kubelet\") pod \"multus-nrss7\" (UID: \"1edd2f99-7d44-4eed-8850-50ca38aa5153\") " pod="openshift-multus/multus-nrss7" Jul 17 18:55:03.169147 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.168696 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1edd2f99-7d44-4eed-8850-50ca38aa5153-etc-kubernetes\") pod \"multus-nrss7\" (UID: \"1edd2f99-7d44-4eed-8850-50ca38aa5153\") " pod="openshift-multus/multus-nrss7" Jul 17 18:55:03.169147 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.168698 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1edd2f99-7d44-4eed-8850-50ca38aa5153-cnibin\") pod \"multus-nrss7\" (UID: \"1edd2f99-7d44-4eed-8850-50ca38aa5153\") " pod="openshift-multus/multus-nrss7" Jul 17 18:55:03.169147 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.168735 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4b162535-066a-471a-aee4-99c07c6e1bb4-node-log\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.169147 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.168736 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1edd2f99-7d44-4eed-8850-50ca38aa5153-host-run-netns\") pod \"multus-nrss7\" (UID: \"1edd2f99-7d44-4eed-8850-50ca38aa5153\") " pod="openshift-multus/multus-nrss7" Jul 17 18:55:03.169147 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.168777 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1edd2f99-7d44-4eed-8850-50ca38aa5153-host-run-netns\") pod \"multus-nrss7\" (UID: \"1edd2f99-7d44-4eed-8850-50ca38aa5153\") " pod="openshift-multus/multus-nrss7" Jul 17 18:55:03.169147 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.168809 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4b162535-066a-471a-aee4-99c07c6e1bb4-host-cni-bin\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.169147 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.168806 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4b162535-066a-471a-aee4-99c07c6e1bb4-host-cni-netd\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.169147 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.168856 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1edd2f99-7d44-4eed-8850-50ca38aa5153-cnibin\") pod \"multus-nrss7\" (UID: \"1edd2f99-7d44-4eed-8850-50ca38aa5153\") " pod="openshift-multus/multus-nrss7" Jul 17 18:55:03.169147 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.168860 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlmhc\" (UniqueName: \"kubernetes.io/projected/2c91e513-4f09-4551-8f50-75a4ba131d12-kube-api-access-mlmhc\") pod \"network-check-target-c54gz\" (UID: \"2c91e513-4f09-4551-8f50-75a4ba131d12\") " pod="openshift-network-diagnostics/network-check-target-c54gz" Jul 17 18:55:03.169147 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.168878 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4b162535-066a-471a-aee4-99c07c6e1bb4-host-cni-bin\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.169147 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.168922 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c649f1a2-c525-4818-bde3-f5c394101099-etc-modprobe-d\") pod \"tuned-kj2t7\" (UID: \"c649f1a2-c525-4818-bde3-f5c394101099\") " pod="openshift-cluster-node-tuning-operator/tuned-kj2t7" Jul 17 18:55:03.169948 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.168979 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/64f94c18-0eab-41bb-88e0-824db8dfa7e4-socket-dir\") pod \"aws-ebs-csi-driver-node-92cjd\" (UID: \"64f94c18-0eab-41bb-88e0-824db8dfa7e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92cjd" Jul 17 18:55:03.169948 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.169037 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn2gs\" (UniqueName: \"kubernetes.io/projected/4b162535-066a-471a-aee4-99c07c6e1bb4-kube-api-access-fn2gs\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.169948 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.169061 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/64f94c18-0eab-41bb-88e0-824db8dfa7e4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-92cjd\" (UID: \"64f94c18-0eab-41bb-88e0-824db8dfa7e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92cjd" Jul 17 18:55:03.169948 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.169084 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/64f94c18-0eab-41bb-88e0-824db8dfa7e4-device-dir\") pod \"aws-ebs-csi-driver-node-92cjd\" (UID: \"64f94c18-0eab-41bb-88e0-824db8dfa7e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92cjd" Jul 17 18:55:03.169948 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.169093 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c649f1a2-c525-4818-bde3-f5c394101099-etc-modprobe-d\") pod \"tuned-kj2t7\" (UID: \"c649f1a2-c525-4818-bde3-f5c394101099\") " pod="openshift-cluster-node-tuning-operator/tuned-kj2t7" Jul 17 18:55:03.169948 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.169111 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsm5p\" (UniqueName: \"kubernetes.io/projected/64f94c18-0eab-41bb-88e0-824db8dfa7e4-kube-api-access-rsm5p\") pod \"aws-ebs-csi-driver-node-92cjd\" (UID: \"64f94c18-0eab-41bb-88e0-824db8dfa7e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92cjd" Jul 17 18:55:03.169948 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.169134 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c649f1a2-c525-4818-bde3-f5c394101099-etc-sysconfig\") pod \"tuned-kj2t7\" (UID: \"c649f1a2-c525-4818-bde3-f5c394101099\") " pod="openshift-cluster-node-tuning-operator/tuned-kj2t7" Jul 17 18:55:03.169948 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.169149 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/64f94c18-0eab-41bb-88e0-824db8dfa7e4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-92cjd\" (UID: \"64f94c18-0eab-41bb-88e0-824db8dfa7e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92cjd" Jul 17 18:55:03.169948 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.169155 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4b162535-066a-471a-aee4-99c07c6e1bb4-log-socket\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.169948 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.169158 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/08c31757-7c13-4c31-a1ad-570aad3cde70-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nb8rb\" (UID: \"08c31757-7c13-4c31-a1ad-570aad3cde70\") " pod="openshift-multus/multus-additional-cni-plugins-nb8rb" Jul 17 18:55:03.169948 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.169181 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4b162535-066a-471a-aee4-99c07c6e1bb4-ovnkube-config\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.169948 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.169186 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4b162535-066a-471a-aee4-99c07c6e1bb4-log-socket\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.169948 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.169184 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3591b9f8-a2ff-4adb-93ec-449225ca6390-original-pull-secret\") pod \"global-pull-secret-syncer-m9wzh\" (UID: \"3591b9f8-a2ff-4adb-93ec-449225ca6390\") " pod="kube-system/global-pull-secret-syncer-m9wzh" Jul 17 18:55:03.169948 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.169229 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08c31757-7c13-4c31-a1ad-570aad3cde70-system-cni-dir\") pod \"multus-additional-cni-plugins-nb8rb\" (UID: \"08c31757-7c13-4c31-a1ad-570aad3cde70\") " pod="openshift-multus/multus-additional-cni-plugins-nb8rb" Jul 17 18:55:03.169948 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.169230 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c649f1a2-c525-4818-bde3-f5c394101099-etc-sysconfig\") pod \"tuned-kj2t7\" (UID: \"c649f1a2-c525-4818-bde3-f5c394101099\") " pod="openshift-cluster-node-tuning-operator/tuned-kj2t7" Jul 17 18:55:03.169948 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:03.169283 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 18:55:03.169948 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.169293 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/08c31757-7c13-4c31-a1ad-570aad3cde70-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nb8rb\" (UID: \"08c31757-7c13-4c31-a1ad-570aad3cde70\") " pod="openshift-multus/multus-additional-cni-plugins-nb8rb" Jul 17 18:55:03.170698 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.169334 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08c31757-7c13-4c31-a1ad-570aad3cde70-system-cni-dir\") pod \"multus-additional-cni-plugins-nb8rb\" (UID: \"08c31757-7c13-4c31-a1ad-570aad3cde70\") " pod="openshift-multus/multus-additional-cni-plugins-nb8rb" Jul 17 18:55:03.170698 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.169326 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/64f94c18-0eab-41bb-88e0-824db8dfa7e4-device-dir\") pod \"aws-ebs-csi-driver-node-92cjd\" (UID: \"64f94c18-0eab-41bb-88e0-824db8dfa7e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92cjd" Jul 17 18:55:03.170698 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:03.169413 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3591b9f8-a2ff-4adb-93ec-449225ca6390-original-pull-secret podName:3591b9f8-a2ff-4adb-93ec-449225ca6390 nodeName:}" failed. No retries permitted until 2026-07-17 18:55:03.669369808 +0000 UTC m=+1.999186009 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3591b9f8-a2ff-4adb-93ec-449225ca6390-original-pull-secret") pod "global-pull-secret-syncer-m9wzh" (UID: "3591b9f8-a2ff-4adb-93ec-449225ca6390") : object "kube-system"/"original-pull-secret" not registered Jul 17 18:55:03.170698 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.169455 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c649f1a2-c525-4818-bde3-f5c394101099-tmp\") pod \"tuned-kj2t7\" (UID: \"c649f1a2-c525-4818-bde3-f5c394101099\") " pod="openshift-cluster-node-tuning-operator/tuned-kj2t7" Jul 17 18:55:03.170698 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.169480 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp8d8\" (UniqueName: \"kubernetes.io/projected/c649f1a2-c525-4818-bde3-f5c394101099-kube-api-access-sp8d8\") pod \"tuned-kj2t7\" (UID: \"c649f1a2-c525-4818-bde3-f5c394101099\") " pod="openshift-cluster-node-tuning-operator/tuned-kj2t7" Jul 17 18:55:03.170698 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.169505 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4b162535-066a-471a-aee4-99c07c6e1bb4-run-systemd\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.170698 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.169552 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjtmm\" (UniqueName: \"kubernetes.io/projected/16db4114-eec4-4eb4-80be-1e6238e5c67b-kube-api-access-kjtmm\") pod \"network-metrics-daemon-7l6k9\" (UID: \"16db4114-eec4-4eb4-80be-1e6238e5c67b\") " pod="openshift-multus/network-metrics-daemon-7l6k9" Jul 17 18:55:03.170698 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.169579 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/64f94c18-0eab-41bb-88e0-824db8dfa7e4-etc-selinux\") pod \"aws-ebs-csi-driver-node-92cjd\" (UID: \"64f94c18-0eab-41bb-88e0-824db8dfa7e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92cjd" Jul 17 18:55:03.170698 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.169606 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/64f94c18-0eab-41bb-88e0-824db8dfa7e4-sys-fs\") pod \"aws-ebs-csi-driver-node-92cjd\" (UID: \"64f94c18-0eab-41bb-88e0-824db8dfa7e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92cjd" Jul 17 18:55:03.170698 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.169616 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4b162535-066a-471a-aee4-99c07c6e1bb4-run-systemd\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.170698 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.169631 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/08c31757-7c13-4c31-a1ad-570aad3cde70-os-release\") pod \"multus-additional-cni-plugins-nb8rb\" (UID: \"08c31757-7c13-4c31-a1ad-570aad3cde70\") " pod="openshift-multus/multus-additional-cni-plugins-nb8rb" Jul 17 18:55:03.170698 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.169655 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/08c31757-7c13-4c31-a1ad-570aad3cde70-cni-binary-copy\") pod \"multus-additional-cni-plugins-nb8rb\" (UID: \"08c31757-7c13-4c31-a1ad-570aad3cde70\") " pod="openshift-multus/multus-additional-cni-plugins-nb8rb" Jul 17 18:55:03.170698 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.169680 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1edd2f99-7d44-4eed-8850-50ca38aa5153-multus-cni-dir\") pod \"multus-nrss7\" (UID: \"1edd2f99-7d44-4eed-8850-50ca38aa5153\") " pod="openshift-multus/multus-nrss7" Jul 17 18:55:03.170698 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.169702 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c649f1a2-c525-4818-bde3-f5c394101099-host\") pod \"tuned-kj2t7\" (UID: \"c649f1a2-c525-4818-bde3-f5c394101099\") " pod="openshift-cluster-node-tuning-operator/tuned-kj2t7" Jul 17 18:55:03.170698 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.169708 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/64f94c18-0eab-41bb-88e0-824db8dfa7e4-sys-fs\") pod \"aws-ebs-csi-driver-node-92cjd\" (UID: \"64f94c18-0eab-41bb-88e0-824db8dfa7e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92cjd" Jul 17 18:55:03.170698 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.169727 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/64f94c18-0eab-41bb-88e0-824db8dfa7e4-etc-selinux\") pod \"aws-ebs-csi-driver-node-92cjd\" (UID: \"64f94c18-0eab-41bb-88e0-824db8dfa7e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92cjd" Jul 17 18:55:03.171240 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.169727 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4b162535-066a-471a-aee4-99c07c6e1bb4-ovn-node-metrics-cert\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.171240 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.169735 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/08c31757-7c13-4c31-a1ad-570aad3cde70-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nb8rb\" (UID: \"08c31757-7c13-4c31-a1ad-570aad3cde70\") " pod="openshift-multus/multus-additional-cni-plugins-nb8rb" Jul 17 18:55:03.171240 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.169763 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/728c6dad-4aae-44c2-ac11-60be3b680cca-tmp-dir\") pod \"node-resolver-pvpbg\" (UID: \"728c6dad-4aae-44c2-ac11-60be3b680cca\") " pod="openshift-dns/node-resolver-pvpbg" Jul 17 18:55:03.171240 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.169815 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1edd2f99-7d44-4eed-8850-50ca38aa5153-host-var-lib-cni-bin\") pod \"multus-nrss7\" (UID: \"1edd2f99-7d44-4eed-8850-50ca38aa5153\") " pod="openshift-multus/multus-nrss7" Jul 17 18:55:03.171240 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.169836 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c649f1a2-c525-4818-bde3-f5c394101099-etc-sysctl-d\") pod \"tuned-kj2t7\" (UID: \"c649f1a2-c525-4818-bde3-f5c394101099\") " pod="openshift-cluster-node-tuning-operator/tuned-kj2t7" Jul 17 18:55:03.171240 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.169877 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1edd2f99-7d44-4eed-8850-50ca38aa5153-host-var-lib-cni-bin\") pod \"multus-nrss7\" (UID: \"1edd2f99-7d44-4eed-8850-50ca38aa5153\") " pod="openshift-multus/multus-nrss7" Jul 17 18:55:03.171240 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.169911 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c649f1a2-c525-4818-bde3-f5c394101099-etc-systemd\") pod \"tuned-kj2t7\" (UID: \"c649f1a2-c525-4818-bde3-f5c394101099\") " pod="openshift-cluster-node-tuning-operator/tuned-kj2t7" Jul 17 18:55:03.171240 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.169933 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c649f1a2-c525-4818-bde3-f5c394101099-etc-sysctl-d\") pod \"tuned-kj2t7\" (UID: \"c649f1a2-c525-4818-bde3-f5c394101099\") " pod="openshift-cluster-node-tuning-operator/tuned-kj2t7" Jul 17 18:55:03.171240 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.169948 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4b162535-066a-471a-aee4-99c07c6e1bb4-systemd-units\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.171240 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.169988 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c649f1a2-c525-4818-bde3-f5c394101099-host\") pod \"tuned-kj2t7\" (UID: \"c649f1a2-c525-4818-bde3-f5c394101099\") " pod="openshift-cluster-node-tuning-operator/tuned-kj2t7" Jul 17 18:55:03.171240 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.170023 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7f5ebd27-cd62-4b2b-9c30-4863ca02f422-host\") pod \"node-ca-n2wtb\" (UID: \"7f5ebd27-cd62-4b2b-9c30-4863ca02f422\") " pod="openshift-image-registry/node-ca-n2wtb" Jul 17 18:55:03.171240 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.170057 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4b162535-066a-471a-aee4-99c07c6e1bb4-systemd-units\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.171240 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.170072 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/728c6dad-4aae-44c2-ac11-60be3b680cca-tmp-dir\") pod \"node-resolver-pvpbg\" (UID: \"728c6dad-4aae-44c2-ac11-60be3b680cca\") " pod="openshift-dns/node-resolver-pvpbg" Jul 17 18:55:03.171240 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.170096 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7f5ebd27-cd62-4b2b-9c30-4863ca02f422-host\") pod \"node-ca-n2wtb\" (UID: \"7f5ebd27-cd62-4b2b-9c30-4863ca02f422\") " pod="openshift-image-registry/node-ca-n2wtb" Jul 17 18:55:03.171240 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.170030 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c649f1a2-c525-4818-bde3-f5c394101099-etc-systemd\") pod \"tuned-kj2t7\" (UID: \"c649f1a2-c525-4818-bde3-f5c394101099\") " pod="openshift-cluster-node-tuning-operator/tuned-kj2t7" Jul 17 18:55:03.171240 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.170195 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1edd2f99-7d44-4eed-8850-50ca38aa5153-cni-binary-copy\") pod \"multus-nrss7\" (UID: \"1edd2f99-7d44-4eed-8850-50ca38aa5153\") " pod="openshift-multus/multus-nrss7" Jul 17 18:55:03.171240 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.170222 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf4zc\" (UniqueName: \"kubernetes.io/projected/1edd2f99-7d44-4eed-8850-50ca38aa5153-kube-api-access-sf4zc\") pod \"multus-nrss7\" (UID: \"1edd2f99-7d44-4eed-8850-50ca38aa5153\") " pod="openshift-multus/multus-nrss7" Jul 17 18:55:03.171240 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.170232 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/08c31757-7c13-4c31-a1ad-570aad3cde70-cni-binary-copy\") pod \"multus-additional-cni-plugins-nb8rb\" (UID: \"08c31757-7c13-4c31-a1ad-570aad3cde70\") " pod="openshift-multus/multus-additional-cni-plugins-nb8rb" Jul 17 18:55:03.172091 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.170243 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c649f1a2-c525-4818-bde3-f5c394101099-etc-kubernetes\") pod \"tuned-kj2t7\" (UID: \"c649f1a2-c525-4818-bde3-f5c394101099\") " pod="openshift-cluster-node-tuning-operator/tuned-kj2t7" Jul 17 18:55:03.172091 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.170265 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c649f1a2-c525-4818-bde3-f5c394101099-lib-modules\") pod \"tuned-kj2t7\" (UID: \"c649f1a2-c525-4818-bde3-f5c394101099\") " pod="openshift-cluster-node-tuning-operator/tuned-kj2t7" Jul 17 18:55:03.172091 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.170295 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/08c31757-7c13-4c31-a1ad-570aad3cde70-os-release\") pod \"multus-additional-cni-plugins-nb8rb\" (UID: \"08c31757-7c13-4c31-a1ad-570aad3cde70\") " pod="openshift-multus/multus-additional-cni-plugins-nb8rb" Jul 17 18:55:03.172091 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.170307 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b162535-066a-471a-aee4-99c07c6e1bb4-host-run-ovn-kubernetes\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.172091 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.170299 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c649f1a2-c525-4818-bde3-f5c394101099-etc-kubernetes\") pod \"tuned-kj2t7\" (UID: \"c649f1a2-c525-4818-bde3-f5c394101099\") " pod="openshift-cluster-node-tuning-operator/tuned-kj2t7" Jul 17 18:55:03.172091 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.170350 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16db4114-eec4-4eb4-80be-1e6238e5c67b-metrics-certs\") pod \"network-metrics-daemon-7l6k9\" (UID: \"16db4114-eec4-4eb4-80be-1e6238e5c67b\") " pod="openshift-multus/network-metrics-daemon-7l6k9" Jul 17 18:55:03.172091 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.170356 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b162535-066a-471a-aee4-99c07c6e1bb4-host-run-ovn-kubernetes\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.172091 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.170195 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1edd2f99-7d44-4eed-8850-50ca38aa5153-multus-cni-dir\") pod \"multus-nrss7\" (UID: \"1edd2f99-7d44-4eed-8850-50ca38aa5153\") " pod="openshift-multus/multus-nrss7" Jul 17 18:55:03.172091 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.170375 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/64f94c18-0eab-41bb-88e0-824db8dfa7e4-registration-dir\") pod \"aws-ebs-csi-driver-node-92cjd\" (UID: \"64f94c18-0eab-41bb-88e0-824db8dfa7e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92cjd" Jul 17 18:55:03.172091 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.170415 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c649f1a2-c525-4818-bde3-f5c394101099-lib-modules\") pod \"tuned-kj2t7\" (UID: \"c649f1a2-c525-4818-bde3-f5c394101099\") " pod="openshift-cluster-node-tuning-operator/tuned-kj2t7" Jul 17 18:55:03.172091 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.170423 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/08c31757-7c13-4c31-a1ad-570aad3cde70-cnibin\") pod \"multus-additional-cni-plugins-nb8rb\" (UID: \"08c31757-7c13-4c31-a1ad-570aad3cde70\") " pod="openshift-multus/multus-additional-cni-plugins-nb8rb" Jul 17 18:55:03.172091 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.170469 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4b162535-066a-471a-aee4-99c07c6e1bb4-var-lib-openvswitch\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.172091 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:03.170475 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 18:55:03.172091 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.170484 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/64f94c18-0eab-41bb-88e0-824db8dfa7e4-registration-dir\") pod \"aws-ebs-csi-driver-node-92cjd\" (UID: \"64f94c18-0eab-41bb-88e0-824db8dfa7e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92cjd" Jul 17 18:55:03.172091 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.170491 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq57r\" (UniqueName: \"kubernetes.io/projected/7f5ebd27-cd62-4b2b-9c30-4863ca02f422-kube-api-access-gq57r\") pod \"node-ca-n2wtb\" (UID: \"7f5ebd27-cd62-4b2b-9c30-4863ca02f422\") " pod="openshift-image-registry/node-ca-n2wtb" Jul 17 18:55:03.172091 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.170555 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/08c31757-7c13-4c31-a1ad-570aad3cde70-cnibin\") pod \"multus-additional-cni-plugins-nb8rb\" (UID: \"08c31757-7c13-4c31-a1ad-570aad3cde70\") " pod="openshift-multus/multus-additional-cni-plugins-nb8rb" Jul 17 18:55:03.172091 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.170553 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s827\" (UniqueName: \"kubernetes.io/projected/728c6dad-4aae-44c2-ac11-60be3b680cca-kube-api-access-5s827\") pod \"node-resolver-pvpbg\" (UID: \"728c6dad-4aae-44c2-ac11-60be3b680cca\") " pod="openshift-dns/node-resolver-pvpbg" Jul 17 18:55:03.172597 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:03.170731 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16db4114-eec4-4eb4-80be-1e6238e5c67b-metrics-certs podName:16db4114-eec4-4eb4-80be-1e6238e5c67b nodeName:}" failed. No retries permitted until 2026-07-17 18:55:03.670704424 +0000 UTC m=+2.000520608 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/16db4114-eec4-4eb4-80be-1e6238e5c67b-metrics-certs") pod "network-metrics-daemon-7l6k9" (UID: "16db4114-eec4-4eb4-80be-1e6238e5c67b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 18:55:03.172597 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.170746 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4b162535-066a-471a-aee4-99c07c6e1bb4-var-lib-openvswitch\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.172597 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.170907 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/08c31757-7c13-4c31-a1ad-570aad3cde70-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nb8rb\" (UID: \"08c31757-7c13-4c31-a1ad-570aad3cde70\") " pod="openshift-multus/multus-additional-cni-plugins-nb8rb" Jul 17 18:55:03.172597 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.170934 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1edd2f99-7d44-4eed-8850-50ca38aa5153-hostroot\") pod \"multus-nrss7\" (UID: \"1edd2f99-7d44-4eed-8850-50ca38aa5153\") " pod="openshift-multus/multus-nrss7" Jul 17 18:55:03.172597 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.170955 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkrll\" (UniqueName: \"kubernetes.io/projected/08c31757-7c13-4c31-a1ad-570aad3cde70-kube-api-access-zkrll\") pod \"multus-additional-cni-plugins-nb8rb\" (UID: \"08c31757-7c13-4c31-a1ad-570aad3cde70\") " pod="openshift-multus/multus-additional-cni-plugins-nb8rb" Jul 17 18:55:03.172597 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.170992 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1edd2f99-7d44-4eed-8850-50ca38aa5153-multus-socket-dir-parent\") pod \"multus-nrss7\" (UID: \"1edd2f99-7d44-4eed-8850-50ca38aa5153\") " pod="openshift-multus/multus-nrss7" Jul 17 18:55:03.172597 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.171018 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c649f1a2-c525-4818-bde3-f5c394101099-run\") pod \"tuned-kj2t7\" (UID: \"c649f1a2-c525-4818-bde3-f5c394101099\") " pod="openshift-cluster-node-tuning-operator/tuned-kj2t7" Jul 17 18:55:03.172597 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.171040 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4b162535-066a-471a-aee4-99c07c6e1bb4-ovnkube-script-lib\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.172597 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.171070 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7f5ebd27-cd62-4b2b-9c30-4863ca02f422-serviceca\") pod \"node-ca-n2wtb\" (UID: \"7f5ebd27-cd62-4b2b-9c30-4863ca02f422\") " pod="openshift-image-registry/node-ca-n2wtb" Jul 17 18:55:03.172597 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.171085 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/08c31757-7c13-4c31-a1ad-570aad3cde70-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nb8rb\" (UID: \"08c31757-7c13-4c31-a1ad-570aad3cde70\") " pod="openshift-multus/multus-additional-cni-plugins-nb8rb" Jul 17 18:55:03.172597 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.171091 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c649f1a2-c525-4818-bde3-f5c394101099-sys\") pod \"tuned-kj2t7\" (UID: \"c649f1a2-c525-4818-bde3-f5c394101099\") " pod="openshift-cluster-node-tuning-operator/tuned-kj2t7" Jul 17 18:55:03.172597 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.171167 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/728c6dad-4aae-44c2-ac11-60be3b680cca-hosts-file\") pod \"node-resolver-pvpbg\" (UID: \"728c6dad-4aae-44c2-ac11-60be3b680cca\") " pod="openshift-dns/node-resolver-pvpbg" Jul 17 18:55:03.172597 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.171174 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c649f1a2-c525-4818-bde3-f5c394101099-run\") pod \"tuned-kj2t7\" (UID: \"c649f1a2-c525-4818-bde3-f5c394101099\") " pod="openshift-cluster-node-tuning-operator/tuned-kj2t7" Jul 17 18:55:03.172597 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.171195 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1edd2f99-7d44-4eed-8850-50ca38aa5153-host-run-k8s-cni-cncf-io\") pod \"multus-nrss7\" (UID: \"1edd2f99-7d44-4eed-8850-50ca38aa5153\") " pod="openshift-multus/multus-nrss7" Jul 17 18:55:03.172597 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.171202 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1edd2f99-7d44-4eed-8850-50ca38aa5153-hostroot\") pod \"multus-nrss7\" (UID: \"1edd2f99-7d44-4eed-8850-50ca38aa5153\") " pod="openshift-multus/multus-nrss7" Jul 17 18:55:03.172597 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.171227 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1edd2f99-7d44-4eed-8850-50ca38aa5153-multus-conf-dir\") pod \"multus-nrss7\" (UID: \"1edd2f99-7d44-4eed-8850-50ca38aa5153\") " pod="openshift-multus/multus-nrss7" Jul 17 18:55:03.172597 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.171230 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1edd2f99-7d44-4eed-8850-50ca38aa5153-multus-socket-dir-parent\") pod \"multus-nrss7\" (UID: \"1edd2f99-7d44-4eed-8850-50ca38aa5153\") " pod="openshift-multus/multus-nrss7" Jul 17 18:55:03.173118 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.171255 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/728c6dad-4aae-44c2-ac11-60be3b680cca-hosts-file\") pod \"node-resolver-pvpbg\" (UID: \"728c6dad-4aae-44c2-ac11-60be3b680cca\") " pod="openshift-dns/node-resolver-pvpbg" Jul 17 18:55:03.173118 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.171252 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1edd2f99-7d44-4eed-8850-50ca38aa5153-multus-daemon-config\") pod \"multus-nrss7\" (UID: \"1edd2f99-7d44-4eed-8850-50ca38aa5153\") " pod="openshift-multus/multus-nrss7" Jul 17 18:55:03.173118 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.171290 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4b162535-066a-471a-aee4-99c07c6e1bb4-host-slash\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.173118 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.171299 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1edd2f99-7d44-4eed-8850-50ca38aa5153-multus-conf-dir\") pod \"multus-nrss7\" (UID: \"1edd2f99-7d44-4eed-8850-50ca38aa5153\") " pod="openshift-multus/multus-nrss7" Jul 17 18:55:03.173118 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.171339 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4b162535-066a-471a-aee4-99c07c6e1bb4-run-ovn\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.173118 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.171337 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c649f1a2-c525-4818-bde3-f5c394101099-sys\") pod \"tuned-kj2t7\" (UID: \"c649f1a2-c525-4818-bde3-f5c394101099\") " pod="openshift-cluster-node-tuning-operator/tuned-kj2t7" Jul 17 18:55:03.173118 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.171362 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1edd2f99-7d44-4eed-8850-50ca38aa5153-host-run-k8s-cni-cncf-io\") pod \"multus-nrss7\" (UID: \"1edd2f99-7d44-4eed-8850-50ca38aa5153\") " pod="openshift-multus/multus-nrss7" Jul 17 18:55:03.173118 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.171366 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4b162535-066a-471a-aee4-99c07c6e1bb4-env-overrides\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.173118 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.171405 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4b162535-066a-471a-aee4-99c07c6e1bb4-host-slash\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.173118 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.171465 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4b162535-066a-471a-aee4-99c07c6e1bb4-run-ovn\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.173118 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.171459 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7f5ebd27-cd62-4b2b-9c30-4863ca02f422-serviceca\") pod \"node-ca-n2wtb\" (UID: \"7f5ebd27-cd62-4b2b-9c30-4863ca02f422\") " pod="openshift-image-registry/node-ca-n2wtb" Jul 17 18:55:03.173118 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.171497 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1edd2f99-7d44-4eed-8850-50ca38aa5153-system-cni-dir\") pod \"multus-nrss7\" (UID: \"1edd2f99-7d44-4eed-8850-50ca38aa5153\") " pod="openshift-multus/multus-nrss7" Jul 17 18:55:03.173118 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.171556 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1edd2f99-7d44-4eed-8850-50ca38aa5153-system-cni-dir\") pod \"multus-nrss7\" (UID: \"1edd2f99-7d44-4eed-8850-50ca38aa5153\") " pod="openshift-multus/multus-nrss7" Jul 17 18:55:03.173118 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.171559 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1edd2f99-7d44-4eed-8850-50ca38aa5153-os-release\") pod \"multus-nrss7\" (UID: \"1edd2f99-7d44-4eed-8850-50ca38aa5153\") " pod="openshift-multus/multus-nrss7" Jul 17 18:55:03.173118 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.171583 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c649f1a2-c525-4818-bde3-f5c394101099-var-lib-kubelet\") pod \"tuned-kj2t7\" (UID: \"c649f1a2-c525-4818-bde3-f5c394101099\") " pod="openshift-cluster-node-tuning-operator/tuned-kj2t7" Jul 17 18:55:03.173118 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.171654 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1edd2f99-7d44-4eed-8850-50ca38aa5153-os-release\") pod \"multus-nrss7\" (UID: \"1edd2f99-7d44-4eed-8850-50ca38aa5153\") " pod="openshift-multus/multus-nrss7" Jul 17 18:55:03.173118 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.171704 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4b162535-066a-471a-aee4-99c07c6e1bb4-etc-openvswitch\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.173118 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.171798 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4b162535-066a-471a-aee4-99c07c6e1bb4-run-openvswitch\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.173720 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.171819 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4b162535-066a-471a-aee4-99c07c6e1bb4-env-overrides\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.173720 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.171838 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b162535-066a-471a-aee4-99c07c6e1bb4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.173720 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.171724 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c649f1a2-c525-4818-bde3-f5c394101099-etc-tuned\") pod \"tuned-kj2t7\" (UID: \"c649f1a2-c525-4818-bde3-f5c394101099\") " pod="openshift-cluster-node-tuning-operator/tuned-kj2t7" Jul 17 18:55:03.173720 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.171763 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4b162535-066a-471a-aee4-99c07c6e1bb4-etc-openvswitch\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.173720 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.171905 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4b162535-066a-471a-aee4-99c07c6e1bb4-run-openvswitch\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.173720 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.171765 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c649f1a2-c525-4818-bde3-f5c394101099-var-lib-kubelet\") pod \"tuned-kj2t7\" (UID: \"c649f1a2-c525-4818-bde3-f5c394101099\") " pod="openshift-cluster-node-tuning-operator/tuned-kj2t7" Jul 17 18:55:03.173720 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.171978 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4b162535-066a-471a-aee4-99c07c6e1bb4-ovnkube-script-lib\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.173720 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.171973 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b162535-066a-471a-aee4-99c07c6e1bb4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.173720 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.172344 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c649f1a2-c525-4818-bde3-f5c394101099-tmp\") pod \"tuned-kj2t7\" (UID: \"c649f1a2-c525-4818-bde3-f5c394101099\") " pod="openshift-cluster-node-tuning-operator/tuned-kj2t7" Jul 17 18:55:03.173720 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.172351 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4b162535-066a-471a-aee4-99c07c6e1bb4-ovn-node-metrics-cert\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.173720 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.172372 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1edd2f99-7d44-4eed-8850-50ca38aa5153-cni-binary-copy\") pod \"multus-nrss7\" (UID: \"1edd2f99-7d44-4eed-8850-50ca38aa5153\") " pod="openshift-multus/multus-nrss7" Jul 17 18:55:03.173720 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.172409 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1edd2f99-7d44-4eed-8850-50ca38aa5153-multus-daemon-config\") pod \"multus-nrss7\" (UID: \"1edd2f99-7d44-4eed-8850-50ca38aa5153\") " pod="openshift-multus/multus-nrss7" Jul 17 18:55:03.185536 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.185504 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsm5p\" (UniqueName: \"kubernetes.io/projected/64f94c18-0eab-41bb-88e0-824db8dfa7e4-kube-api-access-rsm5p\") pod \"aws-ebs-csi-driver-node-92cjd\" (UID: \"64f94c18-0eab-41bb-88e0-824db8dfa7e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92cjd" Jul 17 18:55:03.187828 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:03.187804 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 18:55:03.187828 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:03.187827 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 18:55:03.187931 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:03.187861 2561 projected.go:196] Error preparing data for projected volume kube-api-access-mlmhc for pod openshift-network-diagnostics/network-check-target-c54gz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 18:55:03.187931 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:03.187924 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c91e513-4f09-4551-8f50-75a4ba131d12-kube-api-access-mlmhc podName:2c91e513-4f09-4551-8f50-75a4ba131d12 nodeName:}" failed. No retries permitted until 2026-07-17 18:55:03.687908391 +0000 UTC m=+2.017724593 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mlmhc" (UniqueName: "kubernetes.io/projected/2c91e513-4f09-4551-8f50-75a4ba131d12-kube-api-access-mlmhc") pod "network-check-target-c54gz" (UID: "2c91e513-4f09-4551-8f50-75a4ba131d12") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 18:55:03.189812 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.189789 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn2gs\" (UniqueName: \"kubernetes.io/projected/4b162535-066a-471a-aee4-99c07c6e1bb4-kube-api-access-fn2gs\") pod \"ovnkube-node-hpjzq\" (UID: \"4b162535-066a-471a-aee4-99c07c6e1bb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.196595 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.196573 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf4zc\" (UniqueName: \"kubernetes.io/projected/1edd2f99-7d44-4eed-8850-50ca38aa5153-kube-api-access-sf4zc\") pod \"multus-nrss7\" (UID: \"1edd2f99-7d44-4eed-8850-50ca38aa5153\") " pod="openshift-multus/multus-nrss7" Jul 17 18:55:03.196809 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.196776 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s827\" (UniqueName: \"kubernetes.io/projected/728c6dad-4aae-44c2-ac11-60be3b680cca-kube-api-access-5s827\") pod \"node-resolver-pvpbg\" (UID: \"728c6dad-4aae-44c2-ac11-60be3b680cca\") " pod="openshift-dns/node-resolver-pvpbg" Jul 17 18:55:03.197321 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.197300 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjtmm\" (UniqueName: \"kubernetes.io/projected/16db4114-eec4-4eb4-80be-1e6238e5c67b-kube-api-access-kjtmm\") pod \"network-metrics-daemon-7l6k9\" (UID: \"16db4114-eec4-4eb4-80be-1e6238e5c67b\") " pod="openshift-multus/network-metrics-daemon-7l6k9" Jul 17 18:55:03.197614 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.197598 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkrll\" (UniqueName: \"kubernetes.io/projected/08c31757-7c13-4c31-a1ad-570aad3cde70-kube-api-access-zkrll\") pod \"multus-additional-cni-plugins-nb8rb\" (UID: \"08c31757-7c13-4c31-a1ad-570aad3cde70\") " pod="openshift-multus/multus-additional-cni-plugins-nb8rb" Jul 17 18:55:03.197682 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.197623 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp8d8\" (UniqueName: \"kubernetes.io/projected/c649f1a2-c525-4818-bde3-f5c394101099-kube-api-access-sp8d8\") pod \"tuned-kj2t7\" (UID: \"c649f1a2-c525-4818-bde3-f5c394101099\") " pod="openshift-cluster-node-tuning-operator/tuned-kj2t7" Jul 17 18:55:03.197941 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.197925 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq57r\" (UniqueName: \"kubernetes.io/projected/7f5ebd27-cd62-4b2b-9c30-4863ca02f422-kube-api-access-gq57r\") pod \"node-ca-n2wtb\" (UID: \"7f5ebd27-cd62-4b2b-9c30-4863ca02f422\") " pod="openshift-image-registry/node-ca-n2wtb" Jul 17 18:55:03.279801 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.279727 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2fgtz" Jul 17 18:55:03.300315 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:03.300275 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0135e7e9_6294_4524_ad81_17f251822127.slice/crio-05e3176e6e43f4f15d23eb8873091fa4b75d8a739706dc83294ea3f9ea04388e WatchSource:0}: Error finding container 05e3176e6e43f4f15d23eb8873091fa4b75d8a739706dc83294ea3f9ea04388e: Status 404 returned error can't find the container with id 05e3176e6e43f4f15d23eb8873091fa4b75d8a739706dc83294ea3f9ea04388e Jul 17 18:55:03.301056 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:03.301038 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4f6391d232b2f3f6af57e5daac336ae.slice/crio-dcf757578a5a5bc4c2aed5678f485ca49f11b24693230521e02c5ff6a2408e25 WatchSource:0}: Error finding container dcf757578a5a5bc4c2aed5678f485ca49f11b24693230521e02c5ff6a2408e25: Status 404 returned error can't find the container with id dcf757578a5a5bc4c2aed5678f485ca49f11b24693230521e02c5ff6a2408e25 Jul 17 18:55:03.302321 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.302147 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7g4zx" Jul 17 18:55:03.306154 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.306136 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 18:55:03.311122 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:03.311101 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6b17b15_c9dc_4e3e_9fb3_8477f0a734ed.slice/crio-1aeda12132f19c55c23259d3a24c8dd380574b3d65ebf1402e9a9db6d336ab28 WatchSource:0}: Error finding container 1aeda12132f19c55c23259d3a24c8dd380574b3d65ebf1402e9a9db6d336ab28: Status 404 returned error can't find the container with id 1aeda12132f19c55c23259d3a24c8dd380574b3d65ebf1402e9a9db6d336ab28 Jul 17 18:55:03.319060 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.319039 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-n2wtb" Jul 17 18:55:03.326691 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:03.326659 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f5ebd27_cd62_4b2b_9c30_4863ca02f422.slice/crio-12bb3b4a868ca56011f5ba0603f3769e75568bd1f4aebe2d2a7fb166af8746ee WatchSource:0}: Error finding container 12bb3b4a868ca56011f5ba0603f3769e75568bd1f4aebe2d2a7fb166af8746ee: Status 404 returned error can't find the container with id 12bb3b4a868ca56011f5ba0603f3769e75568bd1f4aebe2d2a7fb166af8746ee Jul 17 18:55:03.328225 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.328209 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pvpbg" Jul 17 18:55:03.335875 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:03.335854 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod728c6dad_4aae_44c2_ac11_60be3b680cca.slice/crio-3f29e7b010faa3311add074eb2aa27baa9dce7f3ddf89422a2d1c6e4aec1ba6f WatchSource:0}: Error finding container 3f29e7b010faa3311add074eb2aa27baa9dce7f3ddf89422a2d1c6e4aec1ba6f: Status 404 returned error can't find the container with id 3f29e7b010faa3311add074eb2aa27baa9dce7f3ddf89422a2d1c6e4aec1ba6f Jul 17 18:55:03.346286 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.346269 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:03.353993 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:03.353959 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b162535_066a_471a_aee4_99c07c6e1bb4.slice/crio-bf8aa5c5bb313410ae6566c74eb3602af7b8958e8c7d463fcca1dae878677fc2 WatchSource:0}: Error finding container bf8aa5c5bb313410ae6566c74eb3602af7b8958e8c7d463fcca1dae878677fc2: Status 404 returned error can't find the container with id bf8aa5c5bb313410ae6566c74eb3602af7b8958e8c7d463fcca1dae878677fc2 Jul 17 18:55:03.370004 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.369975 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nb8rb" Jul 17 18:55:03.378593 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:03.378569 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08c31757_7c13_4c31_a1ad_570aad3cde70.slice/crio-c3e94c9e191d4b09badeb02ecaf576667c70a1937d6120956b945c3ba0317cd8 WatchSource:0}: Error finding container c3e94c9e191d4b09badeb02ecaf576667c70a1937d6120956b945c3ba0317cd8: Status 404 returned error can't find the container with id c3e94c9e191d4b09badeb02ecaf576667c70a1937d6120956b945c3ba0317cd8 Jul 17 18:55:03.381502 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.381486 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nrss7" Jul 17 18:55:03.389330 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:03.389307 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1edd2f99_7d44_4eed_8850_50ca38aa5153.slice/crio-1740f626e776f134b58908faf25ae96472ce46ae3295d437e2052077f878917f WatchSource:0}: Error finding container 1740f626e776f134b58908faf25ae96472ce46ae3295d437e2052077f878917f: Status 404 returned error can't find the container with id 1740f626e776f134b58908faf25ae96472ce46ae3295d437e2052077f878917f Jul 17 18:55:03.393478 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.393457 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92cjd" Jul 17 18:55:03.394371 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.394356 2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 18:55:03.397879 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.397859 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kj2t7" Jul 17 18:55:03.401719 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:03.401699 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64f94c18_0eab_41bb_88e0_824db8dfa7e4.slice/crio-c677cba8ee320e9d0e709be1498be218c02da5c5492438c6ba9311b0cbd14928 WatchSource:0}: Error finding container c677cba8ee320e9d0e709be1498be218c02da5c5492438c6ba9311b0cbd14928: Status 404 returned error can't find the container with id c677cba8ee320e9d0e709be1498be218c02da5c5492438c6ba9311b0cbd14928 Jul 17 18:55:03.407685 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:03.406887 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc649f1a2_c525_4818_bde3_f5c394101099.slice/crio-6f05a84737cdf9ed7e63969a7011129d16c9a60f3ebf8d31d774e99e42dc0967 WatchSource:0}: Error finding container 6f05a84737cdf9ed7e63969a7011129d16c9a60f3ebf8d31d774e99e42dc0967: Status 404 returned error can't find the container with id 6f05a84737cdf9ed7e63969a7011129d16c9a60f3ebf8d31d774e99e42dc0967 Jul 17 18:55:03.672054 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:03.672022 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4055332a3b86c15989d82b71c24d29a9.slice/crio-24db83832b399600a818cfdc203b91c020b6f6cf23a7f29f8964538e675577d2 WatchSource:0}: Error finding container 24db83832b399600a818cfdc203b91c020b6f6cf23a7f29f8964538e675577d2: Status 404 returned error can't find the container with id 24db83832b399600a818cfdc203b91c020b6f6cf23a7f29f8964538e675577d2 Jul 17 18:55:03.676179 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.676161 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16db4114-eec4-4eb4-80be-1e6238e5c67b-metrics-certs\") pod \"network-metrics-daemon-7l6k9\" (UID: \"16db4114-eec4-4eb4-80be-1e6238e5c67b\") " pod="openshift-multus/network-metrics-daemon-7l6k9" Jul 17 18:55:03.676242 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.676208 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3591b9f8-a2ff-4adb-93ec-449225ca6390-original-pull-secret\") pod \"global-pull-secret-syncer-m9wzh\" (UID: \"3591b9f8-a2ff-4adb-93ec-449225ca6390\") " pod="kube-system/global-pull-secret-syncer-m9wzh" Jul 17 18:55:03.676306 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:03.676295 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 18:55:03.676353 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:03.676303 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 18:55:03.676353 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:03.676340 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3591b9f8-a2ff-4adb-93ec-449225ca6390-original-pull-secret podName:3591b9f8-a2ff-4adb-93ec-449225ca6390 nodeName:}" failed. No retries permitted until 2026-07-17 18:55:04.676326087 +0000 UTC m=+3.006142273 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3591b9f8-a2ff-4adb-93ec-449225ca6390-original-pull-secret") pod "global-pull-secret-syncer-m9wzh" (UID: "3591b9f8-a2ff-4adb-93ec-449225ca6390") : object "kube-system"/"original-pull-secret" not registered Jul 17 18:55:03.676353 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:03.676354 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16db4114-eec4-4eb4-80be-1e6238e5c67b-metrics-certs podName:16db4114-eec4-4eb4-80be-1e6238e5c67b nodeName:}" failed. No retries permitted until 2026-07-17 18:55:04.676347982 +0000 UTC m=+3.006164166 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/16db4114-eec4-4eb4-80be-1e6238e5c67b-metrics-certs") pod "network-metrics-daemon-7l6k9" (UID: "16db4114-eec4-4eb4-80be-1e6238e5c67b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 18:55:03.777473 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:03.777438 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlmhc\" (UniqueName: \"kubernetes.io/projected/2c91e513-4f09-4551-8f50-75a4ba131d12-kube-api-access-mlmhc\") pod \"network-check-target-c54gz\" (UID: \"2c91e513-4f09-4551-8f50-75a4ba131d12\") " pod="openshift-network-diagnostics/network-check-target-c54gz" Jul 17 18:55:03.777639 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:03.777625 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 18:55:03.777700 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:03.777649 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 18:55:03.777700 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:03.777661 2561 projected.go:196] Error preparing data for projected volume kube-api-access-mlmhc for pod openshift-network-diagnostics/network-check-target-c54gz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 18:55:03.777808 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:03.777724 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c91e513-4f09-4551-8f50-75a4ba131d12-kube-api-access-mlmhc podName:2c91e513-4f09-4551-8f50-75a4ba131d12 nodeName:}" failed. No retries permitted until 2026-07-17 18:55:04.777705969 +0000 UTC m=+3.107522173 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mlmhc" (UniqueName: "kubernetes.io/projected/2c91e513-4f09-4551-8f50-75a4ba131d12-kube-api-access-mlmhc") pod "network-check-target-c54gz" (UID: "2c91e513-4f09-4551-8f50-75a4ba131d12") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 18:55:04.000491 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:04.000379 2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 18:50:02 +0000 UTC" deadline="2027-12-26 23:22:10.321723534 +0000 UTC" Jul 17 18:55:04.000491 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:04.000438 2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12652h27m6.321290067s" Jul 17 18:55:04.112378 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:04.111923 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kj2t7" event={"ID":"c649f1a2-c525-4818-bde3-f5c394101099","Type":"ContainerStarted","Data":"6f05a84737cdf9ed7e63969a7011129d16c9a60f3ebf8d31d774e99e42dc0967"} Jul 17 18:55:04.119259 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:04.119225 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pvpbg" event={"ID":"728c6dad-4aae-44c2-ac11-60be3b680cca","Type":"ContainerStarted","Data":"3f29e7b010faa3311add074eb2aa27baa9dce7f3ddf89422a2d1c6e4aec1ba6f"} Jul 17 18:55:04.126207 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:04.126172 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-n2wtb" event={"ID":"7f5ebd27-cd62-4b2b-9c30-4863ca02f422","Type":"ContainerStarted","Data":"12bb3b4a868ca56011f5ba0603f3769e75568bd1f4aebe2d2a7fb166af8746ee"} Jul 17 18:55:04.134365 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:04.134329 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7g4zx" event={"ID":"c6b17b15-c9dc-4e3e-9fb3-8477f0a734ed","Type":"ContainerStarted","Data":"1aeda12132f19c55c23259d3a24c8dd380574b3d65ebf1402e9a9db6d336ab28"} Jul 17 18:55:04.145133 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:04.145091 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2fgtz" event={"ID":"0135e7e9-6294-4524-ad81-17f251822127","Type":"ContainerStarted","Data":"05e3176e6e43f4f15d23eb8873091fa4b75d8a739706dc83294ea3f9ea04388e"} Jul 17 18:55:04.158106 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:04.158067 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-137.ec2.internal" event={"ID":"f4f6391d232b2f3f6af57e5daac336ae","Type":"ContainerStarted","Data":"dcf757578a5a5bc4c2aed5678f485ca49f11b24693230521e02c5ff6a2408e25"} Jul 17 18:55:04.175745 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:04.175707 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-137.ec2.internal" event={"ID":"4055332a3b86c15989d82b71c24d29a9","Type":"ContainerStarted","Data":"24db83832b399600a818cfdc203b91c020b6f6cf23a7f29f8964538e675577d2"} Jul 17 18:55:04.181654 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:04.181615 2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 18:55:04.219484 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:04.219435 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92cjd" event={"ID":"64f94c18-0eab-41bb-88e0-824db8dfa7e4","Type":"ContainerStarted","Data":"c677cba8ee320e9d0e709be1498be218c02da5c5492438c6ba9311b0cbd14928"} Jul 17 18:55:04.229747 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:04.229706 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nrss7" event={"ID":"1edd2f99-7d44-4eed-8850-50ca38aa5153","Type":"ContainerStarted","Data":"1740f626e776f134b58908faf25ae96472ce46ae3295d437e2052077f878917f"} Jul 17 18:55:04.258741 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:04.258562 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nb8rb" event={"ID":"08c31757-7c13-4c31-a1ad-570aad3cde70","Type":"ContainerStarted","Data":"c3e94c9e191d4b09badeb02ecaf576667c70a1937d6120956b945c3ba0317cd8"} Jul 17 18:55:04.276265 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:04.276226 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" event={"ID":"4b162535-066a-471a-aee4-99c07c6e1bb4","Type":"ContainerStarted","Data":"bf8aa5c5bb313410ae6566c74eb3602af7b8958e8c7d463fcca1dae878677fc2"} Jul 17 18:55:04.685508 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:04.684922 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3591b9f8-a2ff-4adb-93ec-449225ca6390-original-pull-secret\") pod \"global-pull-secret-syncer-m9wzh\" (UID: \"3591b9f8-a2ff-4adb-93ec-449225ca6390\") " pod="kube-system/global-pull-secret-syncer-m9wzh" Jul 17 18:55:04.685508 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:04.684979 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16db4114-eec4-4eb4-80be-1e6238e5c67b-metrics-certs\") pod \"network-metrics-daemon-7l6k9\" (UID: \"16db4114-eec4-4eb4-80be-1e6238e5c67b\") " pod="openshift-multus/network-metrics-daemon-7l6k9" Jul 17 18:55:04.685508 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:04.685105 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 18:55:04.685508 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:04.685159 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16db4114-eec4-4eb4-80be-1e6238e5c67b-metrics-certs podName:16db4114-eec4-4eb4-80be-1e6238e5c67b nodeName:}" failed. No retries permitted until 2026-07-17 18:55:06.685143228 +0000 UTC m=+5.014959417 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/16db4114-eec4-4eb4-80be-1e6238e5c67b-metrics-certs") pod "network-metrics-daemon-7l6k9" (UID: "16db4114-eec4-4eb4-80be-1e6238e5c67b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 18:55:04.685508 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:04.685487 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 18:55:04.685871 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:04.685561 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3591b9f8-a2ff-4adb-93ec-449225ca6390-original-pull-secret podName:3591b9f8-a2ff-4adb-93ec-449225ca6390 nodeName:}" failed. No retries permitted until 2026-07-17 18:55:06.685545954 +0000 UTC m=+5.015362151 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3591b9f8-a2ff-4adb-93ec-449225ca6390-original-pull-secret") pod "global-pull-secret-syncer-m9wzh" (UID: "3591b9f8-a2ff-4adb-93ec-449225ca6390") : object "kube-system"/"original-pull-secret" not registered Jul 17 18:55:04.786323 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:04.786284 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlmhc\" (UniqueName: \"kubernetes.io/projected/2c91e513-4f09-4551-8f50-75a4ba131d12-kube-api-access-mlmhc\") pod \"network-check-target-c54gz\" (UID: \"2c91e513-4f09-4551-8f50-75a4ba131d12\") " pod="openshift-network-diagnostics/network-check-target-c54gz" Jul 17 18:55:04.786506 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:04.786464 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 18:55:04.786506 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:04.786482 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 18:55:04.786506 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:04.786504 2561 projected.go:196] Error preparing data for projected volume kube-api-access-mlmhc for pod openshift-network-diagnostics/network-check-target-c54gz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 18:55:04.786702 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:04.786578 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c91e513-4f09-4551-8f50-75a4ba131d12-kube-api-access-mlmhc podName:2c91e513-4f09-4551-8f50-75a4ba131d12 nodeName:}" failed. No retries permitted until 2026-07-17 18:55:06.786560734 +0000 UTC m=+5.116376923 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mlmhc" (UniqueName: "kubernetes.io/projected/2c91e513-4f09-4551-8f50-75a4ba131d12-kube-api-access-mlmhc") pod "network-check-target-c54gz" (UID: "2c91e513-4f09-4551-8f50-75a4ba131d12") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 18:55:05.001311 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:05.001221 2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 18:50:02 +0000 UTC" deadline="2027-12-20 01:29:32.786060013 +0000 UTC" Jul 17 18:55:05.001311 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:05.001263 2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12486h34m27.784800974s" Jul 17 18:55:05.094402 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:05.093415 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l6k9" Jul 17 18:55:05.094402 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:05.093576 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7l6k9" podUID="16db4114-eec4-4eb4-80be-1e6238e5c67b" Jul 17 18:55:05.094402 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:05.094032 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m9wzh" Jul 17 18:55:05.094402 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:05.094124 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m9wzh" podUID="3591b9f8-a2ff-4adb-93ec-449225ca6390" Jul 17 18:55:05.094402 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:05.094209 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c54gz" Jul 17 18:55:05.094402 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:05.094287 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c54gz" podUID="2c91e513-4f09-4551-8f50-75a4ba131d12" Jul 17 18:55:05.387494 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:05.387404 2561 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 18:55:06.704027 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:06.703986 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16db4114-eec4-4eb4-80be-1e6238e5c67b-metrics-certs\") pod \"network-metrics-daemon-7l6k9\" (UID: \"16db4114-eec4-4eb4-80be-1e6238e5c67b\") " pod="openshift-multus/network-metrics-daemon-7l6k9" Jul 17 18:55:06.704480 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:06.704088 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3591b9f8-a2ff-4adb-93ec-449225ca6390-original-pull-secret\") pod \"global-pull-secret-syncer-m9wzh\" (UID: \"3591b9f8-a2ff-4adb-93ec-449225ca6390\") " pod="kube-system/global-pull-secret-syncer-m9wzh" Jul 17 18:55:06.704480 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:06.704211 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 18:55:06.704480 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:06.704271 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3591b9f8-a2ff-4adb-93ec-449225ca6390-original-pull-secret podName:3591b9f8-a2ff-4adb-93ec-449225ca6390 nodeName:}" failed. No retries permitted until 2026-07-17 18:55:10.704253725 +0000 UTC m=+9.034069917 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3591b9f8-a2ff-4adb-93ec-449225ca6390-original-pull-secret") pod "global-pull-secret-syncer-m9wzh" (UID: "3591b9f8-a2ff-4adb-93ec-449225ca6390") : object "kube-system"/"original-pull-secret" not registered Jul 17 18:55:06.704480 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:06.704336 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 18:55:06.704480 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:06.704368 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16db4114-eec4-4eb4-80be-1e6238e5c67b-metrics-certs podName:16db4114-eec4-4eb4-80be-1e6238e5c67b nodeName:}" failed. No retries permitted until 2026-07-17 18:55:10.704357256 +0000 UTC m=+9.034173461 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/16db4114-eec4-4eb4-80be-1e6238e5c67b-metrics-certs") pod "network-metrics-daemon-7l6k9" (UID: "16db4114-eec4-4eb4-80be-1e6238e5c67b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 18:55:06.804489 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:06.804446 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlmhc\" (UniqueName: \"kubernetes.io/projected/2c91e513-4f09-4551-8f50-75a4ba131d12-kube-api-access-mlmhc\") pod \"network-check-target-c54gz\" (UID: \"2c91e513-4f09-4551-8f50-75a4ba131d12\") " pod="openshift-network-diagnostics/network-check-target-c54gz" Jul 17 18:55:06.804694 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:06.804665 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 18:55:06.804694 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:06.804689 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 18:55:06.804823 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:06.804702 2561 projected.go:196] Error preparing data for projected volume kube-api-access-mlmhc for pod openshift-network-diagnostics/network-check-target-c54gz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 18:55:06.804823 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:06.804762 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c91e513-4f09-4551-8f50-75a4ba131d12-kube-api-access-mlmhc podName:2c91e513-4f09-4551-8f50-75a4ba131d12 nodeName:}" failed. No retries permitted until 2026-07-17 18:55:10.804743757 +0000 UTC m=+9.134559947 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mlmhc" (UniqueName: "kubernetes.io/projected/2c91e513-4f09-4551-8f50-75a4ba131d12-kube-api-access-mlmhc") pod "network-check-target-c54gz" (UID: "2c91e513-4f09-4551-8f50-75a4ba131d12") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 18:55:07.093742 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:07.093664 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l6k9" Jul 17 18:55:07.093894 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:07.093812 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7l6k9" podUID="16db4114-eec4-4eb4-80be-1e6238e5c67b" Jul 17 18:55:07.093943 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:07.093902 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m9wzh" Jul 17 18:55:07.094014 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:07.093981 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m9wzh" podUID="3591b9f8-a2ff-4adb-93ec-449225ca6390" Jul 17 18:55:07.094130 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:07.094043 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c54gz" Jul 17 18:55:07.094130 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:07.094089 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c54gz" podUID="2c91e513-4f09-4551-8f50-75a4ba131d12" Jul 17 18:55:09.093816 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:09.093785 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m9wzh" Jul 17 18:55:09.093816 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:09.093814 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c54gz" Jul 17 18:55:09.094371 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:09.093784 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l6k9" Jul 17 18:55:09.094371 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:09.093913 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m9wzh" podUID="3591b9f8-a2ff-4adb-93ec-449225ca6390" Jul 17 18:55:09.094371 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:09.094010 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7l6k9" podUID="16db4114-eec4-4eb4-80be-1e6238e5c67b" Jul 17 18:55:09.094371 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:09.094070 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c54gz" podUID="2c91e513-4f09-4551-8f50-75a4ba131d12" Jul 17 18:55:10.740249 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:10.740179 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3591b9f8-a2ff-4adb-93ec-449225ca6390-original-pull-secret\") pod \"global-pull-secret-syncer-m9wzh\" (UID: \"3591b9f8-a2ff-4adb-93ec-449225ca6390\") " pod="kube-system/global-pull-secret-syncer-m9wzh" Jul 17 18:55:10.740726 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:10.740284 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16db4114-eec4-4eb4-80be-1e6238e5c67b-metrics-certs\") pod \"network-metrics-daemon-7l6k9\" (UID: \"16db4114-eec4-4eb4-80be-1e6238e5c67b\") " pod="openshift-multus/network-metrics-daemon-7l6k9" Jul 17 18:55:10.740726 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:10.740457 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 18:55:10.740726 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:10.740519 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16db4114-eec4-4eb4-80be-1e6238e5c67b-metrics-certs podName:16db4114-eec4-4eb4-80be-1e6238e5c67b nodeName:}" failed. No retries permitted until 2026-07-17 18:55:18.740499561 +0000 UTC m=+17.070315767 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/16db4114-eec4-4eb4-80be-1e6238e5c67b-metrics-certs") pod "network-metrics-daemon-7l6k9" (UID: "16db4114-eec4-4eb4-80be-1e6238e5c67b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 18:55:10.740726 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:10.740643 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 18:55:10.740726 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:10.740713 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3591b9f8-a2ff-4adb-93ec-449225ca6390-original-pull-secret podName:3591b9f8-a2ff-4adb-93ec-449225ca6390 nodeName:}" failed. No retries permitted until 2026-07-17 18:55:18.740696313 +0000 UTC m=+17.070512503 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3591b9f8-a2ff-4adb-93ec-449225ca6390-original-pull-secret") pod "global-pull-secret-syncer-m9wzh" (UID: "3591b9f8-a2ff-4adb-93ec-449225ca6390") : object "kube-system"/"original-pull-secret" not registered Jul 17 18:55:10.842282 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:10.841644 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlmhc\" (UniqueName: \"kubernetes.io/projected/2c91e513-4f09-4551-8f50-75a4ba131d12-kube-api-access-mlmhc\") pod \"network-check-target-c54gz\" (UID: \"2c91e513-4f09-4551-8f50-75a4ba131d12\") " pod="openshift-network-diagnostics/network-check-target-c54gz" Jul 17 18:55:10.842282 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:10.841850 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 18:55:10.842282 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:10.841870 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 18:55:10.842282 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:10.841883 2561 projected.go:196] Error preparing data for projected volume kube-api-access-mlmhc for pod openshift-network-diagnostics/network-check-target-c54gz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 18:55:10.842282 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:10.841941 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c91e513-4f09-4551-8f50-75a4ba131d12-kube-api-access-mlmhc podName:2c91e513-4f09-4551-8f50-75a4ba131d12 nodeName:}" failed. No retries permitted until 2026-07-17 18:55:18.841920877 +0000 UTC m=+17.171737083 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mlmhc" (UniqueName: "kubernetes.io/projected/2c91e513-4f09-4551-8f50-75a4ba131d12-kube-api-access-mlmhc") pod "network-check-target-c54gz" (UID: "2c91e513-4f09-4551-8f50-75a4ba131d12") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 18:55:11.093194 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:11.093106 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m9wzh" Jul 17 18:55:11.093194 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:11.093157 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c54gz" Jul 17 18:55:11.093414 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:11.093242 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m9wzh" podUID="3591b9f8-a2ff-4adb-93ec-449225ca6390" Jul 17 18:55:11.093414 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:11.093295 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c54gz" podUID="2c91e513-4f09-4551-8f50-75a4ba131d12" Jul 17 18:55:11.093414 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:11.093319 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l6k9" Jul 17 18:55:11.093414 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:11.093403 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7l6k9" podUID="16db4114-eec4-4eb4-80be-1e6238e5c67b" Jul 17 18:55:13.093682 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:13.093594 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m9wzh" Jul 17 18:55:13.094078 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:13.093711 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m9wzh" podUID="3591b9f8-a2ff-4adb-93ec-449225ca6390" Jul 17 18:55:13.094078 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:13.094017 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c54gz" Jul 17 18:55:13.094146 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:13.094079 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c54gz" podUID="2c91e513-4f09-4551-8f50-75a4ba131d12" Jul 17 18:55:13.094146 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:13.094120 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l6k9" Jul 17 18:55:13.094208 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:13.094169 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7l6k9" podUID="16db4114-eec4-4eb4-80be-1e6238e5c67b" Jul 17 18:55:15.093732 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:15.093699 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m9wzh" Jul 17 18:55:15.093732 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:15.093734 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c54gz" Jul 17 18:55:15.094245 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:15.093741 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l6k9" Jul 17 18:55:15.094245 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:15.093798 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m9wzh" podUID="3591b9f8-a2ff-4adb-93ec-449225ca6390" Jul 17 18:55:15.094245 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:15.093909 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c54gz" podUID="2c91e513-4f09-4551-8f50-75a4ba131d12" Jul 17 18:55:15.094245 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:15.093989 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7l6k9" podUID="16db4114-eec4-4eb4-80be-1e6238e5c67b" Jul 17 18:55:17.093752 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:17.093520 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m9wzh" Jul 17 18:55:17.094171 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:17.093546 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l6k9" Jul 17 18:55:17.094171 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:17.093834 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m9wzh" podUID="3591b9f8-a2ff-4adb-93ec-449225ca6390" Jul 17 18:55:17.094171 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:17.093924 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7l6k9" podUID="16db4114-eec4-4eb4-80be-1e6238e5c67b" Jul 17 18:55:17.094171 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:17.093575 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c54gz" Jul 17 18:55:17.094171 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:17.094013 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c54gz" podUID="2c91e513-4f09-4551-8f50-75a4ba131d12" Jul 17 18:55:18.801845 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:18.801800 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3591b9f8-a2ff-4adb-93ec-449225ca6390-original-pull-secret\") pod \"global-pull-secret-syncer-m9wzh\" (UID: \"3591b9f8-a2ff-4adb-93ec-449225ca6390\") " pod="kube-system/global-pull-secret-syncer-m9wzh" Jul 17 18:55:18.802308 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:18.801864 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16db4114-eec4-4eb4-80be-1e6238e5c67b-metrics-certs\") pod \"network-metrics-daemon-7l6k9\" (UID: \"16db4114-eec4-4eb4-80be-1e6238e5c67b\") " pod="openshift-multus/network-metrics-daemon-7l6k9" Jul 17 18:55:18.802308 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:18.801968 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 18:55:18.802308 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:18.801978 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 18:55:18.802308 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:18.802050 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16db4114-eec4-4eb4-80be-1e6238e5c67b-metrics-certs podName:16db4114-eec4-4eb4-80be-1e6238e5c67b nodeName:}" failed. No retries permitted until 2026-07-17 18:55:34.802029629 +0000 UTC m=+33.131845838 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/16db4114-eec4-4eb4-80be-1e6238e5c67b-metrics-certs") pod "network-metrics-daemon-7l6k9" (UID: "16db4114-eec4-4eb4-80be-1e6238e5c67b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 18:55:18.802308 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:18.802069 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3591b9f8-a2ff-4adb-93ec-449225ca6390-original-pull-secret podName:3591b9f8-a2ff-4adb-93ec-449225ca6390 nodeName:}" failed. No retries permitted until 2026-07-17 18:55:34.80206063 +0000 UTC m=+33.131876816 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3591b9f8-a2ff-4adb-93ec-449225ca6390-original-pull-secret") pod "global-pull-secret-syncer-m9wzh" (UID: "3591b9f8-a2ff-4adb-93ec-449225ca6390") : object "kube-system"/"original-pull-secret" not registered Jul 17 18:55:18.902737 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:18.902694 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlmhc\" (UniqueName: \"kubernetes.io/projected/2c91e513-4f09-4551-8f50-75a4ba131d12-kube-api-access-mlmhc\") pod \"network-check-target-c54gz\" (UID: \"2c91e513-4f09-4551-8f50-75a4ba131d12\") " pod="openshift-network-diagnostics/network-check-target-c54gz" Jul 17 18:55:18.902937 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:18.902850 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 18:55:18.902937 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:18.902878 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 18:55:18.902937 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:18.902890 2561 projected.go:196] Error preparing data for projected volume kube-api-access-mlmhc for pod openshift-network-diagnostics/network-check-target-c54gz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 18:55:18.903092 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:18.902954 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c91e513-4f09-4551-8f50-75a4ba131d12-kube-api-access-mlmhc podName:2c91e513-4f09-4551-8f50-75a4ba131d12 nodeName:}" failed. No retries permitted until 2026-07-17 18:55:34.902936058 +0000 UTC m=+33.232752244 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mlmhc" (UniqueName: "kubernetes.io/projected/2c91e513-4f09-4551-8f50-75a4ba131d12-kube-api-access-mlmhc") pod "network-check-target-c54gz" (UID: "2c91e513-4f09-4551-8f50-75a4ba131d12") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 18:55:19.093370 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:19.093286 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c54gz" Jul 17 18:55:19.093370 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:19.093322 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l6k9" Jul 17 18:55:19.093626 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:19.093287 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m9wzh" Jul 17 18:55:19.093626 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:19.093423 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c54gz" podUID="2c91e513-4f09-4551-8f50-75a4ba131d12" Jul 17 18:55:19.093626 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:19.093510 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7l6k9" podUID="16db4114-eec4-4eb4-80be-1e6238e5c67b" Jul 17 18:55:19.093626 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:19.093603 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m9wzh" podUID="3591b9f8-a2ff-4adb-93ec-449225ca6390" Jul 17 18:55:21.093361 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:21.093322 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m9wzh" Jul 17 18:55:21.093869 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:21.093380 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c54gz" Jul 17 18:55:21.093869 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:21.093437 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l6k9" Jul 17 18:55:21.093869 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:21.093454 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m9wzh" podUID="3591b9f8-a2ff-4adb-93ec-449225ca6390" Jul 17 18:55:21.093869 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:21.093568 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7l6k9" podUID="16db4114-eec4-4eb4-80be-1e6238e5c67b" Jul 17 18:55:21.093869 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:21.093634 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c54gz" podUID="2c91e513-4f09-4551-8f50-75a4ba131d12" Jul 17 18:55:23.093690 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:23.093645 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m9wzh" Jul 17 18:55:23.094115 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:23.093645 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c54gz" Jul 17 18:55:23.094115 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:23.093760 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m9wzh" podUID="3591b9f8-a2ff-4adb-93ec-449225ca6390" Jul 17 18:55:23.094115 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:23.093823 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c54gz" podUID="2c91e513-4f09-4551-8f50-75a4ba131d12" Jul 17 18:55:23.094115 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:23.093663 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l6k9" Jul 17 18:55:23.094115 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:23.093913 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7l6k9" podUID="16db4114-eec4-4eb4-80be-1e6238e5c67b" Jul 17 18:55:24.326415 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:24.326377 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-137.ec2.internal" event={"ID":"4055332a3b86c15989d82b71c24d29a9","Type":"ContainerStarted","Data":"68ec3425bc426609df9d095be62a57cabb33a08105dc350ecae1fae2513e05ce"} Jul 17 18:55:24.327972 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:24.327942 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nrss7" event={"ID":"1edd2f99-7d44-4eed-8850-50ca38aa5153","Type":"ContainerStarted","Data":"70b65a7f94bf03f050aaf686840fab3eed5e42996fc0ff7e156cdbd8a28309ff"} Jul 17 18:55:24.329277 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:24.329254 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kj2t7" event={"ID":"c649f1a2-c525-4818-bde3-f5c394101099","Type":"ContainerStarted","Data":"44a8b7113f9d3b9b8efb21689c0c76650e50912e7ba7b894a7777111bd3412a5"} Jul 17 18:55:24.342951 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:24.342594 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-137.ec2.internal" podStartSLOduration=22.342575522 podStartE2EDuration="22.342575522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:55:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:55:24.341818529 +0000 UTC m=+22.671634738" watchObservedRunningTime="2026-07-17 18:55:24.342575522 +0000 UTC m=+22.672391731" Jul 17 18:55:24.362890 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:24.362699 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-nrss7" podStartSLOduration=1.793530469 podStartE2EDuration="22.36267763s" totalImagesPullingTime="20.569147161s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:55:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:55:24.362416338 +0000 UTC m=+22.692232547" watchObservedRunningTime="2026-07-17 18:55:24.36267763 +0000 UTC m=+22.692493839" Jul 17 18:55:24.379889 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:24.379829 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-kj2t7" podStartSLOduration=1.864622725 podStartE2EDuration="22.379812571s" totalImagesPullingTime="20.515189846s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:55:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:55:24.37955593 +0000 UTC m=+22.709372132" watchObservedRunningTime="2026-07-17 18:55:24.379812571 +0000 UTC m=+22.709628780" Jul 17 18:55:25.093954 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:25.093450 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m9wzh" Jul 17 18:55:25.094073 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:25.093479 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c54gz" Jul 17 18:55:25.094073 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:25.094054 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m9wzh" podUID="3591b9f8-a2ff-4adb-93ec-449225ca6390" Jul 17 18:55:25.094179 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:25.093499 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l6k9" Jul 17 18:55:25.094179 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:25.094120 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c54gz" podUID="2c91e513-4f09-4551-8f50-75a4ba131d12" Jul 17 18:55:25.094246 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:25.094181 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7l6k9" podUID="16db4114-eec4-4eb4-80be-1e6238e5c67b" Jul 17 18:55:25.333326 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:25.333296 2561 generic.go:356] "Generic (PLEG): container finished" podID="08c31757-7c13-4c31-a1ad-570aad3cde70" containerID="04d47753951b5e7ad4f46ff004c1d9400bc72074d069872c364b412fe3d2875c" exitCode=0 Jul 17 18:55:25.334200 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:25.333368 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nb8rb" event={"ID":"08c31757-7c13-4c31-a1ad-570aad3cde70","Type":"ContainerDied","Data":"04d47753951b5e7ad4f46ff004c1d9400bc72074d069872c364b412fe3d2875c"} Jul 17 18:55:25.336378 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:25.336362 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpjzq_4b162535-066a-471a-aee4-99c07c6e1bb4/ovn-acl-logging/0.log" Jul 17 18:55:25.336749 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:25.336727 2561 generic.go:356] "Generic (PLEG): container finished" podID="4b162535-066a-471a-aee4-99c07c6e1bb4" containerID="5ad92766d6f5371b8a13a8585dd5f79a064f364196beba9dbf62764a4fa7db89" exitCode=1 Jul 17 18:55:25.336839 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:25.336778 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" event={"ID":"4b162535-066a-471a-aee4-99c07c6e1bb4","Type":"ContainerStarted","Data":"35d3cfa2f4defae8b150a412bdf9d951a7a66e7e27257287d908573f05d4c532"} Jul 17 18:55:25.336900 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:25.336869 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" event={"ID":"4b162535-066a-471a-aee4-99c07c6e1bb4","Type":"ContainerStarted","Data":"acb675f9459820eef70c2726a4aa0f6d3b0b27821c1cdb767e5939fe2f707743"} Jul 17 18:55:25.336949 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:25.336919 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" event={"ID":"4b162535-066a-471a-aee4-99c07c6e1bb4","Type":"ContainerStarted","Data":"d2693b1b9b876554a4834bb1157ddc2c4613d4bc9e12ba416ed000a953ab75b9"} Jul 17 18:55:25.336983 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:25.336968 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" event={"ID":"4b162535-066a-471a-aee4-99c07c6e1bb4","Type":"ContainerStarted","Data":"124ee802623a4161df451c257203abc6a0d6c725fd62601e6a34d54f900c9707"} Jul 17 18:55:25.337033 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:25.337005 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" event={"ID":"4b162535-066a-471a-aee4-99c07c6e1bb4","Type":"ContainerDied","Data":"5ad92766d6f5371b8a13a8585dd5f79a064f364196beba9dbf62764a4fa7db89"} Jul 17 18:55:25.337083 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:25.337043 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" event={"ID":"4b162535-066a-471a-aee4-99c07c6e1bb4","Type":"ContainerStarted","Data":"a3524fc7c30d867ecc764cbe25f9936e3b0c8f79335a06237e887b713b7dbd5e"} Jul 17 18:55:25.338155 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:25.338135 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pvpbg" event={"ID":"728c6dad-4aae-44c2-ac11-60be3b680cca","Type":"ContainerStarted","Data":"858fa0a4d34629bade609d7e58eb7d27b368e2aae1d66f2d75b664b9148abab8"} Jul 17 18:55:25.339464 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:25.339421 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-n2wtb" event={"ID":"7f5ebd27-cd62-4b2b-9c30-4863ca02f422","Type":"ContainerStarted","Data":"b8c2555b79290c3e5b5b9f1355a762474db3a2fe64989464309f88c9e2ebc114"} Jul 17 18:55:25.341239 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:25.341219 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2fgtz" event={"ID":"0135e7e9-6294-4524-ad81-17f251822127","Type":"ContainerStarted","Data":"a019b3686a11dd12a6ca9a28e45a290941733f9672ad18b06e6d60b30278289b"} Jul 17 18:55:25.342595 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:25.342576 2561 generic.go:356] "Generic (PLEG): container finished" podID="f4f6391d232b2f3f6af57e5daac336ae" containerID="6fb5159104692b0b34245053739241bf9cbf8efc33151034e3f1a4636565dae0" exitCode=0 Jul 17 18:55:25.342679 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:25.342641 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-137.ec2.internal" event={"ID":"f4f6391d232b2f3f6af57e5daac336ae","Type":"ContainerDied","Data":"6fb5159104692b0b34245053739241bf9cbf8efc33151034e3f1a4636565dae0"} Jul 17 18:55:25.344089 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:25.344068 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92cjd" event={"ID":"64f94c18-0eab-41bb-88e0-824db8dfa7e4","Type":"ContainerStarted","Data":"09d89ee743ab139ec3bb86649fb6125bd9e5746531ead0f47aefffe2762354cf"} Jul 17 18:55:25.377713 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:25.377670 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-2fgtz" podStartSLOduration=3.781040104 podStartE2EDuration="24.377655293s" totalImagesPullingTime="20.596615189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:55:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:55:25.377034844 +0000 UTC m=+23.706851051" watchObservedRunningTime="2026-07-17 18:55:25.377655293 +0000 UTC m=+23.707471502" Jul 17 18:55:25.395100 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:25.395041 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-n2wtb" podStartSLOduration=3.8203100660000002 podStartE2EDuration="24.395021176s" totalImagesPullingTime="20.57471111s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:55:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:55:25.394485105 +0000 UTC m=+23.724301315" watchObservedRunningTime="2026-07-17 18:55:25.395021176 +0000 UTC m=+23.724837386" Jul 17 18:55:26.348570 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:26.348488 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-137.ec2.internal" event={"ID":"f4f6391d232b2f3f6af57e5daac336ae","Type":"ContainerStarted","Data":"fa6768cb5e67ef1b3f54ac479657c5694000ca4284204e7555a26ac875825cc1"} Jul 17 18:55:26.350911 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:26.350871 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7g4zx" event={"ID":"c6b17b15-c9dc-4e3e-9fb3-8477f0a734ed","Type":"ContainerStarted","Data":"c7897226cd31b332525e2973bd701aff4fa447da836337309c05aa58ebf17cca"} Jul 17 18:55:26.365628 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:26.365558 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-pvpbg" podStartSLOduration=3.777639111 podStartE2EDuration="24.365517957s" totalImagesPullingTime="20.587878846s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:55:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:55:25.424492273 +0000 UTC m=+23.754308481" watchObservedRunningTime="2026-07-17 18:55:26.365517957 +0000 UTC m=+24.695334168" Jul 17 18:55:26.365780 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:26.365681 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-137.ec2.internal" podStartSLOduration=24.365673701 podStartE2EDuration="24.365673701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:55:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:55:26.365669411 +0000 UTC m=+24.695485622" watchObservedRunningTime="2026-07-17 18:55:26.365673701 +0000 UTC m=+24.695489910" Jul 17 18:55:26.368681 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:26.368656 2561 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 17 18:55:27.036435 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:27.036337 2561 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-17T18:55:26.368672713Z","UUID":"95eeb821-ea37-46b3-bd83-9e4a77c5aeb4","Handler":null,"Name":"","Endpoint":""} Jul 17 18:55:27.037927 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:27.037901 2561 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 17 18:55:27.038041 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:27.037947 2561 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 17 18:55:27.093202 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:27.093169 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m9wzh" Jul 17 18:55:27.093372 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:27.093215 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c54gz" Jul 17 18:55:27.093372 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:27.093292 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m9wzh" podUID="3591b9f8-a2ff-4adb-93ec-449225ca6390" Jul 17 18:55:27.093372 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:27.093335 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c54gz" podUID="2c91e513-4f09-4551-8f50-75a4ba131d12" Jul 17 18:55:27.093558 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:27.093387 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l6k9" Jul 17 18:55:27.093558 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:27.093448 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7l6k9" podUID="16db4114-eec4-4eb4-80be-1e6238e5c67b" Jul 17 18:55:27.355572 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:27.355470 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92cjd" event={"ID":"64f94c18-0eab-41bb-88e0-824db8dfa7e4","Type":"ContainerStarted","Data":"2c978f09e058d07e9d162a72e52565d0abab13e259a8d16d0afc1ab8bfb4c83f"} Jul 17 18:55:27.358986 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:27.358966 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpjzq_4b162535-066a-471a-aee4-99c07c6e1bb4/ovn-acl-logging/0.log" Jul 17 18:55:27.359349 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:27.359323 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" event={"ID":"4b162535-066a-471a-aee4-99c07c6e1bb4","Type":"ContainerStarted","Data":"9b5e35a98bc7d2b6121e674d26c925c583e69cd22f8450b956cdb606f5200ffa"} Jul 17 18:55:28.161008 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:28.160931 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-2fgtz" Jul 17 18:55:28.161603 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:28.161582 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-2fgtz" Jul 17 18:55:28.178785 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:28.178730 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7g4zx" podStartSLOduration=15.056917897 podStartE2EDuration="27.178713746s" totalImagesPullingTime="12.121795849s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:55:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:55:26.383505992 +0000 UTC m=+24.713322201" watchObservedRunningTime="2026-07-17 18:55:28.178713746 +0000 UTC m=+26.508529955" Jul 17 18:55:28.363748 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:28.363714 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92cjd" event={"ID":"64f94c18-0eab-41bb-88e0-824db8dfa7e4","Type":"ContainerStarted","Data":"037fbc924e016f256c588def5fc9f8dd616e1b1ff19114a374acd9640b6c84c9"} Jul 17 18:55:28.381927 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:28.381872 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92cjd" podStartSLOduration=2.983819477 podStartE2EDuration="26.381853785s" totalImagesPullingTime="23.398034308s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:55:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:55:28.38181491 +0000 UTC m=+26.711631131" watchObservedRunningTime="2026-07-17 18:55:28.381853785 +0000 UTC m=+26.711669992" Jul 17 18:55:29.093816 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:29.093779 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m9wzh" Jul 17 18:55:29.093993 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:29.093788 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c54gz" Jul 17 18:55:29.093993 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:29.093917 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m9wzh" podUID="3591b9f8-a2ff-4adb-93ec-449225ca6390" Jul 17 18:55:29.093993 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:29.093788 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l6k9" Jul 17 18:55:29.094159 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:29.093977 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c54gz" podUID="2c91e513-4f09-4551-8f50-75a4ba131d12" Jul 17 18:55:29.094159 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:29.094097 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7l6k9" podUID="16db4114-eec4-4eb4-80be-1e6238e5c67b" Jul 17 18:55:31.093357 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:31.093145 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m9wzh" Jul 17 18:55:31.094032 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:31.093209 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l6k9" Jul 17 18:55:31.094032 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:31.093431 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m9wzh" podUID="3591b9f8-a2ff-4adb-93ec-449225ca6390" Jul 17 18:55:31.094032 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:31.093227 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c54gz" Jul 17 18:55:31.094032 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:31.093520 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7l6k9" podUID="16db4114-eec4-4eb4-80be-1e6238e5c67b" Jul 17 18:55:31.094032 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:31.093770 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c54gz" podUID="2c91e513-4f09-4551-8f50-75a4ba131d12" Jul 17 18:55:31.372658 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:31.372569 2561 generic.go:356] "Generic (PLEG): container finished" podID="08c31757-7c13-4c31-a1ad-570aad3cde70" containerID="aa1d76c908e3c56a1290fc294db433d4b17ed695c8ca6f1c2d309341e9ccd52a" exitCode=0 Jul 17 18:55:31.372847 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:31.372666 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nb8rb" event={"ID":"08c31757-7c13-4c31-a1ad-570aad3cde70","Type":"ContainerDied","Data":"aa1d76c908e3c56a1290fc294db433d4b17ed695c8ca6f1c2d309341e9ccd52a"} Jul 17 18:55:31.376268 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:31.376245 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpjzq_4b162535-066a-471a-aee4-99c07c6e1bb4/ovn-acl-logging/0.log" Jul 17 18:55:31.376652 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:31.376632 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" event={"ID":"4b162535-066a-471a-aee4-99c07c6e1bb4","Type":"ContainerStarted","Data":"64a7df7a56f6a7cd64e2a205581da379d50c0ff787bfc3fcc6797979aa43a746"} Jul 17 18:55:31.376932 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:31.376907 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:31.377111 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:31.377095 2561 scope.go:117] "RemoveContainer" containerID="5ad92766d6f5371b8a13a8585dd5f79a064f364196beba9dbf62764a4fa7db89" Jul 17 18:55:31.394352 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:31.394326 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:32.383044 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:32.383019 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpjzq_4b162535-066a-471a-aee4-99c07c6e1bb4/ovn-acl-logging/0.log" Jul 17 18:55:32.383569 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:32.383392 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" event={"ID":"4b162535-066a-471a-aee4-99c07c6e1bb4","Type":"ContainerStarted","Data":"d768415f2be7986f615875e00856d1976390bdb7f465b23afc6a16e5e0c8cbd0"} Jul 17 18:55:32.383706 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:32.383688 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:32.383779 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:32.383767 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:32.399773 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:32.399745 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:55:32.414772 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:32.414722 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" podStartSLOduration=9.41474797 podStartE2EDuration="30.414709107s" totalImagesPullingTime="20.999961137s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:55:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:55:32.414274403 +0000 UTC m=+30.744090611" watchObservedRunningTime="2026-07-17 18:55:32.414709107 +0000 UTC m=+30.744525316" Jul 17 18:55:32.802892 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:32.802859 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-c54gz"] Jul 17 18:55:32.803089 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:32.802994 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c54gz" Jul 17 18:55:32.803169 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:32.803103 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c54gz" podUID="2c91e513-4f09-4551-8f50-75a4ba131d12" Jul 17 18:55:32.803348 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:32.803327 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-m9wzh"] Jul 17 18:55:32.803418 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:32.803413 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m9wzh" Jul 17 18:55:32.803499 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:32.803476 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m9wzh" podUID="3591b9f8-a2ff-4adb-93ec-449225ca6390" Jul 17 18:55:32.807250 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:32.807221 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7l6k9"] Jul 17 18:55:32.807347 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:32.807338 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l6k9" Jul 17 18:55:32.807472 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:32.807454 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7l6k9" podUID="16db4114-eec4-4eb4-80be-1e6238e5c67b" Jul 17 18:55:32.990832 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:32.990799 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-2fgtz" Jul 17 18:55:32.991022 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:32.990927 2561 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 17 18:55:32.991384 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:32.991349 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2fgtz" Jul 17 18:55:34.093463 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:34.093373 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c54gz" Jul 17 18:55:34.093851 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:34.093490 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c54gz" podUID="2c91e513-4f09-4551-8f50-75a4ba131d12" Jul 17 18:55:34.822771 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:34.822690 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3591b9f8-a2ff-4adb-93ec-449225ca6390-original-pull-secret\") pod \"global-pull-secret-syncer-m9wzh\" (UID: \"3591b9f8-a2ff-4adb-93ec-449225ca6390\") " pod="kube-system/global-pull-secret-syncer-m9wzh" Jul 17 18:55:34.822771 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:34.822741 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16db4114-eec4-4eb4-80be-1e6238e5c67b-metrics-certs\") pod \"network-metrics-daemon-7l6k9\" (UID: \"16db4114-eec4-4eb4-80be-1e6238e5c67b\") " pod="openshift-multus/network-metrics-daemon-7l6k9" Jul 17 18:55:34.822943 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:34.822848 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 18:55:34.822943 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:34.822866 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 18:55:34.822943 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:34.822918 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16db4114-eec4-4eb4-80be-1e6238e5c67b-metrics-certs podName:16db4114-eec4-4eb4-80be-1e6238e5c67b nodeName:}" failed. No retries permitted until 2026-07-17 18:56:06.822901919 +0000 UTC m=+65.152718108 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/16db4114-eec4-4eb4-80be-1e6238e5c67b-metrics-certs") pod "network-metrics-daemon-7l6k9" (UID: "16db4114-eec4-4eb4-80be-1e6238e5c67b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 18:55:34.822943 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:34.822935 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3591b9f8-a2ff-4adb-93ec-449225ca6390-original-pull-secret podName:3591b9f8-a2ff-4adb-93ec-449225ca6390 nodeName:}" failed. No retries permitted until 2026-07-17 18:56:06.822926929 +0000 UTC m=+65.152743129 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3591b9f8-a2ff-4adb-93ec-449225ca6390-original-pull-secret") pod "global-pull-secret-syncer-m9wzh" (UID: "3591b9f8-a2ff-4adb-93ec-449225ca6390") : object "kube-system"/"original-pull-secret" not registered Jul 17 18:55:34.923797 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:34.923758 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlmhc\" (UniqueName: \"kubernetes.io/projected/2c91e513-4f09-4551-8f50-75a4ba131d12-kube-api-access-mlmhc\") pod \"network-check-target-c54gz\" (UID: \"2c91e513-4f09-4551-8f50-75a4ba131d12\") " pod="openshift-network-diagnostics/network-check-target-c54gz" Jul 17 18:55:34.923956 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:34.923938 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 18:55:34.923993 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:34.923959 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 18:55:34.923993 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:34.923970 2561 projected.go:196] Error preparing data for projected volume kube-api-access-mlmhc for pod openshift-network-diagnostics/network-check-target-c54gz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 18:55:34.924056 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:34.924026 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c91e513-4f09-4551-8f50-75a4ba131d12-kube-api-access-mlmhc podName:2c91e513-4f09-4551-8f50-75a4ba131d12 nodeName:}" failed. No retries permitted until 2026-07-17 18:56:06.924013164 +0000 UTC m=+65.253829350 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mlmhc" (UniqueName: "kubernetes.io/projected/2c91e513-4f09-4551-8f50-75a4ba131d12-kube-api-access-mlmhc") pod "network-check-target-c54gz" (UID: "2c91e513-4f09-4551-8f50-75a4ba131d12") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 18:55:35.093840 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:35.093759 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m9wzh" Jul 17 18:55:35.094213 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:35.093859 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m9wzh" podUID="3591b9f8-a2ff-4adb-93ec-449225ca6390" Jul 17 18:55:35.094213 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:35.093914 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l6k9" Jul 17 18:55:35.094213 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:35.094007 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7l6k9" podUID="16db4114-eec4-4eb4-80be-1e6238e5c67b" Jul 17 18:55:36.093372 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:36.093333 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c54gz" Jul 17 18:55:36.093573 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:36.093438 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c54gz" podUID="2c91e513-4f09-4551-8f50-75a4ba131d12" Jul 17 18:55:37.093327 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:37.093288 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m9wzh" Jul 17 18:55:37.093797 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:37.093297 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l6k9" Jul 17 18:55:37.093797 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:37.093387 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m9wzh" podUID="3591b9f8-a2ff-4adb-93ec-449225ca6390" Jul 17 18:55:37.093797 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:37.093513 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7l6k9" podUID="16db4114-eec4-4eb4-80be-1e6238e5c67b" Jul 17 18:55:37.981898 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:37.981821 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-137.ec2.internal" event="NodeReady" Jul 17 18:55:37.982058 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:37.981935 2561 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 17 18:55:38.019084 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.019051 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6cb89cc658-6mn7r"] Jul 17 18:55:38.022134 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.022112 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cb89cc658-6mn7r" Jul 17 18:55:38.024834 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.024811 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 17 18:55:38.024962 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.024891 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 17 18:55:38.024962 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.024901 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zbkpp\"" Jul 17 18:55:38.026117 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.025235 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 17 18:55:38.031456 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.031437 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 17 18:55:38.036863 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.036842 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-crvtk"] Jul 17 18:55:38.040481 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.040463 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ps4bw"] Jul 17 18:55:38.040614 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.040596 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-crvtk" Jul 17 18:55:38.043078 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.043063 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 17 18:55:38.043172 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.043085 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kfsnb\"" Jul 17 18:55:38.043172 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.043085 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 17 18:55:38.043876 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.043862 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6cb89cc658-6mn7r"] Jul 17 18:55:38.043952 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.043941 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ps4bw" Jul 17 18:55:38.046188 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.046170 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-lp744\"" Jul 17 18:55:38.046512 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.046491 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 17 18:55:38.046615 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.046516 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 17 18:55:38.046926 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.046912 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 17 18:55:38.051502 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.051483 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ps4bw"] Jul 17 18:55:38.052392 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.052373 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-crvtk"] Jul 17 18:55:38.093932 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.093898 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c54gz" Jul 17 18:55:38.096742 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.096718 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bqhtg\"" Jul 17 18:55:38.096882 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.096718 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 17 18:55:38.096981 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.096965 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 17 18:55:38.146269 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.146226 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-bound-sa-token\") pod \"image-registry-6cb89cc658-6mn7r\" (UID: \"e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3\") " pod="openshift-image-registry/image-registry-6cb89cc658-6mn7r" Jul 17 18:55:38.146470 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.146296 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-trusted-ca\") pod \"image-registry-6cb89cc658-6mn7r\" (UID: \"e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3\") " pod="openshift-image-registry/image-registry-6cb89cc658-6mn7r" Jul 17 18:55:38.146470 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.146326 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71e3530f-7f67-4f34-a65a-444d1f270c93-metrics-tls\") pod \"dns-default-crvtk\" (UID: \"71e3530f-7f67-4f34-a65a-444d1f270c93\") " pod="openshift-dns/dns-default-crvtk" Jul 17 18:55:38.146470 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.146368 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71e3530f-7f67-4f34-a65a-444d1f270c93-tmp-dir\") pod \"dns-default-crvtk\" (UID: \"71e3530f-7f67-4f34-a65a-444d1f270c93\") " pod="openshift-dns/dns-default-crvtk" Jul 17 18:55:38.146470 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.146422 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-registry-certificates\") pod \"image-registry-6cb89cc658-6mn7r\" (UID: \"e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3\") " pod="openshift-image-registry/image-registry-6cb89cc658-6mn7r" Jul 17 18:55:38.146470 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.146447 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-installation-pull-secrets\") pod \"image-registry-6cb89cc658-6mn7r\" (UID: \"e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3\") " pod="openshift-image-registry/image-registry-6cb89cc658-6mn7r" Jul 17 18:55:38.146717 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.146493 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-image-registry-private-configuration\") pod \"image-registry-6cb89cc658-6mn7r\" (UID: \"e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3\") " pod="openshift-image-registry/image-registry-6cb89cc658-6mn7r" Jul 17 18:55:38.146717 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.146549 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-ca-trust-extracted\") pod \"image-registry-6cb89cc658-6mn7r\" (UID: \"e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3\") " pod="openshift-image-registry/image-registry-6cb89cc658-6mn7r" Jul 17 18:55:38.146717 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.146574 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/71e3530f-7f67-4f34-a65a-444d1f270c93-config-volume\") pod \"dns-default-crvtk\" (UID: \"71e3530f-7f67-4f34-a65a-444d1f270c93\") " pod="openshift-dns/dns-default-crvtk" Jul 17 18:55:38.146717 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.146604 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-registry-tls\") pod \"image-registry-6cb89cc658-6mn7r\" (UID: \"e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3\") " pod="openshift-image-registry/image-registry-6cb89cc658-6mn7r" Jul 17 18:55:38.146717 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.146658 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jbdk\" (UniqueName: \"kubernetes.io/projected/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-kube-api-access-6jbdk\") pod \"image-registry-6cb89cc658-6mn7r\" (UID: \"e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3\") " pod="openshift-image-registry/image-registry-6cb89cc658-6mn7r" Jul 17 18:55:38.146717 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.146682 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vn7c\" (UniqueName: \"kubernetes.io/projected/c7ab0b84-2d5c-4dd8-817a-b6baa962fab6-kube-api-access-4vn7c\") pod \"ingress-canary-ps4bw\" (UID: \"c7ab0b84-2d5c-4dd8-817a-b6baa962fab6\") " pod="openshift-ingress-canary/ingress-canary-ps4bw" Jul 17 18:55:38.146913 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.146726 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c7ab0b84-2d5c-4dd8-817a-b6baa962fab6-cert\") pod \"ingress-canary-ps4bw\" (UID: \"c7ab0b84-2d5c-4dd8-817a-b6baa962fab6\") " pod="openshift-ingress-canary/ingress-canary-ps4bw" Jul 17 18:55:38.146913 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.146746 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz69c\" (UniqueName: \"kubernetes.io/projected/71e3530f-7f67-4f34-a65a-444d1f270c93-kube-api-access-nz69c\") pod \"dns-default-crvtk\" (UID: \"71e3530f-7f67-4f34-a65a-444d1f270c93\") " pod="openshift-dns/dns-default-crvtk" Jul 17 18:55:38.247266 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.247234 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-registry-certificates\") pod \"image-registry-6cb89cc658-6mn7r\" (UID: \"e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3\") " pod="openshift-image-registry/image-registry-6cb89cc658-6mn7r" Jul 17 18:55:38.247266 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.247268 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-installation-pull-secrets\") pod \"image-registry-6cb89cc658-6mn7r\" (UID: \"e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3\") " pod="openshift-image-registry/image-registry-6cb89cc658-6mn7r" Jul 17 18:55:38.247501 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.247301 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-image-registry-private-configuration\") pod \"image-registry-6cb89cc658-6mn7r\" (UID: \"e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3\") " pod="openshift-image-registry/image-registry-6cb89cc658-6mn7r" Jul 17 18:55:38.247501 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.247321 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-ca-trust-extracted\") pod \"image-registry-6cb89cc658-6mn7r\" (UID: \"e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3\") " pod="openshift-image-registry/image-registry-6cb89cc658-6mn7r" Jul 17 18:55:38.247501 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.247346 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/71e3530f-7f67-4f34-a65a-444d1f270c93-config-volume\") pod \"dns-default-crvtk\" (UID: \"71e3530f-7f67-4f34-a65a-444d1f270c93\") " pod="openshift-dns/dns-default-crvtk" Jul 17 18:55:38.247501 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.247375 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-registry-tls\") pod \"image-registry-6cb89cc658-6mn7r\" (UID: \"e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3\") " pod="openshift-image-registry/image-registry-6cb89cc658-6mn7r" Jul 17 18:55:38.247501 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.247416 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jbdk\" (UniqueName: \"kubernetes.io/projected/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-kube-api-access-6jbdk\") pod \"image-registry-6cb89cc658-6mn7r\" (UID: \"e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3\") " pod="openshift-image-registry/image-registry-6cb89cc658-6mn7r" Jul 17 18:55:38.247501 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.247441 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vn7c\" (UniqueName: \"kubernetes.io/projected/c7ab0b84-2d5c-4dd8-817a-b6baa962fab6-kube-api-access-4vn7c\") pod \"ingress-canary-ps4bw\" (UID: \"c7ab0b84-2d5c-4dd8-817a-b6baa962fab6\") " pod="openshift-ingress-canary/ingress-canary-ps4bw" Jul 17 18:55:38.247501 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.247471 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c7ab0b84-2d5c-4dd8-817a-b6baa962fab6-cert\") pod \"ingress-canary-ps4bw\" (UID: \"c7ab0b84-2d5c-4dd8-817a-b6baa962fab6\") " pod="openshift-ingress-canary/ingress-canary-ps4bw" Jul 17 18:55:38.247501 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.247492 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz69c\" (UniqueName: \"kubernetes.io/projected/71e3530f-7f67-4f34-a65a-444d1f270c93-kube-api-access-nz69c\") pod \"dns-default-crvtk\" (UID: \"71e3530f-7f67-4f34-a65a-444d1f270c93\") " pod="openshift-dns/dns-default-crvtk" Jul 17 18:55:38.247923 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:38.247511 2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 17 18:55:38.247923 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.247553 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-bound-sa-token\") pod \"image-registry-6cb89cc658-6mn7r\" (UID: \"e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3\") " pod="openshift-image-registry/image-registry-6cb89cc658-6mn7r" Jul 17 18:55:38.247923 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:38.247565 2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cb89cc658-6mn7r: secret "image-registry-tls" not found Jul 17 18:55:38.247923 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.247595 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-trusted-ca\") pod \"image-registry-6cb89cc658-6mn7r\" (UID: \"e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3\") " pod="openshift-image-registry/image-registry-6cb89cc658-6mn7r" Jul 17 18:55:38.247923 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:38.247644 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-registry-tls podName:e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3 nodeName:}" failed. No retries permitted until 2026-07-17 18:55:38.747607926 +0000 UTC m=+37.077424135 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-registry-tls") pod "image-registry-6cb89cc658-6mn7r" (UID: "e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3") : secret "image-registry-tls" not found Jul 17 18:55:38.247923 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.247667 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71e3530f-7f67-4f34-a65a-444d1f270c93-metrics-tls\") pod \"dns-default-crvtk\" (UID: \"71e3530f-7f67-4f34-a65a-444d1f270c93\") " pod="openshift-dns/dns-default-crvtk" Jul 17 18:55:38.247923 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.247702 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71e3530f-7f67-4f34-a65a-444d1f270c93-tmp-dir\") pod \"dns-default-crvtk\" (UID: \"71e3530f-7f67-4f34-a65a-444d1f270c93\") " pod="openshift-dns/dns-default-crvtk" Jul 17 18:55:38.247923 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.247877 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-ca-trust-extracted\") pod \"image-registry-6cb89cc658-6mn7r\" (UID: \"e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3\") " pod="openshift-image-registry/image-registry-6cb89cc658-6mn7r" Jul 17 18:55:38.248363 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.247967 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71e3530f-7f67-4f34-a65a-444d1f270c93-tmp-dir\") pod \"dns-default-crvtk\" (UID: \"71e3530f-7f67-4f34-a65a-444d1f270c93\") " pod="openshift-dns/dns-default-crvtk" Jul 17 18:55:38.248363 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.248005 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/71e3530f-7f67-4f34-a65a-444d1f270c93-config-volume\") pod \"dns-default-crvtk\" (UID: \"71e3530f-7f67-4f34-a65a-444d1f270c93\") " pod="openshift-dns/dns-default-crvtk" Jul 17 18:55:38.248363 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:38.248039 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 18:55:38.248363 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.248050 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-registry-certificates\") pod \"image-registry-6cb89cc658-6mn7r\" (UID: \"e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3\") " pod="openshift-image-registry/image-registry-6cb89cc658-6mn7r" Jul 17 18:55:38.248363 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:38.248084 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71e3530f-7f67-4f34-a65a-444d1f270c93-metrics-tls podName:71e3530f-7f67-4f34-a65a-444d1f270c93 nodeName:}" failed. No retries permitted until 2026-07-17 18:55:38.74806969 +0000 UTC m=+37.077885879 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/71e3530f-7f67-4f34-a65a-444d1f270c93-metrics-tls") pod "dns-default-crvtk" (UID: "71e3530f-7f67-4f34-a65a-444d1f270c93") : secret "dns-default-metrics-tls" not found Jul 17 18:55:38.248549 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.248451 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-trusted-ca\") pod \"image-registry-6cb89cc658-6mn7r\" (UID: \"e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3\") " pod="openshift-image-registry/image-registry-6cb89cc658-6mn7r" Jul 17 18:55:38.251976 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.251873 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-installation-pull-secrets\") pod \"image-registry-6cb89cc658-6mn7r\" (UID: \"e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3\") " pod="openshift-image-registry/image-registry-6cb89cc658-6mn7r" Jul 17 18:55:38.252079 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.251925 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-image-registry-private-configuration\") pod \"image-registry-6cb89cc658-6mn7r\" (UID: \"e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3\") " pod="openshift-image-registry/image-registry-6cb89cc658-6mn7r" Jul 17 18:55:38.252079 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.251956 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c7ab0b84-2d5c-4dd8-817a-b6baa962fab6-cert\") pod \"ingress-canary-ps4bw\" (UID: \"c7ab0b84-2d5c-4dd8-817a-b6baa962fab6\") " pod="openshift-ingress-canary/ingress-canary-ps4bw" Jul 17 18:55:38.257802 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.257776 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz69c\" (UniqueName: \"kubernetes.io/projected/71e3530f-7f67-4f34-a65a-444d1f270c93-kube-api-access-nz69c\") pod \"dns-default-crvtk\" (UID: \"71e3530f-7f67-4f34-a65a-444d1f270c93\") " pod="openshift-dns/dns-default-crvtk" Jul 17 18:55:38.258130 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.258111 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-bound-sa-token\") pod \"image-registry-6cb89cc658-6mn7r\" (UID: \"e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3\") " pod="openshift-image-registry/image-registry-6cb89cc658-6mn7r" Jul 17 18:55:38.258518 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.258488 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jbdk\" (UniqueName: \"kubernetes.io/projected/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-kube-api-access-6jbdk\") pod \"image-registry-6cb89cc658-6mn7r\" (UID: \"e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3\") " pod="openshift-image-registry/image-registry-6cb89cc658-6mn7r" Jul 17 18:55:38.258721 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.258705 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vn7c\" (UniqueName: \"kubernetes.io/projected/c7ab0b84-2d5c-4dd8-817a-b6baa962fab6-kube-api-access-4vn7c\") pod \"ingress-canary-ps4bw\" (UID: \"c7ab0b84-2d5c-4dd8-817a-b6baa962fab6\") " pod="openshift-ingress-canary/ingress-canary-ps4bw" Jul 17 18:55:38.357549 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.357472 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ps4bw" Jul 17 18:55:38.500320 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.500237 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ps4bw"] Jul 17 18:55:38.503418 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:38.503392 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7ab0b84_2d5c_4dd8_817a_b6baa962fab6.slice/crio-587b508ac2a53675959eed99ba30f1dd0db20b46dd90f67a262bc90510668750 WatchSource:0}: Error finding container 587b508ac2a53675959eed99ba30f1dd0db20b46dd90f67a262bc90510668750: Status 404 returned error can't find the container with id 587b508ac2a53675959eed99ba30f1dd0db20b46dd90f67a262bc90510668750 Jul 17 18:55:38.751606 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.751500 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71e3530f-7f67-4f34-a65a-444d1f270c93-metrics-tls\") pod \"dns-default-crvtk\" (UID: \"71e3530f-7f67-4f34-a65a-444d1f270c93\") " pod="openshift-dns/dns-default-crvtk" Jul 17 18:55:38.751606 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:38.751589 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-registry-tls\") pod \"image-registry-6cb89cc658-6mn7r\" (UID: \"e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3\") " pod="openshift-image-registry/image-registry-6cb89cc658-6mn7r" Jul 17 18:55:38.751815 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:38.751640 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 18:55:38.751815 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:38.751709 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71e3530f-7f67-4f34-a65a-444d1f270c93-metrics-tls podName:71e3530f-7f67-4f34-a65a-444d1f270c93 nodeName:}" failed. No retries permitted until 2026-07-17 18:55:39.751691285 +0000 UTC m=+38.081507472 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/71e3530f-7f67-4f34-a65a-444d1f270c93-metrics-tls") pod "dns-default-crvtk" (UID: "71e3530f-7f67-4f34-a65a-444d1f270c93") : secret "dns-default-metrics-tls" not found Jul 17 18:55:38.751815 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:38.751729 2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 17 18:55:38.751815 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:38.751747 2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cb89cc658-6mn7r: secret "image-registry-tls" not found Jul 17 18:55:38.751815 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:38.751809 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-registry-tls podName:e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3 nodeName:}" failed. No retries permitted until 2026-07-17 18:55:39.751773771 +0000 UTC m=+38.081589988 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-registry-tls") pod "image-registry-6cb89cc658-6mn7r" (UID: "e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3") : secret "image-registry-tls" not found Jul 17 18:55:39.093878 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:39.093793 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l6k9" Jul 17 18:55:39.093878 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:39.093787 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m9wzh" Jul 17 18:55:39.097599 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:39.097569 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 17 18:55:39.098404 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:39.098377 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n57jp\"" Jul 17 18:55:39.098404 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:39.098397 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 17 18:55:39.400701 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:39.400622 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ps4bw" event={"ID":"c7ab0b84-2d5c-4dd8-817a-b6baa962fab6","Type":"ContainerStarted","Data":"587b508ac2a53675959eed99ba30f1dd0db20b46dd90f67a262bc90510668750"} Jul 17 18:55:39.759908 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:39.759875 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71e3530f-7f67-4f34-a65a-444d1f270c93-metrics-tls\") pod \"dns-default-crvtk\" (UID: \"71e3530f-7f67-4f34-a65a-444d1f270c93\") " pod="openshift-dns/dns-default-crvtk" Jul 17 18:55:39.760071 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:39.759931 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-registry-tls\") pod \"image-registry-6cb89cc658-6mn7r\" (UID: \"e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3\") " pod="openshift-image-registry/image-registry-6cb89cc658-6mn7r" Jul 17 18:55:39.760071 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:39.760039 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 18:55:39.760071 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:39.760051 2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 17 18:55:39.760071 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:39.760066 2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cb89cc658-6mn7r: secret "image-registry-tls" not found Jul 17 18:55:39.760247 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:39.760114 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71e3530f-7f67-4f34-a65a-444d1f270c93-metrics-tls podName:71e3530f-7f67-4f34-a65a-444d1f270c93 nodeName:}" failed. No retries permitted until 2026-07-17 18:55:41.760092887 +0000 UTC m=+40.089909074 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/71e3530f-7f67-4f34-a65a-444d1f270c93-metrics-tls") pod "dns-default-crvtk" (UID: "71e3530f-7f67-4f34-a65a-444d1f270c93") : secret "dns-default-metrics-tls" not found Jul 17 18:55:39.760247 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:39.760134 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-registry-tls podName:e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3 nodeName:}" failed. No retries permitted until 2026-07-17 18:55:41.760124957 +0000 UTC m=+40.089941144 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-registry-tls") pod "image-registry-6cb89cc658-6mn7r" (UID: "e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3") : secret "image-registry-tls" not found Jul 17 18:55:40.524696 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:40.524663 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6c6g4"] Jul 17 18:55:40.529863 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:40.529841 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6c6g4" Jul 17 18:55:40.533731 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:40.533711 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-d9hlx\"" Jul 17 18:55:40.534806 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:40.534787 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 17 18:55:40.536176 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:40.536143 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 17 18:55:40.536413 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:40.536389 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 17 18:55:40.536413 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:40.536396 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 17 18:55:40.542990 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:40.542966 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6c6g4"] Jul 17 18:55:40.667306 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:40.667269 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxl55\" (UniqueName: \"kubernetes.io/projected/96280fd2-5cf1-4adc-a0a2-a8f54aef67a9-kube-api-access-sxl55\") pod \"insights-runtime-extractor-6c6g4\" (UID: \"96280fd2-5cf1-4adc-a0a2-a8f54aef67a9\") " pod="openshift-insights/insights-runtime-extractor-6c6g4" Jul 17 18:55:40.667518 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:40.667351 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/96280fd2-5cf1-4adc-a0a2-a8f54aef67a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6c6g4\" (UID: \"96280fd2-5cf1-4adc-a0a2-a8f54aef67a9\") " pod="openshift-insights/insights-runtime-extractor-6c6g4" Jul 17 18:55:40.667518 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:40.667399 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/96280fd2-5cf1-4adc-a0a2-a8f54aef67a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6c6g4\" (UID: \"96280fd2-5cf1-4adc-a0a2-a8f54aef67a9\") " pod="openshift-insights/insights-runtime-extractor-6c6g4" Jul 17 18:55:40.667518 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:40.667492 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/96280fd2-5cf1-4adc-a0a2-a8f54aef67a9-crio-socket\") pod \"insights-runtime-extractor-6c6g4\" (UID: \"96280fd2-5cf1-4adc-a0a2-a8f54aef67a9\") " pod="openshift-insights/insights-runtime-extractor-6c6g4" Jul 17 18:55:40.667668 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:40.667549 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/96280fd2-5cf1-4adc-a0a2-a8f54aef67a9-data-volume\") pod \"insights-runtime-extractor-6c6g4\" (UID: \"96280fd2-5cf1-4adc-a0a2-a8f54aef67a9\") " pod="openshift-insights/insights-runtime-extractor-6c6g4" Jul 17 18:55:40.768886 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:40.768849 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxl55\" (UniqueName: \"kubernetes.io/projected/96280fd2-5cf1-4adc-a0a2-a8f54aef67a9-kube-api-access-sxl55\") pod \"insights-runtime-extractor-6c6g4\" (UID: \"96280fd2-5cf1-4adc-a0a2-a8f54aef67a9\") " pod="openshift-insights/insights-runtime-extractor-6c6g4" Jul 17 18:55:40.769048 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:40.768915 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/96280fd2-5cf1-4adc-a0a2-a8f54aef67a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6c6g4\" (UID: \"96280fd2-5cf1-4adc-a0a2-a8f54aef67a9\") " pod="openshift-insights/insights-runtime-extractor-6c6g4" Jul 17 18:55:40.769048 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:40.768942 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/96280fd2-5cf1-4adc-a0a2-a8f54aef67a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6c6g4\" (UID: \"96280fd2-5cf1-4adc-a0a2-a8f54aef67a9\") " pod="openshift-insights/insights-runtime-extractor-6c6g4" Jul 17 18:55:40.769048 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:40.768994 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/96280fd2-5cf1-4adc-a0a2-a8f54aef67a9-crio-socket\") pod \"insights-runtime-extractor-6c6g4\" (UID: \"96280fd2-5cf1-4adc-a0a2-a8f54aef67a9\") " pod="openshift-insights/insights-runtime-extractor-6c6g4" Jul 17 18:55:40.769048 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:40.769025 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/96280fd2-5cf1-4adc-a0a2-a8f54aef67a9-data-volume\") pod \"insights-runtime-extractor-6c6g4\" (UID: \"96280fd2-5cf1-4adc-a0a2-a8f54aef67a9\") " pod="openshift-insights/insights-runtime-extractor-6c6g4" Jul 17 18:55:40.769217 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:40.769048 2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 17 18:55:40.769217 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:40.769073 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 17 18:55:40.769217 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:40.769120 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/96280fd2-5cf1-4adc-a0a2-a8f54aef67a9-kube-rbac-proxy-cm podName:96280fd2-5cf1-4adc-a0a2-a8f54aef67a9 nodeName:}" failed. No retries permitted until 2026-07-17 18:55:41.269100574 +0000 UTC m=+39.598916761 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/96280fd2-5cf1-4adc-a0a2-a8f54aef67a9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6c6g4" (UID: "96280fd2-5cf1-4adc-a0a2-a8f54aef67a9") : configmap "kube-rbac-proxy" not found Jul 17 18:55:40.769217 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:40.769134 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96280fd2-5cf1-4adc-a0a2-a8f54aef67a9-insights-runtime-extractor-tls podName:96280fd2-5cf1-4adc-a0a2-a8f54aef67a9 nodeName:}" failed. No retries permitted until 2026-07-17 18:55:41.269128139 +0000 UTC m=+39.598944325 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/96280fd2-5cf1-4adc-a0a2-a8f54aef67a9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6c6g4" (UID: "96280fd2-5cf1-4adc-a0a2-a8f54aef67a9") : secret "insights-runtime-extractor-tls" not found Jul 17 18:55:40.769382 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:40.769221 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/96280fd2-5cf1-4adc-a0a2-a8f54aef67a9-crio-socket\") pod \"insights-runtime-extractor-6c6g4\" (UID: \"96280fd2-5cf1-4adc-a0a2-a8f54aef67a9\") " pod="openshift-insights/insights-runtime-extractor-6c6g4" Jul 17 18:55:40.769429 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:40.769378 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/96280fd2-5cf1-4adc-a0a2-a8f54aef67a9-data-volume\") pod \"insights-runtime-extractor-6c6g4\" (UID: \"96280fd2-5cf1-4adc-a0a2-a8f54aef67a9\") " pod="openshift-insights/insights-runtime-extractor-6c6g4" Jul 17 18:55:40.781338 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:40.781274 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxl55\" (UniqueName: \"kubernetes.io/projected/96280fd2-5cf1-4adc-a0a2-a8f54aef67a9-kube-api-access-sxl55\") pod \"insights-runtime-extractor-6c6g4\" (UID: \"96280fd2-5cf1-4adc-a0a2-a8f54aef67a9\") " pod="openshift-insights/insights-runtime-extractor-6c6g4" Jul 17 18:55:41.273024 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:41.272988 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/96280fd2-5cf1-4adc-a0a2-a8f54aef67a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6c6g4\" (UID: \"96280fd2-5cf1-4adc-a0a2-a8f54aef67a9\") " pod="openshift-insights/insights-runtime-extractor-6c6g4" Jul 17 18:55:41.273024 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:41.273026 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/96280fd2-5cf1-4adc-a0a2-a8f54aef67a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6c6g4\" (UID: \"96280fd2-5cf1-4adc-a0a2-a8f54aef67a9\") " pod="openshift-insights/insights-runtime-extractor-6c6g4" Jul 17 18:55:41.273207 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:41.273121 2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 17 18:55:41.273207 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:41.273167 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 17 18:55:41.273207 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:41.273188 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/96280fd2-5cf1-4adc-a0a2-a8f54aef67a9-kube-rbac-proxy-cm podName:96280fd2-5cf1-4adc-a0a2-a8f54aef67a9 nodeName:}" failed. No retries permitted until 2026-07-17 18:55:42.273168387 +0000 UTC m=+40.602984579 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/96280fd2-5cf1-4adc-a0a2-a8f54aef67a9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6c6g4" (UID: "96280fd2-5cf1-4adc-a0a2-a8f54aef67a9") : configmap "kube-rbac-proxy" not found Jul 17 18:55:41.273207 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:41.273204 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96280fd2-5cf1-4adc-a0a2-a8f54aef67a9-insights-runtime-extractor-tls podName:96280fd2-5cf1-4adc-a0a2-a8f54aef67a9 nodeName:}" failed. No retries permitted until 2026-07-17 18:55:42.273194721 +0000 UTC m=+40.603010909 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/96280fd2-5cf1-4adc-a0a2-a8f54aef67a9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6c6g4" (UID: "96280fd2-5cf1-4adc-a0a2-a8f54aef67a9") : secret "insights-runtime-extractor-tls" not found Jul 17 18:55:41.407198 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:41.407163 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ps4bw" event={"ID":"c7ab0b84-2d5c-4dd8-817a-b6baa962fab6","Type":"ContainerStarted","Data":"180210cfedf834fbe636ec0ccbd7aeea24954383984e3d85feb037f0ec138763"} Jul 17 18:55:41.428992 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:41.428941 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ps4bw" podStartSLOduration=0.650697451 podStartE2EDuration="3.428923498s" totalImagesPullingTime="2.778226047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:55:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:55:41.427707761 +0000 UTC m=+39.757523970" watchObservedRunningTime="2026-07-17 18:55:41.428923498 +0000 UTC m=+39.758739684" Jul 17 18:55:41.748389 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:41.748342 2561 log.go:32] "PullImage from image service failed" err="rpc error: code = Unknown desc = unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 504 Gateway Time-out; artifact err: provided artifact is a container image" image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:90f7e288d2e4b5406601a0ea9bdd9dc650bdf03a405ab5b85d67ef386e48e0fc" Jul 17 18:55:41.748813 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:41.748455 2561 kuberuntime_manager.go:1459] "Unhandled Error" err="init container bond-cni-plugin start failed in pod multus-additional-cni-plugins-nb8rb_openshift-multus(08c31757-7c13-4c31-a1ad-570aad3cde70): ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 504 Gateway Time-out; artifact err: provided artifact is a container image" logger="UnhandledError" Jul 17 18:55:41.748813 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:41.748488 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"bond-cni-plugin\" with ErrImagePull: \"unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 504 Gateway Time-out; artifact err: provided artifact is a container image\"" pod="openshift-multus/multus-additional-cni-plugins-nb8rb" podUID="08c31757-7c13-4c31-a1ad-570aad3cde70" Jul 17 18:55:41.777963 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:41.777932 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71e3530f-7f67-4f34-a65a-444d1f270c93-metrics-tls\") pod \"dns-default-crvtk\" (UID: \"71e3530f-7f67-4f34-a65a-444d1f270c93\") " pod="openshift-dns/dns-default-crvtk" Jul 17 18:55:41.778039 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:41.777982 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-registry-tls\") pod \"image-registry-6cb89cc658-6mn7r\" (UID: \"e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3\") " pod="openshift-image-registry/image-registry-6cb89cc658-6mn7r" Jul 17 18:55:41.778105 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:41.778088 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 18:55:41.778105 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:41.778100 2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 17 18:55:41.778192 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:41.778115 2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cb89cc658-6mn7r: secret "image-registry-tls" not found Jul 17 18:55:41.778192 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:41.778156 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-registry-tls podName:e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3 nodeName:}" failed. No retries permitted until 2026-07-17 18:55:45.778139426 +0000 UTC m=+44.107955625 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-registry-tls") pod "image-registry-6cb89cc658-6mn7r" (UID: "e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3") : secret "image-registry-tls" not found Jul 17 18:55:41.778192 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:41.778173 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71e3530f-7f67-4f34-a65a-444d1f270c93-metrics-tls podName:71e3530f-7f67-4f34-a65a-444d1f270c93 nodeName:}" failed. No retries permitted until 2026-07-17 18:55:45.778165423 +0000 UTC m=+44.107981611 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/71e3530f-7f67-4f34-a65a-444d1f270c93-metrics-tls") pod "dns-default-crvtk" (UID: "71e3530f-7f67-4f34-a65a-444d1f270c93") : secret "dns-default-metrics-tls" not found Jul 17 18:55:42.091036 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:42.091004 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pvpbg_728c6dad-4aae-44c2-ac11-60be3b680cca/dns-node-resolver/0.log" Jul 17 18:55:42.282117 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:42.282080 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/96280fd2-5cf1-4adc-a0a2-a8f54aef67a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6c6g4\" (UID: \"96280fd2-5cf1-4adc-a0a2-a8f54aef67a9\") " pod="openshift-insights/insights-runtime-extractor-6c6g4" Jul 17 18:55:42.282117 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:42.282121 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/96280fd2-5cf1-4adc-a0a2-a8f54aef67a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6c6g4\" (UID: \"96280fd2-5cf1-4adc-a0a2-a8f54aef67a9\") " pod="openshift-insights/insights-runtime-extractor-6c6g4" Jul 17 18:55:42.282321 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:42.282224 2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 17 18:55:42.282321 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:42.282236 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 17 18:55:42.282321 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:42.282291 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96280fd2-5cf1-4adc-a0a2-a8f54aef67a9-insights-runtime-extractor-tls podName:96280fd2-5cf1-4adc-a0a2-a8f54aef67a9 nodeName:}" failed. No retries permitted until 2026-07-17 18:55:44.28227791 +0000 UTC m=+42.612094095 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/96280fd2-5cf1-4adc-a0a2-a8f54aef67a9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6c6g4" (UID: "96280fd2-5cf1-4adc-a0a2-a8f54aef67a9") : secret "insights-runtime-extractor-tls" not found Jul 17 18:55:42.282411 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:42.282329 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/96280fd2-5cf1-4adc-a0a2-a8f54aef67a9-kube-rbac-proxy-cm podName:96280fd2-5cf1-4adc-a0a2-a8f54aef67a9 nodeName:}" failed. No retries permitted until 2026-07-17 18:55:44.282321885 +0000 UTC m=+42.612138070 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/96280fd2-5cf1-4adc-a0a2-a8f54aef67a9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6c6g4" (UID: "96280fd2-5cf1-4adc-a0a2-a8f54aef67a9") : configmap "kube-rbac-proxy" not found Jul 17 18:55:42.410577 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:42.410550 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"bond-cni-plugin\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:90f7e288d2e4b5406601a0ea9bdd9dc650bdf03a405ab5b85d67ef386e48e0fc\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 504 Gateway Time-out; artifact err: provided artifact is a container image\"" pod="openshift-multus/multus-additional-cni-plugins-nb8rb" podUID="08c31757-7c13-4c31-a1ad-570aad3cde70" Jul 17 18:55:42.741092 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:42.741064 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-7dwkn"] Jul 17 18:55:42.765217 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:42.765182 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-7dwkn"] Jul 17 18:55:42.765622 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:42.765305 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-7dwkn" Jul 17 18:55:42.768589 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:42.768561 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 17 18:55:42.768729 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:42.768627 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 17 18:55:42.768803 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:42.768750 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-v6n25\"" Jul 17 18:55:42.769042 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:42.769022 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 17 18:55:42.769497 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:42.769477 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 17 18:55:42.886596 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:42.886564 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/43a8551c-9107-4ec5-8d44-3bf3ba7120e8-signing-cabundle\") pod \"service-ca-58598695f7-7dwkn\" (UID: \"43a8551c-9107-4ec5-8d44-3bf3ba7120e8\") " pod="openshift-service-ca/service-ca-58598695f7-7dwkn" Jul 17 18:55:42.886769 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:42.886626 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/43a8551c-9107-4ec5-8d44-3bf3ba7120e8-signing-key\") pod \"service-ca-58598695f7-7dwkn\" (UID: \"43a8551c-9107-4ec5-8d44-3bf3ba7120e8\") " pod="openshift-service-ca/service-ca-58598695f7-7dwkn" Jul 17 18:55:42.886769 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:42.886680 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgmvl\" (UniqueName: \"kubernetes.io/projected/43a8551c-9107-4ec5-8d44-3bf3ba7120e8-kube-api-access-cgmvl\") pod \"service-ca-58598695f7-7dwkn\" (UID: \"43a8551c-9107-4ec5-8d44-3bf3ba7120e8\") " pod="openshift-service-ca/service-ca-58598695f7-7dwkn" Jul 17 18:55:42.987008 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:42.986969 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/43a8551c-9107-4ec5-8d44-3bf3ba7120e8-signing-key\") pod \"service-ca-58598695f7-7dwkn\" (UID: \"43a8551c-9107-4ec5-8d44-3bf3ba7120e8\") " pod="openshift-service-ca/service-ca-58598695f7-7dwkn" Jul 17 18:55:42.987008 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:42.987025 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgmvl\" (UniqueName: \"kubernetes.io/projected/43a8551c-9107-4ec5-8d44-3bf3ba7120e8-kube-api-access-cgmvl\") pod \"service-ca-58598695f7-7dwkn\" (UID: \"43a8551c-9107-4ec5-8d44-3bf3ba7120e8\") " pod="openshift-service-ca/service-ca-58598695f7-7dwkn" Jul 17 18:55:42.987267 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:42.987113 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/43a8551c-9107-4ec5-8d44-3bf3ba7120e8-signing-cabundle\") pod \"service-ca-58598695f7-7dwkn\" (UID: \"43a8551c-9107-4ec5-8d44-3bf3ba7120e8\") " pod="openshift-service-ca/service-ca-58598695f7-7dwkn" Jul 17 18:55:42.987720 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:42.987700 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/43a8551c-9107-4ec5-8d44-3bf3ba7120e8-signing-cabundle\") pod \"service-ca-58598695f7-7dwkn\" (UID: \"43a8551c-9107-4ec5-8d44-3bf3ba7120e8\") " pod="openshift-service-ca/service-ca-58598695f7-7dwkn" Jul 17 18:55:42.989485 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:42.989469 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/43a8551c-9107-4ec5-8d44-3bf3ba7120e8-signing-key\") pod \"service-ca-58598695f7-7dwkn\" (UID: \"43a8551c-9107-4ec5-8d44-3bf3ba7120e8\") " pod="openshift-service-ca/service-ca-58598695f7-7dwkn" Jul 17 18:55:42.996559 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:42.996502 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgmvl\" (UniqueName: \"kubernetes.io/projected/43a8551c-9107-4ec5-8d44-3bf3ba7120e8-kube-api-access-cgmvl\") pod \"service-ca-58598695f7-7dwkn\" (UID: \"43a8551c-9107-4ec5-8d44-3bf3ba7120e8\") " pod="openshift-service-ca/service-ca-58598695f7-7dwkn" Jul 17 18:55:43.074000 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:43.073955 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-7dwkn" Jul 17 18:55:43.220222 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:43.220189 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-7dwkn"] Jul 17 18:55:43.223890 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:43.223858 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43a8551c_9107_4ec5_8d44_3bf3ba7120e8.slice/crio-88f5b2c6989227c402ee63d5d44752ecea0c1d888de457347633c28a5fa8677e WatchSource:0}: Error finding container 88f5b2c6989227c402ee63d5d44752ecea0c1d888de457347633c28a5fa8677e: Status 404 returned error can't find the container with id 88f5b2c6989227c402ee63d5d44752ecea0c1d888de457347633c28a5fa8677e Jul 17 18:55:43.290712 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:43.290634 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-n2wtb_7f5ebd27-cd62-4b2b-9c30-4863ca02f422/node-ca/0.log" Jul 17 18:55:43.412296 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:43.412263 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-7dwkn" event={"ID":"43a8551c-9107-4ec5-8d44-3bf3ba7120e8","Type":"ContainerStarted","Data":"88f5b2c6989227c402ee63d5d44752ecea0c1d888de457347633c28a5fa8677e"} Jul 17 18:55:43.891369 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:43.891339 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ps4bw_c7ab0b84-2d5c-4dd8-817a-b6baa962fab6/serve-healthcheck-canary/0.log" Jul 17 18:55:44.297276 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:44.297230 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/96280fd2-5cf1-4adc-a0a2-a8f54aef67a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6c6g4\" (UID: \"96280fd2-5cf1-4adc-a0a2-a8f54aef67a9\") " pod="openshift-insights/insights-runtime-extractor-6c6g4" Jul 17 18:55:44.297276 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:44.297272 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/96280fd2-5cf1-4adc-a0a2-a8f54aef67a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6c6g4\" (UID: \"96280fd2-5cf1-4adc-a0a2-a8f54aef67a9\") " pod="openshift-insights/insights-runtime-extractor-6c6g4" Jul 17 18:55:44.297498 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:44.297362 2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 17 18:55:44.297498 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:44.297384 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 17 18:55:44.297498 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:44.297429 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/96280fd2-5cf1-4adc-a0a2-a8f54aef67a9-kube-rbac-proxy-cm podName:96280fd2-5cf1-4adc-a0a2-a8f54aef67a9 nodeName:}" failed. No retries permitted until 2026-07-17 18:55:48.29741244 +0000 UTC m=+46.627228628 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/96280fd2-5cf1-4adc-a0a2-a8f54aef67a9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6c6g4" (UID: "96280fd2-5cf1-4adc-a0a2-a8f54aef67a9") : configmap "kube-rbac-proxy" not found Jul 17 18:55:44.297498 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:44.297443 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96280fd2-5cf1-4adc-a0a2-a8f54aef67a9-insights-runtime-extractor-tls podName:96280fd2-5cf1-4adc-a0a2-a8f54aef67a9 nodeName:}" failed. No retries permitted until 2026-07-17 18:55:48.297436974 +0000 UTC m=+46.627253160 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/96280fd2-5cf1-4adc-a0a2-a8f54aef67a9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6c6g4" (UID: "96280fd2-5cf1-4adc-a0a2-a8f54aef67a9") : secret "insights-runtime-extractor-tls" not found Jul 17 18:55:45.809169 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:45.809132 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71e3530f-7f67-4f34-a65a-444d1f270c93-metrics-tls\") pod \"dns-default-crvtk\" (UID: \"71e3530f-7f67-4f34-a65a-444d1f270c93\") " pod="openshift-dns/dns-default-crvtk" Jul 17 18:55:45.809559 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:45.809187 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-registry-tls\") pod \"image-registry-6cb89cc658-6mn7r\" (UID: \"e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3\") " pod="openshift-image-registry/image-registry-6cb89cc658-6mn7r" Jul 17 18:55:45.809559 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:45.809299 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 18:55:45.809559 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:45.809306 2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 17 18:55:45.809559 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:45.809322 2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cb89cc658-6mn7r: secret "image-registry-tls" not found Jul 17 18:55:45.809559 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:45.809368 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71e3530f-7f67-4f34-a65a-444d1f270c93-metrics-tls podName:71e3530f-7f67-4f34-a65a-444d1f270c93 nodeName:}" failed. No retries permitted until 2026-07-17 18:55:53.809348597 +0000 UTC m=+52.139164783 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/71e3530f-7f67-4f34-a65a-444d1f270c93-metrics-tls") pod "dns-default-crvtk" (UID: "71e3530f-7f67-4f34-a65a-444d1f270c93") : secret "dns-default-metrics-tls" not found Jul 17 18:55:45.809559 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:45.809388 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-registry-tls podName:e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3 nodeName:}" failed. No retries permitted until 2026-07-17 18:55:53.809380931 +0000 UTC m=+52.139197118 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-registry-tls") pod "image-registry-6cb89cc658-6mn7r" (UID: "e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3") : secret "image-registry-tls" not found Jul 17 18:55:46.421500 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:46.421465 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-7dwkn" event={"ID":"43a8551c-9107-4ec5-8d44-3bf3ba7120e8","Type":"ContainerStarted","Data":"70aac1b94706b89daf068a0065c64fbaf49e6224156c59270c45685d7f8042a9"} Jul 17 18:55:46.441304 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:46.441258 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-7dwkn" podStartSLOduration=2.091316557 podStartE2EDuration="4.441243644s" totalImagesPullingTime="2.349927087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:55:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:55:46.440870074 +0000 UTC m=+44.770686281" watchObservedRunningTime="2026-07-17 18:55:46.441243644 +0000 UTC m=+44.771059869" Jul 17 18:55:48.331810 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:48.331770 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/96280fd2-5cf1-4adc-a0a2-a8f54aef67a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6c6g4\" (UID: \"96280fd2-5cf1-4adc-a0a2-a8f54aef67a9\") " pod="openshift-insights/insights-runtime-extractor-6c6g4" Jul 17 18:55:48.331810 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:48.331811 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/96280fd2-5cf1-4adc-a0a2-a8f54aef67a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6c6g4\" (UID: \"96280fd2-5cf1-4adc-a0a2-a8f54aef67a9\") " pod="openshift-insights/insights-runtime-extractor-6c6g4" Jul 17 18:55:48.332303 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:48.331926 2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 17 18:55:48.332303 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:48.331981 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 17 18:55:48.332303 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:48.332001 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/96280fd2-5cf1-4adc-a0a2-a8f54aef67a9-kube-rbac-proxy-cm podName:96280fd2-5cf1-4adc-a0a2-a8f54aef67a9 nodeName:}" failed. No retries permitted until 2026-07-17 18:55:56.331981931 +0000 UTC m=+54.661798137 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/96280fd2-5cf1-4adc-a0a2-a8f54aef67a9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6c6g4" (UID: "96280fd2-5cf1-4adc-a0a2-a8f54aef67a9") : configmap "kube-rbac-proxy" not found Jul 17 18:55:48.332303 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:48.332023 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96280fd2-5cf1-4adc-a0a2-a8f54aef67a9-insights-runtime-extractor-tls podName:96280fd2-5cf1-4adc-a0a2-a8f54aef67a9 nodeName:}" failed. No retries permitted until 2026-07-17 18:55:56.332012981 +0000 UTC m=+54.661829166 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/96280fd2-5cf1-4adc-a0a2-a8f54aef67a9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6c6g4" (UID: "96280fd2-5cf1-4adc-a0a2-a8f54aef67a9") : secret "insights-runtime-extractor-tls" not found Jul 17 18:55:50.531582 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:50.531538 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6cb89cc658-6mn7r"] Jul 17 18:55:50.531999 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:50.531795 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-6cb89cc658-6mn7r" podUID="e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3" Jul 17 18:55:50.570964 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:50.570927 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-d5567c6d9-bt97k"] Jul 17 18:55:50.579238 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:50.579213 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d5567c6d9-bt97k" Jul 17 18:55:50.586746 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:50.586721 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d5567c6d9-bt97k"] Jul 17 18:55:50.752733 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:50.752689 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ec8a5e73-e34f-4c8e-b194-56371da5cb81-bound-sa-token\") pod \"image-registry-d5567c6d9-bt97k\" (UID: \"ec8a5e73-e34f-4c8e-b194-56371da5cb81\") " pod="openshift-image-registry/image-registry-d5567c6d9-bt97k" Jul 17 18:55:50.752733 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:50.752734 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ec8a5e73-e34f-4c8e-b194-56371da5cb81-ca-trust-extracted\") pod \"image-registry-d5567c6d9-bt97k\" (UID: \"ec8a5e73-e34f-4c8e-b194-56371da5cb81\") " pod="openshift-image-registry/image-registry-d5567c6d9-bt97k" Jul 17 18:55:50.752984 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:50.752771 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgg22\" (UniqueName: \"kubernetes.io/projected/ec8a5e73-e34f-4c8e-b194-56371da5cb81-kube-api-access-hgg22\") pod \"image-registry-d5567c6d9-bt97k\" (UID: \"ec8a5e73-e34f-4c8e-b194-56371da5cb81\") " pod="openshift-image-registry/image-registry-d5567c6d9-bt97k" Jul 17 18:55:50.752984 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:50.752834 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ec8a5e73-e34f-4c8e-b194-56371da5cb81-trusted-ca\") pod \"image-registry-d5567c6d9-bt97k\" (UID: \"ec8a5e73-e34f-4c8e-b194-56371da5cb81\") " pod="openshift-image-registry/image-registry-d5567c6d9-bt97k" Jul 17 18:55:50.752984 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:50.752905 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ec8a5e73-e34f-4c8e-b194-56371da5cb81-image-registry-private-configuration\") pod \"image-registry-d5567c6d9-bt97k\" (UID: \"ec8a5e73-e34f-4c8e-b194-56371da5cb81\") " pod="openshift-image-registry/image-registry-d5567c6d9-bt97k" Jul 17 18:55:50.752984 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:50.752946 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ec8a5e73-e34f-4c8e-b194-56371da5cb81-registry-certificates\") pod \"image-registry-d5567c6d9-bt97k\" (UID: \"ec8a5e73-e34f-4c8e-b194-56371da5cb81\") " pod="openshift-image-registry/image-registry-d5567c6d9-bt97k" Jul 17 18:55:50.752984 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:50.752977 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ec8a5e73-e34f-4c8e-b194-56371da5cb81-registry-tls\") pod \"image-registry-d5567c6d9-bt97k\" (UID: \"ec8a5e73-e34f-4c8e-b194-56371da5cb81\") " pod="openshift-image-registry/image-registry-d5567c6d9-bt97k" Jul 17 18:55:50.753141 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:50.753044 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ec8a5e73-e34f-4c8e-b194-56371da5cb81-installation-pull-secrets\") pod \"image-registry-d5567c6d9-bt97k\" (UID: \"ec8a5e73-e34f-4c8e-b194-56371da5cb81\") " pod="openshift-image-registry/image-registry-d5567c6d9-bt97k" Jul 17 18:55:50.853743 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:50.853648 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ec8a5e73-e34f-4c8e-b194-56371da5cb81-trusted-ca\") pod \"image-registry-d5567c6d9-bt97k\" (UID: \"ec8a5e73-e34f-4c8e-b194-56371da5cb81\") " pod="openshift-image-registry/image-registry-d5567c6d9-bt97k" Jul 17 18:55:50.853743 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:50.853685 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ec8a5e73-e34f-4c8e-b194-56371da5cb81-image-registry-private-configuration\") pod \"image-registry-d5567c6d9-bt97k\" (UID: \"ec8a5e73-e34f-4c8e-b194-56371da5cb81\") " pod="openshift-image-registry/image-registry-d5567c6d9-bt97k" Jul 17 18:55:50.853743 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:50.853718 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ec8a5e73-e34f-4c8e-b194-56371da5cb81-registry-certificates\") pod \"image-registry-d5567c6d9-bt97k\" (UID: \"ec8a5e73-e34f-4c8e-b194-56371da5cb81\") " pod="openshift-image-registry/image-registry-d5567c6d9-bt97k" Jul 17 18:55:50.854023 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:50.853749 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ec8a5e73-e34f-4c8e-b194-56371da5cb81-registry-tls\") pod \"image-registry-d5567c6d9-bt97k\" (UID: \"ec8a5e73-e34f-4c8e-b194-56371da5cb81\") " pod="openshift-image-registry/image-registry-d5567c6d9-bt97k" Jul 17 18:55:50.854023 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:50.853792 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ec8a5e73-e34f-4c8e-b194-56371da5cb81-installation-pull-secrets\") pod \"image-registry-d5567c6d9-bt97k\" (UID: \"ec8a5e73-e34f-4c8e-b194-56371da5cb81\") " pod="openshift-image-registry/image-registry-d5567c6d9-bt97k" Jul 17 18:55:50.854023 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:50.853839 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ec8a5e73-e34f-4c8e-b194-56371da5cb81-bound-sa-token\") pod \"image-registry-d5567c6d9-bt97k\" (UID: \"ec8a5e73-e34f-4c8e-b194-56371da5cb81\") " pod="openshift-image-registry/image-registry-d5567c6d9-bt97k" Jul 17 18:55:50.854165 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:50.854048 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ec8a5e73-e34f-4c8e-b194-56371da5cb81-ca-trust-extracted\") pod \"image-registry-d5567c6d9-bt97k\" (UID: \"ec8a5e73-e34f-4c8e-b194-56371da5cb81\") " pod="openshift-image-registry/image-registry-d5567c6d9-bt97k" Jul 17 18:55:50.854165 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:50.854106 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgg22\" (UniqueName: \"kubernetes.io/projected/ec8a5e73-e34f-4c8e-b194-56371da5cb81-kube-api-access-hgg22\") pod \"image-registry-d5567c6d9-bt97k\" (UID: \"ec8a5e73-e34f-4c8e-b194-56371da5cb81\") " pod="openshift-image-registry/image-registry-d5567c6d9-bt97k" Jul 17 18:55:50.854415 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:50.854390 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ec8a5e73-e34f-4c8e-b194-56371da5cb81-ca-trust-extracted\") pod \"image-registry-d5567c6d9-bt97k\" (UID: \"ec8a5e73-e34f-4c8e-b194-56371da5cb81\") " pod="openshift-image-registry/image-registry-d5567c6d9-bt97k" Jul 17 18:55:50.854743 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:50.854720 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ec8a5e73-e34f-4c8e-b194-56371da5cb81-registry-certificates\") pod \"image-registry-d5567c6d9-bt97k\" (UID: \"ec8a5e73-e34f-4c8e-b194-56371da5cb81\") " pod="openshift-image-registry/image-registry-d5567c6d9-bt97k" Jul 17 18:55:50.855496 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:50.855473 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ec8a5e73-e34f-4c8e-b194-56371da5cb81-trusted-ca\") pod \"image-registry-d5567c6d9-bt97k\" (UID: \"ec8a5e73-e34f-4c8e-b194-56371da5cb81\") " pod="openshift-image-registry/image-registry-d5567c6d9-bt97k" Jul 17 18:55:50.856435 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:50.856407 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ec8a5e73-e34f-4c8e-b194-56371da5cb81-installation-pull-secrets\") pod \"image-registry-d5567c6d9-bt97k\" (UID: \"ec8a5e73-e34f-4c8e-b194-56371da5cb81\") " pod="openshift-image-registry/image-registry-d5567c6d9-bt97k" Jul 17 18:55:50.856435 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:50.856426 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ec8a5e73-e34f-4c8e-b194-56371da5cb81-image-registry-private-configuration\") pod \"image-registry-d5567c6d9-bt97k\" (UID: \"ec8a5e73-e34f-4c8e-b194-56371da5cb81\") " pod="openshift-image-registry/image-registry-d5567c6d9-bt97k" Jul 17 18:55:50.856968 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:50.856949 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ec8a5e73-e34f-4c8e-b194-56371da5cb81-registry-tls\") pod \"image-registry-d5567c6d9-bt97k\" (UID: \"ec8a5e73-e34f-4c8e-b194-56371da5cb81\") " pod="openshift-image-registry/image-registry-d5567c6d9-bt97k" Jul 17 18:55:50.863313 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:50.863287 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ec8a5e73-e34f-4c8e-b194-56371da5cb81-bound-sa-token\") pod \"image-registry-d5567c6d9-bt97k\" (UID: \"ec8a5e73-e34f-4c8e-b194-56371da5cb81\") " pod="openshift-image-registry/image-registry-d5567c6d9-bt97k" Jul 17 18:55:50.863421 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:50.863366 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgg22\" (UniqueName: \"kubernetes.io/projected/ec8a5e73-e34f-4c8e-b194-56371da5cb81-kube-api-access-hgg22\") pod \"image-registry-d5567c6d9-bt97k\" (UID: \"ec8a5e73-e34f-4c8e-b194-56371da5cb81\") " pod="openshift-image-registry/image-registry-d5567c6d9-bt97k" Jul 17 18:55:50.888375 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:50.888335 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d5567c6d9-bt97k" Jul 17 18:55:51.039799 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:51.039761 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d5567c6d9-bt97k"] Jul 17 18:55:51.043203 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:51.043177 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec8a5e73_e34f_4c8e_b194_56371da5cb81.slice/crio-2677348c5d3245460b929659384b313b57678456504532008dc57aeaa1e89d41 WatchSource:0}: Error finding container 2677348c5d3245460b929659384b313b57678456504532008dc57aeaa1e89d41: Status 404 returned error can't find the container with id 2677348c5d3245460b929659384b313b57678456504532008dc57aeaa1e89d41 Jul 17 18:55:51.436002 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:51.435923 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cb89cc658-6mn7r" Jul 17 18:55:51.436167 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:51.435913 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d5567c6d9-bt97k" event={"ID":"ec8a5e73-e34f-4c8e-b194-56371da5cb81","Type":"ContainerStarted","Data":"914fed5ff559a3ee9d153a4384f0425fba9fd02160f41180c5b03eedced841b7"} Jul 17 18:55:51.436167 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:51.436084 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d5567c6d9-bt97k" event={"ID":"ec8a5e73-e34f-4c8e-b194-56371da5cb81","Type":"ContainerStarted","Data":"2677348c5d3245460b929659384b313b57678456504532008dc57aeaa1e89d41"} Jul 17 18:55:51.452557 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:51.452506 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cb89cc658-6mn7r" Jul 17 18:55:51.477185 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:51.477118 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-d5567c6d9-bt97k" podStartSLOduration=1.4770939 podStartE2EDuration="1.4770939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:55:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:55:51.475739233 +0000 UTC m=+49.805555643" watchObservedRunningTime="2026-07-17 18:55:51.4770939 +0000 UTC m=+49.806910111" Jul 17 18:55:51.560361 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:51.560329 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-ca-trust-extracted\") pod \"e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3\" (UID: \"e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3\") " Jul 17 18:55:51.560361 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:51.560367 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-image-registry-private-configuration\") pod \"e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3\" (UID: \"e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3\") " Jul 17 18:55:51.560886 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:51.560396 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-trusted-ca\") pod \"e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3\" (UID: \"e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3\") " Jul 17 18:55:51.560886 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:51.560426 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-bound-sa-token\") pod \"e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3\" (UID: \"e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3\") " Jul 17 18:55:51.560886 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:51.560487 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6jbdk\" (UniqueName: \"kubernetes.io/projected/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-kube-api-access-6jbdk\") pod \"e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3\" (UID: \"e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3\") " Jul 17 18:55:51.560886 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:51.560520 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-registry-certificates\") pod \"e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3\" (UID: \"e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3\") " Jul 17 18:55:51.560886 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:51.560574 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-installation-pull-secrets\") pod \"e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3\" (UID: \"e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3\") " Jul 17 18:55:51.560886 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:51.560666 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3" (UID: "e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:55:51.561177 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:51.561127 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3" (UID: "e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:55:51.561285 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:51.561256 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3" (UID: "e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:55:51.563318 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:51.562348 2561 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-registry-certificates\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 18:55:51.563318 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:51.562376 2561 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-ca-trust-extracted\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 18:55:51.563318 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:51.562399 2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-trusted-ca\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 18:55:51.564839 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:51.563801 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3" (UID: "e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:55:51.564839 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:51.564049 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3" (UID: "e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:55:51.566511 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:51.565849 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3" (UID: "e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:55:51.566783 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:51.566728 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-kube-api-access-6jbdk" (OuterVolumeSpecName: "kube-api-access-6jbdk") pod "e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3" (UID: "e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3"). InnerVolumeSpecName "kube-api-access-6jbdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:55:51.662910 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:51.662868 2561 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-image-registry-private-configuration\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 18:55:51.662910 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:51.662905 2561 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-bound-sa-token\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 18:55:51.663156 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:51.662921 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jbdk\" (UniqueName: \"kubernetes.io/projected/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-kube-api-access-6jbdk\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 18:55:51.663156 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:51.662936 2561 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-installation-pull-secrets\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 18:55:52.437933 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:52.437906 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cb89cc658-6mn7r" Jul 17 18:55:52.438097 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:52.438029 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-d5567c6d9-bt97k" Jul 17 18:55:52.479157 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:52.479123 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6cb89cc658-6mn7r"] Jul 17 18:55:52.485245 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:52.485217 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6cb89cc658-6mn7r"] Jul 17 18:55:52.672682 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:52.672648 2561 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3-registry-tls\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 18:55:53.883063 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:53.883024 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71e3530f-7f67-4f34-a65a-444d1f270c93-metrics-tls\") pod \"dns-default-crvtk\" (UID: \"71e3530f-7f67-4f34-a65a-444d1f270c93\") " pod="openshift-dns/dns-default-crvtk" Jul 17 18:55:53.885504 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:53.885474 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71e3530f-7f67-4f34-a65a-444d1f270c93-metrics-tls\") pod \"dns-default-crvtk\" (UID: \"71e3530f-7f67-4f34-a65a-444d1f270c93\") " pod="openshift-dns/dns-default-crvtk" Jul 17 18:55:53.950031 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:53.949981 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-crvtk" Jul 17 18:55:54.099146 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:54.099115 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3" path="/var/lib/kubelet/pods/e6a12356-d9b9-4a98-93aa-bbfdd7a3e0f3/volumes" Jul 17 18:55:54.112254 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:54.112215 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-crvtk"] Jul 17 18:55:54.115144 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:55:54.115118 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71e3530f_7f67_4f34_a65a_444d1f270c93.slice/crio-ebd655ad7ff2249607a23393a680ddc5721a6300876483c8613782589b53e774 WatchSource:0}: Error finding container ebd655ad7ff2249607a23393a680ddc5721a6300876483c8613782589b53e774: Status 404 returned error can't find the container with id ebd655ad7ff2249607a23393a680ddc5721a6300876483c8613782589b53e774 Jul 17 18:55:54.443244 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:54.443211 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-crvtk" event={"ID":"71e3530f-7f67-4f34-a65a-444d1f270c93","Type":"ContainerStarted","Data":"ebd655ad7ff2249607a23393a680ddc5721a6300876483c8613782589b53e774"} Jul 17 18:55:56.405245 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:56.405210 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/96280fd2-5cf1-4adc-a0a2-a8f54aef67a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6c6g4\" (UID: \"96280fd2-5cf1-4adc-a0a2-a8f54aef67a9\") " pod="openshift-insights/insights-runtime-extractor-6c6g4" Jul 17 18:55:56.409586 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:56.405314 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/96280fd2-5cf1-4adc-a0a2-a8f54aef67a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6c6g4\" (UID: \"96280fd2-5cf1-4adc-a0a2-a8f54aef67a9\") " pod="openshift-insights/insights-runtime-extractor-6c6g4" Jul 17 18:55:56.409586 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:56.405454 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 17 18:55:56.409586 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:55:56.405520 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96280fd2-5cf1-4adc-a0a2-a8f54aef67a9-insights-runtime-extractor-tls podName:96280fd2-5cf1-4adc-a0a2-a8f54aef67a9 nodeName:}" failed. No retries permitted until 2026-07-17 18:56:12.405492322 +0000 UTC m=+70.735308508 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/96280fd2-5cf1-4adc-a0a2-a8f54aef67a9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6c6g4" (UID: "96280fd2-5cf1-4adc-a0a2-a8f54aef67a9") : secret "insights-runtime-extractor-tls" not found Jul 17 18:55:56.409586 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:56.406083 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/96280fd2-5cf1-4adc-a0a2-a8f54aef67a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6c6g4\" (UID: \"96280fd2-5cf1-4adc-a0a2-a8f54aef67a9\") " pod="openshift-insights/insights-runtime-extractor-6c6g4" Jul 17 18:55:56.449665 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:56.449629 2561 generic.go:356] "Generic (PLEG): container finished" podID="08c31757-7c13-4c31-a1ad-570aad3cde70" containerID="49789d2db9adbdfa6bd9e7c04e24eb8c2e7b42b973e6d70cb89b845c951099a1" exitCode=0 Jul 17 18:55:56.449839 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:56.449700 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nb8rb" event={"ID":"08c31757-7c13-4c31-a1ad-570aad3cde70","Type":"ContainerDied","Data":"49789d2db9adbdfa6bd9e7c04e24eb8c2e7b42b973e6d70cb89b845c951099a1"} Jul 17 18:55:56.451270 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:56.451220 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-crvtk" event={"ID":"71e3530f-7f67-4f34-a65a-444d1f270c93","Type":"ContainerStarted","Data":"2333cf97d89603bd35bf205ea0518fdd30eaf59334667ae8ca108c12579a03a1"} Jul 17 18:55:57.455993 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:57.455954 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-crvtk" event={"ID":"71e3530f-7f67-4f34-a65a-444d1f270c93","Type":"ContainerStarted","Data":"0caaac2610e79a5d6a6b714a23fca3dc26606b52bd7da14bee6404c10396b6ad"} Jul 17 18:55:57.456400 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:57.456027 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-crvtk" Jul 17 18:55:57.458409 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:57.458383 2561 generic.go:356] "Generic (PLEG): container finished" podID="08c31757-7c13-4c31-a1ad-570aad3cde70" containerID="c7203685ba2b59e83bae3e7e5f1bad409349c9f09b25dcec9755925df2013325" exitCode=0 Jul 17 18:55:57.458557 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:57.458422 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nb8rb" event={"ID":"08c31757-7c13-4c31-a1ad-570aad3cde70","Type":"ContainerDied","Data":"c7203685ba2b59e83bae3e7e5f1bad409349c9f09b25dcec9755925df2013325"} Jul 17 18:55:57.474741 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:55:57.474702 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-crvtk" podStartSLOduration=17.311312798 podStartE2EDuration="19.474688259s" totalImagesPullingTime="2.163375461s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:55:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:55:57.474274506 +0000 UTC m=+55.804090715" watchObservedRunningTime="2026-07-17 18:55:57.474688259 +0000 UTC m=+55.804504473" Jul 17 18:56:02.475972 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:02.475916 2561 generic.go:356] "Generic (PLEG): container finished" podID="08c31757-7c13-4c31-a1ad-570aad3cde70" containerID="1b9278a3e927be2cbc4b69c8b507c359876810226dddcf7353ec083316e09f28" exitCode=0 Jul 17 18:56:02.476770 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:02.476005 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nb8rb" event={"ID":"08c31757-7c13-4c31-a1ad-570aad3cde70","Type":"ContainerDied","Data":"1b9278a3e927be2cbc4b69c8b507c359876810226dddcf7353ec083316e09f28"} Jul 17 18:56:03.482021 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:03.481989 2561 generic.go:356] "Generic (PLEG): container finished" podID="08c31757-7c13-4c31-a1ad-570aad3cde70" containerID="892c5bc4d49afc311789ae3f30add43fe7810dc5a5e859e00f61794052b433bb" exitCode=0 Jul 17 18:56:03.482401 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:03.482058 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nb8rb" event={"ID":"08c31757-7c13-4c31-a1ad-570aad3cde70","Type":"ContainerDied","Data":"892c5bc4d49afc311789ae3f30add43fe7810dc5a5e859e00f61794052b433bb"} Jul 17 18:56:04.399946 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:04.399919 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hpjzq" Jul 17 18:56:04.488895 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:04.488856 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nb8rb" event={"ID":"08c31757-7c13-4c31-a1ad-570aad3cde70","Type":"ContainerStarted","Data":"d753ffb022241c4d4c493d351829fc915e696f1f07622e9267f84487eb3a9bad"} Jul 17 18:56:04.511585 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:04.511490 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-nb8rb" podStartSLOduration=6.159178693 podStartE2EDuration="1m2.511472752s" totalImagesPullingTime="56.352294059s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:55:02 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=1 observedRunningTime="2026-07-17 18:56:04.511102595 +0000 UTC m=+62.840918804" watchObservedRunningTime="2026-07-17 18:56:04.511472752 +0000 UTC m=+62.841288959" Jul 17 18:56:06.875584 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:06.875549 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16db4114-eec4-4eb4-80be-1e6238e5c67b-metrics-certs\") pod \"network-metrics-daemon-7l6k9\" (UID: \"16db4114-eec4-4eb4-80be-1e6238e5c67b\") " pod="openshift-multus/network-metrics-daemon-7l6k9" Jul 17 18:56:06.875986 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:06.875607 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3591b9f8-a2ff-4adb-93ec-449225ca6390-original-pull-secret\") pod \"global-pull-secret-syncer-m9wzh\" (UID: \"3591b9f8-a2ff-4adb-93ec-449225ca6390\") " pod="kube-system/global-pull-secret-syncer-m9wzh" Jul 17 18:56:06.878462 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:06.878441 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 17 18:56:06.879114 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:06.879099 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 17 18:56:06.888449 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:06.888418 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3591b9f8-a2ff-4adb-93ec-449225ca6390-original-pull-secret\") pod \"global-pull-secret-syncer-m9wzh\" (UID: \"3591b9f8-a2ff-4adb-93ec-449225ca6390\") " pod="kube-system/global-pull-secret-syncer-m9wzh" Jul 17 18:56:06.888586 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:06.888506 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16db4114-eec4-4eb4-80be-1e6238e5c67b-metrics-certs\") pod \"network-metrics-daemon-7l6k9\" (UID: \"16db4114-eec4-4eb4-80be-1e6238e5c67b\") " pod="openshift-multus/network-metrics-daemon-7l6k9" Jul 17 18:56:06.976084 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:06.976049 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlmhc\" (UniqueName: \"kubernetes.io/projected/2c91e513-4f09-4551-8f50-75a4ba131d12-kube-api-access-mlmhc\") pod \"network-check-target-c54gz\" (UID: \"2c91e513-4f09-4551-8f50-75a4ba131d12\") " pod="openshift-network-diagnostics/network-check-target-c54gz" Jul 17 18:56:06.979019 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:06.978995 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 17 18:56:06.988428 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:06.988401 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 17 18:56:06.998949 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:06.998920 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlmhc\" (UniqueName: \"kubernetes.io/projected/2c91e513-4f09-4551-8f50-75a4ba131d12-kube-api-access-mlmhc\") pod \"network-check-target-c54gz\" (UID: \"2c91e513-4f09-4551-8f50-75a4ba131d12\") " pod="openshift-network-diagnostics/network-check-target-c54gz" Jul 17 18:56:07.008967 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:07.008942 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n57jp\"" Jul 17 18:56:07.011869 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:07.011853 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m9wzh" Jul 17 18:56:07.017096 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:07.017080 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7l6k9" Jul 17 18:56:07.143698 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:07.143620 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-m9wzh"] Jul 17 18:56:07.146267 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:56:07.146240 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3591b9f8_a2ff_4adb_93ec_449225ca6390.slice/crio-34b4f336e6aa7586653c062fa14dfb2a495b392ce201492fa4013622e5646590 WatchSource:0}: Error finding container 34b4f336e6aa7586653c062fa14dfb2a495b392ce201492fa4013622e5646590: Status 404 returned error can't find the container with id 34b4f336e6aa7586653c062fa14dfb2a495b392ce201492fa4013622e5646590 Jul 17 18:56:07.159799 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:07.159772 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7l6k9"] Jul 17 18:56:07.162724 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:56:07.162703 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16db4114_eec4_4eb4_80be_1e6238e5c67b.slice/crio-e3bbe14a855fcf76955cf9c44bc6c9eaf4ec09e02d1736a2d14dae665d502d5f WatchSource:0}: Error finding container e3bbe14a855fcf76955cf9c44bc6c9eaf4ec09e02d1736a2d14dae665d502d5f: Status 404 returned error can't find the container with id e3bbe14a855fcf76955cf9c44bc6c9eaf4ec09e02d1736a2d14dae665d502d5f Jul 17 18:56:07.206045 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:07.206013 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bqhtg\"" Jul 17 18:56:07.213409 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:07.213387 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c54gz" Jul 17 18:56:07.343485 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:07.343453 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-c54gz"] Jul 17 18:56:07.346049 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:56:07.346017 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c91e513_4f09_4551_8f50_75a4ba131d12.slice/crio-55b989f59b230af768c845b1d9d8783f4980e9b16a2f472b64d283b4c473668f WatchSource:0}: Error finding container 55b989f59b230af768c845b1d9d8783f4980e9b16a2f472b64d283b4c473668f: Status 404 returned error can't find the container with id 55b989f59b230af768c845b1d9d8783f4980e9b16a2f472b64d283b4c473668f Jul 17 18:56:07.463602 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:07.463509 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-crvtk" Jul 17 18:56:07.500734 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:07.500696 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7l6k9" event={"ID":"16db4114-eec4-4eb4-80be-1e6238e5c67b","Type":"ContainerStarted","Data":"e3bbe14a855fcf76955cf9c44bc6c9eaf4ec09e02d1736a2d14dae665d502d5f"} Jul 17 18:56:07.501868 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:07.501847 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-c54gz" event={"ID":"2c91e513-4f09-4551-8f50-75a4ba131d12","Type":"ContainerStarted","Data":"55b989f59b230af768c845b1d9d8783f4980e9b16a2f472b64d283b4c473668f"} Jul 17 18:56:07.502968 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:07.502950 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-m9wzh" event={"ID":"3591b9f8-a2ff-4adb-93ec-449225ca6390","Type":"ContainerStarted","Data":"34b4f336e6aa7586653c062fa14dfb2a495b392ce201492fa4013622e5646590"} Jul 17 18:56:09.513894 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:09.513849 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7l6k9" event={"ID":"16db4114-eec4-4eb4-80be-1e6238e5c67b","Type":"ContainerStarted","Data":"9c1ab6b83bd1e67017450a73c04d2d88fa631e3f508ef7e764e2275b59355dca"} Jul 17 18:56:09.514265 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:09.513922 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7l6k9" event={"ID":"16db4114-eec4-4eb4-80be-1e6238e5c67b","Type":"ContainerStarted","Data":"4222949a0d73275078050b59262d3f65d1182ffcf371370cf0ebe5ce26a61982"} Jul 17 18:56:09.532318 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:09.532271 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-7l6k9" podStartSLOduration=66.187246622 podStartE2EDuration="1m7.532255144s" totalImagesPullingTime="1.345008522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:55:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:56:09.531288 +0000 UTC m=+67.861104209" watchObservedRunningTime="2026-07-17 18:56:09.532255144 +0000 UTC m=+67.862071352" Jul 17 18:56:10.409789 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.409754 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-649d554676-clrbv"] Jul 17 18:56:10.412897 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.412872 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-649d554676-clrbv" Jul 17 18:56:10.416219 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.416195 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-rlptx\"" Jul 17 18:56:10.416344 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.416326 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 17 18:56:10.417651 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.417475 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 17 18:56:10.417651 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.417476 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 17 18:56:10.417651 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.417476 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 17 18:56:10.430084 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.430055 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7848588d47-zbx9t"] Jul 17 18:56:10.433568 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.433542 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-649d554676-clrbv"] Jul 17 18:56:10.433737 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.433639 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7848588d47-zbx9t" Jul 17 18:56:10.436729 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.436708 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 17 18:56:10.448345 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.448316 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7848588d47-zbx9t"] Jul 17 18:56:10.489341 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.489302 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5bc67dcd46-xbsqk"] Jul 17 18:56:10.492619 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.492593 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-hw2f9"] Jul 17 18:56:10.492876 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.492854 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-xbsqk" Jul 17 18:56:10.495414 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.495391 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-6p5gk"] Jul 17 18:56:10.495519 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.495503 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-hw2f9" Jul 17 18:56:10.496092 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.496072 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 17 18:56:10.496663 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.496646 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 17 18:56:10.496735 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.496673 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-tscnt\"" Jul 17 18:56:10.498435 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.498417 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-6p5gk" Jul 17 18:56:10.499355 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.499335 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-tmnhs\"" Jul 17 18:56:10.499687 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.499666 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 17 18:56:10.500919 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.500687 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 17 18:56:10.500919 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.500756 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-ft2sr\"" Jul 17 18:56:10.501075 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.500999 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 17 18:56:10.505494 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.504849 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpg65\" (UniqueName: \"kubernetes.io/projected/b8ae4f97-aeeb-4e21-b036-d49cb650bd59-kube-api-access-fpg65\") pod \"klusterlet-addon-workmgr-7848588d47-zbx9t\" (UID: \"b8ae4f97-aeeb-4e21-b036-d49cb650bd59\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7848588d47-zbx9t" Jul 17 18:56:10.505494 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.504895 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/35250a09-3a42-4c8a-b8ea-8d87c2602dcd-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-649d554676-clrbv\" (UID: \"35250a09-3a42-4c8a-b8ea-8d87c2602dcd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-649d554676-clrbv" Jul 17 18:56:10.505494 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.504926 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bkvf\" (UniqueName: \"kubernetes.io/projected/35250a09-3a42-4c8a-b8ea-8d87c2602dcd-kube-api-access-4bkvf\") pod \"managed-serviceaccount-addon-agent-649d554676-clrbv\" (UID: \"35250a09-3a42-4c8a-b8ea-8d87c2602dcd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-649d554676-clrbv" Jul 17 18:56:10.505494 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.504963 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8ae4f97-aeeb-4e21-b036-d49cb650bd59-tmp\") pod \"klusterlet-addon-workmgr-7848588d47-zbx9t\" (UID: \"b8ae4f97-aeeb-4e21-b036-d49cb650bd59\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7848588d47-zbx9t" Jul 17 18:56:10.505494 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.504996 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b8ae4f97-aeeb-4e21-b036-d49cb650bd59-klusterlet-config\") pod \"klusterlet-addon-workmgr-7848588d47-zbx9t\" (UID: \"b8ae4f97-aeeb-4e21-b036-d49cb650bd59\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7848588d47-zbx9t" Jul 17 18:56:10.508552 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.507586 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-xbsqk"] Jul 17 18:56:10.509147 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.509127 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-6p5gk"] Jul 17 18:56:10.509977 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.509956 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-hw2f9"] Jul 17 18:56:10.606182 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.606144 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8ae4f97-aeeb-4e21-b036-d49cb650bd59-tmp\") pod \"klusterlet-addon-workmgr-7848588d47-zbx9t\" (UID: \"b8ae4f97-aeeb-4e21-b036-d49cb650bd59\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7848588d47-zbx9t" Jul 17 18:56:10.606649 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.606187 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlpmk\" (UniqueName: \"kubernetes.io/projected/67b653fc-1593-4bee-a23f-3f54197879c0-kube-api-access-vlpmk\") pod \"downloads-5bc67dcd46-xbsqk\" (UID: \"67b653fc-1593-4bee-a23f-3f54197879c0\") " pod="openshift-console/downloads-5bc67dcd46-xbsqk" Jul 17 18:56:10.606649 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.606221 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b8ae4f97-aeeb-4e21-b036-d49cb650bd59-klusterlet-config\") pod \"klusterlet-addon-workmgr-7848588d47-zbx9t\" (UID: \"b8ae4f97-aeeb-4e21-b036-d49cb650bd59\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7848588d47-zbx9t" Jul 17 18:56:10.606649 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.606248 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxvg6\" (UniqueName: \"kubernetes.io/projected/18235a43-e5c1-4146-876c-55c96b2cbb5c-kube-api-access-mxvg6\") pod \"migrator-7cc9c7ffd7-6p5gk\" (UID: \"18235a43-e5c1-4146-876c-55c96b2cbb5c\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-6p5gk" Jul 17 18:56:10.606649 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.606317 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/24f74548-eab9-4297-acc1-03b123c45161-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-hw2f9\" (UID: \"24f74548-eab9-4297-acc1-03b123c45161\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-hw2f9" Jul 17 18:56:10.606649 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.606376 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpg65\" (UniqueName: \"kubernetes.io/projected/b8ae4f97-aeeb-4e21-b036-d49cb650bd59-kube-api-access-fpg65\") pod \"klusterlet-addon-workmgr-7848588d47-zbx9t\" (UID: \"b8ae4f97-aeeb-4e21-b036-d49cb650bd59\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7848588d47-zbx9t" Jul 17 18:56:10.606649 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.606416 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/35250a09-3a42-4c8a-b8ea-8d87c2602dcd-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-649d554676-clrbv\" (UID: \"35250a09-3a42-4c8a-b8ea-8d87c2602dcd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-649d554676-clrbv" Jul 17 18:56:10.606649 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.606444 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bkvf\" (UniqueName: \"kubernetes.io/projected/35250a09-3a42-4c8a-b8ea-8d87c2602dcd-kube-api-access-4bkvf\") pod \"managed-serviceaccount-addon-agent-649d554676-clrbv\" (UID: \"35250a09-3a42-4c8a-b8ea-8d87c2602dcd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-649d554676-clrbv" Jul 17 18:56:10.606649 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.606624 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8ae4f97-aeeb-4e21-b036-d49cb650bd59-tmp\") pod \"klusterlet-addon-workmgr-7848588d47-zbx9t\" (UID: \"b8ae4f97-aeeb-4e21-b036-d49cb650bd59\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7848588d47-zbx9t" Jul 17 18:56:10.609338 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.609309 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b8ae4f97-aeeb-4e21-b036-d49cb650bd59-klusterlet-config\") pod \"klusterlet-addon-workmgr-7848588d47-zbx9t\" (UID: \"b8ae4f97-aeeb-4e21-b036-d49cb650bd59\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7848588d47-zbx9t" Jul 17 18:56:10.609604 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.609581 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/35250a09-3a42-4c8a-b8ea-8d87c2602dcd-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-649d554676-clrbv\" (UID: \"35250a09-3a42-4c8a-b8ea-8d87c2602dcd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-649d554676-clrbv" Jul 17 18:56:10.616958 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.616933 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpg65\" (UniqueName: \"kubernetes.io/projected/b8ae4f97-aeeb-4e21-b036-d49cb650bd59-kube-api-access-fpg65\") pod \"klusterlet-addon-workmgr-7848588d47-zbx9t\" (UID: \"b8ae4f97-aeeb-4e21-b036-d49cb650bd59\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7848588d47-zbx9t" Jul 17 18:56:10.617322 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.617295 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bkvf\" (UniqueName: \"kubernetes.io/projected/35250a09-3a42-4c8a-b8ea-8d87c2602dcd-kube-api-access-4bkvf\") pod \"managed-serviceaccount-addon-agent-649d554676-clrbv\" (UID: \"35250a09-3a42-4c8a-b8ea-8d87c2602dcd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-649d554676-clrbv" Jul 17 18:56:10.707044 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.706960 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/24f74548-eab9-4297-acc1-03b123c45161-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-hw2f9\" (UID: \"24f74548-eab9-4297-acc1-03b123c45161\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-hw2f9" Jul 17 18:56:10.707044 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.707045 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlpmk\" (UniqueName: \"kubernetes.io/projected/67b653fc-1593-4bee-a23f-3f54197879c0-kube-api-access-vlpmk\") pod \"downloads-5bc67dcd46-xbsqk\" (UID: \"67b653fc-1593-4bee-a23f-3f54197879c0\") " pod="openshift-console/downloads-5bc67dcd46-xbsqk" Jul 17 18:56:10.707246 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.707069 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxvg6\" (UniqueName: \"kubernetes.io/projected/18235a43-e5c1-4146-876c-55c96b2cbb5c-kube-api-access-mxvg6\") pod \"migrator-7cc9c7ffd7-6p5gk\" (UID: \"18235a43-e5c1-4146-876c-55c96b2cbb5c\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-6p5gk" Jul 17 18:56:10.710038 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.710010 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/24f74548-eab9-4297-acc1-03b123c45161-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-hw2f9\" (UID: \"24f74548-eab9-4297-acc1-03b123c45161\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-hw2f9" Jul 17 18:56:10.715927 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.715900 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlpmk\" (UniqueName: \"kubernetes.io/projected/67b653fc-1593-4bee-a23f-3f54197879c0-kube-api-access-vlpmk\") pod \"downloads-5bc67dcd46-xbsqk\" (UID: \"67b653fc-1593-4bee-a23f-3f54197879c0\") " pod="openshift-console/downloads-5bc67dcd46-xbsqk" Jul 17 18:56:10.716050 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.715963 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxvg6\" (UniqueName: \"kubernetes.io/projected/18235a43-e5c1-4146-876c-55c96b2cbb5c-kube-api-access-mxvg6\") pod \"migrator-7cc9c7ffd7-6p5gk\" (UID: \"18235a43-e5c1-4146-876c-55c96b2cbb5c\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-6p5gk" Jul 17 18:56:10.737932 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.737905 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-649d554676-clrbv" Jul 17 18:56:10.748357 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.748323 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7848588d47-zbx9t" Jul 17 18:56:10.806155 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.806112 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-xbsqk" Jul 17 18:56:10.819140 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.819115 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-hw2f9" Jul 17 18:56:10.825954 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.825925 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-6p5gk" Jul 17 18:56:10.893782 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.893725 2561 patch_prober.go:28] interesting pod/image-registry-d5567c6d9-bt97k container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 17 18:56:10.893980 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:10.893785 2561 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-d5567c6d9-bt97k" podUID="ec8a5e73-e34f-4c8e-b194-56371da5cb81" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 17 18:56:12.421213 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:12.421166 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/96280fd2-5cf1-4adc-a0a2-a8f54aef67a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6c6g4\" (UID: \"96280fd2-5cf1-4adc-a0a2-a8f54aef67a9\") " pod="openshift-insights/insights-runtime-extractor-6c6g4" Jul 17 18:56:12.450937 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:12.449783 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/96280fd2-5cf1-4adc-a0a2-a8f54aef67a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6c6g4\" (UID: \"96280fd2-5cf1-4adc-a0a2-a8f54aef67a9\") " pod="openshift-insights/insights-runtime-extractor-6c6g4" Jul 17 18:56:12.522776 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:12.522658 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-6p5gk"] Jul 17 18:56:12.533553 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:12.533407 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-c54gz" event={"ID":"2c91e513-4f09-4551-8f50-75a4ba131d12","Type":"ContainerStarted","Data":"db322ed60bba356d71f3a7953571070cf528bbf3af2eeb2e11981a9fe51033b8"} Jul 17 18:56:12.533826 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:12.533739 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-c54gz" Jul 17 18:56:12.535347 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:12.535226 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-m9wzh" event={"ID":"3591b9f8-a2ff-4adb-93ec-449225ca6390","Type":"ContainerStarted","Data":"75a1ccce7b16ca49a588790109e59cc3d3a1c5ed78eeff63d40713cb6fa4cdd4"} Jul 17 18:56:12.552341 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:12.552290 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-c54gz" podStartSLOduration=65.569000006 podStartE2EDuration="1m10.552273526s" totalImagesPullingTime="4.98327352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:55:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:56:12.550785773 +0000 UTC m=+70.880601980" watchObservedRunningTime="2026-07-17 18:56:12.552273526 +0000 UTC m=+70.882089773" Jul 17 18:56:12.566863 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:12.566767 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-m9wzh" podStartSLOduration=65.392537878 podStartE2EDuration="1m10.566751655s" totalImagesPullingTime="5.174213777s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:55:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:56:12.566088343 +0000 UTC m=+70.895904550" watchObservedRunningTime="2026-07-17 18:56:12.566751655 +0000 UTC m=+70.896567894" Jul 17 18:56:12.643004 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:12.642966 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-d9hlx\"" Jul 17 18:56:12.651118 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:12.651096 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6c6g4" Jul 17 18:56:12.740989 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:12.740960 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-hw2f9"] Jul 17 18:56:12.745198 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:56:12.745166 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24f74548_eab9_4297_acc1_03b123c45161.slice/crio-d2c988d7d4ee5e143d971daa41ddb36b1b6cc0e6e4a1df01b21fc359519ff263 WatchSource:0}: Error finding container d2c988d7d4ee5e143d971daa41ddb36b1b6cc0e6e4a1df01b21fc359519ff263: Status 404 returned error can't find the container with id d2c988d7d4ee5e143d971daa41ddb36b1b6cc0e6e4a1df01b21fc359519ff263 Jul 17 18:56:12.749538 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:12.749481 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-xbsqk"] Jul 17 18:56:12.753184 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:56:12.753159 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67b653fc_1593_4bee_a23f_3f54197879c0.slice/crio-1b0bfc07640471ee699b474ad886e69b1cf5544fbf750a27d4718a76c7a52962 WatchSource:0}: Error finding container 1b0bfc07640471ee699b474ad886e69b1cf5544fbf750a27d4718a76c7a52962: Status 404 returned error can't find the container with id 1b0bfc07640471ee699b474ad886e69b1cf5544fbf750a27d4718a76c7a52962 Jul 17 18:56:12.788485 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:12.788455 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6c6g4"] Jul 17 18:56:12.789245 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:12.789226 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7848588d47-zbx9t"] Jul 17 18:56:12.789920 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:12.789895 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-649d554676-clrbv"] Jul 17 18:56:12.790318 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:56:12.790291 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8ae4f97_aeeb_4e21_b036_d49cb650bd59.slice/crio-8d0bd92ee33d35ecc806d8ba692916707d124eaf3d6d263771dcccda2917fe1e WatchSource:0}: Error finding container 8d0bd92ee33d35ecc806d8ba692916707d124eaf3d6d263771dcccda2917fe1e: Status 404 returned error can't find the container with id 8d0bd92ee33d35ecc806d8ba692916707d124eaf3d6d263771dcccda2917fe1e Jul 17 18:56:12.791027 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:56:12.790989 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96280fd2_5cf1_4adc_a0a2_a8f54aef67a9.slice/crio-54cc6d239dfdb8b866c44c18561a019b279d79c64f6ece784470a06c7a2e558d WatchSource:0}: Error finding container 54cc6d239dfdb8b866c44c18561a019b279d79c64f6ece784470a06c7a2e558d: Status 404 returned error can't find the container with id 54cc6d239dfdb8b866c44c18561a019b279d79c64f6ece784470a06c7a2e558d Jul 17 18:56:12.791881 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:56:12.791859 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35250a09_3a42_4c8a_b8ea_8d87c2602dcd.slice/crio-7f4ea2fa62f0f00421360d975985010c551b2bc26b5e5143120ae47ee6622349 WatchSource:0}: Error finding container 7f4ea2fa62f0f00421360d975985010c551b2bc26b5e5143120ae47ee6622349: Status 404 returned error can't find the container with id 7f4ea2fa62f0f00421360d975985010c551b2bc26b5e5143120ae47ee6622349 Jul 17 18:56:13.447654 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:13.447620 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-d5567c6d9-bt97k" Jul 17 18:56:13.549498 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:13.549408 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6c6g4" event={"ID":"96280fd2-5cf1-4adc-a0a2-a8f54aef67a9","Type":"ContainerStarted","Data":"54cc6d239dfdb8b866c44c18561a019b279d79c64f6ece784470a06c7a2e558d"} Jul 17 18:56:13.553412 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:13.553327 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-hw2f9" event={"ID":"24f74548-eab9-4297-acc1-03b123c45161","Type":"ContainerStarted","Data":"d2c988d7d4ee5e143d971daa41ddb36b1b6cc0e6e4a1df01b21fc359519ff263"} Jul 17 18:56:13.558139 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:13.558102 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-xbsqk" event={"ID":"67b653fc-1593-4bee-a23f-3f54197879c0","Type":"ContainerStarted","Data":"1b0bfc07640471ee699b474ad886e69b1cf5544fbf750a27d4718a76c7a52962"} Jul 17 18:56:13.559903 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:13.559874 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-649d554676-clrbv" event={"ID":"35250a09-3a42-4c8a-b8ea-8d87c2602dcd","Type":"ContainerStarted","Data":"7f4ea2fa62f0f00421360d975985010c551b2bc26b5e5143120ae47ee6622349"} Jul 17 18:56:13.563334 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:13.563306 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-6p5gk" event={"ID":"18235a43-e5c1-4146-876c-55c96b2cbb5c","Type":"ContainerStarted","Data":"0a7e606f6e5932093bac0a8f9a918d830b275281fc7b0c429e033f9d38312e26"} Jul 17 18:56:13.568864 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:13.568832 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7848588d47-zbx9t" event={"ID":"b8ae4f97-aeeb-4e21-b036-d49cb650bd59","Type":"ContainerStarted","Data":"8d0bd92ee33d35ecc806d8ba692916707d124eaf3d6d263771dcccda2917fe1e"} Jul 17 18:56:22.614705 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:22.614559 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-649d554676-clrbv" event={"ID":"35250a09-3a42-4c8a-b8ea-8d87c2602dcd","Type":"ContainerStarted","Data":"f1f84e20108525ff9dc561b86264845bf25b5496be65311fcb6ab8551ec0ce1d"} Jul 17 18:56:22.617894 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:22.617855 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-6p5gk" event={"ID":"18235a43-e5c1-4146-876c-55c96b2cbb5c","Type":"ContainerStarted","Data":"b44d773c36cf724fb82916ecd678fe6891a9ef76b8ebc2975a0645f9abeb3785"} Jul 17 18:56:22.618041 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:22.617921 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-6p5gk" event={"ID":"18235a43-e5c1-4146-876c-55c96b2cbb5c","Type":"ContainerStarted","Data":"9a5d1cc3846815ab4a7f91e8c6e928b9bebafaf966fcc944353476d8467904bd"} Jul 17 18:56:22.622170 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:22.620173 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7848588d47-zbx9t" event={"ID":"b8ae4f97-aeeb-4e21-b036-d49cb650bd59","Type":"ContainerStarted","Data":"a615c49620f1c7e924839efd7c9399caeb22ce05ea8a7bab9f5deb105acec44f"} Jul 17 18:56:22.622170 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:22.620906 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7848588d47-zbx9t" Jul 17 18:56:22.623196 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:22.623035 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7848588d47-zbx9t" Jul 17 18:56:22.624156 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:22.624087 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6c6g4" event={"ID":"96280fd2-5cf1-4adc-a0a2-a8f54aef67a9","Type":"ContainerStarted","Data":"973411ae687c38ddf28990e52d44d5661aa705dc6213ca2eaff14aeb1092b248"} Jul 17 18:56:22.626494 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:22.626470 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-hw2f9" event={"ID":"24f74548-eab9-4297-acc1-03b123c45161","Type":"ContainerStarted","Data":"09d4cbb1a2e140016de06ec568f25c1e7e6911474b8afef1caf1c23ef84541ae"} Jul 17 18:56:22.628112 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:22.626795 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-hw2f9" Jul 17 18:56:22.633934 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:22.633478 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-649d554676-clrbv" podStartSLOduration=3.947548179 podStartE2EDuration="12.633459158s" totalImagesPullingTime="8.685910979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:56:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:56:22.632312426 +0000 UTC m=+80.962128635" watchObservedRunningTime="2026-07-17 18:56:22.633459158 +0000 UTC m=+80.963275367" Jul 17 18:56:22.634370 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:22.634310 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-hw2f9" Jul 17 18:56:22.648306 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:22.648247 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-hw2f9" podStartSLOduration=4.67674825 podStartE2EDuration="12.648227687s" totalImagesPullingTime="7.971479437s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:56:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:56:22.647652168 +0000 UTC m=+80.977468386" watchObservedRunningTime="2026-07-17 18:56:22.648227687 +0000 UTC m=+80.978043898" Jul 17 18:56:22.665476 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:22.665410 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7848588d47-zbx9t" podStartSLOduration=3.977939364 podStartE2EDuration="12.665390782s" totalImagesPullingTime="8.687451418s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:56:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:56:22.663765633 +0000 UTC m=+80.993581843" watchObservedRunningTime="2026-07-17 18:56:22.665390782 +0000 UTC m=+80.995206992" Jul 17 18:56:22.686636 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:22.686444 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-6p5gk" podStartSLOduration=8.768976106 podStartE2EDuration="12.68642984s" totalImagesPullingTime="3.917453734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:56:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:56:22.685225559 +0000 UTC m=+81.015041767" watchObservedRunningTime="2026-07-17 18:56:22.68642984 +0000 UTC m=+81.016246043" Jul 17 18:56:29.751264 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:29.751229 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-cfkrj"] Jul 17 18:56:29.815211 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:29.815173 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cfkrj" Jul 17 18:56:29.817997 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:29.817970 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-sj6hh\"" Jul 17 18:56:29.818687 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:29.818507 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 17 18:56:29.819030 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:29.818848 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 17 18:56:29.819030 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:29.818885 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 17 18:56:29.819178 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:29.819091 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 17 18:56:29.819478 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:29.819282 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 17 18:56:29.819478 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:29.819420 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 17 18:56:29.862230 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:29.862194 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9jt7\" (UniqueName: \"kubernetes.io/projected/7051876f-4c82-48f1-9ab9-0e2b64818bc5-kube-api-access-r9jt7\") pod \"node-exporter-cfkrj\" (UID: \"7051876f-4c82-48f1-9ab9-0e2b64818bc5\") " pod="openshift-monitoring/node-exporter-cfkrj" Jul 17 18:56:29.862392 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:29.862254 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7051876f-4c82-48f1-9ab9-0e2b64818bc5-node-exporter-tls\") pod \"node-exporter-cfkrj\" (UID: \"7051876f-4c82-48f1-9ab9-0e2b64818bc5\") " pod="openshift-monitoring/node-exporter-cfkrj" Jul 17 18:56:29.862392 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:29.862282 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7051876f-4c82-48f1-9ab9-0e2b64818bc5-node-exporter-wtmp\") pod \"node-exporter-cfkrj\" (UID: \"7051876f-4c82-48f1-9ab9-0e2b64818bc5\") " pod="openshift-monitoring/node-exporter-cfkrj" Jul 17 18:56:29.862392 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:29.862309 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7051876f-4c82-48f1-9ab9-0e2b64818bc5-sys\") pod \"node-exporter-cfkrj\" (UID: \"7051876f-4c82-48f1-9ab9-0e2b64818bc5\") " pod="openshift-monitoring/node-exporter-cfkrj" Jul 17 18:56:29.862392 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:29.862336 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7051876f-4c82-48f1-9ab9-0e2b64818bc5-node-exporter-textfile\") pod \"node-exporter-cfkrj\" (UID: \"7051876f-4c82-48f1-9ab9-0e2b64818bc5\") " pod="openshift-monitoring/node-exporter-cfkrj" Jul 17 18:56:29.862392 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:29.862362 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7051876f-4c82-48f1-9ab9-0e2b64818bc5-root\") pod \"node-exporter-cfkrj\" (UID: \"7051876f-4c82-48f1-9ab9-0e2b64818bc5\") " pod="openshift-monitoring/node-exporter-cfkrj" Jul 17 18:56:29.862604 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:29.862399 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7051876f-4c82-48f1-9ab9-0e2b64818bc5-metrics-client-ca\") pod \"node-exporter-cfkrj\" (UID: \"7051876f-4c82-48f1-9ab9-0e2b64818bc5\") " pod="openshift-monitoring/node-exporter-cfkrj" Jul 17 18:56:29.862604 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:29.862429 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7051876f-4c82-48f1-9ab9-0e2b64818bc5-node-exporter-accelerators-collector-config\") pod \"node-exporter-cfkrj\" (UID: \"7051876f-4c82-48f1-9ab9-0e2b64818bc5\") " pod="openshift-monitoring/node-exporter-cfkrj" Jul 17 18:56:29.862604 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:29.862464 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7051876f-4c82-48f1-9ab9-0e2b64818bc5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cfkrj\" (UID: \"7051876f-4c82-48f1-9ab9-0e2b64818bc5\") " pod="openshift-monitoring/node-exporter-cfkrj" Jul 17 18:56:29.962915 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:29.962854 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7051876f-4c82-48f1-9ab9-0e2b64818bc5-node-exporter-accelerators-collector-config\") pod \"node-exporter-cfkrj\" (UID: \"7051876f-4c82-48f1-9ab9-0e2b64818bc5\") " pod="openshift-monitoring/node-exporter-cfkrj" Jul 17 18:56:29.962915 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:29.962910 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7051876f-4c82-48f1-9ab9-0e2b64818bc5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cfkrj\" (UID: \"7051876f-4c82-48f1-9ab9-0e2b64818bc5\") " pod="openshift-monitoring/node-exporter-cfkrj" Jul 17 18:56:29.963109 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:29.962943 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9jt7\" (UniqueName: \"kubernetes.io/projected/7051876f-4c82-48f1-9ab9-0e2b64818bc5-kube-api-access-r9jt7\") pod \"node-exporter-cfkrj\" (UID: \"7051876f-4c82-48f1-9ab9-0e2b64818bc5\") " pod="openshift-monitoring/node-exporter-cfkrj" Jul 17 18:56:29.963109 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:29.962989 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7051876f-4c82-48f1-9ab9-0e2b64818bc5-node-exporter-tls\") pod \"node-exporter-cfkrj\" (UID: \"7051876f-4c82-48f1-9ab9-0e2b64818bc5\") " pod="openshift-monitoring/node-exporter-cfkrj" Jul 17 18:56:29.963109 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:29.963012 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7051876f-4c82-48f1-9ab9-0e2b64818bc5-node-exporter-wtmp\") pod \"node-exporter-cfkrj\" (UID: \"7051876f-4c82-48f1-9ab9-0e2b64818bc5\") " pod="openshift-monitoring/node-exporter-cfkrj" Jul 17 18:56:29.963109 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:29.963035 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7051876f-4c82-48f1-9ab9-0e2b64818bc5-sys\") pod \"node-exporter-cfkrj\" (UID: \"7051876f-4c82-48f1-9ab9-0e2b64818bc5\") " pod="openshift-monitoring/node-exporter-cfkrj" Jul 17 18:56:29.963109 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:29.963063 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7051876f-4c82-48f1-9ab9-0e2b64818bc5-node-exporter-textfile\") pod \"node-exporter-cfkrj\" (UID: \"7051876f-4c82-48f1-9ab9-0e2b64818bc5\") " pod="openshift-monitoring/node-exporter-cfkrj" Jul 17 18:56:29.963109 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:29.963092 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7051876f-4c82-48f1-9ab9-0e2b64818bc5-root\") pod \"node-exporter-cfkrj\" (UID: \"7051876f-4c82-48f1-9ab9-0e2b64818bc5\") " pod="openshift-monitoring/node-exporter-cfkrj" Jul 17 18:56:29.963281 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:29.963137 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7051876f-4c82-48f1-9ab9-0e2b64818bc5-metrics-client-ca\") pod \"node-exporter-cfkrj\" (UID: \"7051876f-4c82-48f1-9ab9-0e2b64818bc5\") " pod="openshift-monitoring/node-exporter-cfkrj" Jul 17 18:56:29.963510 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:56:29.963411 2561 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 17 18:56:29.963510 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:56:29.963483 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7051876f-4c82-48f1-9ab9-0e2b64818bc5-node-exporter-tls podName:7051876f-4c82-48f1-9ab9-0e2b64818bc5 nodeName:}" failed. No retries permitted until 2026-07-17 18:56:30.463462756 +0000 UTC m=+88.793278947 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/7051876f-4c82-48f1-9ab9-0e2b64818bc5-node-exporter-tls") pod "node-exporter-cfkrj" (UID: "7051876f-4c82-48f1-9ab9-0e2b64818bc5") : secret "node-exporter-tls" not found Jul 17 18:56:29.964657 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:29.963788 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7051876f-4c82-48f1-9ab9-0e2b64818bc5-metrics-client-ca\") pod \"node-exporter-cfkrj\" (UID: \"7051876f-4c82-48f1-9ab9-0e2b64818bc5\") " pod="openshift-monitoring/node-exporter-cfkrj" Jul 17 18:56:29.964657 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:29.963938 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7051876f-4c82-48f1-9ab9-0e2b64818bc5-node-exporter-wtmp\") pod \"node-exporter-cfkrj\" (UID: \"7051876f-4c82-48f1-9ab9-0e2b64818bc5\") " pod="openshift-monitoring/node-exporter-cfkrj" Jul 17 18:56:29.964657 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:29.963980 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7051876f-4c82-48f1-9ab9-0e2b64818bc5-sys\") pod \"node-exporter-cfkrj\" (UID: \"7051876f-4c82-48f1-9ab9-0e2b64818bc5\") " pod="openshift-monitoring/node-exporter-cfkrj" Jul 17 18:56:29.964657 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:29.964236 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7051876f-4c82-48f1-9ab9-0e2b64818bc5-node-exporter-textfile\") pod \"node-exporter-cfkrj\" (UID: \"7051876f-4c82-48f1-9ab9-0e2b64818bc5\") " pod="openshift-monitoring/node-exporter-cfkrj" Jul 17 18:56:29.964657 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:29.964243 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7051876f-4c82-48f1-9ab9-0e2b64818bc5-node-exporter-accelerators-collector-config\") pod \"node-exporter-cfkrj\" (UID: \"7051876f-4c82-48f1-9ab9-0e2b64818bc5\") " pod="openshift-monitoring/node-exporter-cfkrj" Jul 17 18:56:29.964657 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:29.964296 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7051876f-4c82-48f1-9ab9-0e2b64818bc5-root\") pod \"node-exporter-cfkrj\" (UID: \"7051876f-4c82-48f1-9ab9-0e2b64818bc5\") " pod="openshift-monitoring/node-exporter-cfkrj" Jul 17 18:56:29.971092 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:29.971062 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7051876f-4c82-48f1-9ab9-0e2b64818bc5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cfkrj\" (UID: \"7051876f-4c82-48f1-9ab9-0e2b64818bc5\") " pod="openshift-monitoring/node-exporter-cfkrj" Jul 17 18:56:29.979058 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:29.979001 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9jt7\" (UniqueName: \"kubernetes.io/projected/7051876f-4c82-48f1-9ab9-0e2b64818bc5-kube-api-access-r9jt7\") pod \"node-exporter-cfkrj\" (UID: \"7051876f-4c82-48f1-9ab9-0e2b64818bc5\") " pod="openshift-monitoring/node-exporter-cfkrj" Jul 17 18:56:30.468045 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:30.468002 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7051876f-4c82-48f1-9ab9-0e2b64818bc5-node-exporter-tls\") pod \"node-exporter-cfkrj\" (UID: \"7051876f-4c82-48f1-9ab9-0e2b64818bc5\") " pod="openshift-monitoring/node-exporter-cfkrj" Jul 17 18:56:30.470663 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:30.470637 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7051876f-4c82-48f1-9ab9-0e2b64818bc5-node-exporter-tls\") pod \"node-exporter-cfkrj\" (UID: \"7051876f-4c82-48f1-9ab9-0e2b64818bc5\") " pod="openshift-monitoring/node-exporter-cfkrj" Jul 17 18:56:30.729486 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:30.729404 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cfkrj" Jul 17 18:56:32.305430 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:32.305396 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-d5567c6d9-bt97k"] Jul 17 18:56:33.527987 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:56:33.527949 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7051876f_4c82_48f1_9ab9_0e2b64818bc5.slice/crio-150f7ea3a0ed8ecce3ddabcb135742fc6618211572cd131c4c0d5a7cf15d1d03 WatchSource:0}: Error finding container 150f7ea3a0ed8ecce3ddabcb135742fc6618211572cd131c4c0d5a7cf15d1d03: Status 404 returned error can't find the container with id 150f7ea3a0ed8ecce3ddabcb135742fc6618211572cd131c4c0d5a7cf15d1d03 Jul 17 18:56:33.673305 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:33.673273 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cfkrj" event={"ID":"7051876f-4c82-48f1-9ab9-0e2b64818bc5","Type":"ContainerStarted","Data":"150f7ea3a0ed8ecce3ddabcb135742fc6618211572cd131c4c0d5a7cf15d1d03"} Jul 17 18:56:34.680725 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:34.680679 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6c6g4" event={"ID":"96280fd2-5cf1-4adc-a0a2-a8f54aef67a9","Type":"ContainerStarted","Data":"fa3f64fbc3284eee22c99533deacc20ec036a6be50147e3b3ffd7e625ecf6659"} Jul 17 18:56:34.682727 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:34.682627 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-xbsqk" event={"ID":"67b653fc-1593-4bee-a23f-3f54197879c0","Type":"ContainerStarted","Data":"c80d7cdaec5f9fbc2377a0dadfb1c53c9d2bf58bf65103b7454bd5cc59a6a98c"} Jul 17 18:56:34.683043 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:34.682993 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5bc67dcd46-xbsqk" Jul 17 18:56:34.695624 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:34.695594 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5bc67dcd46-xbsqk" Jul 17 18:56:34.708180 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:34.708124 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5bc67dcd46-xbsqk" podStartSLOduration=3.892469885 podStartE2EDuration="24.708104005s" totalImagesPullingTime="20.81563412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:56:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:56:34.707358539 +0000 UTC m=+93.037174752" watchObservedRunningTime="2026-07-17 18:56:34.708104005 +0000 UTC m=+93.037920233" Jul 17 18:56:35.690520 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:35.689875 2561 generic.go:356] "Generic (PLEG): container finished" podID="7051876f-4c82-48f1-9ab9-0e2b64818bc5" containerID="3187a21825b617217b3e6394f38e2023c8e12471aa0c59c2f8e24c299198720b" exitCode=0 Jul 17 18:56:35.690520 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:35.690036 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cfkrj" event={"ID":"7051876f-4c82-48f1-9ab9-0e2b64818bc5","Type":"ContainerDied","Data":"3187a21825b617217b3e6394f38e2023c8e12471aa0c59c2f8e24c299198720b"} Jul 17 18:56:36.697500 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:36.697459 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cfkrj" event={"ID":"7051876f-4c82-48f1-9ab9-0e2b64818bc5","Type":"ContainerStarted","Data":"911410910c4730d10deb137e59c8519eca561bba3760d19b658995cf41e8cfca"} Jul 17 18:56:36.697970 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:36.697511 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cfkrj" event={"ID":"7051876f-4c82-48f1-9ab9-0e2b64818bc5","Type":"ContainerStarted","Data":"82f3f3106cb3c9d201f7e5d998b976b038a82d887a2c4b411581abbec27b26f4"} Jul 17 18:56:36.723407 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:36.723338 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-cfkrj" podStartSLOduration=5.978681092 podStartE2EDuration="7.723316855s" totalImagesPullingTime="1.622668502s" totalInitContainerRuntime="121.967261ms" isStatefulPod=true podCreationTimestamp="2026-07-17 18:56:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:56:36.721954165 +0000 UTC m=+95.051770375" watchObservedRunningTime="2026-07-17 18:56:36.723316855 +0000 UTC m=+95.053133072" Jul 17 18:56:39.711875 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:39.711828 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6c6g4" event={"ID":"96280fd2-5cf1-4adc-a0a2-a8f54aef67a9","Type":"ContainerStarted","Data":"c59c670ace3d578f1b30e37ca7832dc77e10a6494cddd6fb0ed121830a19e219"} Jul 17 18:56:39.729914 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:39.729852 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-6c6g4" podStartSLOduration=33.611243931 podStartE2EDuration="59.729831069s" totalImagesPullingTime="26.118587138s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:55:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:56:39.72904712 +0000 UTC m=+98.058863328" watchObservedRunningTime="2026-07-17 18:56:39.729831069 +0000 UTC m=+98.059647278" Jul 17 18:56:40.152767 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:40.152728 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-56bc97c86b-nj8fx"] Jul 17 18:56:40.174655 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:40.174620 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56bc97c86b-nj8fx"] Jul 17 18:56:40.174824 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:40.174753 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56bc97c86b-nj8fx" Jul 17 18:56:40.178111 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:40.177331 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 17 18:56:40.178111 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:40.177341 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 17 18:56:40.178111 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:40.177682 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 17 18:56:40.178111 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:40.177686 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 17 18:56:40.178111 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:40.177687 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-bdvh2\"" Jul 17 18:56:40.178111 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:40.177739 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 17 18:56:40.182384 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:40.182363 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 17 18:56:40.257213 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:40.257177 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa-console-oauth-config\") pod \"console-56bc97c86b-nj8fx\" (UID: \"ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa\") " pod="openshift-console/console-56bc97c86b-nj8fx" Jul 17 18:56:40.257397 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:40.257227 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvlb9\" (UniqueName: \"kubernetes.io/projected/ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa-kube-api-access-rvlb9\") pod \"console-56bc97c86b-nj8fx\" (UID: \"ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa\") " pod="openshift-console/console-56bc97c86b-nj8fx" Jul 17 18:56:40.257397 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:40.257254 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa-console-config\") pod \"console-56bc97c86b-nj8fx\" (UID: \"ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa\") " pod="openshift-console/console-56bc97c86b-nj8fx" Jul 17 18:56:40.257397 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:40.257341 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa-console-serving-cert\") pod \"console-56bc97c86b-nj8fx\" (UID: \"ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa\") " pod="openshift-console/console-56bc97c86b-nj8fx" Jul 17 18:56:40.257397 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:40.257383 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa-trusted-ca-bundle\") pod \"console-56bc97c86b-nj8fx\" (UID: \"ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa\") " pod="openshift-console/console-56bc97c86b-nj8fx" Jul 17 18:56:40.257658 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:40.257410 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa-service-ca\") pod \"console-56bc97c86b-nj8fx\" (UID: \"ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa\") " pod="openshift-console/console-56bc97c86b-nj8fx" Jul 17 18:56:40.257658 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:40.257439 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa-oauth-serving-cert\") pod \"console-56bc97c86b-nj8fx\" (UID: \"ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa\") " pod="openshift-console/console-56bc97c86b-nj8fx" Jul 17 18:56:40.358145 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:40.358099 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa-console-oauth-config\") pod \"console-56bc97c86b-nj8fx\" (UID: \"ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa\") " pod="openshift-console/console-56bc97c86b-nj8fx" Jul 17 18:56:40.358329 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:40.358162 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvlb9\" (UniqueName: \"kubernetes.io/projected/ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa-kube-api-access-rvlb9\") pod \"console-56bc97c86b-nj8fx\" (UID: \"ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa\") " pod="openshift-console/console-56bc97c86b-nj8fx" Jul 17 18:56:40.358329 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:40.358189 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa-console-config\") pod \"console-56bc97c86b-nj8fx\" (UID: \"ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa\") " pod="openshift-console/console-56bc97c86b-nj8fx" Jul 17 18:56:40.358329 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:40.358237 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa-console-serving-cert\") pod \"console-56bc97c86b-nj8fx\" (UID: \"ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa\") " pod="openshift-console/console-56bc97c86b-nj8fx" Jul 17 18:56:40.358495 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:40.358476 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa-trusted-ca-bundle\") pod \"console-56bc97c86b-nj8fx\" (UID: \"ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa\") " pod="openshift-console/console-56bc97c86b-nj8fx" Jul 17 18:56:40.359133 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:40.359107 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa-console-config\") pod \"console-56bc97c86b-nj8fx\" (UID: \"ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa\") " pod="openshift-console/console-56bc97c86b-nj8fx" Jul 17 18:56:40.359266 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:40.359236 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa-trusted-ca-bundle\") pod \"console-56bc97c86b-nj8fx\" (UID: \"ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa\") " pod="openshift-console/console-56bc97c86b-nj8fx" Jul 17 18:56:40.359331 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:40.359313 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa-service-ca\") pod \"console-56bc97c86b-nj8fx\" (UID: \"ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa\") " pod="openshift-console/console-56bc97c86b-nj8fx" Jul 17 18:56:40.359393 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:40.359337 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa-oauth-serving-cert\") pod \"console-56bc97c86b-nj8fx\" (UID: \"ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa\") " pod="openshift-console/console-56bc97c86b-nj8fx" Jul 17 18:56:40.359996 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:40.359977 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa-oauth-serving-cert\") pod \"console-56bc97c86b-nj8fx\" (UID: \"ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa\") " pod="openshift-console/console-56bc97c86b-nj8fx" Jul 17 18:56:40.360072 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:40.360028 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa-service-ca\") pod \"console-56bc97c86b-nj8fx\" (UID: \"ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa\") " pod="openshift-console/console-56bc97c86b-nj8fx" Jul 17 18:56:40.361380 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:40.361357 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa-console-serving-cert\") pod \"console-56bc97c86b-nj8fx\" (UID: \"ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa\") " pod="openshift-console/console-56bc97c86b-nj8fx" Jul 17 18:56:40.361479 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:40.361389 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa-console-oauth-config\") pod \"console-56bc97c86b-nj8fx\" (UID: \"ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa\") " pod="openshift-console/console-56bc97c86b-nj8fx" Jul 17 18:56:40.368273 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:40.368252 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvlb9\" (UniqueName: \"kubernetes.io/projected/ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa-kube-api-access-rvlb9\") pod \"console-56bc97c86b-nj8fx\" (UID: \"ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa\") " pod="openshift-console/console-56bc97c86b-nj8fx" Jul 17 18:56:40.485937 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:40.485845 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56bc97c86b-nj8fx" Jul 17 18:56:40.686210 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:40.686173 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56bc97c86b-nj8fx"] Jul 17 18:56:40.690507 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:56:40.690476 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec11d3bb_5bc3_4205_8a54_fdc5062cb7fa.slice/crio-fe779b12d39be901bd785b9cb51d6b78eee5cf2dd5ef4008348bfd648d021d6b WatchSource:0}: Error finding container fe779b12d39be901bd785b9cb51d6b78eee5cf2dd5ef4008348bfd648d021d6b: Status 404 returned error can't find the container with id fe779b12d39be901bd785b9cb51d6b78eee5cf2dd5ef4008348bfd648d021d6b Jul 17 18:56:40.716776 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:40.716739 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56bc97c86b-nj8fx" event={"ID":"ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa","Type":"ContainerStarted","Data":"fe779b12d39be901bd785b9cb51d6b78eee5cf2dd5ef4008348bfd648d021d6b"} Jul 17 18:56:43.572863 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:43.572829 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-c54gz" Jul 17 18:56:44.737046 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:44.737016 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56bc97c86b-nj8fx" event={"ID":"ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa","Type":"ContainerStarted","Data":"cd5ead79fe8691c2719ea4a90c9b3602e0622207b564add2a61c88d42f7d77ed"} Jul 17 18:56:44.755676 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:44.755624 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-56bc97c86b-nj8fx" podStartSLOduration=1.172523127 podStartE2EDuration="4.755608825s" totalImagesPullingTime="3.583085698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:56:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:56:44.754478991 +0000 UTC m=+103.084295199" watchObservedRunningTime="2026-07-17 18:56:44.755608825 +0000 UTC m=+103.085425032" Jul 17 18:56:50.486248 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:50.486189 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-56bc97c86b-nj8fx" Jul 17 18:56:50.486837 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:50.486292 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-56bc97c86b-nj8fx" Jul 17 18:56:50.491052 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:50.491032 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-56bc97c86b-nj8fx" Jul 17 18:56:50.762050 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:50.761975 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-56bc97c86b-nj8fx" Jul 17 18:56:57.326097 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:57.326040 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-d5567c6d9-bt97k" podUID="ec8a5e73-e34f-4c8e-b194-56371da5cb81" containerName="registry" containerID="cri-o://914fed5ff559a3ee9d153a4384f0425fba9fd02160f41180c5b03eedced841b7" gracePeriod=30 Jul 17 18:56:57.610048 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:57.610017 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d5567c6d9-bt97k" Jul 17 18:56:57.708485 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:57.708441 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ec8a5e73-e34f-4c8e-b194-56371da5cb81-installation-pull-secrets\") pod \"ec8a5e73-e34f-4c8e-b194-56371da5cb81\" (UID: \"ec8a5e73-e34f-4c8e-b194-56371da5cb81\") " Jul 17 18:56:57.708711 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:57.708518 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ec8a5e73-e34f-4c8e-b194-56371da5cb81-image-registry-private-configuration\") pod \"ec8a5e73-e34f-4c8e-b194-56371da5cb81\" (UID: \"ec8a5e73-e34f-4c8e-b194-56371da5cb81\") " Jul 17 18:56:57.708711 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:57.708575 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ec8a5e73-e34f-4c8e-b194-56371da5cb81-bound-sa-token\") pod \"ec8a5e73-e34f-4c8e-b194-56371da5cb81\" (UID: \"ec8a5e73-e34f-4c8e-b194-56371da5cb81\") " Jul 17 18:56:57.708711 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:57.708607 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ec8a5e73-e34f-4c8e-b194-56371da5cb81-trusted-ca\") pod \"ec8a5e73-e34f-4c8e-b194-56371da5cb81\" (UID: \"ec8a5e73-e34f-4c8e-b194-56371da5cb81\") " Jul 17 18:56:57.708711 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:57.708627 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ec8a5e73-e34f-4c8e-b194-56371da5cb81-registry-tls\") pod \"ec8a5e73-e34f-4c8e-b194-56371da5cb81\" (UID: \"ec8a5e73-e34f-4c8e-b194-56371da5cb81\") " Jul 17 18:56:57.708711 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:57.708654 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ec8a5e73-e34f-4c8e-b194-56371da5cb81-ca-trust-extracted\") pod \"ec8a5e73-e34f-4c8e-b194-56371da5cb81\" (UID: \"ec8a5e73-e34f-4c8e-b194-56371da5cb81\") " Jul 17 18:56:57.708711 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:57.708683 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ec8a5e73-e34f-4c8e-b194-56371da5cb81-registry-certificates\") pod \"ec8a5e73-e34f-4c8e-b194-56371da5cb81\" (UID: \"ec8a5e73-e34f-4c8e-b194-56371da5cb81\") " Jul 17 18:56:57.708996 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:57.708719 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hgg22\" (UniqueName: \"kubernetes.io/projected/ec8a5e73-e34f-4c8e-b194-56371da5cb81-kube-api-access-hgg22\") pod \"ec8a5e73-e34f-4c8e-b194-56371da5cb81\" (UID: \"ec8a5e73-e34f-4c8e-b194-56371da5cb81\") " Jul 17 18:56:57.709224 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:57.709145 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec8a5e73-e34f-4c8e-b194-56371da5cb81-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "ec8a5e73-e34f-4c8e-b194-56371da5cb81" (UID: "ec8a5e73-e34f-4c8e-b194-56371da5cb81"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:56:57.709224 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:57.709233 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec8a5e73-e34f-4c8e-b194-56371da5cb81-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "ec8a5e73-e34f-4c8e-b194-56371da5cb81" (UID: "ec8a5e73-e34f-4c8e-b194-56371da5cb81"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:56:57.711392 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:57.711351 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec8a5e73-e34f-4c8e-b194-56371da5cb81-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "ec8a5e73-e34f-4c8e-b194-56371da5cb81" (UID: "ec8a5e73-e34f-4c8e-b194-56371da5cb81"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:56:57.711512 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:57.711388 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec8a5e73-e34f-4c8e-b194-56371da5cb81-kube-api-access-hgg22" (OuterVolumeSpecName: "kube-api-access-hgg22") pod "ec8a5e73-e34f-4c8e-b194-56371da5cb81" (UID: "ec8a5e73-e34f-4c8e-b194-56371da5cb81"). InnerVolumeSpecName "kube-api-access-hgg22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:56:57.711615 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:57.711591 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec8a5e73-e34f-4c8e-b194-56371da5cb81-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "ec8a5e73-e34f-4c8e-b194-56371da5cb81" (UID: "ec8a5e73-e34f-4c8e-b194-56371da5cb81"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:56:57.711657 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:57.711625 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec8a5e73-e34f-4c8e-b194-56371da5cb81-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "ec8a5e73-e34f-4c8e-b194-56371da5cb81" (UID: "ec8a5e73-e34f-4c8e-b194-56371da5cb81"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:56:57.711847 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:57.711826 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec8a5e73-e34f-4c8e-b194-56371da5cb81-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "ec8a5e73-e34f-4c8e-b194-56371da5cb81" (UID: "ec8a5e73-e34f-4c8e-b194-56371da5cb81"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:56:57.717344 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:57.717318 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec8a5e73-e34f-4c8e-b194-56371da5cb81-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "ec8a5e73-e34f-4c8e-b194-56371da5cb81" (UID: "ec8a5e73-e34f-4c8e-b194-56371da5cb81"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:56:57.784359 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:57.784326 2561 generic.go:356] "Generic (PLEG): container finished" podID="ec8a5e73-e34f-4c8e-b194-56371da5cb81" containerID="914fed5ff559a3ee9d153a4384f0425fba9fd02160f41180c5b03eedced841b7" exitCode=0 Jul 17 18:56:57.784562 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:57.784371 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d5567c6d9-bt97k" Jul 17 18:56:57.784562 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:57.784400 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d5567c6d9-bt97k" event={"ID":"ec8a5e73-e34f-4c8e-b194-56371da5cb81","Type":"ContainerDied","Data":"914fed5ff559a3ee9d153a4384f0425fba9fd02160f41180c5b03eedced841b7"} Jul 17 18:56:57.784562 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:57.784440 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d5567c6d9-bt97k" event={"ID":"ec8a5e73-e34f-4c8e-b194-56371da5cb81","Type":"ContainerDied","Data":"2677348c5d3245460b929659384b313b57678456504532008dc57aeaa1e89d41"} Jul 17 18:56:57.784562 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:57.784458 2561 scope.go:117] "RemoveContainer" containerID="914fed5ff559a3ee9d153a4384f0425fba9fd02160f41180c5b03eedced841b7" Jul 17 18:56:57.793179 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:57.793160 2561 scope.go:117] "RemoveContainer" containerID="914fed5ff559a3ee9d153a4384f0425fba9fd02160f41180c5b03eedced841b7" Jul 17 18:56:57.793472 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:56:57.793454 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"914fed5ff559a3ee9d153a4384f0425fba9fd02160f41180c5b03eedced841b7\": container with ID starting with 914fed5ff559a3ee9d153a4384f0425fba9fd02160f41180c5b03eedced841b7 not found: ID does not exist" containerID="914fed5ff559a3ee9d153a4384f0425fba9fd02160f41180c5b03eedced841b7" Jul 17 18:56:57.793539 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:57.793483 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"914fed5ff559a3ee9d153a4384f0425fba9fd02160f41180c5b03eedced841b7"} err="failed to get container status \"914fed5ff559a3ee9d153a4384f0425fba9fd02160f41180c5b03eedced841b7\": rpc error: code = NotFound desc = could not find container \"914fed5ff559a3ee9d153a4384f0425fba9fd02160f41180c5b03eedced841b7\": container with ID starting with 914fed5ff559a3ee9d153a4384f0425fba9fd02160f41180c5b03eedced841b7 not found: ID does not exist" Jul 17 18:56:57.805633 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:57.805597 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-d5567c6d9-bt97k"] Jul 17 18:56:57.808885 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:57.808858 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-d5567c6d9-bt97k"] Jul 17 18:56:57.809143 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:57.809129 2561 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ec8a5e73-e34f-4c8e-b194-56371da5cb81-installation-pull-secrets\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 18:56:57.809187 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:57.809146 2561 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ec8a5e73-e34f-4c8e-b194-56371da5cb81-image-registry-private-configuration\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 18:56:57.809187 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:57.809156 2561 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ec8a5e73-e34f-4c8e-b194-56371da5cb81-bound-sa-token\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 18:56:57.809187 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:57.809165 2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ec8a5e73-e34f-4c8e-b194-56371da5cb81-trusted-ca\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 18:56:57.809187 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:57.809172 2561 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ec8a5e73-e34f-4c8e-b194-56371da5cb81-registry-tls\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 18:56:57.809187 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:57.809179 2561 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ec8a5e73-e34f-4c8e-b194-56371da5cb81-ca-trust-extracted\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 18:56:57.809187 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:57.809187 2561 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ec8a5e73-e34f-4c8e-b194-56371da5cb81-registry-certificates\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 18:56:57.809369 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:57.809195 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgg22\" (UniqueName: \"kubernetes.io/projected/ec8a5e73-e34f-4c8e-b194-56371da5cb81-kube-api-access-hgg22\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 18:56:58.100491 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:56:58.100450 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec8a5e73-e34f-4c8e-b194-56371da5cb81" path="/var/lib/kubelet/pods/ec8a5e73-e34f-4c8e-b194-56371da5cb81/volumes" Jul 17 18:57:52.183996 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:57:52.183959 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-58f9457988-8cnbf"] Jul 17 18:57:52.184466 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:57:52.184205 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec8a5e73-e34f-4c8e-b194-56371da5cb81" containerName="registry" Jul 17 18:57:52.184466 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:57:52.184215 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec8a5e73-e34f-4c8e-b194-56371da5cb81" containerName="registry" Jul 17 18:57:52.184466 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:57:52.184261 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec8a5e73-e34f-4c8e-b194-56371da5cb81" containerName="registry" Jul 17 18:57:52.187019 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:57:52.187001 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58f9457988-8cnbf" Jul 17 18:57:52.199571 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:57:52.199515 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58f9457988-8cnbf"] Jul 17 18:57:52.300789 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:57:52.300748 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ed9a0cc3-1335-4b5f-ba3a-85ff27da8974-service-ca\") pod \"console-58f9457988-8cnbf\" (UID: \"ed9a0cc3-1335-4b5f-ba3a-85ff27da8974\") " pod="openshift-console/console-58f9457988-8cnbf" Jul 17 18:57:52.300789 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:57:52.300789 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5kg4\" (UniqueName: \"kubernetes.io/projected/ed9a0cc3-1335-4b5f-ba3a-85ff27da8974-kube-api-access-f5kg4\") pod \"console-58f9457988-8cnbf\" (UID: \"ed9a0cc3-1335-4b5f-ba3a-85ff27da8974\") " pod="openshift-console/console-58f9457988-8cnbf" Jul 17 18:57:52.301054 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:57:52.300816 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ed9a0cc3-1335-4b5f-ba3a-85ff27da8974-console-serving-cert\") pod \"console-58f9457988-8cnbf\" (UID: \"ed9a0cc3-1335-4b5f-ba3a-85ff27da8974\") " pod="openshift-console/console-58f9457988-8cnbf" Jul 17 18:57:52.301054 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:57:52.300895 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ed9a0cc3-1335-4b5f-ba3a-85ff27da8974-console-config\") pod \"console-58f9457988-8cnbf\" (UID: \"ed9a0cc3-1335-4b5f-ba3a-85ff27da8974\") " pod="openshift-console/console-58f9457988-8cnbf" Jul 17 18:57:52.301054 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:57:52.300945 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ed9a0cc3-1335-4b5f-ba3a-85ff27da8974-oauth-serving-cert\") pod \"console-58f9457988-8cnbf\" (UID: \"ed9a0cc3-1335-4b5f-ba3a-85ff27da8974\") " pod="openshift-console/console-58f9457988-8cnbf" Jul 17 18:57:52.301054 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:57:52.300986 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed9a0cc3-1335-4b5f-ba3a-85ff27da8974-trusted-ca-bundle\") pod \"console-58f9457988-8cnbf\" (UID: \"ed9a0cc3-1335-4b5f-ba3a-85ff27da8974\") " pod="openshift-console/console-58f9457988-8cnbf" Jul 17 18:57:52.301054 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:57:52.301032 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ed9a0cc3-1335-4b5f-ba3a-85ff27da8974-console-oauth-config\") pod \"console-58f9457988-8cnbf\" (UID: \"ed9a0cc3-1335-4b5f-ba3a-85ff27da8974\") " pod="openshift-console/console-58f9457988-8cnbf" Jul 17 18:57:52.401786 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:57:52.401754 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ed9a0cc3-1335-4b5f-ba3a-85ff27da8974-console-oauth-config\") pod \"console-58f9457988-8cnbf\" (UID: \"ed9a0cc3-1335-4b5f-ba3a-85ff27da8974\") " pod="openshift-console/console-58f9457988-8cnbf" Jul 17 18:57:52.401944 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:57:52.401803 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ed9a0cc3-1335-4b5f-ba3a-85ff27da8974-service-ca\") pod \"console-58f9457988-8cnbf\" (UID: \"ed9a0cc3-1335-4b5f-ba3a-85ff27da8974\") " pod="openshift-console/console-58f9457988-8cnbf" Jul 17 18:57:52.401944 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:57:52.401907 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5kg4\" (UniqueName: \"kubernetes.io/projected/ed9a0cc3-1335-4b5f-ba3a-85ff27da8974-kube-api-access-f5kg4\") pod \"console-58f9457988-8cnbf\" (UID: \"ed9a0cc3-1335-4b5f-ba3a-85ff27da8974\") " pod="openshift-console/console-58f9457988-8cnbf" Jul 17 18:57:52.402057 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:57:52.401949 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ed9a0cc3-1335-4b5f-ba3a-85ff27da8974-console-serving-cert\") pod \"console-58f9457988-8cnbf\" (UID: \"ed9a0cc3-1335-4b5f-ba3a-85ff27da8974\") " pod="openshift-console/console-58f9457988-8cnbf" Jul 17 18:57:52.402057 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:57:52.401972 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ed9a0cc3-1335-4b5f-ba3a-85ff27da8974-console-config\") pod \"console-58f9457988-8cnbf\" (UID: \"ed9a0cc3-1335-4b5f-ba3a-85ff27da8974\") " pod="openshift-console/console-58f9457988-8cnbf" Jul 17 18:57:52.402057 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:57:52.402014 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ed9a0cc3-1335-4b5f-ba3a-85ff27da8974-oauth-serving-cert\") pod \"console-58f9457988-8cnbf\" (UID: \"ed9a0cc3-1335-4b5f-ba3a-85ff27da8974\") " pod="openshift-console/console-58f9457988-8cnbf" Jul 17 18:57:52.402057 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:57:52.402039 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed9a0cc3-1335-4b5f-ba3a-85ff27da8974-trusted-ca-bundle\") pod \"console-58f9457988-8cnbf\" (UID: \"ed9a0cc3-1335-4b5f-ba3a-85ff27da8974\") " pod="openshift-console/console-58f9457988-8cnbf" Jul 17 18:57:52.402637 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:57:52.402611 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ed9a0cc3-1335-4b5f-ba3a-85ff27da8974-console-config\") pod \"console-58f9457988-8cnbf\" (UID: \"ed9a0cc3-1335-4b5f-ba3a-85ff27da8974\") " pod="openshift-console/console-58f9457988-8cnbf" Jul 17 18:57:52.402759 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:57:52.402613 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ed9a0cc3-1335-4b5f-ba3a-85ff27da8974-service-ca\") pod \"console-58f9457988-8cnbf\" (UID: \"ed9a0cc3-1335-4b5f-ba3a-85ff27da8974\") " pod="openshift-console/console-58f9457988-8cnbf" Jul 17 18:57:52.402840 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:57:52.402821 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ed9a0cc3-1335-4b5f-ba3a-85ff27da8974-oauth-serving-cert\") pod \"console-58f9457988-8cnbf\" (UID: \"ed9a0cc3-1335-4b5f-ba3a-85ff27da8974\") " pod="openshift-console/console-58f9457988-8cnbf" Jul 17 18:57:52.403411 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:57:52.403389 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed9a0cc3-1335-4b5f-ba3a-85ff27da8974-trusted-ca-bundle\") pod \"console-58f9457988-8cnbf\" (UID: \"ed9a0cc3-1335-4b5f-ba3a-85ff27da8974\") " pod="openshift-console/console-58f9457988-8cnbf" Jul 17 18:57:52.404301 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:57:52.404284 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ed9a0cc3-1335-4b5f-ba3a-85ff27da8974-console-oauth-config\") pod \"console-58f9457988-8cnbf\" (UID: \"ed9a0cc3-1335-4b5f-ba3a-85ff27da8974\") " pod="openshift-console/console-58f9457988-8cnbf" Jul 17 18:57:52.404644 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:57:52.404623 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ed9a0cc3-1335-4b5f-ba3a-85ff27da8974-console-serving-cert\") pod \"console-58f9457988-8cnbf\" (UID: \"ed9a0cc3-1335-4b5f-ba3a-85ff27da8974\") " pod="openshift-console/console-58f9457988-8cnbf" Jul 17 18:57:52.410418 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:57:52.410395 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5kg4\" (UniqueName: \"kubernetes.io/projected/ed9a0cc3-1335-4b5f-ba3a-85ff27da8974-kube-api-access-f5kg4\") pod \"console-58f9457988-8cnbf\" (UID: \"ed9a0cc3-1335-4b5f-ba3a-85ff27da8974\") " pod="openshift-console/console-58f9457988-8cnbf" Jul 17 18:57:52.495553 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:57:52.495504 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58f9457988-8cnbf" Jul 17 18:57:52.623668 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:57:52.623639 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58f9457988-8cnbf"] Jul 17 18:57:52.626804 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:57:52.626663 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded9a0cc3_1335_4b5f_ba3a_85ff27da8974.slice/crio-cfc0b26101e44a09ed7be2011016de5fd73d8f25b5b8278031663be37b00b177 WatchSource:0}: Error finding container cfc0b26101e44a09ed7be2011016de5fd73d8f25b5b8278031663be37b00b177: Status 404 returned error can't find the container with id cfc0b26101e44a09ed7be2011016de5fd73d8f25b5b8278031663be37b00b177 Jul 17 18:57:52.967143 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:57:52.967052 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58f9457988-8cnbf" event={"ID":"ed9a0cc3-1335-4b5f-ba3a-85ff27da8974","Type":"ContainerStarted","Data":"84e9a1ba3e739fb89ba7a43c0aa7331c649445efaa38efc8a766e69bb2147f6e"} Jul 17 18:57:52.967143 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:57:52.967102 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58f9457988-8cnbf" event={"ID":"ed9a0cc3-1335-4b5f-ba3a-85ff27da8974","Type":"ContainerStarted","Data":"cfc0b26101e44a09ed7be2011016de5fd73d8f25b5b8278031663be37b00b177"} Jul 17 18:57:52.986361 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:57:52.986302 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-58f9457988-8cnbf" podStartSLOduration=0.986286547 podStartE2EDuration="986.286547ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:57:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:57:52.985207945 +0000 UTC m=+171.315024164" watchObservedRunningTime="2026-07-17 18:57:52.986286547 +0000 UTC m=+171.316102754" Jul 17 18:58:02.495675 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:02.495639 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-58f9457988-8cnbf" Jul 17 18:58:02.496207 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:02.495729 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-58f9457988-8cnbf" Jul 17 18:58:02.500798 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:02.500772 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-58f9457988-8cnbf" Jul 17 18:58:03.006121 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:03.006097 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-58f9457988-8cnbf" Jul 17 18:58:03.056752 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:03.056719 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56bc97c86b-nj8fx"] Jul 17 18:58:28.077030 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:28.076966 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-56bc97c86b-nj8fx" podUID="ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa" containerName="console" containerID="cri-o://cd5ead79fe8691c2719ea4a90c9b3602e0622207b564add2a61c88d42f7d77ed" gracePeriod=15 Jul 17 18:58:28.315034 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:28.315006 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56bc97c86b-nj8fx" Jul 17 18:58:28.359729 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:28.359648 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvlb9\" (UniqueName: \"kubernetes.io/projected/ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa-kube-api-access-rvlb9\") pod \"ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa\" (UID: \"ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa\") " Jul 17 18:58:28.359729 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:28.359684 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa-console-serving-cert\") pod \"ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa\" (UID: \"ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa\") " Jul 17 18:58:28.359729 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:28.359708 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa-console-config\") pod \"ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa\" (UID: \"ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa\") " Jul 17 18:58:28.359985 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:28.359733 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa-console-oauth-config\") pod \"ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa\" (UID: \"ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa\") " Jul 17 18:58:28.359985 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:28.359870 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa-oauth-serving-cert\") pod \"ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa\" (UID: \"ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa\") " Jul 17 18:58:28.359985 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:28.359926 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa-service-ca\") pod \"ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa\" (UID: \"ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa\") " Jul 17 18:58:28.360124 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:28.359998 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa-trusted-ca-bundle\") pod \"ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa\" (UID: \"ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa\") " Jul 17 18:58:28.360221 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:28.360194 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa-console-config" (OuterVolumeSpecName: "console-config") pod "ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa" (UID: "ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:58:28.360285 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:28.360222 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa" (UID: "ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:58:28.360451 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:28.360318 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa-service-ca" (OuterVolumeSpecName: "service-ca") pod "ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa" (UID: "ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:58:28.360560 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:28.360514 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa" (UID: "ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:58:28.362109 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:28.362089 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa" (UID: "ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:58:28.362180 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:28.362111 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa-kube-api-access-rvlb9" (OuterVolumeSpecName: "kube-api-access-rvlb9") pod "ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa" (UID: "ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa"). InnerVolumeSpecName "kube-api-access-rvlb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:58:28.362180 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:28.362155 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa" (UID: "ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:58:28.460990 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:28.460937 2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa-console-oauth-config\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 18:58:28.460990 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:28.460983 2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa-oauth-serving-cert\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 18:58:28.460990 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:28.460993 2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa-service-ca\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 18:58:28.460990 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:28.461001 2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa-trusted-ca-bundle\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 18:58:28.460990 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:28.461010 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvlb9\" (UniqueName: \"kubernetes.io/projected/ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa-kube-api-access-rvlb9\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 18:58:28.461273 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:28.461019 2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa-console-serving-cert\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 18:58:28.461273 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:28.461028 2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa-console-config\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 18:58:29.096166 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:29.096126 2561 generic.go:356] "Generic (PLEG): container finished" podID="ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa" containerID="cd5ead79fe8691c2719ea4a90c9b3602e0622207b564add2a61c88d42f7d77ed" exitCode=0 Jul 17 18:58:29.096166 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:29.096168 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56bc97c86b-nj8fx" Jul 17 18:58:29.096665 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:29.096177 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56bc97c86b-nj8fx" event={"ID":"ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa","Type":"ContainerDied","Data":"cd5ead79fe8691c2719ea4a90c9b3602e0622207b564add2a61c88d42f7d77ed"} Jul 17 18:58:29.096665 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:29.096219 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56bc97c86b-nj8fx" event={"ID":"ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa","Type":"ContainerDied","Data":"fe779b12d39be901bd785b9cb51d6b78eee5cf2dd5ef4008348bfd648d021d6b"} Jul 17 18:58:29.096665 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:29.096246 2561 scope.go:117] "RemoveContainer" containerID="cd5ead79fe8691c2719ea4a90c9b3602e0622207b564add2a61c88d42f7d77ed" Jul 17 18:58:29.108797 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:29.108778 2561 scope.go:117] "RemoveContainer" containerID="cd5ead79fe8691c2719ea4a90c9b3602e0622207b564add2a61c88d42f7d77ed" Jul 17 18:58:29.109077 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:58:29.109057 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd5ead79fe8691c2719ea4a90c9b3602e0622207b564add2a61c88d42f7d77ed\": container with ID starting with cd5ead79fe8691c2719ea4a90c9b3602e0622207b564add2a61c88d42f7d77ed not found: ID does not exist" containerID="cd5ead79fe8691c2719ea4a90c9b3602e0622207b564add2a61c88d42f7d77ed" Jul 17 18:58:29.109119 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:29.109088 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd5ead79fe8691c2719ea4a90c9b3602e0622207b564add2a61c88d42f7d77ed"} err="failed to get container status \"cd5ead79fe8691c2719ea4a90c9b3602e0622207b564add2a61c88d42f7d77ed\": rpc error: code = NotFound desc = could not find container \"cd5ead79fe8691c2719ea4a90c9b3602e0622207b564add2a61c88d42f7d77ed\": container with ID starting with cd5ead79fe8691c2719ea4a90c9b3602e0622207b564add2a61c88d42f7d77ed not found: ID does not exist" Jul 17 18:58:29.122542 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:29.122503 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56bc97c86b-nj8fx"] Jul 17 18:58:29.127201 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:29.127174 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-56bc97c86b-nj8fx"] Jul 17 18:58:30.097724 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:30.097683 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa" path="/var/lib/kubelet/pods/ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa/volumes" Jul 17 18:58:56.514177 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:56.514141 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qzg5r"] Jul 17 18:58:56.514656 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:56.514374 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa" containerName="console" Jul 17 18:58:56.514656 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:56.514385 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa" containerName="console" Jul 17 18:58:56.514656 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:56.514437 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec11d3bb-5bc3-4205-8a54-fdc5062cb7fa" containerName="console" Jul 17 18:58:56.517407 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:56.517390 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qzg5r" Jul 17 18:58:56.522394 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:56.522369 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 17 18:58:56.522537 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:56.522426 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 17 18:58:56.522537 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:56.522470 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-hwr9j\"" Jul 17 18:58:56.531240 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:56.531218 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qzg5r"] Jul 17 18:58:56.564573 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:56.564514 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9afb492f-3ce1-4550-bc9b-07cbea05d81f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qzg5r\" (UID: \"9afb492f-3ce1-4550-bc9b-07cbea05d81f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qzg5r" Jul 17 18:58:56.564726 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:56.564629 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9afb492f-3ce1-4550-bc9b-07cbea05d81f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qzg5r\" (UID: \"9afb492f-3ce1-4550-bc9b-07cbea05d81f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qzg5r" Jul 17 18:58:56.564726 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:56.564691 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc78h\" (UniqueName: \"kubernetes.io/projected/9afb492f-3ce1-4550-bc9b-07cbea05d81f-kube-api-access-rc78h\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qzg5r\" (UID: \"9afb492f-3ce1-4550-bc9b-07cbea05d81f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qzg5r" Jul 17 18:58:56.665863 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:56.665827 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc78h\" (UniqueName: \"kubernetes.io/projected/9afb492f-3ce1-4550-bc9b-07cbea05d81f-kube-api-access-rc78h\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qzg5r\" (UID: \"9afb492f-3ce1-4550-bc9b-07cbea05d81f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qzg5r" Jul 17 18:58:56.666025 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:56.665880 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9afb492f-3ce1-4550-bc9b-07cbea05d81f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qzg5r\" (UID: \"9afb492f-3ce1-4550-bc9b-07cbea05d81f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qzg5r" Jul 17 18:58:56.666025 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:56.665913 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9afb492f-3ce1-4550-bc9b-07cbea05d81f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qzg5r\" (UID: \"9afb492f-3ce1-4550-bc9b-07cbea05d81f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qzg5r" Jul 17 18:58:56.666240 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:56.666223 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9afb492f-3ce1-4550-bc9b-07cbea05d81f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qzg5r\" (UID: \"9afb492f-3ce1-4550-bc9b-07cbea05d81f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qzg5r" Jul 17 18:58:56.666314 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:56.666269 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9afb492f-3ce1-4550-bc9b-07cbea05d81f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qzg5r\" (UID: \"9afb492f-3ce1-4550-bc9b-07cbea05d81f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qzg5r" Jul 17 18:58:56.677052 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:56.677024 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc78h\" (UniqueName: \"kubernetes.io/projected/9afb492f-3ce1-4550-bc9b-07cbea05d81f-kube-api-access-rc78h\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qzg5r\" (UID: \"9afb492f-3ce1-4550-bc9b-07cbea05d81f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qzg5r" Jul 17 18:58:56.825872 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:56.825782 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qzg5r" Jul 17 18:58:56.960185 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:56.960164 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qzg5r"] Jul 17 18:58:56.962797 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:58:56.962766 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9afb492f_3ce1_4550_bc9b_07cbea05d81f.slice/crio-bf523bcb90f7905c97743fcdcbb3beca7bdfb376ae9f33266b0d96ba05ab8487 WatchSource:0}: Error finding container bf523bcb90f7905c97743fcdcbb3beca7bdfb376ae9f33266b0d96ba05ab8487: Status 404 returned error can't find the container with id bf523bcb90f7905c97743fcdcbb3beca7bdfb376ae9f33266b0d96ba05ab8487 Jul 17 18:58:57.196338 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:58:57.196243 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qzg5r" event={"ID":"9afb492f-3ce1-4550-bc9b-07cbea05d81f","Type":"ContainerStarted","Data":"bf523bcb90f7905c97743fcdcbb3beca7bdfb376ae9f33266b0d96ba05ab8487"} Jul 17 18:59:04.231333 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:04.231292 2561 generic.go:356] "Generic (PLEG): container finished" podID="9afb492f-3ce1-4550-bc9b-07cbea05d81f" containerID="211f18eb30280e69886c9e0c3ee5bf55b5d98f572bb23a6182fbae931d447e85" exitCode=0 Jul 17 18:59:04.231782 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:04.231370 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qzg5r" event={"ID":"9afb492f-3ce1-4550-bc9b-07cbea05d81f","Type":"ContainerDied","Data":"211f18eb30280e69886c9e0c3ee5bf55b5d98f572bb23a6182fbae931d447e85"} Jul 17 18:59:07.244417 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:07.244376 2561 generic.go:356] "Generic (PLEG): container finished" podID="9afb492f-3ce1-4550-bc9b-07cbea05d81f" containerID="bc9bea1280c8cb7f794104f60d9e8212b88e7b8a063070d750ab3c2cb428609c" exitCode=0 Jul 17 18:59:07.244789 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:07.244428 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qzg5r" event={"ID":"9afb492f-3ce1-4550-bc9b-07cbea05d81f","Type":"ContainerDied","Data":"bc9bea1280c8cb7f794104f60d9e8212b88e7b8a063070d750ab3c2cb428609c"} Jul 17 18:59:15.274041 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:15.274000 2561 generic.go:356] "Generic (PLEG): container finished" podID="9afb492f-3ce1-4550-bc9b-07cbea05d81f" containerID="c5158e94b0b650a2b26957fa4ed70a5b66b9da2167794c3a50d6d2b4c09a1e44" exitCode=0 Jul 17 18:59:15.274414 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:15.274059 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qzg5r" event={"ID":"9afb492f-3ce1-4550-bc9b-07cbea05d81f","Type":"ContainerDied","Data":"c5158e94b0b650a2b26957fa4ed70a5b66b9da2167794c3a50d6d2b4c09a1e44"} Jul 17 18:59:16.434727 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:16.434693 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qzg5r" Jul 17 18:59:16.521571 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:16.521505 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9afb492f-3ce1-4550-bc9b-07cbea05d81f-util\") pod \"9afb492f-3ce1-4550-bc9b-07cbea05d81f\" (UID: \"9afb492f-3ce1-4550-bc9b-07cbea05d81f\") " Jul 17 18:59:16.521766 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:16.521597 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rc78h\" (UniqueName: \"kubernetes.io/projected/9afb492f-3ce1-4550-bc9b-07cbea05d81f-kube-api-access-rc78h\") pod \"9afb492f-3ce1-4550-bc9b-07cbea05d81f\" (UID: \"9afb492f-3ce1-4550-bc9b-07cbea05d81f\") " Jul 17 18:59:16.521766 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:16.521637 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9afb492f-3ce1-4550-bc9b-07cbea05d81f-bundle\") pod \"9afb492f-3ce1-4550-bc9b-07cbea05d81f\" (UID: \"9afb492f-3ce1-4550-bc9b-07cbea05d81f\") " Jul 17 18:59:16.522283 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:16.522257 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9afb492f-3ce1-4550-bc9b-07cbea05d81f-bundle" (OuterVolumeSpecName: "bundle") pod "9afb492f-3ce1-4550-bc9b-07cbea05d81f" (UID: "9afb492f-3ce1-4550-bc9b-07cbea05d81f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:59:16.523973 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:16.523952 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9afb492f-3ce1-4550-bc9b-07cbea05d81f-kube-api-access-rc78h" (OuterVolumeSpecName: "kube-api-access-rc78h") pod "9afb492f-3ce1-4550-bc9b-07cbea05d81f" (UID: "9afb492f-3ce1-4550-bc9b-07cbea05d81f"). InnerVolumeSpecName "kube-api-access-rc78h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:59:16.526554 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:16.526519 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9afb492f-3ce1-4550-bc9b-07cbea05d81f-util" (OuterVolumeSpecName: "util") pod "9afb492f-3ce1-4550-bc9b-07cbea05d81f" (UID: "9afb492f-3ce1-4550-bc9b-07cbea05d81f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:59:16.623136 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:16.623047 2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9afb492f-3ce1-4550-bc9b-07cbea05d81f-util\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 18:59:16.623136 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:16.623081 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc78h\" (UniqueName: \"kubernetes.io/projected/9afb492f-3ce1-4550-bc9b-07cbea05d81f-kube-api-access-rc78h\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 18:59:16.623136 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:16.623091 2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9afb492f-3ce1-4550-bc9b-07cbea05d81f-bundle\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 18:59:17.283216 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:17.283183 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qzg5r" event={"ID":"9afb492f-3ce1-4550-bc9b-07cbea05d81f","Type":"ContainerDied","Data":"bf523bcb90f7905c97743fcdcbb3beca7bdfb376ae9f33266b0d96ba05ab8487"} Jul 17 18:59:17.283216 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:17.283205 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qzg5r" Jul 17 18:59:17.283416 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:17.283226 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf523bcb90f7905c97743fcdcbb3beca7bdfb376ae9f33266b0d96ba05ab8487" Jul 17 18:59:24.526028 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:24.525714 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-22v7j"] Jul 17 18:59:24.526458 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:24.526242 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9afb492f-3ce1-4550-bc9b-07cbea05d81f" containerName="extract" Jul 17 18:59:24.526458 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:24.526255 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9afb492f-3ce1-4550-bc9b-07cbea05d81f" containerName="extract" Jul 17 18:59:24.526458 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:24.526274 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9afb492f-3ce1-4550-bc9b-07cbea05d81f" containerName="pull" Jul 17 18:59:24.526458 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:24.526280 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9afb492f-3ce1-4550-bc9b-07cbea05d81f" containerName="pull" Jul 17 18:59:24.526458 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:24.526287 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9afb492f-3ce1-4550-bc9b-07cbea05d81f" containerName="util" Jul 17 18:59:24.526458 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:24.526292 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9afb492f-3ce1-4550-bc9b-07cbea05d81f" containerName="util" Jul 17 18:59:24.526458 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:24.526336 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9afb492f-3ce1-4550-bc9b-07cbea05d81f" containerName="extract" Jul 17 18:59:24.529105 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:24.529083 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-22v7j" Jul 17 18:59:24.535571 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:24.534962 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-qvzbb\"" Jul 17 18:59:24.535571 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:24.535249 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 17 18:59:24.535571 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:24.535484 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 17 18:59:24.543600 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:24.543573 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-22v7j"] Jul 17 18:59:24.687724 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:24.687693 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/783dbe17-f6ed-4cee-826a-b5f5de220a06-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-22v7j\" (UID: \"783dbe17-f6ed-4cee-826a-b5f5de220a06\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-22v7j" Jul 17 18:59:24.687932 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:24.687734 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhh9d\" (UniqueName: \"kubernetes.io/projected/783dbe17-f6ed-4cee-826a-b5f5de220a06-kube-api-access-zhh9d\") pod \"cert-manager-operator-controller-manager-55c444bf86-22v7j\" (UID: \"783dbe17-f6ed-4cee-826a-b5f5de220a06\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-22v7j" Jul 17 18:59:24.788680 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:24.788585 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/783dbe17-f6ed-4cee-826a-b5f5de220a06-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-22v7j\" (UID: \"783dbe17-f6ed-4cee-826a-b5f5de220a06\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-22v7j" Jul 17 18:59:24.788680 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:24.788630 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhh9d\" (UniqueName: \"kubernetes.io/projected/783dbe17-f6ed-4cee-826a-b5f5de220a06-kube-api-access-zhh9d\") pod \"cert-manager-operator-controller-manager-55c444bf86-22v7j\" (UID: \"783dbe17-f6ed-4cee-826a-b5f5de220a06\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-22v7j" Jul 17 18:59:24.789059 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:24.789039 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/783dbe17-f6ed-4cee-826a-b5f5de220a06-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-22v7j\" (UID: \"783dbe17-f6ed-4cee-826a-b5f5de220a06\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-22v7j" Jul 17 18:59:24.798182 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:24.798132 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhh9d\" (UniqueName: \"kubernetes.io/projected/783dbe17-f6ed-4cee-826a-b5f5de220a06-kube-api-access-zhh9d\") pod \"cert-manager-operator-controller-manager-55c444bf86-22v7j\" (UID: \"783dbe17-f6ed-4cee-826a-b5f5de220a06\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-22v7j" Jul 17 18:59:24.843106 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:24.843059 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-22v7j" Jul 17 18:59:25.010253 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:25.010216 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-22v7j"] Jul 17 18:59:25.016817 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:59:25.016788 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod783dbe17_f6ed_4cee_826a_b5f5de220a06.slice/crio-3e0ab071cf73d4689d6c58c9494b3640ee861b1571583fa4f8fe0bacdcb586e2 WatchSource:0}: Error finding container 3e0ab071cf73d4689d6c58c9494b3640ee861b1571583fa4f8fe0bacdcb586e2: Status 404 returned error can't find the container with id 3e0ab071cf73d4689d6c58c9494b3640ee861b1571583fa4f8fe0bacdcb586e2 Jul 17 18:59:25.312573 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:25.312514 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-22v7j" event={"ID":"783dbe17-f6ed-4cee-826a-b5f5de220a06","Type":"ContainerStarted","Data":"3e0ab071cf73d4689d6c58c9494b3640ee861b1571583fa4f8fe0bacdcb586e2"} Jul 17 18:59:27.322579 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:27.322546 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-22v7j" event={"ID":"783dbe17-f6ed-4cee-826a-b5f5de220a06","Type":"ContainerStarted","Data":"0099d58c24363a093ebcc67ee987f69040754f5837fdeba833f36b04d18dc7aa"} Jul 17 18:59:27.345791 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:27.345731 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-22v7j" podStartSLOduration=1.657177627 podStartE2EDuration="3.345710442s" totalImagesPullingTime="1.688532815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:59:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:59:27.344559833 +0000 UTC m=+265.674376041" watchObservedRunningTime="2026-07-17 18:59:27.345710442 +0000 UTC m=+265.675526628" Jul 17 18:59:28.417099 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:28.417063 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64fm8"] Jul 17 18:59:28.420360 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:28.420343 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64fm8" Jul 17 18:59:28.423243 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:28.423220 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 17 18:59:28.424411 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:28.424394 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-hwr9j\"" Jul 17 18:59:28.424469 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:28.424414 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 17 18:59:28.430614 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:28.430591 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64fm8"] Jul 17 18:59:28.517269 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:28.517236 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a45ef875-61e3-4fb9-9c03-992917ba0dc9-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64fm8\" (UID: \"a45ef875-61e3-4fb9-9c03-992917ba0dc9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64fm8" Jul 17 18:59:28.517451 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:28.517293 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bhjc\" (UniqueName: \"kubernetes.io/projected/a45ef875-61e3-4fb9-9c03-992917ba0dc9-kube-api-access-8bhjc\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64fm8\" (UID: \"a45ef875-61e3-4fb9-9c03-992917ba0dc9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64fm8" Jul 17 18:59:28.517451 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:28.517326 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a45ef875-61e3-4fb9-9c03-992917ba0dc9-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64fm8\" (UID: \"a45ef875-61e3-4fb9-9c03-992917ba0dc9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64fm8" Jul 17 18:59:28.618236 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:28.618203 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a45ef875-61e3-4fb9-9c03-992917ba0dc9-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64fm8\" (UID: \"a45ef875-61e3-4fb9-9c03-992917ba0dc9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64fm8" Jul 17 18:59:28.618236 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:28.618243 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bhjc\" (UniqueName: \"kubernetes.io/projected/a45ef875-61e3-4fb9-9c03-992917ba0dc9-kube-api-access-8bhjc\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64fm8\" (UID: \"a45ef875-61e3-4fb9-9c03-992917ba0dc9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64fm8" Jul 17 18:59:28.618485 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:28.618263 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a45ef875-61e3-4fb9-9c03-992917ba0dc9-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64fm8\" (UID: \"a45ef875-61e3-4fb9-9c03-992917ba0dc9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64fm8" Jul 17 18:59:28.618697 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:28.618675 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a45ef875-61e3-4fb9-9c03-992917ba0dc9-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64fm8\" (UID: \"a45ef875-61e3-4fb9-9c03-992917ba0dc9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64fm8" Jul 17 18:59:28.618697 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:28.618693 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a45ef875-61e3-4fb9-9c03-992917ba0dc9-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64fm8\" (UID: \"a45ef875-61e3-4fb9-9c03-992917ba0dc9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64fm8" Jul 17 18:59:28.636998 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:28.636964 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bhjc\" (UniqueName: \"kubernetes.io/projected/a45ef875-61e3-4fb9-9c03-992917ba0dc9-kube-api-access-8bhjc\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64fm8\" (UID: \"a45ef875-61e3-4fb9-9c03-992917ba0dc9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64fm8" Jul 17 18:59:28.730163 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:28.730074 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64fm8" Jul 17 18:59:28.864487 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:28.864462 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64fm8"] Jul 17 18:59:28.866745 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:59:28.866718 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda45ef875_61e3_4fb9_9c03_992917ba0dc9.slice/crio-7a6560b774d41301e7664ed8da2b29ab21deb472d2915362dabccf23bd6bb668 WatchSource:0}: Error finding container 7a6560b774d41301e7664ed8da2b29ab21deb472d2915362dabccf23bd6bb668: Status 404 returned error can't find the container with id 7a6560b774d41301e7664ed8da2b29ab21deb472d2915362dabccf23bd6bb668 Jul 17 18:59:29.037686 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:29.037656 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-m2drj"] Jul 17 18:59:29.040734 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:29.040710 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-m2drj" Jul 17 18:59:29.043197 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:29.043175 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-4tmlb\"" Jul 17 18:59:29.043337 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:29.043177 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 17 18:59:29.043337 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:29.043177 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 17 18:59:29.048696 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:29.048675 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-m2drj"] Jul 17 18:59:29.123034 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:29.123002 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d3e5d9b2-2d1f-485a-b598-0423e820e6e0-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-m2drj\" (UID: \"d3e5d9b2-2d1f-485a-b598-0423e820e6e0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-m2drj" Jul 17 18:59:29.123216 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:29.123055 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppg7d\" (UniqueName: \"kubernetes.io/projected/d3e5d9b2-2d1f-485a-b598-0423e820e6e0-kube-api-access-ppg7d\") pod \"cert-manager-webhook-6486ffd997-m2drj\" (UID: \"d3e5d9b2-2d1f-485a-b598-0423e820e6e0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-m2drj" Jul 17 18:59:29.223549 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:29.223489 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppg7d\" (UniqueName: \"kubernetes.io/projected/d3e5d9b2-2d1f-485a-b598-0423e820e6e0-kube-api-access-ppg7d\") pod \"cert-manager-webhook-6486ffd997-m2drj\" (UID: \"d3e5d9b2-2d1f-485a-b598-0423e820e6e0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-m2drj" Jul 17 18:59:29.223752 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:29.223599 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d3e5d9b2-2d1f-485a-b598-0423e820e6e0-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-m2drj\" (UID: \"d3e5d9b2-2d1f-485a-b598-0423e820e6e0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-m2drj" Jul 17 18:59:29.232379 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:29.232342 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d3e5d9b2-2d1f-485a-b598-0423e820e6e0-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-m2drj\" (UID: \"d3e5d9b2-2d1f-485a-b598-0423e820e6e0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-m2drj" Jul 17 18:59:29.232506 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:29.232441 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppg7d\" (UniqueName: \"kubernetes.io/projected/d3e5d9b2-2d1f-485a-b598-0423e820e6e0-kube-api-access-ppg7d\") pod \"cert-manager-webhook-6486ffd997-m2drj\" (UID: \"d3e5d9b2-2d1f-485a-b598-0423e820e6e0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-m2drj" Jul 17 18:59:29.335868 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:29.335775 2561 generic.go:356] "Generic (PLEG): container finished" podID="a45ef875-61e3-4fb9-9c03-992917ba0dc9" containerID="4b5e96719dad4cb6b53958029f6ff32fcde214e2e2c7c9366851e8b37b0c1bbe" exitCode=0 Jul 17 18:59:29.336027 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:29.335865 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64fm8" event={"ID":"a45ef875-61e3-4fb9-9c03-992917ba0dc9","Type":"ContainerDied","Data":"4b5e96719dad4cb6b53958029f6ff32fcde214e2e2c7c9366851e8b37b0c1bbe"} Jul 17 18:59:29.336027 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:29.335939 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64fm8" event={"ID":"a45ef875-61e3-4fb9-9c03-992917ba0dc9","Type":"ContainerStarted","Data":"7a6560b774d41301e7664ed8da2b29ab21deb472d2915362dabccf23bd6bb668"} Jul 17 18:59:29.350046 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:29.350009 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-m2drj" Jul 17 18:59:29.479057 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:29.478923 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-m2drj"] Jul 17 18:59:29.481486 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:59:29.481461 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3e5d9b2_2d1f_485a_b598_0423e820e6e0.slice/crio-957300b6c5c0b6849abe8f9949c1f723998af8a49d3c8ecb1b9dc2c6298213b1 WatchSource:0}: Error finding container 957300b6c5c0b6849abe8f9949c1f723998af8a49d3c8ecb1b9dc2c6298213b1: Status 404 returned error can't find the container with id 957300b6c5c0b6849abe8f9949c1f723998af8a49d3c8ecb1b9dc2c6298213b1 Jul 17 18:59:30.343077 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:30.343035 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-m2drj" event={"ID":"d3e5d9b2-2d1f-485a-b598-0423e820e6e0","Type":"ContainerStarted","Data":"957300b6c5c0b6849abe8f9949c1f723998af8a49d3c8ecb1b9dc2c6298213b1"} Jul 17 18:59:32.786137 ip-10-0-134-137 kubenswrapper[2561]: E0717 18:59:32.786105 2561 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda45ef875_61e3_4fb9_9c03_992917ba0dc9.slice/crio-15b5048227e322e211355c8e378c9d2adc54616b4486b975e48438425b69951b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda45ef875_61e3_4fb9_9c03_992917ba0dc9.slice/crio-conmon-15b5048227e322e211355c8e378c9d2adc54616b4486b975e48438425b69951b.scope\": RecentStats: unable to find data in memory cache]" Jul 17 18:59:33.357724 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:33.357690 2561 generic.go:356] "Generic (PLEG): container finished" podID="a45ef875-61e3-4fb9-9c03-992917ba0dc9" containerID="15b5048227e322e211355c8e378c9d2adc54616b4486b975e48438425b69951b" exitCode=0 Jul 17 18:59:33.357980 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:33.357774 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64fm8" event={"ID":"a45ef875-61e3-4fb9-9c03-992917ba0dc9","Type":"ContainerDied","Data":"15b5048227e322e211355c8e378c9d2adc54616b4486b975e48438425b69951b"} Jul 17 18:59:33.359268 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:33.359243 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-m2drj" event={"ID":"d3e5d9b2-2d1f-485a-b598-0423e820e6e0","Type":"ContainerStarted","Data":"c0c16c1194adb19184665a91277de9f4d6df3ba3265c3071a82fd573ece279b5"} Jul 17 18:59:33.359359 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:33.359294 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-m2drj" Jul 17 18:59:33.399161 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:33.399117 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-m2drj" podStartSLOduration=1.239321305 podStartE2EDuration="4.399101178s" totalImagesPullingTime="3.159779873s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:59:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:59:33.398461427 +0000 UTC m=+271.728277635" watchObservedRunningTime="2026-07-17 18:59:33.399101178 +0000 UTC m=+271.728917377" Jul 17 18:59:34.364879 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:34.364845 2561 generic.go:356] "Generic (PLEG): container finished" podID="a45ef875-61e3-4fb9-9c03-992917ba0dc9" containerID="1f3c63483f0a3ef323950e37c00c011b19a2a40ebde0c7b971d6298203f84503" exitCode=0 Jul 17 18:59:34.365249 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:34.364931 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64fm8" event={"ID":"a45ef875-61e3-4fb9-9c03-992917ba0dc9","Type":"ContainerDied","Data":"1f3c63483f0a3ef323950e37c00c011b19a2a40ebde0c7b971d6298203f84503"} Jul 17 18:59:35.492272 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:35.492245 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64fm8" Jul 17 18:59:35.677358 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:35.677262 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8bhjc\" (UniqueName: \"kubernetes.io/projected/a45ef875-61e3-4fb9-9c03-992917ba0dc9-kube-api-access-8bhjc\") pod \"a45ef875-61e3-4fb9-9c03-992917ba0dc9\" (UID: \"a45ef875-61e3-4fb9-9c03-992917ba0dc9\") " Jul 17 18:59:35.677358 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:35.677313 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a45ef875-61e3-4fb9-9c03-992917ba0dc9-bundle\") pod \"a45ef875-61e3-4fb9-9c03-992917ba0dc9\" (UID: \"a45ef875-61e3-4fb9-9c03-992917ba0dc9\") " Jul 17 18:59:35.677611 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:35.677395 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a45ef875-61e3-4fb9-9c03-992917ba0dc9-util\") pod \"a45ef875-61e3-4fb9-9c03-992917ba0dc9\" (UID: \"a45ef875-61e3-4fb9-9c03-992917ba0dc9\") " Jul 17 18:59:35.677775 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:35.677744 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a45ef875-61e3-4fb9-9c03-992917ba0dc9-bundle" (OuterVolumeSpecName: "bundle") pod "a45ef875-61e3-4fb9-9c03-992917ba0dc9" (UID: "a45ef875-61e3-4fb9-9c03-992917ba0dc9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:59:35.679635 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:35.679605 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a45ef875-61e3-4fb9-9c03-992917ba0dc9-kube-api-access-8bhjc" (OuterVolumeSpecName: "kube-api-access-8bhjc") pod "a45ef875-61e3-4fb9-9c03-992917ba0dc9" (UID: "a45ef875-61e3-4fb9-9c03-992917ba0dc9"). InnerVolumeSpecName "kube-api-access-8bhjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:59:35.681627 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:35.681596 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a45ef875-61e3-4fb9-9c03-992917ba0dc9-util" (OuterVolumeSpecName: "util") pod "a45ef875-61e3-4fb9-9c03-992917ba0dc9" (UID: "a45ef875-61e3-4fb9-9c03-992917ba0dc9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:59:35.778618 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:35.778579 2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a45ef875-61e3-4fb9-9c03-992917ba0dc9-util\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 18:59:35.778618 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:35.778611 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bhjc\" (UniqueName: \"kubernetes.io/projected/a45ef875-61e3-4fb9-9c03-992917ba0dc9-kube-api-access-8bhjc\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 18:59:35.778618 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:35.778621 2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a45ef875-61e3-4fb9-9c03-992917ba0dc9-bundle\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 18:59:36.374870 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:36.374778 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64fm8" event={"ID":"a45ef875-61e3-4fb9-9c03-992917ba0dc9","Type":"ContainerDied","Data":"7a6560b774d41301e7664ed8da2b29ab21deb472d2915362dabccf23bd6bb668"} Jul 17 18:59:36.374870 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:36.374804 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64fm8" Jul 17 18:59:36.374870 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:36.374835 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a6560b774d41301e7664ed8da2b29ab21deb472d2915362dabccf23bd6bb668" Jul 17 18:59:39.367797 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:39.367769 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-m2drj" Jul 17 18:59:39.834654 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:39.834616 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-ct8hj"] Jul 17 18:59:39.834880 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:39.834869 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a45ef875-61e3-4fb9-9c03-992917ba0dc9" containerName="util" Jul 17 18:59:39.834937 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:39.834881 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a45ef875-61e3-4fb9-9c03-992917ba0dc9" containerName="util" Jul 17 18:59:39.834937 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:39.834889 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a45ef875-61e3-4fb9-9c03-992917ba0dc9" containerName="pull" Jul 17 18:59:39.834937 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:39.834895 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a45ef875-61e3-4fb9-9c03-992917ba0dc9" containerName="pull" Jul 17 18:59:39.834937 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:39.834906 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a45ef875-61e3-4fb9-9c03-992917ba0dc9" containerName="extract" Jul 17 18:59:39.834937 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:39.834912 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a45ef875-61e3-4fb9-9c03-992917ba0dc9" containerName="extract" Jul 17 18:59:39.835078 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:39.834955 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a45ef875-61e3-4fb9-9c03-992917ba0dc9" containerName="extract" Jul 17 18:59:39.839303 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:39.839287 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-ct8hj" Jul 17 18:59:39.841598 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:39.841565 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-gbvrj\"" Jul 17 18:59:39.848263 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:39.848235 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-ct8hj"] Jul 17 18:59:40.007247 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:40.007195 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/69785b05-3ca4-4b4e-8480-0c0449ff70a5-bound-sa-token\") pod \"cert-manager-55b4d954d5-ct8hj\" (UID: \"69785b05-3ca4-4b4e-8480-0c0449ff70a5\") " pod="cert-manager/cert-manager-55b4d954d5-ct8hj" Jul 17 18:59:40.007247 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:40.007246 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx76q\" (UniqueName: \"kubernetes.io/projected/69785b05-3ca4-4b4e-8480-0c0449ff70a5-kube-api-access-tx76q\") pod \"cert-manager-55b4d954d5-ct8hj\" (UID: \"69785b05-3ca4-4b4e-8480-0c0449ff70a5\") " pod="cert-manager/cert-manager-55b4d954d5-ct8hj" Jul 17 18:59:40.107897 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:40.107809 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/69785b05-3ca4-4b4e-8480-0c0449ff70a5-bound-sa-token\") pod \"cert-manager-55b4d954d5-ct8hj\" (UID: \"69785b05-3ca4-4b4e-8480-0c0449ff70a5\") " pod="cert-manager/cert-manager-55b4d954d5-ct8hj" Jul 17 18:59:40.107897 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:40.107845 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx76q\" (UniqueName: \"kubernetes.io/projected/69785b05-3ca4-4b4e-8480-0c0449ff70a5-kube-api-access-tx76q\") pod \"cert-manager-55b4d954d5-ct8hj\" (UID: \"69785b05-3ca4-4b4e-8480-0c0449ff70a5\") " pod="cert-manager/cert-manager-55b4d954d5-ct8hj" Jul 17 18:59:40.116459 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:40.116426 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/69785b05-3ca4-4b4e-8480-0c0449ff70a5-bound-sa-token\") pod \"cert-manager-55b4d954d5-ct8hj\" (UID: \"69785b05-3ca4-4b4e-8480-0c0449ff70a5\") " pod="cert-manager/cert-manager-55b4d954d5-ct8hj" Jul 17 18:59:40.116607 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:40.116574 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx76q\" (UniqueName: \"kubernetes.io/projected/69785b05-3ca4-4b4e-8480-0c0449ff70a5-kube-api-access-tx76q\") pod \"cert-manager-55b4d954d5-ct8hj\" (UID: \"69785b05-3ca4-4b4e-8480-0c0449ff70a5\") " pod="cert-manager/cert-manager-55b4d954d5-ct8hj" Jul 17 18:59:40.148379 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:40.148340 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-ct8hj" Jul 17 18:59:40.235331 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:40.235300 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-lspt7"] Jul 17 18:59:40.242228 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:40.242204 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-lspt7" Jul 17 18:59:40.244891 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:40.244857 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-7kvx9\"" Jul 17 18:59:40.246812 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:40.246784 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-lspt7"] Jul 17 18:59:40.285338 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:40.285311 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-ct8hj"] Jul 17 18:59:40.288149 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:59:40.288118 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69785b05_3ca4_4b4e_8480_0c0449ff70a5.slice/crio-03a9c0c82fc2d23ce3d1316c301df63a7b5584a99bb0f57da69524f714b260dd WatchSource:0}: Error finding container 03a9c0c82fc2d23ce3d1316c301df63a7b5584a99bb0f57da69524f714b260dd: Status 404 returned error can't find the container with id 03a9c0c82fc2d23ce3d1316c301df63a7b5584a99bb0f57da69524f714b260dd Jul 17 18:59:40.394307 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:40.394228 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-ct8hj" event={"ID":"69785b05-3ca4-4b4e-8480-0c0449ff70a5","Type":"ContainerStarted","Data":"eeb3cf571c170031a1e8ca899ac715dcee1947efaf75c0cdc8ae3a3cbb8221f3"} Jul 17 18:59:40.394307 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:40.394269 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-ct8hj" event={"ID":"69785b05-3ca4-4b4e-8480-0c0449ff70a5","Type":"ContainerStarted","Data":"03a9c0c82fc2d23ce3d1316c301df63a7b5584a99bb0f57da69524f714b260dd"} Jul 17 18:59:40.409892 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:40.409860 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f938504b-1443-4db1-8d13-5239fa3af1a5-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-lspt7\" (UID: \"f938504b-1443-4db1-8d13-5239fa3af1a5\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-lspt7" Jul 17 18:59:40.409892 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:40.409894 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjk6q\" (UniqueName: \"kubernetes.io/projected/f938504b-1443-4db1-8d13-5239fa3af1a5-kube-api-access-zjk6q\") pod \"cert-manager-cainjector-559cdd8c87-lspt7\" (UID: \"f938504b-1443-4db1-8d13-5239fa3af1a5\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-lspt7" Jul 17 18:59:40.412863 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:40.412817 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-ct8hj" podStartSLOduration=1.412802577 podStartE2EDuration="1.412802577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:59:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:59:40.411758215 +0000 UTC m=+278.741574422" watchObservedRunningTime="2026-07-17 18:59:40.412802577 +0000 UTC m=+278.742618784" Jul 17 18:59:40.510681 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:40.510637 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f938504b-1443-4db1-8d13-5239fa3af1a5-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-lspt7\" (UID: \"f938504b-1443-4db1-8d13-5239fa3af1a5\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-lspt7" Jul 17 18:59:40.510681 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:40.510681 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjk6q\" (UniqueName: \"kubernetes.io/projected/f938504b-1443-4db1-8d13-5239fa3af1a5-kube-api-access-zjk6q\") pod \"cert-manager-cainjector-559cdd8c87-lspt7\" (UID: \"f938504b-1443-4db1-8d13-5239fa3af1a5\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-lspt7" Jul 17 18:59:40.519315 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:40.519290 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f938504b-1443-4db1-8d13-5239fa3af1a5-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-lspt7\" (UID: \"f938504b-1443-4db1-8d13-5239fa3af1a5\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-lspt7" Jul 17 18:59:40.519435 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:40.519406 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjk6q\" (UniqueName: \"kubernetes.io/projected/f938504b-1443-4db1-8d13-5239fa3af1a5-kube-api-access-zjk6q\") pod \"cert-manager-cainjector-559cdd8c87-lspt7\" (UID: \"f938504b-1443-4db1-8d13-5239fa3af1a5\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-lspt7" Jul 17 18:59:40.554225 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:40.554183 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-lspt7" Jul 17 18:59:40.681332 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:40.681241 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-lspt7"] Jul 17 18:59:40.683896 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:59:40.683869 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf938504b_1443_4db1_8d13_5239fa3af1a5.slice/crio-5573ca53a7fb8924a579bd8eec93c1e5290dd2fb386bf241a02dcd237c680907 WatchSource:0}: Error finding container 5573ca53a7fb8924a579bd8eec93c1e5290dd2fb386bf241a02dcd237c680907: Status 404 returned error can't find the container with id 5573ca53a7fb8924a579bd8eec93c1e5290dd2fb386bf241a02dcd237c680907 Jul 17 18:59:41.399597 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:41.399560 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-lspt7" event={"ID":"f938504b-1443-4db1-8d13-5239fa3af1a5","Type":"ContainerStarted","Data":"0d74dfbb609df831a65729f61bf0b6e1fccbc69c6696c3947d44d166db16a152"} Jul 17 18:59:41.399597 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:41.399603 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-lspt7" event={"ID":"f938504b-1443-4db1-8d13-5239fa3af1a5","Type":"ContainerStarted","Data":"5573ca53a7fb8924a579bd8eec93c1e5290dd2fb386bf241a02dcd237c680907"} Jul 17 18:59:41.417923 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:41.417869 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-lspt7" podStartSLOduration=1.417852156 podStartE2EDuration="1.417852156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:59:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:59:41.416520207 +0000 UTC m=+279.746336414" watchObservedRunningTime="2026-07-17 18:59:41.417852156 +0000 UTC m=+279.747668364" Jul 17 18:59:57.312539 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:57.312496 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnccq2"] Jul 17 18:59:57.315800 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:57.315782 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnccq2" Jul 17 18:59:57.318314 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:57.318285 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 17 18:59:57.319130 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:57.319111 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-hwr9j\"" Jul 17 18:59:57.319233 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:57.319136 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 17 18:59:57.324895 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:57.324871 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnccq2"] Jul 17 18:59:57.326676 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:57.326655 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b596f5bd-daa2-4c0d-86ce-42ed83ccff68-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnccq2\" (UID: \"b596f5bd-daa2-4c0d-86ce-42ed83ccff68\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnccq2" Jul 17 18:59:57.326789 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:57.326705 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltwt8\" (UniqueName: \"kubernetes.io/projected/b596f5bd-daa2-4c0d-86ce-42ed83ccff68-kube-api-access-ltwt8\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnccq2\" (UID: \"b596f5bd-daa2-4c0d-86ce-42ed83ccff68\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnccq2" Jul 17 18:59:57.326846 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:57.326777 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b596f5bd-daa2-4c0d-86ce-42ed83ccff68-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnccq2\" (UID: \"b596f5bd-daa2-4c0d-86ce-42ed83ccff68\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnccq2" Jul 17 18:59:57.427132 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:57.427089 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b596f5bd-daa2-4c0d-86ce-42ed83ccff68-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnccq2\" (UID: \"b596f5bd-daa2-4c0d-86ce-42ed83ccff68\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnccq2" Jul 17 18:59:57.427132 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:57.427135 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b596f5bd-daa2-4c0d-86ce-42ed83ccff68-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnccq2\" (UID: \"b596f5bd-daa2-4c0d-86ce-42ed83ccff68\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnccq2" Jul 17 18:59:57.427357 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:57.427177 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltwt8\" (UniqueName: \"kubernetes.io/projected/b596f5bd-daa2-4c0d-86ce-42ed83ccff68-kube-api-access-ltwt8\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnccq2\" (UID: \"b596f5bd-daa2-4c0d-86ce-42ed83ccff68\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnccq2" Jul 17 18:59:57.427499 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:57.427477 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b596f5bd-daa2-4c0d-86ce-42ed83ccff68-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnccq2\" (UID: \"b596f5bd-daa2-4c0d-86ce-42ed83ccff68\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnccq2" Jul 17 18:59:57.427565 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:57.427516 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b596f5bd-daa2-4c0d-86ce-42ed83ccff68-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnccq2\" (UID: \"b596f5bd-daa2-4c0d-86ce-42ed83ccff68\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnccq2" Jul 17 18:59:57.436462 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:57.436434 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltwt8\" (UniqueName: \"kubernetes.io/projected/b596f5bd-daa2-4c0d-86ce-42ed83ccff68-kube-api-access-ltwt8\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnccq2\" (UID: \"b596f5bd-daa2-4c0d-86ce-42ed83ccff68\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnccq2" Jul 17 18:59:57.626290 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:57.626205 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnccq2" Jul 17 18:59:57.756015 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:57.755958 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnccq2"] Jul 17 18:59:57.758265 ip-10-0-134-137 kubenswrapper[2561]: W0717 18:59:57.758239 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb596f5bd_daa2_4c0d_86ce_42ed83ccff68.slice/crio-9d6b63fc81f09d5994bc0ae3fc34afe21313429f4d968e7af996122afa62700e WatchSource:0}: Error finding container 9d6b63fc81f09d5994bc0ae3fc34afe21313429f4d968e7af996122afa62700e: Status 404 returned error can't find the container with id 9d6b63fc81f09d5994bc0ae3fc34afe21313429f4d968e7af996122afa62700e Jul 17 18:59:58.467171 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:58.467136 2561 generic.go:356] "Generic (PLEG): container finished" podID="b596f5bd-daa2-4c0d-86ce-42ed83ccff68" containerID="17a3fc11eabd7839edb92d30686efdd3e3cc6da59ebdd917cf63524731577c12" exitCode=0 Jul 17 18:59:58.467543 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:58.467181 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnccq2" event={"ID":"b596f5bd-daa2-4c0d-86ce-42ed83ccff68","Type":"ContainerDied","Data":"17a3fc11eabd7839edb92d30686efdd3e3cc6da59ebdd917cf63524731577c12"} Jul 17 18:59:58.467543 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:58.467216 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnccq2" event={"ID":"b596f5bd-daa2-4c0d-86ce-42ed83ccff68","Type":"ContainerStarted","Data":"9d6b63fc81f09d5994bc0ae3fc34afe21313429f4d968e7af996122afa62700e"} Jul 17 18:59:59.473250 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:59.473154 2561 generic.go:356] "Generic (PLEG): container finished" podID="b596f5bd-daa2-4c0d-86ce-42ed83ccff68" containerID="57dca12e5b6ecab5d244daf31cf42abf4067225674f8e81e906db08d241a67c2" exitCode=0 Jul 17 18:59:59.473250 ip-10-0-134-137 kubenswrapper[2561]: I0717 18:59:59.473211 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnccq2" event={"ID":"b596f5bd-daa2-4c0d-86ce-42ed83ccff68","Type":"ContainerDied","Data":"57dca12e5b6ecab5d244daf31cf42abf4067225674f8e81e906db08d241a67c2"} Jul 17 19:00:00.479674 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:00.479638 2561 generic.go:356] "Generic (PLEG): container finished" podID="b596f5bd-daa2-4c0d-86ce-42ed83ccff68" containerID="62ca78c3f470d130668f10891aacbc740e49c051b6e79533181949a6fc3e8d4b" exitCode=0 Jul 17 19:00:00.480060 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:00.479705 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnccq2" event={"ID":"b596f5bd-daa2-4c0d-86ce-42ed83ccff68","Type":"ContainerDied","Data":"62ca78c3f470d130668f10891aacbc740e49c051b6e79533181949a6fc3e8d4b"} Jul 17 19:00:01.612941 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:01.612918 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnccq2" Jul 17 19:00:01.657428 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:01.657392 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ltwt8\" (UniqueName: \"kubernetes.io/projected/b596f5bd-daa2-4c0d-86ce-42ed83ccff68-kube-api-access-ltwt8\") pod \"b596f5bd-daa2-4c0d-86ce-42ed83ccff68\" (UID: \"b596f5bd-daa2-4c0d-86ce-42ed83ccff68\") " Jul 17 19:00:01.657627 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:01.657446 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b596f5bd-daa2-4c0d-86ce-42ed83ccff68-bundle\") pod \"b596f5bd-daa2-4c0d-86ce-42ed83ccff68\" (UID: \"b596f5bd-daa2-4c0d-86ce-42ed83ccff68\") " Jul 17 19:00:01.657627 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:01.657479 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b596f5bd-daa2-4c0d-86ce-42ed83ccff68-util\") pod \"b596f5bd-daa2-4c0d-86ce-42ed83ccff68\" (UID: \"b596f5bd-daa2-4c0d-86ce-42ed83ccff68\") " Jul 17 19:00:01.658304 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:01.658269 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b596f5bd-daa2-4c0d-86ce-42ed83ccff68-bundle" (OuterVolumeSpecName: "bundle") pod "b596f5bd-daa2-4c0d-86ce-42ed83ccff68" (UID: "b596f5bd-daa2-4c0d-86ce-42ed83ccff68"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 19:00:01.659730 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:01.659706 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b596f5bd-daa2-4c0d-86ce-42ed83ccff68-kube-api-access-ltwt8" (OuterVolumeSpecName: "kube-api-access-ltwt8") pod "b596f5bd-daa2-4c0d-86ce-42ed83ccff68" (UID: "b596f5bd-daa2-4c0d-86ce-42ed83ccff68"). InnerVolumeSpecName "kube-api-access-ltwt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 19:00:01.666398 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:01.666366 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b596f5bd-daa2-4c0d-86ce-42ed83ccff68-util" (OuterVolumeSpecName: "util") pod "b596f5bd-daa2-4c0d-86ce-42ed83ccff68" (UID: "b596f5bd-daa2-4c0d-86ce-42ed83ccff68"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 19:00:01.758163 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:01.758117 2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b596f5bd-daa2-4c0d-86ce-42ed83ccff68-bundle\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:00:01.758163 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:01.758160 2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b596f5bd-daa2-4c0d-86ce-42ed83ccff68-util\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:00:01.758163 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:01.758171 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltwt8\" (UniqueName: \"kubernetes.io/projected/b596f5bd-daa2-4c0d-86ce-42ed83ccff68-kube-api-access-ltwt8\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:00:01.983405 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:01.983372 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpjzq_4b162535-066a-471a-aee4-99c07c6e1bb4/ovn-acl-logging/0.log" Jul 17 19:00:01.986365 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:01.986339 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpjzq_4b162535-066a-471a-aee4-99c07c6e1bb4/ovn-acl-logging/0.log" Jul 17 19:00:01.987319 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:01.987295 2561 kubelet.go:1648] "Image garbage collection succeeded" Jul 17 19:00:02.489670 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:02.489641 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnccq2" Jul 17 19:00:02.489865 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:02.489633 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnccq2" event={"ID":"b596f5bd-daa2-4c0d-86ce-42ed83ccff68","Type":"ContainerDied","Data":"9d6b63fc81f09d5994bc0ae3fc34afe21313429f4d968e7af996122afa62700e"} Jul 17 19:00:02.489865 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:02.489734 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d6b63fc81f09d5994bc0ae3fc34afe21313429f4d968e7af996122afa62700e" Jul 17 19:00:13.021181 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:13.021141 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b55dc8c5f-24x2m"] Jul 17 19:00:13.021608 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:13.021423 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b596f5bd-daa2-4c0d-86ce-42ed83ccff68" containerName="extract" Jul 17 19:00:13.021608 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:13.021434 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b596f5bd-daa2-4c0d-86ce-42ed83ccff68" containerName="extract" Jul 17 19:00:13.021608 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:13.021457 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b596f5bd-daa2-4c0d-86ce-42ed83ccff68" containerName="util" Jul 17 19:00:13.021608 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:13.021464 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b596f5bd-daa2-4c0d-86ce-42ed83ccff68" containerName="util" Jul 17 19:00:13.021608 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:13.021478 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b596f5bd-daa2-4c0d-86ce-42ed83ccff68" containerName="pull" Jul 17 19:00:13.021608 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:13.021483 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b596f5bd-daa2-4c0d-86ce-42ed83ccff68" containerName="pull" Jul 17 19:00:13.021608 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:13.021549 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b596f5bd-daa2-4c0d-86ce-42ed83ccff68" containerName="extract" Jul 17 19:00:13.024755 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:13.024737 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-b55dc8c5f-24x2m" Jul 17 19:00:13.027291 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:13.027264 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 17 19:00:13.027291 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:13.027285 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 17 19:00:13.028005 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:13.027985 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 17 19:00:13.028114 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:13.028098 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-p5gcz\"" Jul 17 19:00:13.028276 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:13.028258 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 17 19:00:13.032083 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:13.032065 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b186dd68-8d8e-4f7a-9565-fed517965a70-webhook-cert\") pod \"opendatahub-operator-controller-manager-b55dc8c5f-24x2m\" (UID: \"b186dd68-8d8e-4f7a-9565-fed517965a70\") " pod="opendatahub/opendatahub-operator-controller-manager-b55dc8c5f-24x2m" Jul 17 19:00:13.032165 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:13.032098 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b186dd68-8d8e-4f7a-9565-fed517965a70-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b55dc8c5f-24x2m\" (UID: \"b186dd68-8d8e-4f7a-9565-fed517965a70\") " pod="opendatahub/opendatahub-operator-controller-manager-b55dc8c5f-24x2m" Jul 17 19:00:13.032165 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:13.032135 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b186dd68-8d8e-4f7a-9565-fed517965a70-tmp\") pod \"opendatahub-operator-controller-manager-b55dc8c5f-24x2m\" (UID: \"b186dd68-8d8e-4f7a-9565-fed517965a70\") " pod="opendatahub/opendatahub-operator-controller-manager-b55dc8c5f-24x2m" Jul 17 19:00:13.032257 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:13.032174 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b186dd68-8d8e-4f7a-9565-fed517965a70-manifests\") pod \"opendatahub-operator-controller-manager-b55dc8c5f-24x2m\" (UID: \"b186dd68-8d8e-4f7a-9565-fed517965a70\") " pod="opendatahub/opendatahub-operator-controller-manager-b55dc8c5f-24x2m" Jul 17 19:00:13.032257 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:13.032196 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6256\" (UniqueName: \"kubernetes.io/projected/b186dd68-8d8e-4f7a-9565-fed517965a70-kube-api-access-r6256\") pod \"opendatahub-operator-controller-manager-b55dc8c5f-24x2m\" (UID: \"b186dd68-8d8e-4f7a-9565-fed517965a70\") " pod="opendatahub/opendatahub-operator-controller-manager-b55dc8c5f-24x2m" Jul 17 19:00:13.046247 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:13.046218 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b55dc8c5f-24x2m"] Jul 17 19:00:13.132660 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:13.132631 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b186dd68-8d8e-4f7a-9565-fed517965a70-manifests\") pod \"opendatahub-operator-controller-manager-b55dc8c5f-24x2m\" (UID: \"b186dd68-8d8e-4f7a-9565-fed517965a70\") " pod="opendatahub/opendatahub-operator-controller-manager-b55dc8c5f-24x2m" Jul 17 19:00:13.132660 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:13.132671 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6256\" (UniqueName: \"kubernetes.io/projected/b186dd68-8d8e-4f7a-9565-fed517965a70-kube-api-access-r6256\") pod \"opendatahub-operator-controller-manager-b55dc8c5f-24x2m\" (UID: \"b186dd68-8d8e-4f7a-9565-fed517965a70\") " pod="opendatahub/opendatahub-operator-controller-manager-b55dc8c5f-24x2m" Jul 17 19:00:13.132920 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:13.132749 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b186dd68-8d8e-4f7a-9565-fed517965a70-webhook-cert\") pod \"opendatahub-operator-controller-manager-b55dc8c5f-24x2m\" (UID: \"b186dd68-8d8e-4f7a-9565-fed517965a70\") " pod="opendatahub/opendatahub-operator-controller-manager-b55dc8c5f-24x2m" Jul 17 19:00:13.132920 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:13.132782 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b186dd68-8d8e-4f7a-9565-fed517965a70-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b55dc8c5f-24x2m\" (UID: \"b186dd68-8d8e-4f7a-9565-fed517965a70\") " pod="opendatahub/opendatahub-operator-controller-manager-b55dc8c5f-24x2m" Jul 17 19:00:13.132920 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:13.132847 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b186dd68-8d8e-4f7a-9565-fed517965a70-tmp\") pod \"opendatahub-operator-controller-manager-b55dc8c5f-24x2m\" (UID: \"b186dd68-8d8e-4f7a-9565-fed517965a70\") " pod="opendatahub/opendatahub-operator-controller-manager-b55dc8c5f-24x2m" Jul 17 19:00:13.133105 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:13.133083 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b186dd68-8d8e-4f7a-9565-fed517965a70-manifests\") pod \"opendatahub-operator-controller-manager-b55dc8c5f-24x2m\" (UID: \"b186dd68-8d8e-4f7a-9565-fed517965a70\") " pod="opendatahub/opendatahub-operator-controller-manager-b55dc8c5f-24x2m" Jul 17 19:00:13.135253 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:13.135222 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b186dd68-8d8e-4f7a-9565-fed517965a70-tmp\") pod \"opendatahub-operator-controller-manager-b55dc8c5f-24x2m\" (UID: \"b186dd68-8d8e-4f7a-9565-fed517965a70\") " pod="opendatahub/opendatahub-operator-controller-manager-b55dc8c5f-24x2m" Jul 17 19:00:13.135378 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:13.135361 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b186dd68-8d8e-4f7a-9565-fed517965a70-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b55dc8c5f-24x2m\" (UID: \"b186dd68-8d8e-4f7a-9565-fed517965a70\") " pod="opendatahub/opendatahub-operator-controller-manager-b55dc8c5f-24x2m" Jul 17 19:00:13.135449 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:13.135427 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b186dd68-8d8e-4f7a-9565-fed517965a70-webhook-cert\") pod \"opendatahub-operator-controller-manager-b55dc8c5f-24x2m\" (UID: \"b186dd68-8d8e-4f7a-9565-fed517965a70\") " pod="opendatahub/opendatahub-operator-controller-manager-b55dc8c5f-24x2m" Jul 17 19:00:13.144255 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:13.144227 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6256\" (UniqueName: \"kubernetes.io/projected/b186dd68-8d8e-4f7a-9565-fed517965a70-kube-api-access-r6256\") pod \"opendatahub-operator-controller-manager-b55dc8c5f-24x2m\" (UID: \"b186dd68-8d8e-4f7a-9565-fed517965a70\") " pod="opendatahub/opendatahub-operator-controller-manager-b55dc8c5f-24x2m" Jul 17 19:00:13.334657 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:13.334560 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-b55dc8c5f-24x2m" Jul 17 19:00:13.481809 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:13.481788 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b55dc8c5f-24x2m"] Jul 17 19:00:13.483806 ip-10-0-134-137 kubenswrapper[2561]: W0717 19:00:13.483777 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb186dd68_8d8e_4f7a_9565_fed517965a70.slice/crio-fef1c6c0f02f244ef449ca3a5053932ff13c8acda7cf3505def85b92510c7461 WatchSource:0}: Error finding container fef1c6c0f02f244ef449ca3a5053932ff13c8acda7cf3505def85b92510c7461: Status 404 returned error can't find the container with id fef1c6c0f02f244ef449ca3a5053932ff13c8acda7cf3505def85b92510c7461 Jul 17 19:00:13.485664 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:13.485648 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 19:00:13.535495 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:13.535459 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b55dc8c5f-24x2m" event={"ID":"b186dd68-8d8e-4f7a-9565-fed517965a70","Type":"ContainerStarted","Data":"fef1c6c0f02f244ef449ca3a5053932ff13c8acda7cf3505def85b92510c7461"} Jul 17 19:00:17.562071 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:17.562027 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b55dc8c5f-24x2m" event={"ID":"b186dd68-8d8e-4f7a-9565-fed517965a70","Type":"ContainerStarted","Data":"5394113c34590a2df03cbf154bb128a92295b5fcba68e1a930e7bc44ca8ca458"} Jul 17 19:00:18.567446 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:18.567413 2561 generic.go:356] "Generic (PLEG): container finished" podID="b186dd68-8d8e-4f7a-9565-fed517965a70" containerID="5394113c34590a2df03cbf154bb128a92295b5fcba68e1a930e7bc44ca8ca458" exitCode=0 Jul 17 19:00:18.567928 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:18.567481 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b55dc8c5f-24x2m" event={"ID":"b186dd68-8d8e-4f7a-9565-fed517965a70","Type":"ContainerDied","Data":"5394113c34590a2df03cbf154bb128a92295b5fcba68e1a930e7bc44ca8ca458"} Jul 17 19:00:19.574167 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:19.574131 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b55dc8c5f-24x2m" event={"ID":"b186dd68-8d8e-4f7a-9565-fed517965a70","Type":"ContainerStarted","Data":"83b69a469586d0a8a36808de69d1a54ef5aefec66650a9026cdd218c7caf3695"} Jul 17 19:00:19.574566 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:19.574272 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-b55dc8c5f-24x2m" Jul 17 19:00:19.600314 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:19.600260 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-b55dc8c5f-24x2m" podStartSLOduration=3.094971987 podStartE2EDuration="7.600245963s" totalImagesPullingTime="3.450944389s" totalInitContainerRuntime="1.054329587s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:00:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:00:19.598238552 +0000 UTC m=+317.928054760" watchObservedRunningTime="2026-07-17 19:00:19.600245963 +0000 UTC m=+317.930062170" Jul 17 19:00:30.581155 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:30.581126 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-b55dc8c5f-24x2m" Jul 17 19:00:30.823380 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:30.823348 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-64d7f9f4cf-t44gk"] Jul 17 19:00:30.828389 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:30.828368 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-64d7f9f4cf-t44gk" Jul 17 19:00:30.831384 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:30.831319 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 17 19:00:30.831384 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:30.831319 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-ksw94\"" Jul 17 19:00:30.831384 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:30.831369 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 17 19:00:30.831625 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:30.831369 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 17 19:00:30.831625 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:30.831395 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 17 19:00:30.831625 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:30.831319 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 17 19:00:30.835880 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:30.835856 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-64d7f9f4cf-t44gk"] Jul 17 19:00:30.861827 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:30.861795 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2748a3b1-0239-4a12-8181-4949c5b7aa84-metrics-cert\") pod \"lws-controller-manager-64d7f9f4cf-t44gk\" (UID: \"2748a3b1-0239-4a12-8181-4949c5b7aa84\") " pod="openshift-lws-operator/lws-controller-manager-64d7f9f4cf-t44gk" Jul 17 19:00:30.861977 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:30.861849 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2748a3b1-0239-4a12-8181-4949c5b7aa84-manager-config\") pod \"lws-controller-manager-64d7f9f4cf-t44gk\" (UID: \"2748a3b1-0239-4a12-8181-4949c5b7aa84\") " pod="openshift-lws-operator/lws-controller-manager-64d7f9f4cf-t44gk" Jul 17 19:00:30.861977 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:30.861896 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blzqj\" (UniqueName: \"kubernetes.io/projected/2748a3b1-0239-4a12-8181-4949c5b7aa84-kube-api-access-blzqj\") pod \"lws-controller-manager-64d7f9f4cf-t44gk\" (UID: \"2748a3b1-0239-4a12-8181-4949c5b7aa84\") " pod="openshift-lws-operator/lws-controller-manager-64d7f9f4cf-t44gk" Jul 17 19:00:30.861977 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:30.861914 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2748a3b1-0239-4a12-8181-4949c5b7aa84-cert\") pod \"lws-controller-manager-64d7f9f4cf-t44gk\" (UID: \"2748a3b1-0239-4a12-8181-4949c5b7aa84\") " pod="openshift-lws-operator/lws-controller-manager-64d7f9f4cf-t44gk" Jul 17 19:00:30.963111 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:30.963077 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2748a3b1-0239-4a12-8181-4949c5b7aa84-cert\") pod \"lws-controller-manager-64d7f9f4cf-t44gk\" (UID: \"2748a3b1-0239-4a12-8181-4949c5b7aa84\") " pod="openshift-lws-operator/lws-controller-manager-64d7f9f4cf-t44gk" Jul 17 19:00:30.963269 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:30.963128 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2748a3b1-0239-4a12-8181-4949c5b7aa84-metrics-cert\") pod \"lws-controller-manager-64d7f9f4cf-t44gk\" (UID: \"2748a3b1-0239-4a12-8181-4949c5b7aa84\") " pod="openshift-lws-operator/lws-controller-manager-64d7f9f4cf-t44gk" Jul 17 19:00:30.963269 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:30.963175 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2748a3b1-0239-4a12-8181-4949c5b7aa84-manager-config\") pod \"lws-controller-manager-64d7f9f4cf-t44gk\" (UID: \"2748a3b1-0239-4a12-8181-4949c5b7aa84\") " pod="openshift-lws-operator/lws-controller-manager-64d7f9f4cf-t44gk" Jul 17 19:00:30.963269 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:30.963230 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blzqj\" (UniqueName: \"kubernetes.io/projected/2748a3b1-0239-4a12-8181-4949c5b7aa84-kube-api-access-blzqj\") pod \"lws-controller-manager-64d7f9f4cf-t44gk\" (UID: \"2748a3b1-0239-4a12-8181-4949c5b7aa84\") " pod="openshift-lws-operator/lws-controller-manager-64d7f9f4cf-t44gk" Jul 17 19:00:30.963953 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:30.963927 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2748a3b1-0239-4a12-8181-4949c5b7aa84-manager-config\") pod \"lws-controller-manager-64d7f9f4cf-t44gk\" (UID: \"2748a3b1-0239-4a12-8181-4949c5b7aa84\") " pod="openshift-lws-operator/lws-controller-manager-64d7f9f4cf-t44gk" Jul 17 19:00:30.965869 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:30.965840 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2748a3b1-0239-4a12-8181-4949c5b7aa84-metrics-cert\") pod \"lws-controller-manager-64d7f9f4cf-t44gk\" (UID: \"2748a3b1-0239-4a12-8181-4949c5b7aa84\") " pod="openshift-lws-operator/lws-controller-manager-64d7f9f4cf-t44gk" Jul 17 19:00:30.966027 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:30.966007 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2748a3b1-0239-4a12-8181-4949c5b7aa84-cert\") pod \"lws-controller-manager-64d7f9f4cf-t44gk\" (UID: \"2748a3b1-0239-4a12-8181-4949c5b7aa84\") " pod="openshift-lws-operator/lws-controller-manager-64d7f9f4cf-t44gk" Jul 17 19:00:30.971768 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:30.971746 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blzqj\" (UniqueName: \"kubernetes.io/projected/2748a3b1-0239-4a12-8181-4949c5b7aa84-kube-api-access-blzqj\") pod \"lws-controller-manager-64d7f9f4cf-t44gk\" (UID: \"2748a3b1-0239-4a12-8181-4949c5b7aa84\") " pod="openshift-lws-operator/lws-controller-manager-64d7f9f4cf-t44gk" Jul 17 19:00:31.138328 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:31.138239 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-64d7f9f4cf-t44gk" Jul 17 19:00:31.278790 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:31.278759 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-64d7f9f4cf-t44gk"] Jul 17 19:00:31.280941 ip-10-0-134-137 kubenswrapper[2561]: W0717 19:00:31.280911 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2748a3b1_0239_4a12_8181_4949c5b7aa84.slice/crio-029f959c3c218ad34591c244fde39bfdbe835831ba0d3a0a58e4576a94854abc WatchSource:0}: Error finding container 029f959c3c218ad34591c244fde39bfdbe835831ba0d3a0a58e4576a94854abc: Status 404 returned error can't find the container with id 029f959c3c218ad34591c244fde39bfdbe835831ba0d3a0a58e4576a94854abc Jul 17 19:00:31.625659 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:31.625625 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-64d7f9f4cf-t44gk" event={"ID":"2748a3b1-0239-4a12-8181-4949c5b7aa84","Type":"ContainerStarted","Data":"029f959c3c218ad34591c244fde39bfdbe835831ba0d3a0a58e4576a94854abc"} Jul 17 19:00:33.642946 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:33.642909 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-64d7f9f4cf-t44gk" event={"ID":"2748a3b1-0239-4a12-8181-4949c5b7aa84","Type":"ContainerStarted","Data":"f327bcb411ac8c0016bd4d1b3a82fe643e049369a6f011b6a66ca9ea39c4d62c"} Jul 17 19:00:33.643396 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:33.643263 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-64d7f9f4cf-t44gk" Jul 17 19:00:33.715495 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:33.715428 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-64d7f9f4cf-t44gk" podStartSLOduration=2.094834758 podStartE2EDuration="3.715410641s" totalImagesPullingTime="1.620575883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:00:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:00:33.715198488 +0000 UTC m=+332.045014697" watchObservedRunningTime="2026-07-17 19:00:33.715410641 +0000 UTC m=+332.045226899" Jul 17 19:00:41.646739 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:41.646700 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kz8sh"] Jul 17 19:00:41.649933 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:41.649915 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kz8sh" Jul 17 19:00:41.655274 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:41.655241 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 17 19:00:41.655274 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:41.655263 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 17 19:00:41.655475 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:41.655292 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 17 19:00:41.655475 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:41.655252 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 17 19:00:41.655475 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:41.655327 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-sfxgm\"" Jul 17 19:00:41.655643 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:41.655629 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 17 19:00:41.656369 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:41.656051 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 17 19:00:41.666830 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:41.666805 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kz8sh"] Jul 17 19:00:41.755847 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:41.755815 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/724470c2-a23a-496e-8246-e1a73870190b-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-kz8sh\" (UID: \"724470c2-a23a-496e-8246-e1a73870190b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kz8sh" Jul 17 19:00:41.755847 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:41.755851 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/724470c2-a23a-496e-8246-e1a73870190b-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-kz8sh\" (UID: \"724470c2-a23a-496e-8246-e1a73870190b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kz8sh" Jul 17 19:00:41.756056 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:41.755874 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/724470c2-a23a-496e-8246-e1a73870190b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-kz8sh\" (UID: \"724470c2-a23a-496e-8246-e1a73870190b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kz8sh" Jul 17 19:00:41.756056 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:41.755899 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkv9c\" (UniqueName: \"kubernetes.io/projected/724470c2-a23a-496e-8246-e1a73870190b-kube-api-access-jkv9c\") pod \"istiod-openshift-gateway-94bb8fbfd-kz8sh\" (UID: \"724470c2-a23a-496e-8246-e1a73870190b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kz8sh" Jul 17 19:00:41.756056 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:41.755941 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/724470c2-a23a-496e-8246-e1a73870190b-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-kz8sh\" (UID: \"724470c2-a23a-496e-8246-e1a73870190b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kz8sh" Jul 17 19:00:41.756056 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:41.755962 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/724470c2-a23a-496e-8246-e1a73870190b-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-kz8sh\" (UID: \"724470c2-a23a-496e-8246-e1a73870190b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kz8sh" Jul 17 19:00:41.756178 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:41.756053 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/724470c2-a23a-496e-8246-e1a73870190b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-kz8sh\" (UID: \"724470c2-a23a-496e-8246-e1a73870190b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kz8sh" Jul 17 19:00:41.857203 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:41.857166 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/724470c2-a23a-496e-8246-e1a73870190b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-kz8sh\" (UID: \"724470c2-a23a-496e-8246-e1a73870190b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kz8sh" Jul 17 19:00:41.857369 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:41.857218 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/724470c2-a23a-496e-8246-e1a73870190b-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-kz8sh\" (UID: \"724470c2-a23a-496e-8246-e1a73870190b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kz8sh" Jul 17 19:00:41.857369 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:41.857242 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/724470c2-a23a-496e-8246-e1a73870190b-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-kz8sh\" (UID: \"724470c2-a23a-496e-8246-e1a73870190b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kz8sh" Jul 17 19:00:41.857369 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:41.857329 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/724470c2-a23a-496e-8246-e1a73870190b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-kz8sh\" (UID: \"724470c2-a23a-496e-8246-e1a73870190b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kz8sh" Jul 17 19:00:41.857369 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:41.857362 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkv9c\" (UniqueName: \"kubernetes.io/projected/724470c2-a23a-496e-8246-e1a73870190b-kube-api-access-jkv9c\") pod \"istiod-openshift-gateway-94bb8fbfd-kz8sh\" (UID: \"724470c2-a23a-496e-8246-e1a73870190b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kz8sh" Jul 17 19:00:41.857596 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:41.857405 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/724470c2-a23a-496e-8246-e1a73870190b-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-kz8sh\" (UID: \"724470c2-a23a-496e-8246-e1a73870190b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kz8sh" Jul 17 19:00:41.857596 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:41.857502 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/724470c2-a23a-496e-8246-e1a73870190b-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-kz8sh\" (UID: \"724470c2-a23a-496e-8246-e1a73870190b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kz8sh" Jul 17 19:00:41.857917 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:41.857891 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/724470c2-a23a-496e-8246-e1a73870190b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-kz8sh\" (UID: \"724470c2-a23a-496e-8246-e1a73870190b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kz8sh" Jul 17 19:00:41.859885 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:41.859853 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/724470c2-a23a-496e-8246-e1a73870190b-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-kz8sh\" (UID: \"724470c2-a23a-496e-8246-e1a73870190b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kz8sh" Jul 17 19:00:41.859990 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:41.859928 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/724470c2-a23a-496e-8246-e1a73870190b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-kz8sh\" (UID: \"724470c2-a23a-496e-8246-e1a73870190b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kz8sh" Jul 17 19:00:41.859990 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:41.859977 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/724470c2-a23a-496e-8246-e1a73870190b-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-kz8sh\" (UID: \"724470c2-a23a-496e-8246-e1a73870190b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kz8sh" Jul 17 19:00:41.860064 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:41.860023 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/724470c2-a23a-496e-8246-e1a73870190b-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-kz8sh\" (UID: \"724470c2-a23a-496e-8246-e1a73870190b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kz8sh" Jul 17 19:00:41.879660 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:41.879628 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/724470c2-a23a-496e-8246-e1a73870190b-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-kz8sh\" (UID: \"724470c2-a23a-496e-8246-e1a73870190b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kz8sh" Jul 17 19:00:41.879839 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:41.879820 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkv9c\" (UniqueName: \"kubernetes.io/projected/724470c2-a23a-496e-8246-e1a73870190b-kube-api-access-jkv9c\") pod \"istiod-openshift-gateway-94bb8fbfd-kz8sh\" (UID: \"724470c2-a23a-496e-8246-e1a73870190b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kz8sh" Jul 17 19:00:41.959053 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:41.958967 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kz8sh" Jul 17 19:00:42.117506 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:42.117483 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kz8sh"] Jul 17 19:00:42.119555 ip-10-0-134-137 kubenswrapper[2561]: W0717 19:00:42.119511 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod724470c2_a23a_496e_8246_e1a73870190b.slice/crio-2d551fb57a2e6c80690424a4a8c86860fd705d37963bfe0183fa1b86aec0a51f WatchSource:0}: Error finding container 2d551fb57a2e6c80690424a4a8c86860fd705d37963bfe0183fa1b86aec0a51f: Status 404 returned error can't find the container with id 2d551fb57a2e6c80690424a4a8c86860fd705d37963bfe0183fa1b86aec0a51f Jul 17 19:00:42.682336 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:42.682305 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kz8sh" event={"ID":"724470c2-a23a-496e-8246-e1a73870190b","Type":"ContainerStarted","Data":"2d551fb57a2e6c80690424a4a8c86860fd705d37963bfe0183fa1b86aec0a51f"} Jul 17 19:00:44.651258 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:44.651227 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-64d7f9f4cf-t44gk" Jul 17 19:00:45.325661 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:45.325617 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 17 19:00:45.325784 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:45.325700 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 17 19:00:45.698583 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:45.698473 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kz8sh" event={"ID":"724470c2-a23a-496e-8246-e1a73870190b","Type":"ContainerStarted","Data":"449576f768149d60ac3b9cdc9f93f8d63218e2ce8d184603746dfff26a466715"} Jul 17 19:00:45.698583 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:45.698569 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kz8sh" Jul 17 19:00:45.779409 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:45.779346 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kz8sh" podStartSLOduration=1.575451051 podStartE2EDuration="4.779330508s" totalImagesPullingTime="3.203879457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:00:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:00:45.774683545 +0000 UTC m=+344.104499753" watchObservedRunningTime="2026-07-17 19:00:45.779330508 +0000 UTC m=+344.109146716" Jul 17 19:00:46.704991 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:00:46.704964 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-kz8sh" Jul 17 19:01:49.646297 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:49.646260 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzwzv"] Jul 17 19:01:49.649677 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:49.649660 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzwzv" Jul 17 19:01:49.652459 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:49.652437 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 17 19:01:49.652568 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:49.652437 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 17 19:01:49.653400 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:49.653382 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-w799f\"" Jul 17 19:01:49.658661 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:49.658641 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzwzv"] Jul 17 19:01:49.712777 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:49.712745 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8nj9\" (UniqueName: \"kubernetes.io/projected/b4c09057-4e4c-464c-9985-05025e1e0e09-kube-api-access-v8nj9\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzwzv\" (UID: \"b4c09057-4e4c-464c-9985-05025e1e0e09\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzwzv" Jul 17 19:01:49.712981 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:49.712801 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4c09057-4e4c-464c-9985-05025e1e0e09-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzwzv\" (UID: \"b4c09057-4e4c-464c-9985-05025e1e0e09\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzwzv" Jul 17 19:01:49.712981 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:49.712888 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4c09057-4e4c-464c-9985-05025e1e0e09-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzwzv\" (UID: \"b4c09057-4e4c-464c-9985-05025e1e0e09\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzwzv" Jul 17 19:01:49.813941 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:49.813903 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4c09057-4e4c-464c-9985-05025e1e0e09-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzwzv\" (UID: \"b4c09057-4e4c-464c-9985-05025e1e0e09\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzwzv" Jul 17 19:01:49.814125 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:49.813956 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4c09057-4e4c-464c-9985-05025e1e0e09-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzwzv\" (UID: \"b4c09057-4e4c-464c-9985-05025e1e0e09\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzwzv" Jul 17 19:01:49.814125 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:49.813992 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8nj9\" (UniqueName: \"kubernetes.io/projected/b4c09057-4e4c-464c-9985-05025e1e0e09-kube-api-access-v8nj9\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzwzv\" (UID: \"b4c09057-4e4c-464c-9985-05025e1e0e09\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzwzv" Jul 17 19:01:49.814301 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:49.814279 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4c09057-4e4c-464c-9985-05025e1e0e09-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzwzv\" (UID: \"b4c09057-4e4c-464c-9985-05025e1e0e09\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzwzv" Jul 17 19:01:49.814379 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:49.814357 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4c09057-4e4c-464c-9985-05025e1e0e09-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzwzv\" (UID: \"b4c09057-4e4c-464c-9985-05025e1e0e09\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzwzv" Jul 17 19:01:49.824112 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:49.824075 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8nj9\" (UniqueName: \"kubernetes.io/projected/b4c09057-4e4c-464c-9985-05025e1e0e09-kube-api-access-v8nj9\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzwzv\" (UID: \"b4c09057-4e4c-464c-9985-05025e1e0e09\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzwzv" Jul 17 19:01:49.959567 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:49.959448 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzwzv" Jul 17 19:01:50.102259 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:50.102215 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzwzv"] Jul 17 19:01:50.106690 ip-10-0-134-137 kubenswrapper[2561]: W0717 19:01:50.106660 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4c09057_4e4c_464c_9985_05025e1e0e09.slice/crio-25c2723dbfb9b61ce364d09722c927a6babc70acf5e1790e3cc4b8c30b493234 WatchSource:0}: Error finding container 25c2723dbfb9b61ce364d09722c927a6babc70acf5e1790e3cc4b8c30b493234: Status 404 returned error can't find the container with id 25c2723dbfb9b61ce364d09722c927a6babc70acf5e1790e3cc4b8c30b493234 Jul 17 19:01:50.294028 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:50.293984 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tgnfr"] Jul 17 19:01:50.297410 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:50.297386 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tgnfr" Jul 17 19:01:50.318009 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:50.317975 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tgnfr"] Jul 17 19:01:50.419481 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:50.419413 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvml6\" (UniqueName: \"kubernetes.io/projected/3f7df721-5719-4fe3-be5c-cf1f8d92ab2c-kube-api-access-pvml6\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tgnfr\" (UID: \"3f7df721-5719-4fe3-be5c-cf1f8d92ab2c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tgnfr" Jul 17 19:01:50.419682 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:50.419511 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f7df721-5719-4fe3-be5c-cf1f8d92ab2c-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tgnfr\" (UID: \"3f7df721-5719-4fe3-be5c-cf1f8d92ab2c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tgnfr" Jul 17 19:01:50.419682 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:50.419568 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f7df721-5719-4fe3-be5c-cf1f8d92ab2c-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tgnfr\" (UID: \"3f7df721-5719-4fe3-be5c-cf1f8d92ab2c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tgnfr" Jul 17 19:01:50.520689 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:50.520655 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f7df721-5719-4fe3-be5c-cf1f8d92ab2c-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tgnfr\" (UID: \"3f7df721-5719-4fe3-be5c-cf1f8d92ab2c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tgnfr" Jul 17 19:01:50.520843 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:50.520705 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvml6\" (UniqueName: \"kubernetes.io/projected/3f7df721-5719-4fe3-be5c-cf1f8d92ab2c-kube-api-access-pvml6\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tgnfr\" (UID: \"3f7df721-5719-4fe3-be5c-cf1f8d92ab2c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tgnfr" Jul 17 19:01:50.520843 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:50.520747 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f7df721-5719-4fe3-be5c-cf1f8d92ab2c-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tgnfr\" (UID: \"3f7df721-5719-4fe3-be5c-cf1f8d92ab2c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tgnfr" Jul 17 19:01:50.521101 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:50.521080 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f7df721-5719-4fe3-be5c-cf1f8d92ab2c-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tgnfr\" (UID: \"3f7df721-5719-4fe3-be5c-cf1f8d92ab2c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tgnfr" Jul 17 19:01:50.521162 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:50.521090 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f7df721-5719-4fe3-be5c-cf1f8d92ab2c-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tgnfr\" (UID: \"3f7df721-5719-4fe3-be5c-cf1f8d92ab2c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tgnfr" Jul 17 19:01:50.530402 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:50.530371 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvml6\" (UniqueName: \"kubernetes.io/projected/3f7df721-5719-4fe3-be5c-cf1f8d92ab2c-kube-api-access-pvml6\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tgnfr\" (UID: \"3f7df721-5719-4fe3-be5c-cf1f8d92ab2c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tgnfr" Jul 17 19:01:50.606240 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:50.606208 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tgnfr" Jul 17 19:01:50.666704 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:50.666673 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nww7x"] Jul 17 19:01:50.670203 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:50.670183 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nww7x" Jul 17 19:01:50.683545 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:50.683503 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nww7x"] Jul 17 19:01:50.722411 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:50.722379 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nww7x\" (UID: \"a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nww7x" Jul 17 19:01:50.722597 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:50.722421 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hbrs\" (UniqueName: \"kubernetes.io/projected/a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499-kube-api-access-7hbrs\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nww7x\" (UID: \"a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nww7x" Jul 17 19:01:50.722597 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:50.722474 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nww7x\" (UID: \"a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nww7x" Jul 17 19:01:50.751600 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:50.751564 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tgnfr"] Jul 17 19:01:50.753231 ip-10-0-134-137 kubenswrapper[2561]: W0717 19:01:50.753204 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f7df721_5719_4fe3_be5c_cf1f8d92ab2c.slice/crio-2d793f24d65c96113480199ea5788d86072bd9bdde040623558e9bef7f5a1eb8 WatchSource:0}: Error finding container 2d793f24d65c96113480199ea5788d86072bd9bdde040623558e9bef7f5a1eb8: Status 404 returned error can't find the container with id 2d793f24d65c96113480199ea5788d86072bd9bdde040623558e9bef7f5a1eb8 Jul 17 19:01:50.823589 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:50.823552 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nww7x\" (UID: \"a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nww7x" Jul 17 19:01:50.823720 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:50.823684 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nww7x\" (UID: \"a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nww7x" Jul 17 19:01:50.823777 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:50.823735 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hbrs\" (UniqueName: \"kubernetes.io/projected/a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499-kube-api-access-7hbrs\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nww7x\" (UID: \"a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nww7x" Jul 17 19:01:50.823838 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:50.823824 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nww7x\" (UID: \"a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nww7x" Jul 17 19:01:50.824024 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:50.824001 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nww7x\" (UID: \"a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nww7x" Jul 17 19:01:50.834432 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:50.834405 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hbrs\" (UniqueName: \"kubernetes.io/projected/a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499-kube-api-access-7hbrs\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nww7x\" (UID: \"a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nww7x" Jul 17 19:01:50.979278 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:50.979236 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nww7x" Jul 17 19:01:50.982516 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:50.982022 2561 generic.go:356] "Generic (PLEG): container finished" podID="b4c09057-4e4c-464c-9985-05025e1e0e09" containerID="210e2c76167d19f7fcbdb194619b3553d58f547a80155e1a20d9a79551f27f00" exitCode=0 Jul 17 19:01:50.982516 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:50.982157 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzwzv" event={"ID":"b4c09057-4e4c-464c-9985-05025e1e0e09","Type":"ContainerDied","Data":"210e2c76167d19f7fcbdb194619b3553d58f547a80155e1a20d9a79551f27f00"} Jul 17 19:01:50.982516 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:50.982204 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzwzv" event={"ID":"b4c09057-4e4c-464c-9985-05025e1e0e09","Type":"ContainerStarted","Data":"25c2723dbfb9b61ce364d09722c927a6babc70acf5e1790e3cc4b8c30b493234"} Jul 17 19:01:50.983946 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:50.983923 2561 generic.go:356] "Generic (PLEG): container finished" podID="3f7df721-5719-4fe3-be5c-cf1f8d92ab2c" containerID="5efd1b8519422c1ab0326527d7e1ec3fa753e36a4c32b6c939666006ac0c768e" exitCode=0 Jul 17 19:01:50.984041 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:50.983977 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tgnfr" event={"ID":"3f7df721-5719-4fe3-be5c-cf1f8d92ab2c","Type":"ContainerDied","Data":"5efd1b8519422c1ab0326527d7e1ec3fa753e36a4c32b6c939666006ac0c768e"} Jul 17 19:01:50.984041 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:50.984009 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tgnfr" event={"ID":"3f7df721-5719-4fe3-be5c-cf1f8d92ab2c","Type":"ContainerStarted","Data":"2d793f24d65c96113480199ea5788d86072bd9bdde040623558e9bef7f5a1eb8"} Jul 17 19:01:51.054076 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:51.053997 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mfkhp"] Jul 17 19:01:51.058085 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:51.058067 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mfkhp" Jul 17 19:01:51.068317 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:51.068293 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mfkhp"] Jul 17 19:01:51.123110 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:51.123069 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nww7x"] Jul 17 19:01:51.126204 ip-10-0-134-137 kubenswrapper[2561]: W0717 19:01:51.126176 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda12f9ca0_8182_4b4a_b96c_d5c8a7c1e499.slice/crio-fa16426b5e77b5dbc0c664051463b49fa29a5b3e37c79628b4d21235eb6e1e8a WatchSource:0}: Error finding container fa16426b5e77b5dbc0c664051463b49fa29a5b3e37c79628b4d21235eb6e1e8a: Status 404 returned error can't find the container with id fa16426b5e77b5dbc0c664051463b49fa29a5b3e37c79628b4d21235eb6e1e8a Jul 17 19:01:51.126319 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:51.126292 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfrl6\" (UniqueName: \"kubernetes.io/projected/47ae49e7-0d98-49e5-beb7-dc7d972803b8-kube-api-access-wfrl6\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mfkhp\" (UID: \"47ae49e7-0d98-49e5-beb7-dc7d972803b8\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mfkhp" Jul 17 19:01:51.126390 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:51.126359 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/47ae49e7-0d98-49e5-beb7-dc7d972803b8-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mfkhp\" (UID: \"47ae49e7-0d98-49e5-beb7-dc7d972803b8\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mfkhp" Jul 17 19:01:51.126430 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:51.126396 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/47ae49e7-0d98-49e5-beb7-dc7d972803b8-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mfkhp\" (UID: \"47ae49e7-0d98-49e5-beb7-dc7d972803b8\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mfkhp" Jul 17 19:01:51.227963 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:51.227931 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfrl6\" (UniqueName: \"kubernetes.io/projected/47ae49e7-0d98-49e5-beb7-dc7d972803b8-kube-api-access-wfrl6\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mfkhp\" (UID: \"47ae49e7-0d98-49e5-beb7-dc7d972803b8\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mfkhp" Jul 17 19:01:51.228086 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:51.227995 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/47ae49e7-0d98-49e5-beb7-dc7d972803b8-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mfkhp\" (UID: \"47ae49e7-0d98-49e5-beb7-dc7d972803b8\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mfkhp" Jul 17 19:01:51.228086 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:51.228027 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/47ae49e7-0d98-49e5-beb7-dc7d972803b8-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mfkhp\" (UID: \"47ae49e7-0d98-49e5-beb7-dc7d972803b8\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mfkhp" Jul 17 19:01:51.228339 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:51.228319 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/47ae49e7-0d98-49e5-beb7-dc7d972803b8-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mfkhp\" (UID: \"47ae49e7-0d98-49e5-beb7-dc7d972803b8\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mfkhp" Jul 17 19:01:51.228396 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:51.228349 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/47ae49e7-0d98-49e5-beb7-dc7d972803b8-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mfkhp\" (UID: \"47ae49e7-0d98-49e5-beb7-dc7d972803b8\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mfkhp" Jul 17 19:01:51.239328 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:51.239309 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfrl6\" (UniqueName: \"kubernetes.io/projected/47ae49e7-0d98-49e5-beb7-dc7d972803b8-kube-api-access-wfrl6\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mfkhp\" (UID: \"47ae49e7-0d98-49e5-beb7-dc7d972803b8\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mfkhp" Jul 17 19:01:51.369002 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:51.368924 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mfkhp" Jul 17 19:01:51.507028 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:51.507002 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mfkhp"] Jul 17 19:01:51.508775 ip-10-0-134-137 kubenswrapper[2561]: W0717 19:01:51.508754 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47ae49e7_0d98_49e5_beb7_dc7d972803b8.slice/crio-e4267b462eb40c39be21e8f8155ba56ba1f471f62c176cb1f697a3042760fcdf WatchSource:0}: Error finding container e4267b462eb40c39be21e8f8155ba56ba1f471f62c176cb1f697a3042760fcdf: Status 404 returned error can't find the container with id e4267b462eb40c39be21e8f8155ba56ba1f471f62c176cb1f697a3042760fcdf Jul 17 19:01:51.990063 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:51.990024 2561 generic.go:356] "Generic (PLEG): container finished" podID="a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499" containerID="c6245c725a47f473c2afa45db00cbc92ccff8cbc9b03575f372db88f84df86a5" exitCode=0 Jul 17 19:01:51.990492 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:51.990104 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nww7x" event={"ID":"a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499","Type":"ContainerDied","Data":"c6245c725a47f473c2afa45db00cbc92ccff8cbc9b03575f372db88f84df86a5"} Jul 17 19:01:51.990492 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:51.990154 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nww7x" event={"ID":"a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499","Type":"ContainerStarted","Data":"fa16426b5e77b5dbc0c664051463b49fa29a5b3e37c79628b4d21235eb6e1e8a"} Jul 17 19:01:51.991761 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:51.991737 2561 generic.go:356] "Generic (PLEG): container finished" podID="47ae49e7-0d98-49e5-beb7-dc7d972803b8" containerID="b207738f31cc518970d6bda183b8769ecf99179866660a96252c85364e1b20d6" exitCode=0 Jul 17 19:01:51.991896 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:51.991823 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mfkhp" event={"ID":"47ae49e7-0d98-49e5-beb7-dc7d972803b8","Type":"ContainerDied","Data":"b207738f31cc518970d6bda183b8769ecf99179866660a96252c85364e1b20d6"} Jul 17 19:01:51.991896 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:51.991872 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mfkhp" event={"ID":"47ae49e7-0d98-49e5-beb7-dc7d972803b8","Type":"ContainerStarted","Data":"e4267b462eb40c39be21e8f8155ba56ba1f471f62c176cb1f697a3042760fcdf"} Jul 17 19:01:52.997630 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:52.997577 2561 generic.go:356] "Generic (PLEG): container finished" podID="a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499" containerID="95a9941f1e16674b03066eef917c2be67526aec25e9d17e5e3c85362a2a9fa19" exitCode=0 Jul 17 19:01:52.998122 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:52.997657 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nww7x" event={"ID":"a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499","Type":"ContainerDied","Data":"95a9941f1e16674b03066eef917c2be67526aec25e9d17e5e3c85362a2a9fa19"} Jul 17 19:01:52.999625 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:52.999595 2561 generic.go:356] "Generic (PLEG): container finished" podID="b4c09057-4e4c-464c-9985-05025e1e0e09" containerID="fbe7910720807fda35f66298ee019f762d1a680828524bae5112ce3a471c8aa6" exitCode=0 Jul 17 19:01:52.999712 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:52.999643 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzwzv" event={"ID":"b4c09057-4e4c-464c-9985-05025e1e0e09","Type":"ContainerDied","Data":"fbe7910720807fda35f66298ee019f762d1a680828524bae5112ce3a471c8aa6"} Jul 17 19:01:53.001520 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:53.001497 2561 generic.go:356] "Generic (PLEG): container finished" podID="3f7df721-5719-4fe3-be5c-cf1f8d92ab2c" containerID="4997b843e293b509a1c13039098cd70817301b36fbd7705cd4fde80a9b0d7e15" exitCode=0 Jul 17 19:01:53.001645 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:53.001561 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tgnfr" event={"ID":"3f7df721-5719-4fe3-be5c-cf1f8d92ab2c","Type":"ContainerDied","Data":"4997b843e293b509a1c13039098cd70817301b36fbd7705cd4fde80a9b0d7e15"} Jul 17 19:01:54.008246 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:54.008211 2561 generic.go:356] "Generic (PLEG): container finished" podID="b4c09057-4e4c-464c-9985-05025e1e0e09" containerID="42894742ac1122a819e7bc8b7349a9268190f06acce4912613a0e64aef7e1d1b" exitCode=0 Jul 17 19:01:54.008680 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:54.008294 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzwzv" event={"ID":"b4c09057-4e4c-464c-9985-05025e1e0e09","Type":"ContainerDied","Data":"42894742ac1122a819e7bc8b7349a9268190f06acce4912613a0e64aef7e1d1b"} Jul 17 19:01:54.010413 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:54.010389 2561 generic.go:356] "Generic (PLEG): container finished" podID="3f7df721-5719-4fe3-be5c-cf1f8d92ab2c" containerID="f32f91250d7039c826dbf4f5631172bbb733afce31003e67011be997ea83e00e" exitCode=0 Jul 17 19:01:54.010514 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:54.010473 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tgnfr" event={"ID":"3f7df721-5719-4fe3-be5c-cf1f8d92ab2c","Type":"ContainerDied","Data":"f32f91250d7039c826dbf4f5631172bbb733afce31003e67011be997ea83e00e"} Jul 17 19:01:54.012411 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:54.012383 2561 generic.go:356] "Generic (PLEG): container finished" podID="a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499" containerID="296fd9f8dba6452d9e686dda5e3634faedc56ec9b1b2b687b1072381a6c1a0f8" exitCode=0 Jul 17 19:01:54.012555 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:54.012471 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nww7x" event={"ID":"a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499","Type":"ContainerDied","Data":"296fd9f8dba6452d9e686dda5e3634faedc56ec9b1b2b687b1072381a6c1a0f8"} Jul 17 19:01:54.014224 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:54.014207 2561 generic.go:356] "Generic (PLEG): container finished" podID="47ae49e7-0d98-49e5-beb7-dc7d972803b8" containerID="426d09d1b6b1154e1e2f5c3cda618051786b427dfe6515d32cf221b697d05fad" exitCode=0 Jul 17 19:01:54.014294 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:54.014236 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mfkhp" event={"ID":"47ae49e7-0d98-49e5-beb7-dc7d972803b8","Type":"ContainerDied","Data":"426d09d1b6b1154e1e2f5c3cda618051786b427dfe6515d32cf221b697d05fad"} Jul 17 19:01:55.021003 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:55.020963 2561 generic.go:356] "Generic (PLEG): container finished" podID="47ae49e7-0d98-49e5-beb7-dc7d972803b8" containerID="3933a52d03d30e56c830574693287a5fb12f4ca18e3422e26538f1a1881e5016" exitCode=0 Jul 17 19:01:55.021404 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:55.021047 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mfkhp" event={"ID":"47ae49e7-0d98-49e5-beb7-dc7d972803b8","Type":"ContainerDied","Data":"3933a52d03d30e56c830574693287a5fb12f4ca18e3422e26538f1a1881e5016"} Jul 17 19:01:55.177860 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:55.177834 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nww7x" Jul 17 19:01:55.181273 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:55.181250 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzwzv" Jul 17 19:01:55.212861 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:55.212830 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tgnfr" Jul 17 19:01:55.265802 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:55.265768 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hbrs\" (UniqueName: \"kubernetes.io/projected/a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499-kube-api-access-7hbrs\") pod \"a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499\" (UID: \"a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499\") " Jul 17 19:01:55.265802 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:55.265807 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4c09057-4e4c-464c-9985-05025e1e0e09-bundle\") pod \"b4c09057-4e4c-464c-9985-05025e1e0e09\" (UID: \"b4c09057-4e4c-464c-9985-05025e1e0e09\") " Jul 17 19:01:55.266061 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:55.265829 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499-util\") pod \"a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499\" (UID: \"a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499\") " Jul 17 19:01:55.266061 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:55.265886 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499-bundle\") pod \"a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499\" (UID: \"a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499\") " Jul 17 19:01:55.266061 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:55.265915 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8nj9\" (UniqueName: \"kubernetes.io/projected/b4c09057-4e4c-464c-9985-05025e1e0e09-kube-api-access-v8nj9\") pod \"b4c09057-4e4c-464c-9985-05025e1e0e09\" (UID: \"b4c09057-4e4c-464c-9985-05025e1e0e09\") " Jul 17 19:01:55.266061 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:55.265929 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4c09057-4e4c-464c-9985-05025e1e0e09-util\") pod \"b4c09057-4e4c-464c-9985-05025e1e0e09\" (UID: \"b4c09057-4e4c-464c-9985-05025e1e0e09\") " Jul 17 19:01:55.266463 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:55.266426 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c09057-4e4c-464c-9985-05025e1e0e09-bundle" (OuterVolumeSpecName: "bundle") pod "b4c09057-4e4c-464c-9985-05025e1e0e09" (UID: "b4c09057-4e4c-464c-9985-05025e1e0e09"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 19:01:55.266463 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:55.266428 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499-bundle" (OuterVolumeSpecName: "bundle") pod "a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499" (UID: "a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 19:01:55.268234 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:55.268210 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499-kube-api-access-7hbrs" (OuterVolumeSpecName: "kube-api-access-7hbrs") pod "a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499" (UID: "a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499"). InnerVolumeSpecName "kube-api-access-7hbrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 19:01:55.268318 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:55.268252 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4c09057-4e4c-464c-9985-05025e1e0e09-kube-api-access-v8nj9" (OuterVolumeSpecName: "kube-api-access-v8nj9") pod "b4c09057-4e4c-464c-9985-05025e1e0e09" (UID: "b4c09057-4e4c-464c-9985-05025e1e0e09"). InnerVolumeSpecName "kube-api-access-v8nj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 19:01:55.271780 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:55.271717 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499-util" (OuterVolumeSpecName: "util") pod "a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499" (UID: "a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 19:01:55.272032 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:55.272009 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c09057-4e4c-464c-9985-05025e1e0e09-util" (OuterVolumeSpecName: "util") pod "b4c09057-4e4c-464c-9985-05025e1e0e09" (UID: "b4c09057-4e4c-464c-9985-05025e1e0e09"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 19:01:55.366768 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:55.366734 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f7df721-5719-4fe3-be5c-cf1f8d92ab2c-util\") pod \"3f7df721-5719-4fe3-be5c-cf1f8d92ab2c\" (UID: \"3f7df721-5719-4fe3-be5c-cf1f8d92ab2c\") " Jul 17 19:01:55.366949 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:55.366790 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f7df721-5719-4fe3-be5c-cf1f8d92ab2c-bundle\") pod \"3f7df721-5719-4fe3-be5c-cf1f8d92ab2c\" (UID: \"3f7df721-5719-4fe3-be5c-cf1f8d92ab2c\") " Jul 17 19:01:55.366949 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:55.366826 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pvml6\" (UniqueName: \"kubernetes.io/projected/3f7df721-5719-4fe3-be5c-cf1f8d92ab2c-kube-api-access-pvml6\") pod \"3f7df721-5719-4fe3-be5c-cf1f8d92ab2c\" (UID: \"3f7df721-5719-4fe3-be5c-cf1f8d92ab2c\") " Jul 17 19:01:55.367071 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:55.367049 2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499-bundle\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:01:55.367128 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:55.367076 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8nj9\" (UniqueName: \"kubernetes.io/projected/b4c09057-4e4c-464c-9985-05025e1e0e09-kube-api-access-v8nj9\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:01:55.367128 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:55.367091 2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4c09057-4e4c-464c-9985-05025e1e0e09-util\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:01:55.367128 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:55.367106 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hbrs\" (UniqueName: \"kubernetes.io/projected/a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499-kube-api-access-7hbrs\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:01:55.367128 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:55.367119 2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4c09057-4e4c-464c-9985-05025e1e0e09-bundle\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:01:55.367256 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:55.367132 2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499-util\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:01:55.367386 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:55.367359 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f7df721-5719-4fe3-be5c-cf1f8d92ab2c-bundle" (OuterVolumeSpecName: "bundle") pod "3f7df721-5719-4fe3-be5c-cf1f8d92ab2c" (UID: "3f7df721-5719-4fe3-be5c-cf1f8d92ab2c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 19:01:55.369117 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:55.369092 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f7df721-5719-4fe3-be5c-cf1f8d92ab2c-kube-api-access-pvml6" (OuterVolumeSpecName: "kube-api-access-pvml6") pod "3f7df721-5719-4fe3-be5c-cf1f8d92ab2c" (UID: "3f7df721-5719-4fe3-be5c-cf1f8d92ab2c"). InnerVolumeSpecName "kube-api-access-pvml6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 19:01:55.372244 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:55.372213 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f7df721-5719-4fe3-be5c-cf1f8d92ab2c-util" (OuterVolumeSpecName: "util") pod "3f7df721-5719-4fe3-be5c-cf1f8d92ab2c" (UID: "3f7df721-5719-4fe3-be5c-cf1f8d92ab2c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 19:01:55.468200 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:55.468166 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvml6\" (UniqueName: \"kubernetes.io/projected/3f7df721-5719-4fe3-be5c-cf1f8d92ab2c-kube-api-access-pvml6\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:01:55.468200 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:55.468196 2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f7df721-5719-4fe3-be5c-cf1f8d92ab2c-util\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:01:55.468200 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:55.468208 2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f7df721-5719-4fe3-be5c-cf1f8d92ab2c-bundle\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:01:56.028249 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:56.028225 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzwzv" Jul 17 19:01:56.028668 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:56.028224 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzwzv" event={"ID":"b4c09057-4e4c-464c-9985-05025e1e0e09","Type":"ContainerDied","Data":"25c2723dbfb9b61ce364d09722c927a6babc70acf5e1790e3cc4b8c30b493234"} Jul 17 19:01:56.028668 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:56.028356 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25c2723dbfb9b61ce364d09722c927a6babc70acf5e1790e3cc4b8c30b493234" Jul 17 19:01:56.030169 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:56.030146 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tgnfr" Jul 17 19:01:56.030169 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:56.030152 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tgnfr" event={"ID":"3f7df721-5719-4fe3-be5c-cf1f8d92ab2c","Type":"ContainerDied","Data":"2d793f24d65c96113480199ea5788d86072bd9bdde040623558e9bef7f5a1eb8"} Jul 17 19:01:56.030345 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:56.030186 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d793f24d65c96113480199ea5788d86072bd9bdde040623558e9bef7f5a1eb8" Jul 17 19:01:56.032131 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:56.032103 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nww7x" event={"ID":"a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499","Type":"ContainerDied","Data":"fa16426b5e77b5dbc0c664051463b49fa29a5b3e37c79628b4d21235eb6e1e8a"} Jul 17 19:01:56.032247 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:56.032147 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa16426b5e77b5dbc0c664051463b49fa29a5b3e37c79628b4d21235eb6e1e8a" Jul 17 19:01:56.032312 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:56.032278 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nww7x" Jul 17 19:01:56.154382 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:56.154358 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mfkhp" Jul 17 19:01:56.275030 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:56.274986 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/47ae49e7-0d98-49e5-beb7-dc7d972803b8-util\") pod \"47ae49e7-0d98-49e5-beb7-dc7d972803b8\" (UID: \"47ae49e7-0d98-49e5-beb7-dc7d972803b8\") " Jul 17 19:01:56.275172 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:56.275066 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wfrl6\" (UniqueName: \"kubernetes.io/projected/47ae49e7-0d98-49e5-beb7-dc7d972803b8-kube-api-access-wfrl6\") pod \"47ae49e7-0d98-49e5-beb7-dc7d972803b8\" (UID: \"47ae49e7-0d98-49e5-beb7-dc7d972803b8\") " Jul 17 19:01:56.275172 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:56.275112 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/47ae49e7-0d98-49e5-beb7-dc7d972803b8-bundle\") pod \"47ae49e7-0d98-49e5-beb7-dc7d972803b8\" (UID: \"47ae49e7-0d98-49e5-beb7-dc7d972803b8\") " Jul 17 19:01:56.275701 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:56.275676 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47ae49e7-0d98-49e5-beb7-dc7d972803b8-bundle" (OuterVolumeSpecName: "bundle") pod "47ae49e7-0d98-49e5-beb7-dc7d972803b8" (UID: "47ae49e7-0d98-49e5-beb7-dc7d972803b8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 19:01:56.277265 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:56.277239 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47ae49e7-0d98-49e5-beb7-dc7d972803b8-kube-api-access-wfrl6" (OuterVolumeSpecName: "kube-api-access-wfrl6") pod "47ae49e7-0d98-49e5-beb7-dc7d972803b8" (UID: "47ae49e7-0d98-49e5-beb7-dc7d972803b8"). InnerVolumeSpecName "kube-api-access-wfrl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 19:01:56.280384 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:56.280324 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47ae49e7-0d98-49e5-beb7-dc7d972803b8-util" (OuterVolumeSpecName: "util") pod "47ae49e7-0d98-49e5-beb7-dc7d972803b8" (UID: "47ae49e7-0d98-49e5-beb7-dc7d972803b8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 19:01:56.376502 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:56.376461 2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/47ae49e7-0d98-49e5-beb7-dc7d972803b8-bundle\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:01:56.376502 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:56.376494 2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/47ae49e7-0d98-49e5-beb7-dc7d972803b8-util\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:01:56.376502 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:56.376504 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfrl6\" (UniqueName: \"kubernetes.io/projected/47ae49e7-0d98-49e5-beb7-dc7d972803b8-kube-api-access-wfrl6\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:01:57.039021 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:57.038987 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mfkhp" Jul 17 19:01:57.039021 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:57.038994 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mfkhp" event={"ID":"47ae49e7-0d98-49e5-beb7-dc7d972803b8","Type":"ContainerDied","Data":"e4267b462eb40c39be21e8f8155ba56ba1f471f62c176cb1f697a3042760fcdf"} Jul 17 19:01:57.039475 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:01:57.039074 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4267b462eb40c39be21e8f8155ba56ba1f471f62c176cb1f697a3042760fcdf" Jul 17 19:02:09.768289 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:09.768248 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58f9457988-8cnbf"] Jul 17 19:02:34.789275 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:34.789156 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-58f9457988-8cnbf" podUID="ed9a0cc3-1335-4b5f-ba3a-85ff27da8974" containerName="console" containerID="cri-o://84e9a1ba3e739fb89ba7a43c0aa7331c649445efaa38efc8a766e69bb2147f6e" gracePeriod=15 Jul 17 19:02:35.031209 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.031183 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58f9457988-8cnbf" Jul 17 19:02:35.076130 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.076044 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ed9a0cc3-1335-4b5f-ba3a-85ff27da8974-oauth-serving-cert\") pod \"ed9a0cc3-1335-4b5f-ba3a-85ff27da8974\" (UID: \"ed9a0cc3-1335-4b5f-ba3a-85ff27da8974\") " Jul 17 19:02:35.076130 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.076082 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ed9a0cc3-1335-4b5f-ba3a-85ff27da8974-console-oauth-config\") pod \"ed9a0cc3-1335-4b5f-ba3a-85ff27da8974\" (UID: \"ed9a0cc3-1335-4b5f-ba3a-85ff27da8974\") " Jul 17 19:02:35.076130 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.076118 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ed9a0cc3-1335-4b5f-ba3a-85ff27da8974-console-config\") pod \"ed9a0cc3-1335-4b5f-ba3a-85ff27da8974\" (UID: \"ed9a0cc3-1335-4b5f-ba3a-85ff27da8974\") " Jul 17 19:02:35.076390 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.076145 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f5kg4\" (UniqueName: \"kubernetes.io/projected/ed9a0cc3-1335-4b5f-ba3a-85ff27da8974-kube-api-access-f5kg4\") pod \"ed9a0cc3-1335-4b5f-ba3a-85ff27da8974\" (UID: \"ed9a0cc3-1335-4b5f-ba3a-85ff27da8974\") " Jul 17 19:02:35.076390 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.076272 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed9a0cc3-1335-4b5f-ba3a-85ff27da8974-trusted-ca-bundle\") pod \"ed9a0cc3-1335-4b5f-ba3a-85ff27da8974\" (UID: \"ed9a0cc3-1335-4b5f-ba3a-85ff27da8974\") " Jul 17 19:02:35.076390 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.076342 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ed9a0cc3-1335-4b5f-ba3a-85ff27da8974-console-serving-cert\") pod \"ed9a0cc3-1335-4b5f-ba3a-85ff27da8974\" (UID: \"ed9a0cc3-1335-4b5f-ba3a-85ff27da8974\") " Jul 17 19:02:35.076572 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.076410 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ed9a0cc3-1335-4b5f-ba3a-85ff27da8974-service-ca\") pod \"ed9a0cc3-1335-4b5f-ba3a-85ff27da8974\" (UID: \"ed9a0cc3-1335-4b5f-ba3a-85ff27da8974\") " Jul 17 19:02:35.076572 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.076427 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed9a0cc3-1335-4b5f-ba3a-85ff27da8974-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ed9a0cc3-1335-4b5f-ba3a-85ff27da8974" (UID: "ed9a0cc3-1335-4b5f-ba3a-85ff27da8974"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 19:02:35.076696 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.076667 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed9a0cc3-1335-4b5f-ba3a-85ff27da8974-console-config" (OuterVolumeSpecName: "console-config") pod "ed9a0cc3-1335-4b5f-ba3a-85ff27da8974" (UID: "ed9a0cc3-1335-4b5f-ba3a-85ff27da8974"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 19:02:35.076741 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.076693 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed9a0cc3-1335-4b5f-ba3a-85ff27da8974-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ed9a0cc3-1335-4b5f-ba3a-85ff27da8974" (UID: "ed9a0cc3-1335-4b5f-ba3a-85ff27da8974"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 19:02:35.076795 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.076736 2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ed9a0cc3-1335-4b5f-ba3a-85ff27da8974-oauth-serving-cert\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:02:35.077230 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.077207 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed9a0cc3-1335-4b5f-ba3a-85ff27da8974-service-ca" (OuterVolumeSpecName: "service-ca") pod "ed9a0cc3-1335-4b5f-ba3a-85ff27da8974" (UID: "ed9a0cc3-1335-4b5f-ba3a-85ff27da8974"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 19:02:35.078790 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.078765 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed9a0cc3-1335-4b5f-ba3a-85ff27da8974-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ed9a0cc3-1335-4b5f-ba3a-85ff27da8974" (UID: "ed9a0cc3-1335-4b5f-ba3a-85ff27da8974"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 19:02:35.078874 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.078796 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed9a0cc3-1335-4b5f-ba3a-85ff27da8974-kube-api-access-f5kg4" (OuterVolumeSpecName: "kube-api-access-f5kg4") pod "ed9a0cc3-1335-4b5f-ba3a-85ff27da8974" (UID: "ed9a0cc3-1335-4b5f-ba3a-85ff27da8974"). InnerVolumeSpecName "kube-api-access-f5kg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 19:02:35.078874 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.078834 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed9a0cc3-1335-4b5f-ba3a-85ff27da8974-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ed9a0cc3-1335-4b5f-ba3a-85ff27da8974" (UID: "ed9a0cc3-1335-4b5f-ba3a-85ff27da8974"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 19:02:35.177568 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.177506 2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ed9a0cc3-1335-4b5f-ba3a-85ff27da8974-console-oauth-config\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:02:35.177568 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.177568 2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ed9a0cc3-1335-4b5f-ba3a-85ff27da8974-console-config\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:02:35.177771 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.177579 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5kg4\" (UniqueName: \"kubernetes.io/projected/ed9a0cc3-1335-4b5f-ba3a-85ff27da8974-kube-api-access-f5kg4\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:02:35.177771 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.177591 2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed9a0cc3-1335-4b5f-ba3a-85ff27da8974-trusted-ca-bundle\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:02:35.177771 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.177600 2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ed9a0cc3-1335-4b5f-ba3a-85ff27da8974-console-serving-cert\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:02:35.177771 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.177609 2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ed9a0cc3-1335-4b5f-ba3a-85ff27da8974-service-ca\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:02:35.216208 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.216171 2561 generic.go:356] "Generic (PLEG): container finished" podID="ed9a0cc3-1335-4b5f-ba3a-85ff27da8974" containerID="84e9a1ba3e739fb89ba7a43c0aa7331c649445efaa38efc8a766e69bb2147f6e" exitCode=0 Jul 17 19:02:35.216208 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.216213 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58f9457988-8cnbf" Jul 17 19:02:35.216394 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.216256 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58f9457988-8cnbf" event={"ID":"ed9a0cc3-1335-4b5f-ba3a-85ff27da8974","Type":"ContainerDied","Data":"84e9a1ba3e739fb89ba7a43c0aa7331c649445efaa38efc8a766e69bb2147f6e"} Jul 17 19:02:35.216394 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.216306 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58f9457988-8cnbf" event={"ID":"ed9a0cc3-1335-4b5f-ba3a-85ff27da8974","Type":"ContainerDied","Data":"cfc0b26101e44a09ed7be2011016de5fd73d8f25b5b8278031663be37b00b177"} Jul 17 19:02:35.216394 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.216327 2561 scope.go:117] "RemoveContainer" containerID="84e9a1ba3e739fb89ba7a43c0aa7331c649445efaa38efc8a766e69bb2147f6e" Jul 17 19:02:35.226109 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.226087 2561 scope.go:117] "RemoveContainer" containerID="84e9a1ba3e739fb89ba7a43c0aa7331c649445efaa38efc8a766e69bb2147f6e" Jul 17 19:02:35.226393 ip-10-0-134-137 kubenswrapper[2561]: E0717 19:02:35.226377 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84e9a1ba3e739fb89ba7a43c0aa7331c649445efaa38efc8a766e69bb2147f6e\": container with ID starting with 84e9a1ba3e739fb89ba7a43c0aa7331c649445efaa38efc8a766e69bb2147f6e not found: ID does not exist" containerID="84e9a1ba3e739fb89ba7a43c0aa7331c649445efaa38efc8a766e69bb2147f6e" Jul 17 19:02:35.226440 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.226402 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84e9a1ba3e739fb89ba7a43c0aa7331c649445efaa38efc8a766e69bb2147f6e"} err="failed to get container status \"84e9a1ba3e739fb89ba7a43c0aa7331c649445efaa38efc8a766e69bb2147f6e\": rpc error: code = NotFound desc = could not find container \"84e9a1ba3e739fb89ba7a43c0aa7331c649445efaa38efc8a766e69bb2147f6e\": container with ID starting with 84e9a1ba3e739fb89ba7a43c0aa7331c649445efaa38efc8a766e69bb2147f6e not found: ID does not exist" Jul 17 19:02:35.238054 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.238026 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58f9457988-8cnbf"] Jul 17 19:02:35.243608 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.243586 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-58f9457988-8cnbf"] Jul 17 19:02:35.589803 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.589764 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2wsmc"] Jul 17 19:02:35.590112 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.590095 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47ae49e7-0d98-49e5-beb7-dc7d972803b8" containerName="extract" Jul 17 19:02:35.590198 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.590115 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="47ae49e7-0d98-49e5-beb7-dc7d972803b8" containerName="extract" Jul 17 19:02:35.590198 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.590130 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f7df721-5719-4fe3-be5c-cf1f8d92ab2c" containerName="util" Jul 17 19:02:35.590198 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.590137 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f7df721-5719-4fe3-be5c-cf1f8d92ab2c" containerName="util" Jul 17 19:02:35.590198 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.590148 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed9a0cc3-1335-4b5f-ba3a-85ff27da8974" containerName="console" Jul 17 19:02:35.590198 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.590157 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed9a0cc3-1335-4b5f-ba3a-85ff27da8974" containerName="console" Jul 17 19:02:35.590198 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.590168 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4c09057-4e4c-464c-9985-05025e1e0e09" containerName="util" Jul 17 19:02:35.590198 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.590177 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4c09057-4e4c-464c-9985-05025e1e0e09" containerName="util" Jul 17 19:02:35.590198 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.590188 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f7df721-5719-4fe3-be5c-cf1f8d92ab2c" containerName="pull" Jul 17 19:02:35.590198 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.590196 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f7df721-5719-4fe3-be5c-cf1f8d92ab2c" containerName="pull" Jul 17 19:02:35.590652 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.590210 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47ae49e7-0d98-49e5-beb7-dc7d972803b8" containerName="util" Jul 17 19:02:35.590652 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.590218 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="47ae49e7-0d98-49e5-beb7-dc7d972803b8" containerName="util" Jul 17 19:02:35.590652 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.590228 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f7df721-5719-4fe3-be5c-cf1f8d92ab2c" containerName="extract" Jul 17 19:02:35.590652 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.590236 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f7df721-5719-4fe3-be5c-cf1f8d92ab2c" containerName="extract" Jul 17 19:02:35.590652 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.590246 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47ae49e7-0d98-49e5-beb7-dc7d972803b8" containerName="pull" Jul 17 19:02:35.590652 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.590254 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="47ae49e7-0d98-49e5-beb7-dc7d972803b8" containerName="pull" Jul 17 19:02:35.590652 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.590272 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499" containerName="util" Jul 17 19:02:35.590652 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.590280 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499" containerName="util" Jul 17 19:02:35.590652 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.590292 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499" containerName="pull" Jul 17 19:02:35.590652 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.590300 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499" containerName="pull" Jul 17 19:02:35.590652 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.590310 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4c09057-4e4c-464c-9985-05025e1e0e09" containerName="pull" Jul 17 19:02:35.590652 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.590316 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4c09057-4e4c-464c-9985-05025e1e0e09" containerName="pull" Jul 17 19:02:35.590652 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.590328 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499" containerName="extract" Jul 17 19:02:35.590652 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.590336 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499" containerName="extract" Jul 17 19:02:35.590652 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.590344 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4c09057-4e4c-464c-9985-05025e1e0e09" containerName="extract" Jul 17 19:02:35.590652 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.590352 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4c09057-4e4c-464c-9985-05025e1e0e09" containerName="extract" Jul 17 19:02:35.590652 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.590424 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f7df721-5719-4fe3-be5c-cf1f8d92ab2c" containerName="extract" Jul 17 19:02:35.590652 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.590438 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="47ae49e7-0d98-49e5-beb7-dc7d972803b8" containerName="extract" Jul 17 19:02:35.590652 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.590447 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed9a0cc3-1335-4b5f-ba3a-85ff27da8974" containerName="console" Jul 17 19:02:35.590652 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.590457 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499" containerName="extract" Jul 17 19:02:35.590652 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.590466 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4c09057-4e4c-464c-9985-05025e1e0e09" containerName="extract" Jul 17 19:02:35.595117 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.595092 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2wsmc" Jul 17 19:02:35.598157 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.598094 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 17 19:02:35.598157 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.598151 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 17 19:02:35.598349 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.598095 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-b2sv6\"" Jul 17 19:02:35.607033 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.606991 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2wsmc"] Jul 17 19:02:35.681800 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.681767 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dlqt\" (UniqueName: \"kubernetes.io/projected/acc4a4ab-11a0-4286-b6cd-4763a7dd51a1-kube-api-access-8dlqt\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-2wsmc\" (UID: \"acc4a4ab-11a0-4286-b6cd-4763a7dd51a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2wsmc" Jul 17 19:02:35.681994 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.681834 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/acc4a4ab-11a0-4286-b6cd-4763a7dd51a1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-2wsmc\" (UID: \"acc4a4ab-11a0-4286-b6cd-4763a7dd51a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2wsmc" Jul 17 19:02:35.782980 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.782935 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/acc4a4ab-11a0-4286-b6cd-4763a7dd51a1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-2wsmc\" (UID: \"acc4a4ab-11a0-4286-b6cd-4763a7dd51a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2wsmc" Jul 17 19:02:35.783178 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.782995 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dlqt\" (UniqueName: \"kubernetes.io/projected/acc4a4ab-11a0-4286-b6cd-4763a7dd51a1-kube-api-access-8dlqt\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-2wsmc\" (UID: \"acc4a4ab-11a0-4286-b6cd-4763a7dd51a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2wsmc" Jul 17 19:02:35.783312 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.783289 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/acc4a4ab-11a0-4286-b6cd-4763a7dd51a1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-2wsmc\" (UID: \"acc4a4ab-11a0-4286-b6cd-4763a7dd51a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2wsmc" Jul 17 19:02:35.792093 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.792066 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dlqt\" (UniqueName: \"kubernetes.io/projected/acc4a4ab-11a0-4286-b6cd-4763a7dd51a1-kube-api-access-8dlqt\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-2wsmc\" (UID: \"acc4a4ab-11a0-4286-b6cd-4763a7dd51a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2wsmc" Jul 17 19:02:35.908449 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:35.908351 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2wsmc" Jul 17 19:02:36.051626 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:36.051602 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2wsmc"] Jul 17 19:02:36.053853 ip-10-0-134-137 kubenswrapper[2561]: W0717 19:02:36.053814 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacc4a4ab_11a0_4286_b6cd_4763a7dd51a1.slice/crio-36c162ad94364a5565428e97630857a193b099e0b9c50173a40f40b04eac09c4 WatchSource:0}: Error finding container 36c162ad94364a5565428e97630857a193b099e0b9c50173a40f40b04eac09c4: Status 404 returned error can't find the container with id 36c162ad94364a5565428e97630857a193b099e0b9c50173a40f40b04eac09c4 Jul 17 19:02:36.099019 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:36.098986 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed9a0cc3-1335-4b5f-ba3a-85ff27da8974" path="/var/lib/kubelet/pods/ed9a0cc3-1335-4b5f-ba3a-85ff27da8974/volumes" Jul 17 19:02:36.222758 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:36.222665 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2wsmc" event={"ID":"acc4a4ab-11a0-4286-b6cd-4763a7dd51a1","Type":"ContainerStarted","Data":"36c162ad94364a5565428e97630857a193b099e0b9c50173a40f40b04eac09c4"} Jul 17 19:02:42.267118 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:42.267083 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2wsmc" event={"ID":"acc4a4ab-11a0-4286-b6cd-4763a7dd51a1","Type":"ContainerStarted","Data":"2671b356655815f80281fb39b12223ae8964242d86baeb7cb195a4b85d6752cf"} Jul 17 19:02:42.267516 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:42.267164 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2wsmc" Jul 17 19:02:42.292249 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:42.292192 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2wsmc" podStartSLOduration=1.921199692 podStartE2EDuration="7.292176981s" totalImagesPullingTime="5.370977289s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:02:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:02:42.291425121 +0000 UTC m=+460.621241332" watchObservedRunningTime="2026-07-17 19:02:42.292176981 +0000 UTC m=+460.621993188" Jul 17 19:02:53.274580 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:02:53.274543 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2wsmc" Jul 17 19:03:10.022636 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:03:10.022603 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2wsmc"] Jul 17 19:03:10.023129 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:03:10.022830 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2wsmc" podUID="acc4a4ab-11a0-4286-b6cd-4763a7dd51a1" containerName="manager" containerID="cri-o://2671b356655815f80281fb39b12223ae8964242d86baeb7cb195a4b85d6752cf" gracePeriod=10 Jul 17 19:03:10.290150 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:03:10.289843 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2wsmc" Jul 17 19:03:10.385043 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:03:10.385013 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/acc4a4ab-11a0-4286-b6cd-4763a7dd51a1-extensions-socket-volume\") pod \"acc4a4ab-11a0-4286-b6cd-4763a7dd51a1\" (UID: \"acc4a4ab-11a0-4286-b6cd-4763a7dd51a1\") " Jul 17 19:03:10.385218 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:03:10.385140 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8dlqt\" (UniqueName: \"kubernetes.io/projected/acc4a4ab-11a0-4286-b6cd-4763a7dd51a1-kube-api-access-8dlqt\") pod \"acc4a4ab-11a0-4286-b6cd-4763a7dd51a1\" (UID: \"acc4a4ab-11a0-4286-b6cd-4763a7dd51a1\") " Jul 17 19:03:10.385423 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:03:10.385396 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acc4a4ab-11a0-4286-b6cd-4763a7dd51a1-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "acc4a4ab-11a0-4286-b6cd-4763a7dd51a1" (UID: "acc4a4ab-11a0-4286-b6cd-4763a7dd51a1"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 19:03:10.387347 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:03:10.387322 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acc4a4ab-11a0-4286-b6cd-4763a7dd51a1-kube-api-access-8dlqt" (OuterVolumeSpecName: "kube-api-access-8dlqt") pod "acc4a4ab-11a0-4286-b6cd-4763a7dd51a1" (UID: "acc4a4ab-11a0-4286-b6cd-4763a7dd51a1"). InnerVolumeSpecName "kube-api-access-8dlqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 19:03:10.402574 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:03:10.402546 2561 generic.go:356] "Generic (PLEG): container finished" podID="acc4a4ab-11a0-4286-b6cd-4763a7dd51a1" containerID="2671b356655815f80281fb39b12223ae8964242d86baeb7cb195a4b85d6752cf" exitCode=0 Jul 17 19:03:10.402678 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:03:10.402591 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2wsmc" Jul 17 19:03:10.402678 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:03:10.402630 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2wsmc" event={"ID":"acc4a4ab-11a0-4286-b6cd-4763a7dd51a1","Type":"ContainerDied","Data":"2671b356655815f80281fb39b12223ae8964242d86baeb7cb195a4b85d6752cf"} Jul 17 19:03:10.402751 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:03:10.402683 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2wsmc" event={"ID":"acc4a4ab-11a0-4286-b6cd-4763a7dd51a1","Type":"ContainerDied","Data":"36c162ad94364a5565428e97630857a193b099e0b9c50173a40f40b04eac09c4"} Jul 17 19:03:10.402751 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:03:10.402706 2561 scope.go:117] "RemoveContainer" containerID="2671b356655815f80281fb39b12223ae8964242d86baeb7cb195a4b85d6752cf" Jul 17 19:03:10.411591 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:03:10.411575 2561 scope.go:117] "RemoveContainer" containerID="2671b356655815f80281fb39b12223ae8964242d86baeb7cb195a4b85d6752cf" Jul 17 19:03:10.411882 ip-10-0-134-137 kubenswrapper[2561]: E0717 19:03:10.411863 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2671b356655815f80281fb39b12223ae8964242d86baeb7cb195a4b85d6752cf\": container with ID starting with 2671b356655815f80281fb39b12223ae8964242d86baeb7cb195a4b85d6752cf not found: ID does not exist" containerID="2671b356655815f80281fb39b12223ae8964242d86baeb7cb195a4b85d6752cf" Jul 17 19:03:10.411930 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:03:10.411891 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2671b356655815f80281fb39b12223ae8964242d86baeb7cb195a4b85d6752cf"} err="failed to get container status \"2671b356655815f80281fb39b12223ae8964242d86baeb7cb195a4b85d6752cf\": rpc error: code = NotFound desc = could not find container \"2671b356655815f80281fb39b12223ae8964242d86baeb7cb195a4b85d6752cf\": container with ID starting with 2671b356655815f80281fb39b12223ae8964242d86baeb7cb195a4b85d6752cf not found: ID does not exist" Jul 17 19:03:10.427311 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:03:10.427287 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2wsmc"] Jul 17 19:03:10.431716 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:03:10.431691 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2wsmc"] Jul 17 19:03:10.486230 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:03:10.486195 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dlqt\" (UniqueName: \"kubernetes.io/projected/acc4a4ab-11a0-4286-b6cd-4763a7dd51a1-kube-api-access-8dlqt\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:03:10.486230 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:03:10.486224 2561 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/acc4a4ab-11a0-4286-b6cd-4763a7dd51a1-extensions-socket-volume\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:03:12.100903 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:03:12.100867 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acc4a4ab-11a0-4286-b6cd-4763a7dd51a1" path="/var/lib/kubelet/pods/acc4a4ab-11a0-4286-b6cd-4763a7dd51a1/volumes" Jul 17 19:03:16.490052 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:03:16.490015 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:03:16.490603 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:03:16.490432 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acc4a4ab-11a0-4286-b6cd-4763a7dd51a1" containerName="manager" Jul 17 19:03:16.490603 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:03:16.490452 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="acc4a4ab-11a0-4286-b6cd-4763a7dd51a1" containerName="manager" Jul 17 19:03:16.490603 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:03:16.490550 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="acc4a4ab-11a0-4286-b6cd-4763a7dd51a1" containerName="manager" Jul 17 19:03:16.496205 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:03:16.496179 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-cqg68" Jul 17 19:03:16.499009 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:03:16.498989 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 17 19:03:16.499009 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:03:16.498993 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-w799f\"" Jul 17 19:03:16.500006 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:03:16.499988 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 17 19:03:16.500110 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:03:16.499992 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 17 19:03:16.504243 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:03:16.504225 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:03:16.534613 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:03:16.534561 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:03:16.636056 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:03:16.636020 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f1a73fff-acd0-4416-ba1a-844a75c62607-config-file\") pod \"limitador-limitador-54d6d988d5-cqg68\" (UID: \"f1a73fff-acd0-4416-ba1a-844a75c62607\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-cqg68" Jul 17 19:03:16.636056 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:03:16.636058 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8xhv\" (UniqueName: \"kubernetes.io/projected/f1a73fff-acd0-4416-ba1a-844a75c62607-kube-api-access-q8xhv\") pod \"limitador-limitador-54d6d988d5-cqg68\" (UID: \"f1a73fff-acd0-4416-ba1a-844a75c62607\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-cqg68" Jul 17 19:03:16.737462 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:03:16.737421 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f1a73fff-acd0-4416-ba1a-844a75c62607-config-file\") pod \"limitador-limitador-54d6d988d5-cqg68\" (UID: \"f1a73fff-acd0-4416-ba1a-844a75c62607\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-cqg68" Jul 17 19:03:16.737462 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:03:16.737457 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8xhv\" (UniqueName: \"kubernetes.io/projected/f1a73fff-acd0-4416-ba1a-844a75c62607-kube-api-access-q8xhv\") pod \"limitador-limitador-54d6d988d5-cqg68\" (UID: \"f1a73fff-acd0-4416-ba1a-844a75c62607\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-cqg68" Jul 17 19:03:16.738121 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:03:16.738100 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f1a73fff-acd0-4416-ba1a-844a75c62607-config-file\") pod \"limitador-limitador-54d6d988d5-cqg68\" (UID: \"f1a73fff-acd0-4416-ba1a-844a75c62607\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-cqg68" Jul 17 19:03:16.746805 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:03:16.746756 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8xhv\" (UniqueName: \"kubernetes.io/projected/f1a73fff-acd0-4416-ba1a-844a75c62607-kube-api-access-q8xhv\") pod \"limitador-limitador-54d6d988d5-cqg68\" (UID: \"f1a73fff-acd0-4416-ba1a-844a75c62607\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-cqg68" Jul 17 19:03:16.807657 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:03:16.807620 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-cqg68" Jul 17 19:03:16.942678 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:03:16.942654 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:03:16.944782 ip-10-0-134-137 kubenswrapper[2561]: W0717 19:03:16.944745 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1a73fff_acd0_4416_ba1a_844a75c62607.slice/crio-8fc9e0730c1ca07379bac5e4e0c9e425ab551d7cf5d32ea37e962d7fd938af39 WatchSource:0}: Error finding container 8fc9e0730c1ca07379bac5e4e0c9e425ab551d7cf5d32ea37e962d7fd938af39: Status 404 returned error can't find the container with id 8fc9e0730c1ca07379bac5e4e0c9e425ab551d7cf5d32ea37e962d7fd938af39 Jul 17 19:03:17.437579 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:03:17.437520 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-cqg68" event={"ID":"f1a73fff-acd0-4416-ba1a-844a75c62607","Type":"ContainerStarted","Data":"8fc9e0730c1ca07379bac5e4e0c9e425ab551d7cf5d32ea37e962d7fd938af39"} Jul 17 19:03:20.454725 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:03:20.454681 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-cqg68" event={"ID":"f1a73fff-acd0-4416-ba1a-844a75c62607","Type":"ContainerStarted","Data":"c042908996d5d29106d66aff6ed9733c3d1d641eedd130325597c9f5c111189a"} Jul 17 19:03:20.455177 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:03:20.454754 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-cqg68" Jul 17 19:03:20.472304 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:03:20.472257 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-cqg68" podStartSLOduration=1.914424007 podStartE2EDuration="4.472241695s" totalImagesPullingTime="2.557817688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:03:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:03:20.470103285 +0000 UTC m=+498.799919494" watchObservedRunningTime="2026-07-17 19:03:20.472241695 +0000 UTC m=+498.802057902" Jul 17 19:03:31.460891 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:03:31.460859 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-cqg68" Jul 17 19:04:28.269487 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:04:28.269445 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:04:44.889105 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:04:44.889066 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:04:55.508456 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:04:55.508420 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-s7f4n"] Jul 17 19:04:55.511965 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:04:55.511949 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s7f4n" Jul 17 19:04:55.515482 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:04:55.515460 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-4fjw6\"" Jul 17 19:04:55.515636 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:04:55.515504 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 17 19:04:55.515636 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:04:55.515543 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 17 19:04:55.515636 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:04:55.515590 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 17 19:04:55.520744 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:04:55.520721 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-s7f4n"] Jul 17 19:04:55.609971 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:04:55.609934 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ea241aa-9bb1-4b1f-9096-6db6421a11c4-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s7f4n\" (UID: \"4ea241aa-9bb1-4b1f-9096-6db6421a11c4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s7f4n" Jul 17 19:04:55.610140 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:04:55.609976 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ea241aa-9bb1-4b1f-9096-6db6421a11c4-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s7f4n\" (UID: \"4ea241aa-9bb1-4b1f-9096-6db6421a11c4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s7f4n" Jul 17 19:04:55.610140 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:04:55.610032 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxxf5\" (UniqueName: \"kubernetes.io/projected/4ea241aa-9bb1-4b1f-9096-6db6421a11c4-kube-api-access-sxxf5\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s7f4n\" (UID: \"4ea241aa-9bb1-4b1f-9096-6db6421a11c4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s7f4n" Jul 17 19:04:55.610140 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:04:55.610074 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ea241aa-9bb1-4b1f-9096-6db6421a11c4-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s7f4n\" (UID: \"4ea241aa-9bb1-4b1f-9096-6db6421a11c4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s7f4n" Jul 17 19:04:55.610140 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:04:55.610107 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ea241aa-9bb1-4b1f-9096-6db6421a11c4-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s7f4n\" (UID: \"4ea241aa-9bb1-4b1f-9096-6db6421a11c4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s7f4n" Jul 17 19:04:55.610140 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:04:55.610125 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ea241aa-9bb1-4b1f-9096-6db6421a11c4-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s7f4n\" (UID: \"4ea241aa-9bb1-4b1f-9096-6db6421a11c4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s7f4n" Jul 17 19:04:55.710722 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:04:55.710690 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ea241aa-9bb1-4b1f-9096-6db6421a11c4-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s7f4n\" (UID: \"4ea241aa-9bb1-4b1f-9096-6db6421a11c4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s7f4n" Jul 17 19:04:55.710888 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:04:55.710727 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxxf5\" (UniqueName: \"kubernetes.io/projected/4ea241aa-9bb1-4b1f-9096-6db6421a11c4-kube-api-access-sxxf5\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s7f4n\" (UID: \"4ea241aa-9bb1-4b1f-9096-6db6421a11c4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s7f4n" Jul 17 19:04:55.710888 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:04:55.710789 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ea241aa-9bb1-4b1f-9096-6db6421a11c4-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s7f4n\" (UID: \"4ea241aa-9bb1-4b1f-9096-6db6421a11c4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s7f4n" Jul 17 19:04:55.710888 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:04:55.710829 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ea241aa-9bb1-4b1f-9096-6db6421a11c4-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s7f4n\" (UID: \"4ea241aa-9bb1-4b1f-9096-6db6421a11c4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s7f4n" Jul 17 19:04:55.710888 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:04:55.710854 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ea241aa-9bb1-4b1f-9096-6db6421a11c4-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s7f4n\" (UID: \"4ea241aa-9bb1-4b1f-9096-6db6421a11c4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s7f4n" Jul 17 19:04:55.711056 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:04:55.710931 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ea241aa-9bb1-4b1f-9096-6db6421a11c4-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s7f4n\" (UID: \"4ea241aa-9bb1-4b1f-9096-6db6421a11c4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s7f4n" Jul 17 19:04:55.711201 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:04:55.711180 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ea241aa-9bb1-4b1f-9096-6db6421a11c4-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s7f4n\" (UID: \"4ea241aa-9bb1-4b1f-9096-6db6421a11c4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s7f4n" Jul 17 19:04:55.711283 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:04:55.711216 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ea241aa-9bb1-4b1f-9096-6db6421a11c4-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s7f4n\" (UID: \"4ea241aa-9bb1-4b1f-9096-6db6421a11c4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s7f4n" Jul 17 19:04:55.711283 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:04:55.711241 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ea241aa-9bb1-4b1f-9096-6db6421a11c4-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s7f4n\" (UID: \"4ea241aa-9bb1-4b1f-9096-6db6421a11c4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s7f4n" Jul 17 19:04:55.713319 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:04:55.713287 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ea241aa-9bb1-4b1f-9096-6db6421a11c4-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s7f4n\" (UID: \"4ea241aa-9bb1-4b1f-9096-6db6421a11c4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s7f4n" Jul 17 19:04:55.713422 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:04:55.713327 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ea241aa-9bb1-4b1f-9096-6db6421a11c4-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s7f4n\" (UID: \"4ea241aa-9bb1-4b1f-9096-6db6421a11c4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s7f4n" Jul 17 19:04:55.718822 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:04:55.718802 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxxf5\" (UniqueName: \"kubernetes.io/projected/4ea241aa-9bb1-4b1f-9096-6db6421a11c4-kube-api-access-sxxf5\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s7f4n\" (UID: \"4ea241aa-9bb1-4b1f-9096-6db6421a11c4\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s7f4n" Jul 17 19:04:55.823050 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:04:55.822967 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s7f4n" Jul 17 19:04:55.958047 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:04:55.958024 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-s7f4n"] Jul 17 19:04:55.960066 ip-10-0-134-137 kubenswrapper[2561]: W0717 19:04:55.960041 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ea241aa_9bb1_4b1f_9096_6db6421a11c4.slice/crio-2ffc0234c426bdc7fbb6952db28cf32b22ba85ed26900c3d58c36780594581ac WatchSource:0}: Error finding container 2ffc0234c426bdc7fbb6952db28cf32b22ba85ed26900c3d58c36780594581ac: Status 404 returned error can't find the container with id 2ffc0234c426bdc7fbb6952db28cf32b22ba85ed26900c3d58c36780594581ac Jul 17 19:04:56.383640 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:04:56.383584 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:04:56.894218 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:04:56.894177 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s7f4n" event={"ID":"4ea241aa-9bb1-4b1f-9096-6db6421a11c4","Type":"ContainerStarted","Data":"2ffc0234c426bdc7fbb6952db28cf32b22ba85ed26900c3d58c36780594581ac"} Jul 17 19:04:57.900939 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:04:57.900905 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s7f4n" event={"ID":"4ea241aa-9bb1-4b1f-9096-6db6421a11c4","Type":"ContainerStarted","Data":"88971df692cc97b3f05b153b9bc3c44673614e477efc64efa9d70e4522d5962e"} Jul 17 19:04:57.922110 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:04:57.922057 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s7f4n" podStartSLOduration=1.7594868030000002 podStartE2EDuration="2.922040119s" totalImagesPullingTime="1.162553316s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:04:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:04:57.920839819 +0000 UTC m=+596.250656027" watchObservedRunningTime="2026-07-17 19:04:57.922040119 +0000 UTC m=+596.251856327" Jul 17 19:05:02.015176 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:05:02.015147 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpjzq_4b162535-066a-471a-aee4-99c07c6e1bb4/ovn-acl-logging/0.log" Jul 17 19:05:02.015787 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:05:02.015329 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpjzq_4b162535-066a-471a-aee4-99c07c6e1bb4/ovn-acl-logging/0.log" Jul 17 19:05:05.823557 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:05:05.823492 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s7f4n" Jul 17 19:05:05.825992 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:05:05.823610 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s7f4n" Jul 17 19:05:05.836047 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:05:05.836026 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s7f4n" Jul 17 19:05:05.949475 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:05:05.949446 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s7f4n" Jul 17 19:05:06.881350 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:05:06.881311 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:05:13.786825 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:05:13.786787 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:05:17.975591 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:05:17.975557 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:05:26.183158 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:05:26.183081 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:06:59.879244 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:06:59.879158 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:07:10.173914 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:07:10.173878 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:07:20.185819 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:07:20.185781 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:09:00.684616 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:09:00.684587 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:09:12.878898 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:09:12.878861 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:09:46.180697 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:09:46.180657 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:09:57.876343 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:09:57.876256 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:10:02.043252 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:10:02.043220 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpjzq_4b162535-066a-471a-aee4-99c07c6e1bb4/ovn-acl-logging/0.log" Jul 17 19:10:02.045042 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:10:02.045018 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpjzq_4b162535-066a-471a-aee4-99c07c6e1bb4/ovn-acl-logging/0.log" Jul 17 19:10:06.583462 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:10:06.583426 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:10:20.974378 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:10:20.974336 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:10:48.477559 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:10:48.477510 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:10:58.879662 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:10:58.879621 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:11:23.488379 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:11:23.488288 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:11:33.079329 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:11:33.079294 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:11:55.383242 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:11:55.383205 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:12:05.675217 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:12:05.675179 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:12:22.600199 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:12:22.600156 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:12:31.688678 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:12:31.688640 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:12:41.583042 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:12:41.583002 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:12:50.897402 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:12:50.897360 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:13:01.980369 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:13:01.980285 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:14:48.177228 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:14:48.177143 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:14:57.583086 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:14:57.583040 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:15:00.153673 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:15:00.153640 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738595-9jpgd"] Jul 17 19:15:00.156784 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:15:00.156768 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738595-9jpgd" Jul 17 19:15:00.159646 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:15:00.159627 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lbvd4\"" Jul 17 19:15:00.170506 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:15:00.168501 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738595-9jpgd" Jul 17 19:15:00.170506 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:15:00.168773 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738595-9jpgd"] Jul 17 19:15:00.302413 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:15:00.302388 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738595-9jpgd"] Jul 17 19:15:00.304125 ip-10-0-134-137 kubenswrapper[2561]: W0717 19:15:00.304099 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58028cbe_d060_46c5_82ea_5a94b55906bb.slice/crio-954d5f6dff0c75f2c036e53d30fa9a260eecbdb4155eb6c8d870ea59dfc5d010 WatchSource:0}: Error finding container 954d5f6dff0c75f2c036e53d30fa9a260eecbdb4155eb6c8d870ea59dfc5d010: Status 404 returned error can't find the container with id 954d5f6dff0c75f2c036e53d30fa9a260eecbdb4155eb6c8d870ea59dfc5d010 Jul 17 19:15:00.305873 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:15:00.305856 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 19:15:00.721148 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:15:00.721110 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738595-9jpgd" event={"ID":"58028cbe-d060-46c5-82ea-5a94b55906bb","Type":"ContainerStarted","Data":"954d5f6dff0c75f2c036e53d30fa9a260eecbdb4155eb6c8d870ea59dfc5d010"} Jul 17 19:15:02.077460 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:15:02.077426 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpjzq_4b162535-066a-471a-aee4-99c07c6e1bb4/ovn-acl-logging/0.log" Jul 17 19:15:02.079928 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:15:02.079899 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpjzq_4b162535-066a-471a-aee4-99c07c6e1bb4/ovn-acl-logging/0.log" Jul 17 19:15:03.739439 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:15:03.739395 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738595-9jpgd" event={"ID":"58028cbe-d060-46c5-82ea-5a94b55906bb","Type":"ContainerStarted","Data":"8cba30681a89741abc16e1edb044f4ee5f44cf96d0c1d4057cb2085296df5741"} Jul 17 19:15:03.758851 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:15:03.758796 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738595-9jpgd" podStartSLOduration=1.147593686 podStartE2EDuration="3.758780491s" totalImagesPullingTime="2.611186805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:15:03.757442161 +0000 UTC m=+1202.087258362" watchObservedRunningTime="2026-07-17 19:15:03.758780491 +0000 UTC m=+1202.088596706" Jul 17 19:15:04.745182 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:15:04.745144 2561 generic.go:356] "Generic (PLEG): container finished" podID="58028cbe-d060-46c5-82ea-5a94b55906bb" containerID="8cba30681a89741abc16e1edb044f4ee5f44cf96d0c1d4057cb2085296df5741" exitCode=0 Jul 17 19:15:04.745669 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:15:04.745225 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738595-9jpgd" event={"ID":"58028cbe-d060-46c5-82ea-5a94b55906bb","Type":"ContainerDied","Data":"8cba30681a89741abc16e1edb044f4ee5f44cf96d0c1d4057cb2085296df5741"} Jul 17 19:15:05.881609 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:15:05.881581 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738595-9jpgd" Jul 17 19:15:06.756626 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:15:06.756587 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738595-9jpgd" event={"ID":"58028cbe-d060-46c5-82ea-5a94b55906bb","Type":"ContainerDied","Data":"954d5f6dff0c75f2c036e53d30fa9a260eecbdb4155eb6c8d870ea59dfc5d010"} Jul 17 19:15:06.758707 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:15:06.756649 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="954d5f6dff0c75f2c036e53d30fa9a260eecbdb4155eb6c8d870ea59dfc5d010" Jul 17 19:15:06.758707 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:15:06.756659 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738595-9jpgd" Jul 17 19:15:10.609043 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:15:10.609008 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:15:23.112476 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:15:23.112440 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:15:32.927605 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:15:32.927572 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:15:43.998802 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:15:43.998766 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:15:52.201439 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:15:52.201405 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:16:03.201419 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:16:03.201327 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:16:14.180088 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:16:14.180051 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:16:23.289999 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:16:23.289964 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:16:34.979145 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:16:34.979106 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:16:55.891485 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:16:55.891447 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:17:05.184445 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:17:05.184412 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:17:19.883504 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:17:19.883463 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:17:31.984636 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:17:31.984552 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:17:52.284910 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:17:52.284872 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:18:03.989937 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:18:03.989901 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:18:12.770563 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:18:12.770514 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:18:26.676549 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:18:26.676492 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:18:35.883158 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:18:35.883120 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:18:49.378253 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:18:49.378216 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:18:58.483143 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:18:58.483048 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:19:08.384144 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:19:08.384104 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:19:18.783435 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:19:18.783394 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:19:30.085519 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:19:30.085474 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:19:37.105061 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:19:37.105014 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:19:54.885959 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:19:54.885912 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:20:02.109303 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:20:02.109275 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpjzq_4b162535-066a-471a-aee4-99c07c6e1bb4/ovn-acl-logging/0.log" Jul 17 19:20:02.119429 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:20:02.119405 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpjzq_4b162535-066a-471a-aee4-99c07c6e1bb4/ovn-acl-logging/0.log" Jul 17 19:20:09.477755 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:20:09.477721 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:20:20.376116 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:20:20.376078 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:20:33.181085 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:20:33.180993 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:20:43.285482 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:20:43.285440 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:20:55.478126 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:20:55.478092 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:20:58.683362 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:20:58.683322 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4acef77-md8cn"] Jul 17 19:20:58.683767 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:20:58.683658 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58028cbe-d060-46c5-82ea-5a94b55906bb" containerName="cleanup" Jul 17 19:20:58.683767 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:20:58.683673 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="58028cbe-d060-46c5-82ea-5a94b55906bb" containerName="cleanup" Jul 17 19:20:58.683767 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:20:58.683725 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="58028cbe-d060-46c5-82ea-5a94b55906bb" containerName="cleanup" Jul 17 19:20:58.686765 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:20:58.686748 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4acef77-md8cn" Jul 17 19:20:58.689838 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:20:58.689816 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lbvd4\"" Jul 17 19:20:58.690116 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:20:58.690100 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 19:20:58.696118 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:20:58.696088 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4acef77-md8cn"] Jul 17 19:20:58.716393 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:20:58.716362 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ecf5346-dcd7-4fb4-9ec3-36a8b9e99d54-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a4acef77-md8cn\" (UID: \"8ecf5346-dcd7-4fb4-9ec3-36a8b9e99d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4acef77-md8cn" Jul 17 19:20:58.817850 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:20:58.817809 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ecf5346-dcd7-4fb4-9ec3-36a8b9e99d54-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a4acef77-md8cn\" (UID: \"8ecf5346-dcd7-4fb4-9ec3-36a8b9e99d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4acef77-md8cn" Jul 17 19:20:58.818481 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:20:58.818452 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ecf5346-dcd7-4fb4-9ec3-36a8b9e99d54-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a4acef77-md8cn\" (UID: \"8ecf5346-dcd7-4fb4-9ec3-36a8b9e99d54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4acef77-md8cn" Jul 17 19:20:58.998266 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:20:58.998216 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4acef77-md8cn" Jul 17 19:20:59.132860 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:20:59.132830 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4acef77-md8cn"] Jul 17 19:20:59.134720 ip-10-0-134-137 kubenswrapper[2561]: W0717 19:20:59.134691 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ecf5346_dcd7_4fb4_9ec3_36a8b9e99d54.slice/crio-676f8ca85e1afd8cfa230ba4677170af4e027abbfb354af2a4408f137655997b WatchSource:0}: Error finding container 676f8ca85e1afd8cfa230ba4677170af4e027abbfb354af2a4408f137655997b: Status 404 returned error can't find the container with id 676f8ca85e1afd8cfa230ba4677170af4e027abbfb354af2a4408f137655997b Jul 17 19:20:59.397810 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:20:59.397775 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4acef77-md8cn" event={"ID":"8ecf5346-dcd7-4fb4-9ec3-36a8b9e99d54","Type":"ContainerStarted","Data":"2ef1f1bea100796a225c355bf48231ced7be314067f479c5c00fc7f4a3216725"} Jul 17 19:20:59.398014 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:20:59.397831 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4acef77-md8cn" event={"ID":"8ecf5346-dcd7-4fb4-9ec3-36a8b9e99d54","Type":"ContainerStarted","Data":"676f8ca85e1afd8cfa230ba4677170af4e027abbfb354af2a4408f137655997b"} Jul 17 19:20:59.413647 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:20:59.413586 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4acef77-md8cn" podStartSLOduration=1.413570252 podStartE2EDuration="1.413570252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:20:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:20:59.413138789 +0000 UTC m=+1557.742954998" watchObservedRunningTime="2026-07-17 19:20:59.413570252 +0000 UTC m=+1557.743386460" Jul 17 19:21:00.404379 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:21:00.404341 2561 generic.go:356] "Generic (PLEG): container finished" podID="8ecf5346-dcd7-4fb4-9ec3-36a8b9e99d54" containerID="2ef1f1bea100796a225c355bf48231ced7be314067f479c5c00fc7f4a3216725" exitCode=7 Jul 17 19:21:00.404846 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:21:00.404435 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4acef77-md8cn" event={"ID":"8ecf5346-dcd7-4fb4-9ec3-36a8b9e99d54","Type":"ContainerDied","Data":"2ef1f1bea100796a225c355bf48231ced7be314067f479c5c00fc7f4a3216725"} Jul 17 19:21:00.404846 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:21:00.404616 2561 scope.go:117] "RemoveContainer" containerID="2ef1f1bea100796a225c355bf48231ced7be314067f479c5c00fc7f4a3216725" Jul 17 19:21:01.410729 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:21:01.410698 2561 generic.go:356] "Generic (PLEG): container finished" podID="8ecf5346-dcd7-4fb4-9ec3-36a8b9e99d54" containerID="b72bf38441009e237308f669d6d8bffd8a3394d12ef6399c05a332afa543377d" exitCode=7 Jul 17 19:21:01.411114 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:21:01.410765 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4acef77-md8cn" event={"ID":"8ecf5346-dcd7-4fb4-9ec3-36a8b9e99d54","Type":"ContainerDied","Data":"b72bf38441009e237308f669d6d8bffd8a3394d12ef6399c05a332afa543377d"} Jul 17 19:21:01.411114 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:21:01.410823 2561 scope.go:117] "RemoveContainer" containerID="2ef1f1bea100796a225c355bf48231ced7be314067f479c5c00fc7f4a3216725" Jul 17 19:21:01.411114 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:21:01.410989 2561 scope.go:117] "RemoveContainer" containerID="b72bf38441009e237308f669d6d8bffd8a3394d12ef6399c05a332afa543377d" Jul 17 19:21:01.411233 ip-10-0-134-137 kubenswrapper[2561]: E0717 19:21:01.411201 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a4acef77-md8cn_odh-ai-gateway-infra(8ecf5346-dcd7-4fb4-9ec3-36a8b9e99d54)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4acef77-md8cn" podUID="8ecf5346-dcd7-4fb4-9ec3-36a8b9e99d54" Jul 17 19:21:02.417421 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:21:02.417392 2561 scope.go:117] "RemoveContainer" containerID="b72bf38441009e237308f669d6d8bffd8a3394d12ef6399c05a332afa543377d" Jul 17 19:21:02.417853 ip-10-0-134-137 kubenswrapper[2561]: E0717 19:21:02.417611 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a4acef77-md8cn_odh-ai-gateway-infra(8ecf5346-dcd7-4fb4-9ec3-36a8b9e99d54)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4acef77-md8cn" podUID="8ecf5346-dcd7-4fb4-9ec3-36a8b9e99d54" Jul 17 19:21:05.890148 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:21:05.890111 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:21:16.093733 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:21:16.093698 2561 scope.go:117] "RemoveContainer" containerID="b72bf38441009e237308f669d6d8bffd8a3394d12ef6399c05a332afa543377d" Jul 17 19:21:16.485228 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:21:16.485142 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4acef77-md8cn" event={"ID":"8ecf5346-dcd7-4fb4-9ec3-36a8b9e99d54","Type":"ContainerStarted","Data":"5c245ccfe4e43d144ac44c525b1e446877cd723fd6da4ff0758513a47c9f8ab9"} Jul 17 19:21:17.491702 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:21:17.491670 2561 generic.go:356] "Generic (PLEG): container finished" podID="8ecf5346-dcd7-4fb4-9ec3-36a8b9e99d54" containerID="5c245ccfe4e43d144ac44c525b1e446877cd723fd6da4ff0758513a47c9f8ab9" exitCode=0 Jul 17 19:21:17.492299 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:21:17.491725 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4acef77-md8cn" event={"ID":"8ecf5346-dcd7-4fb4-9ec3-36a8b9e99d54","Type":"ContainerDied","Data":"5c245ccfe4e43d144ac44c525b1e446877cd723fd6da4ff0758513a47c9f8ab9"} Jul 17 19:21:17.492299 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:21:17.491771 2561 scope.go:117] "RemoveContainer" containerID="b72bf38441009e237308f669d6d8bffd8a3394d12ef6399c05a332afa543377d" Jul 17 19:21:17.513623 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:21:17.513593 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4acef77-md8cn"] Jul 17 19:21:18.628519 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:21:18.628492 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4acef77-md8cn" Jul 17 19:21:18.664759 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:21:18.664728 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ecf5346-dcd7-4fb4-9ec3-36a8b9e99d54-maas-api-service-ca\") pod \"8ecf5346-dcd7-4fb4-9ec3-36a8b9e99d54\" (UID: \"8ecf5346-dcd7-4fb4-9ec3-36a8b9e99d54\") " Jul 17 19:21:18.665088 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:21:18.665067 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ecf5346-dcd7-4fb4-9ec3-36a8b9e99d54-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8ecf5346-dcd7-4fb4-9ec3-36a8b9e99d54" (UID: "8ecf5346-dcd7-4fb4-9ec3-36a8b9e99d54"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 19:21:18.765665 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:21:18.765557 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ecf5346-dcd7-4fb4-9ec3-36a8b9e99d54-maas-api-service-ca\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:21:19.503252 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:21:19.503220 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4acef77-md8cn" Jul 17 19:21:19.503442 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:21:19.503221 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4acef77-md8cn" event={"ID":"8ecf5346-dcd7-4fb4-9ec3-36a8b9e99d54","Type":"ContainerDied","Data":"676f8ca85e1afd8cfa230ba4677170af4e027abbfb354af2a4408f137655997b"} Jul 17 19:21:19.503442 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:21:19.503363 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="676f8ca85e1afd8cfa230ba4677170af4e027abbfb354af2a4408f137655997b" Jul 17 19:21:19.531493 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:21:19.531464 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4acef77-md8cn"] Jul 17 19:21:19.537243 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:21:19.537217 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4acef77-md8cn"] Jul 17 19:21:20.098305 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:21:20.098263 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ecf5346-dcd7-4fb4-9ec3-36a8b9e99d54" path="/var/lib/kubelet/pods/8ecf5346-dcd7-4fb4-9ec3-36a8b9e99d54/volumes" Jul 17 19:21:58.727173 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:21:58.727092 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4acef77-s28rl"] Jul 17 19:21:58.727644 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:21:58.727429 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ecf5346-dcd7-4fb4-9ec3-36a8b9e99d54" containerName="revoke-keys" Jul 17 19:21:58.727644 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:21:58.727444 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ecf5346-dcd7-4fb4-9ec3-36a8b9e99d54" containerName="revoke-keys" Jul 17 19:21:58.727644 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:21:58.727452 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ecf5346-dcd7-4fb4-9ec3-36a8b9e99d54" containerName="revoke-keys" Jul 17 19:21:58.727644 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:21:58.727458 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ecf5346-dcd7-4fb4-9ec3-36a8b9e99d54" containerName="revoke-keys" Jul 17 19:21:58.727644 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:21:58.727516 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ecf5346-dcd7-4fb4-9ec3-36a8b9e99d54" containerName="revoke-keys" Jul 17 19:21:58.727644 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:21:58.727547 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ecf5346-dcd7-4fb4-9ec3-36a8b9e99d54" containerName="revoke-keys" Jul 17 19:21:58.730509 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:21:58.730489 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4acef77-s28rl" Jul 17 19:21:58.733105 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:21:58.733082 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lbvd4\"" Jul 17 19:21:58.733239 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:21:58.733170 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 19:21:58.739463 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:21:58.739432 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4acef77-s28rl"] Jul 17 19:21:58.900743 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:21:58.900700 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2476029-0002-4134-9c1f-9c3461563a1b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a4acef77-s28rl\" (UID: \"a2476029-0002-4134-9c1f-9c3461563a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4acef77-s28rl" Jul 17 19:21:59.002102 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:21:59.002070 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2476029-0002-4134-9c1f-9c3461563a1b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a4acef77-s28rl\" (UID: \"a2476029-0002-4134-9c1f-9c3461563a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4acef77-s28rl" Jul 17 19:21:59.002702 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:21:59.002682 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2476029-0002-4134-9c1f-9c3461563a1b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a4acef77-s28rl\" (UID: \"a2476029-0002-4134-9c1f-9c3461563a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4acef77-s28rl" Jul 17 19:21:59.042148 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:21:59.042115 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4acef77-s28rl" Jul 17 19:21:59.171252 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:21:59.171211 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4acef77-s28rl"] Jul 17 19:21:59.173512 ip-10-0-134-137 kubenswrapper[2561]: W0717 19:21:59.173488 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2476029_0002_4134_9c1f_9c3461563a1b.slice/crio-6140c8138ab4cc85c960386adfc8bfe5b4d32424d366c8a69feb3df2d0d6854b WatchSource:0}: Error finding container 6140c8138ab4cc85c960386adfc8bfe5b4d32424d366c8a69feb3df2d0d6854b: Status 404 returned error can't find the container with id 6140c8138ab4cc85c960386adfc8bfe5b4d32424d366c8a69feb3df2d0d6854b Jul 17 19:21:59.700953 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:21:59.700867 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4acef77-s28rl" event={"ID":"a2476029-0002-4134-9c1f-9c3461563a1b","Type":"ContainerStarted","Data":"6469e78509ca1749215ef79947760c98466cca531303bbcad3314f92ecf7fd1b"} Jul 17 19:21:59.700953 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:21:59.700918 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4acef77-s28rl" event={"ID":"a2476029-0002-4134-9c1f-9c3461563a1b","Type":"ContainerStarted","Data":"6140c8138ab4cc85c960386adfc8bfe5b4d32424d366c8a69feb3df2d0d6854b"} Jul 17 19:21:59.717829 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:21:59.717780 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4acef77-s28rl" podStartSLOduration=1.7177647459999998 podStartE2EDuration="1.717764746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:21:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:21:59.71650727 +0000 UTC m=+1618.046323480" watchObservedRunningTime="2026-07-17 19:21:59.717764746 +0000 UTC m=+1618.047580932" Jul 17 19:22:00.707017 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:00.706981 2561 generic.go:356] "Generic (PLEG): container finished" podID="a2476029-0002-4134-9c1f-9c3461563a1b" containerID="6469e78509ca1749215ef79947760c98466cca531303bbcad3314f92ecf7fd1b" exitCode=0 Jul 17 19:22:00.707424 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:00.707077 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4acef77-s28rl" event={"ID":"a2476029-0002-4134-9c1f-9c3461563a1b","Type":"ContainerDied","Data":"6469e78509ca1749215ef79947760c98466cca531303bbcad3314f92ecf7fd1b"} Jul 17 19:22:01.842879 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:01.842854 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4acef77-s28rl" Jul 17 19:22:02.028519 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:02.028484 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2476029-0002-4134-9c1f-9c3461563a1b-maas-api-service-ca\") pod \"a2476029-0002-4134-9c1f-9c3461563a1b\" (UID: \"a2476029-0002-4134-9c1f-9c3461563a1b\") " Jul 17 19:22:02.028887 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:02.028861 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2476029-0002-4134-9c1f-9c3461563a1b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a2476029-0002-4134-9c1f-9c3461563a1b" (UID: "a2476029-0002-4134-9c1f-9c3461563a1b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 19:22:02.129962 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:02.129934 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2476029-0002-4134-9c1f-9c3461563a1b-maas-api-service-ca\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:22:02.719253 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:02.719224 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4acef77-s28rl" Jul 17 19:22:02.719420 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:02.719263 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4acef77-s28rl" event={"ID":"a2476029-0002-4134-9c1f-9c3461563a1b","Type":"ContainerDied","Data":"6140c8138ab4cc85c960386adfc8bfe5b4d32424d366c8a69feb3df2d0d6854b"} Jul 17 19:22:02.719420 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:02.719348 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6140c8138ab4cc85c960386adfc8bfe5b4d32424d366c8a69feb3df2d0d6854b" Jul 17 19:22:12.277257 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:12.277218 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:22:25.867659 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:25.867622 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-59b00c87-7468697dfc-hxdk6"] Jul 17 19:22:25.868168 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:25.867929 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2476029-0002-4134-9c1f-9c3461563a1b" containerName="revoke-keys" Jul 17 19:22:25.868168 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:25.867939 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2476029-0002-4134-9c1f-9c3461563a1b" containerName="revoke-keys" Jul 17 19:22:25.868168 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:25.867952 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ecf5346-dcd7-4fb4-9ec3-36a8b9e99d54" containerName="revoke-keys" Jul 17 19:22:25.868168 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:25.867958 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ecf5346-dcd7-4fb4-9ec3-36a8b9e99d54" containerName="revoke-keys" Jul 17 19:22:25.868168 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:25.868019 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2476029-0002-4134-9c1f-9c3461563a1b" containerName="revoke-keys" Jul 17 19:22:25.868168 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:25.868027 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ecf5346-dcd7-4fb4-9ec3-36a8b9e99d54" containerName="revoke-keys" Jul 17 19:22:25.871224 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:25.871203 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59b00c87-7468697dfc-hxdk6" Jul 17 19:22:25.873739 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:25.873716 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-59b00c87-serving-cert\"" Jul 17 19:22:25.873836 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:25.873754 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 17 19:22:25.874632 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:25.874604 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-mhzxx\"" Jul 17 19:22:25.874716 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:25.874639 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 19:22:25.882128 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:25.882102 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-59b00c87-7468697dfc-hxdk6"] Jul 17 19:22:25.912282 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:25.912252 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db022d02-3aa3-43b0-80a6-9d9c73a03b46-openshift-service-ca\") pod \"maas-api-e2e-migrate-59b00c87-7468697dfc-hxdk6\" (UID: \"db022d02-3aa3-43b0-80a6-9d9c73a03b46\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59b00c87-7468697dfc-hxdk6" Jul 17 19:22:25.912451 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:25.912291 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db022d02-3aa3-43b0-80a6-9d9c73a03b46-maas-api-tls\") pod \"maas-api-e2e-migrate-59b00c87-7468697dfc-hxdk6\" (UID: \"db022d02-3aa3-43b0-80a6-9d9c73a03b46\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59b00c87-7468697dfc-hxdk6" Jul 17 19:22:25.912451 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:25.912315 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p57b\" (UniqueName: \"kubernetes.io/projected/db022d02-3aa3-43b0-80a6-9d9c73a03b46-kube-api-access-2p57b\") pod \"maas-api-e2e-migrate-59b00c87-7468697dfc-hxdk6\" (UID: \"db022d02-3aa3-43b0-80a6-9d9c73a03b46\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59b00c87-7468697dfc-hxdk6" Jul 17 19:22:26.013750 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:26.013717 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db022d02-3aa3-43b0-80a6-9d9c73a03b46-openshift-service-ca\") pod \"maas-api-e2e-migrate-59b00c87-7468697dfc-hxdk6\" (UID: \"db022d02-3aa3-43b0-80a6-9d9c73a03b46\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59b00c87-7468697dfc-hxdk6" Jul 17 19:22:26.013944 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:26.013760 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db022d02-3aa3-43b0-80a6-9d9c73a03b46-maas-api-tls\") pod \"maas-api-e2e-migrate-59b00c87-7468697dfc-hxdk6\" (UID: \"db022d02-3aa3-43b0-80a6-9d9c73a03b46\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59b00c87-7468697dfc-hxdk6" Jul 17 19:22:26.013944 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:26.013789 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p57b\" (UniqueName: \"kubernetes.io/projected/db022d02-3aa3-43b0-80a6-9d9c73a03b46-kube-api-access-2p57b\") pod \"maas-api-e2e-migrate-59b00c87-7468697dfc-hxdk6\" (UID: \"db022d02-3aa3-43b0-80a6-9d9c73a03b46\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59b00c87-7468697dfc-hxdk6" Jul 17 19:22:26.013944 ip-10-0-134-137 kubenswrapper[2561]: E0717 19:22:26.013892 2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-59b00c87-serving-cert: secret "maas-api-e2e-migrate-59b00c87-serving-cert" not found Jul 17 19:22:26.014102 ip-10-0-134-137 kubenswrapper[2561]: E0717 19:22:26.013974 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db022d02-3aa3-43b0-80a6-9d9c73a03b46-maas-api-tls podName:db022d02-3aa3-43b0-80a6-9d9c73a03b46 nodeName:}" failed. No retries permitted until 2026-07-17 19:22:26.513954703 +0000 UTC m=+1644.843770900 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/db022d02-3aa3-43b0-80a6-9d9c73a03b46-maas-api-tls") pod "maas-api-e2e-migrate-59b00c87-7468697dfc-hxdk6" (UID: "db022d02-3aa3-43b0-80a6-9d9c73a03b46") : secret "maas-api-e2e-migrate-59b00c87-serving-cert" not found Jul 17 19:22:26.014477 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:26.014459 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db022d02-3aa3-43b0-80a6-9d9c73a03b46-openshift-service-ca\") pod \"maas-api-e2e-migrate-59b00c87-7468697dfc-hxdk6\" (UID: \"db022d02-3aa3-43b0-80a6-9d9c73a03b46\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59b00c87-7468697dfc-hxdk6" Jul 17 19:22:26.022201 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:26.022176 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p57b\" (UniqueName: \"kubernetes.io/projected/db022d02-3aa3-43b0-80a6-9d9c73a03b46-kube-api-access-2p57b\") pod \"maas-api-e2e-migrate-59b00c87-7468697dfc-hxdk6\" (UID: \"db022d02-3aa3-43b0-80a6-9d9c73a03b46\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59b00c87-7468697dfc-hxdk6" Jul 17 19:22:26.518442 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:26.518406 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db022d02-3aa3-43b0-80a6-9d9c73a03b46-maas-api-tls\") pod \"maas-api-e2e-migrate-59b00c87-7468697dfc-hxdk6\" (UID: \"db022d02-3aa3-43b0-80a6-9d9c73a03b46\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59b00c87-7468697dfc-hxdk6" Jul 17 19:22:26.521115 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:26.521090 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db022d02-3aa3-43b0-80a6-9d9c73a03b46-maas-api-tls\") pod \"maas-api-e2e-migrate-59b00c87-7468697dfc-hxdk6\" (UID: \"db022d02-3aa3-43b0-80a6-9d9c73a03b46\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59b00c87-7468697dfc-hxdk6" Jul 17 19:22:26.782483 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:26.782383 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59b00c87-7468697dfc-hxdk6" Jul 17 19:22:26.922193 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:26.922163 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-59b00c87-7468697dfc-hxdk6"] Jul 17 19:22:26.923705 ip-10-0-134-137 kubenswrapper[2561]: W0717 19:22:26.923677 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb022d02_3aa3_43b0_80a6_9d9c73a03b46.slice/crio-0e5ccfd48b433286dd35b98c4981e4d0efc62116843438315cf6ce50a35d63e6 WatchSource:0}: Error finding container 0e5ccfd48b433286dd35b98c4981e4d0efc62116843438315cf6ce50a35d63e6: Status 404 returned error can't find the container with id 0e5ccfd48b433286dd35b98c4981e4d0efc62116843438315cf6ce50a35d63e6 Jul 17 19:22:26.925018 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:26.925000 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 19:22:27.843837 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:27.843759 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59b00c87-7468697dfc-hxdk6" event={"ID":"db022d02-3aa3-43b0-80a6-9d9c73a03b46","Type":"ContainerStarted","Data":"0e5ccfd48b433286dd35b98c4981e4d0efc62116843438315cf6ce50a35d63e6"} Jul 17 19:22:29.854697 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:29.854655 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59b00c87-7468697dfc-hxdk6" event={"ID":"db022d02-3aa3-43b0-80a6-9d9c73a03b46","Type":"ContainerStarted","Data":"7400678190a07e815f60ae3c25d8177294b34d7b9d552e7c6885318fa7c31f72"} Jul 17 19:22:29.855124 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:29.854873 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59b00c87-7468697dfc-hxdk6" Jul 17 19:22:29.878459 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:29.878409 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59b00c87-7468697dfc-hxdk6" podStartSLOduration=2.6439635150000003 podStartE2EDuration="4.878392449s" totalImagesPullingTime="2.234428934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:22:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:22:29.876464824 +0000 UTC m=+1648.206281033" watchObservedRunningTime="2026-07-17 19:22:29.878392449 +0000 UTC m=+1648.208208657" Jul 17 19:22:30.280342 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:30.280301 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59b00c87-nrd4t"] Jul 17 19:22:30.283797 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:30.283776 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59b00c87-nrd4t" Jul 17 19:22:30.286255 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:30.286229 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lbvd4\"" Jul 17 19:22:30.291212 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:30.291177 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59b00c87-nrd4t"] Jul 17 19:22:30.351918 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:30.351886 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65b04be2-813a-4ed7-a766-04c3d8a42a8f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-59b00c87-nrd4t\" (UID: \"65b04be2-813a-4ed7-a766-04c3d8a42a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59b00c87-nrd4t" Jul 17 19:22:30.452642 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:30.452602 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65b04be2-813a-4ed7-a766-04c3d8a42a8f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-59b00c87-nrd4t\" (UID: \"65b04be2-813a-4ed7-a766-04c3d8a42a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59b00c87-nrd4t" Jul 17 19:22:30.453290 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:30.453264 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65b04be2-813a-4ed7-a766-04c3d8a42a8f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-59b00c87-nrd4t\" (UID: \"65b04be2-813a-4ed7-a766-04c3d8a42a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59b00c87-nrd4t" Jul 17 19:22:30.595676 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:30.595593 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59b00c87-nrd4t" Jul 17 19:22:30.730474 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:30.730448 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59b00c87-nrd4t"] Jul 17 19:22:30.731846 ip-10-0-134-137 kubenswrapper[2561]: W0717 19:22:30.731817 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65b04be2_813a_4ed7_a766_04c3d8a42a8f.slice/crio-b0b6990fdb8be931e1924bb3a50895112e78393cb353894f2b79ede8fd717552 WatchSource:0}: Error finding container b0b6990fdb8be931e1924bb3a50895112e78393cb353894f2b79ede8fd717552: Status 404 returned error can't find the container with id b0b6990fdb8be931e1924bb3a50895112e78393cb353894f2b79ede8fd717552 Jul 17 19:22:30.861344 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:30.861256 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59b00c87-nrd4t" event={"ID":"65b04be2-813a-4ed7-a766-04c3d8a42a8f","Type":"ContainerStarted","Data":"b0b6990fdb8be931e1924bb3a50895112e78393cb353894f2b79ede8fd717552"} Jul 17 19:22:31.867379 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:31.867340 2561 generic.go:356] "Generic (PLEG): container finished" podID="65b04be2-813a-4ed7-a766-04c3d8a42a8f" containerID="52128530c75563f1d8b4b9b09615df7c5f74617cbb0b623b65d0a67f353e562c" exitCode=7 Jul 17 19:22:31.867815 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:31.867434 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59b00c87-nrd4t" event={"ID":"65b04be2-813a-4ed7-a766-04c3d8a42a8f","Type":"ContainerDied","Data":"52128530c75563f1d8b4b9b09615df7c5f74617cbb0b623b65d0a67f353e562c"} Jul 17 19:22:31.867815 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:31.867683 2561 scope.go:117] "RemoveContainer" containerID="52128530c75563f1d8b4b9b09615df7c5f74617cbb0b623b65d0a67f353e562c" Jul 17 19:22:32.686259 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:32.686224 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:22:32.874292 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:32.874207 2561 generic.go:356] "Generic (PLEG): container finished" podID="65b04be2-813a-4ed7-a766-04c3d8a42a8f" containerID="601a3c498edc0f2bf9d218b029c7679abcc4c4154401b774c5458a87ea31e7aa" exitCode=7 Jul 17 19:22:32.874292 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:32.874275 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59b00c87-nrd4t" event={"ID":"65b04be2-813a-4ed7-a766-04c3d8a42a8f","Type":"ContainerDied","Data":"601a3c498edc0f2bf9d218b029c7679abcc4c4154401b774c5458a87ea31e7aa"} Jul 17 19:22:32.874776 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:32.874328 2561 scope.go:117] "RemoveContainer" containerID="52128530c75563f1d8b4b9b09615df7c5f74617cbb0b623b65d0a67f353e562c" Jul 17 19:22:32.874776 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:32.874509 2561 scope.go:117] "RemoveContainer" containerID="601a3c498edc0f2bf9d218b029c7679abcc4c4154401b774c5458a87ea31e7aa" Jul 17 19:22:32.874776 ip-10-0-134-137 kubenswrapper[2561]: E0717 19:22:32.874714 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-59b00c87-nrd4t_odh-ai-gateway-infra(65b04be2-813a-4ed7-a766-04c3d8a42a8f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59b00c87-nrd4t" podUID="65b04be2-813a-4ed7-a766-04c3d8a42a8f" Jul 17 19:22:33.881325 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:33.881287 2561 scope.go:117] "RemoveContainer" containerID="601a3c498edc0f2bf9d218b029c7679abcc4c4154401b774c5458a87ea31e7aa" Jul 17 19:22:33.881740 ip-10-0-134-137 kubenswrapper[2561]: E0717 19:22:33.881500 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-59b00c87-nrd4t_odh-ai-gateway-infra(65b04be2-813a-4ed7-a766-04c3d8a42a8f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59b00c87-nrd4t" podUID="65b04be2-813a-4ed7-a766-04c3d8a42a8f" Jul 17 19:22:35.866726 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:35.866694 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59b00c87-7468697dfc-hxdk6" Jul 17 19:22:46.099363 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:46.099334 2561 scope.go:117] "RemoveContainer" containerID="601a3c498edc0f2bf9d218b029c7679abcc4c4154401b774c5458a87ea31e7aa" Jul 17 19:22:46.949237 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:46.949195 2561 generic.go:356] "Generic (PLEG): container finished" podID="65b04be2-813a-4ed7-a766-04c3d8a42a8f" containerID="b499094f2f4cf8ce599c20e1764dfea6700ae3b5b31123f07dc5612a06a34cd5" exitCode=0 Jul 17 19:22:46.949408 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:46.949259 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59b00c87-nrd4t" event={"ID":"65b04be2-813a-4ed7-a766-04c3d8a42a8f","Type":"ContainerDied","Data":"b499094f2f4cf8ce599c20e1764dfea6700ae3b5b31123f07dc5612a06a34cd5"} Jul 17 19:22:46.949408 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:46.949325 2561 scope.go:117] "RemoveContainer" containerID="601a3c498edc0f2bf9d218b029c7679abcc4c4154401b774c5458a87ea31e7aa" Jul 17 19:22:47.975845 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:47.975806 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59b00c87-nrd4t"] Jul 17 19:22:48.091717 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:48.091690 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59b00c87-nrd4t" Jul 17 19:22:48.209774 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:48.209686 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65b04be2-813a-4ed7-a766-04c3d8a42a8f-maas-api-service-ca\") pod \"65b04be2-813a-4ed7-a766-04c3d8a42a8f\" (UID: \"65b04be2-813a-4ed7-a766-04c3d8a42a8f\") " Jul 17 19:22:48.210063 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:48.210037 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65b04be2-813a-4ed7-a766-04c3d8a42a8f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "65b04be2-813a-4ed7-a766-04c3d8a42a8f" (UID: "65b04be2-813a-4ed7-a766-04c3d8a42a8f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 19:22:48.311207 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:48.311176 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65b04be2-813a-4ed7-a766-04c3d8a42a8f-maas-api-service-ca\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:22:48.962328 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:48.962288 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59b00c87-nrd4t" event={"ID":"65b04be2-813a-4ed7-a766-04c3d8a42a8f","Type":"ContainerDied","Data":"b0b6990fdb8be931e1924bb3a50895112e78393cb353894f2b79ede8fd717552"} Jul 17 19:22:48.962328 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:48.962327 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59b00c87-nrd4t" Jul 17 19:22:48.962590 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:48.962347 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0b6990fdb8be931e1924bb3a50895112e78393cb353894f2b79ede8fd717552" Jul 17 19:22:48.992005 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:48.991968 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59b00c87-nrd4t"] Jul 17 19:22:48.997250 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:48.997223 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59b00c87-nrd4t"] Jul 17 19:22:50.098944 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:22:50.098910 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65b04be2-813a-4ed7-a766-04c3d8a42a8f" path="/var/lib/kubelet/pods/65b04be2-813a-4ed7-a766-04c3d8a42a8f/volumes" Jul 17 19:23:20.322775 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:20.322736 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59b00c87-g965g"] Jul 17 19:23:20.323198 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:20.323051 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65b04be2-813a-4ed7-a766-04c3d8a42a8f" containerName="revoke-keys" Jul 17 19:23:20.323198 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:20.323063 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="65b04be2-813a-4ed7-a766-04c3d8a42a8f" containerName="revoke-keys" Jul 17 19:23:20.323198 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:20.323071 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65b04be2-813a-4ed7-a766-04c3d8a42a8f" containerName="revoke-keys" Jul 17 19:23:20.323198 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:20.323076 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="65b04be2-813a-4ed7-a766-04c3d8a42a8f" containerName="revoke-keys" Jul 17 19:23:20.323198 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:20.323127 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="65b04be2-813a-4ed7-a766-04c3d8a42a8f" containerName="revoke-keys" Jul 17 19:23:20.323198 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:20.323137 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="65b04be2-813a-4ed7-a766-04c3d8a42a8f" containerName="revoke-keys" Jul 17 19:23:20.327553 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:20.326756 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59b00c87-g965g" Jul 17 19:23:20.329704 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:20.329679 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lbvd4\"" Jul 17 19:23:20.335124 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:20.335095 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59b00c87-g965g"] Jul 17 19:23:20.387060 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:20.387026 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcf0e9f0-0268-41a5-8efe-d025c18d49f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-59b00c87-g965g\" (UID: \"dcf0e9f0-0268-41a5-8efe-d025c18d49f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59b00c87-g965g" Jul 17 19:23:20.487693 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:20.487656 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcf0e9f0-0268-41a5-8efe-d025c18d49f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-59b00c87-g965g\" (UID: \"dcf0e9f0-0268-41a5-8efe-d025c18d49f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59b00c87-g965g" Jul 17 19:23:20.488265 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:20.488243 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcf0e9f0-0268-41a5-8efe-d025c18d49f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-59b00c87-g965g\" (UID: \"dcf0e9f0-0268-41a5-8efe-d025c18d49f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59b00c87-g965g" Jul 17 19:23:20.641443 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:20.641362 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59b00c87-g965g" Jul 17 19:23:20.776864 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:20.776838 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59b00c87-g965g"] Jul 17 19:23:20.779217 ip-10-0-134-137 kubenswrapper[2561]: W0717 19:23:20.779190 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcf0e9f0_0268_41a5_8efe_d025c18d49f1.slice/crio-0110b33d52cb10d04453406a8c1c850a567b9eb02cbe5c666b856da097e11701 WatchSource:0}: Error finding container 0110b33d52cb10d04453406a8c1c850a567b9eb02cbe5c666b856da097e11701: Status 404 returned error can't find the container with id 0110b33d52cb10d04453406a8c1c850a567b9eb02cbe5c666b856da097e11701 Jul 17 19:23:21.123977 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:21.123936 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59b00c87-g965g" event={"ID":"dcf0e9f0-0268-41a5-8efe-d025c18d49f1","Type":"ContainerStarted","Data":"3206e98ed1f24a9e7d1d1771bfd12abae5f3e7e3696663daa3182ee1e71d90f5"} Jul 17 19:23:21.124177 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:21.124003 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59b00c87-g965g" event={"ID":"dcf0e9f0-0268-41a5-8efe-d025c18d49f1","Type":"ContainerStarted","Data":"0110b33d52cb10d04453406a8c1c850a567b9eb02cbe5c666b856da097e11701"} Jul 17 19:23:21.142350 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:21.142301 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59b00c87-g965g" podStartSLOduration=1.142282764 podStartE2EDuration="1.142282764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:23:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:23:21.141100564 +0000 UTC m=+1699.470916774" watchObservedRunningTime="2026-07-17 19:23:21.142282764 +0000 UTC m=+1699.472098980" Jul 17 19:23:22.130829 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:22.130795 2561 generic.go:356] "Generic (PLEG): container finished" podID="dcf0e9f0-0268-41a5-8efe-d025c18d49f1" containerID="3206e98ed1f24a9e7d1d1771bfd12abae5f3e7e3696663daa3182ee1e71d90f5" exitCode=0 Jul 17 19:23:22.131206 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:22.130877 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59b00c87-g965g" event={"ID":"dcf0e9f0-0268-41a5-8efe-d025c18d49f1","Type":"ContainerDied","Data":"3206e98ed1f24a9e7d1d1771bfd12abae5f3e7e3696663daa3182ee1e71d90f5"} Jul 17 19:23:23.268139 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:23.268114 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59b00c87-g965g" Jul 17 19:23:23.412630 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:23.412518 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcf0e9f0-0268-41a5-8efe-d025c18d49f1-maas-api-service-ca\") pod \"dcf0e9f0-0268-41a5-8efe-d025c18d49f1\" (UID: \"dcf0e9f0-0268-41a5-8efe-d025c18d49f1\") " Jul 17 19:23:23.412856 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:23.412835 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcf0e9f0-0268-41a5-8efe-d025c18d49f1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dcf0e9f0-0268-41a5-8efe-d025c18d49f1" (UID: "dcf0e9f0-0268-41a5-8efe-d025c18d49f1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 19:23:23.513678 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:23.513639 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcf0e9f0-0268-41a5-8efe-d025c18d49f1-maas-api-service-ca\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:23:24.143233 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:24.143202 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59b00c87-g965g" Jul 17 19:23:24.143233 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:24.143219 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59b00c87-g965g" event={"ID":"dcf0e9f0-0268-41a5-8efe-d025c18d49f1","Type":"ContainerDied","Data":"0110b33d52cb10d04453406a8c1c850a567b9eb02cbe5c666b856da097e11701"} Jul 17 19:23:24.143420 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:24.143266 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0110b33d52cb10d04453406a8c1c850a567b9eb02cbe5c666b856da097e11701" Jul 17 19:23:30.750198 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:30.750166 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-59b00c87-7468697dfc-hxdk6"] Jul 17 19:23:30.750649 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:30.750413 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59b00c87-7468697dfc-hxdk6" podUID="db022d02-3aa3-43b0-80a6-9d9c73a03b46" containerName="maas-api" containerID="cri-o://7400678190a07e815f60ae3c25d8177294b34d7b9d552e7c6885318fa7c31f72" gracePeriod=30 Jul 17 19:23:30.861768 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:30.861725 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59b00c87-7468697dfc-hxdk6" podUID="db022d02-3aa3-43b0-80a6-9d9c73a03b46" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.39:8443/health\": dial tcp 10.133.0.39:8443: connect: connection refused" Jul 17 19:23:31.001387 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:31.001326 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59b00c87-7468697dfc-hxdk6" Jul 17 19:23:31.079884 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:31.079843 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db022d02-3aa3-43b0-80a6-9d9c73a03b46-openshift-service-ca\") pod \"db022d02-3aa3-43b0-80a6-9d9c73a03b46\" (UID: \"db022d02-3aa3-43b0-80a6-9d9c73a03b46\") " Jul 17 19:23:31.080079 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:31.079922 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2p57b\" (UniqueName: \"kubernetes.io/projected/db022d02-3aa3-43b0-80a6-9d9c73a03b46-kube-api-access-2p57b\") pod \"db022d02-3aa3-43b0-80a6-9d9c73a03b46\" (UID: \"db022d02-3aa3-43b0-80a6-9d9c73a03b46\") " Jul 17 19:23:31.080079 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:31.079956 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db022d02-3aa3-43b0-80a6-9d9c73a03b46-maas-api-tls\") pod \"db022d02-3aa3-43b0-80a6-9d9c73a03b46\" (UID: \"db022d02-3aa3-43b0-80a6-9d9c73a03b46\") " Jul 17 19:23:31.080227 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:31.080211 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db022d02-3aa3-43b0-80a6-9d9c73a03b46-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "db022d02-3aa3-43b0-80a6-9d9c73a03b46" (UID: "db022d02-3aa3-43b0-80a6-9d9c73a03b46"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 19:23:31.082273 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:31.082245 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db022d02-3aa3-43b0-80a6-9d9c73a03b46-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "db022d02-3aa3-43b0-80a6-9d9c73a03b46" (UID: "db022d02-3aa3-43b0-80a6-9d9c73a03b46"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 19:23:31.082411 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:31.082391 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db022d02-3aa3-43b0-80a6-9d9c73a03b46-kube-api-access-2p57b" (OuterVolumeSpecName: "kube-api-access-2p57b") pod "db022d02-3aa3-43b0-80a6-9d9c73a03b46" (UID: "db022d02-3aa3-43b0-80a6-9d9c73a03b46"). InnerVolumeSpecName "kube-api-access-2p57b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 19:23:31.178059 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:31.178025 2561 generic.go:356] "Generic (PLEG): container finished" podID="db022d02-3aa3-43b0-80a6-9d9c73a03b46" containerID="7400678190a07e815f60ae3c25d8177294b34d7b9d552e7c6885318fa7c31f72" exitCode=0 Jul 17 19:23:31.178220 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:31.178070 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59b00c87-7468697dfc-hxdk6" Jul 17 19:23:31.178220 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:31.178106 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59b00c87-7468697dfc-hxdk6" event={"ID":"db022d02-3aa3-43b0-80a6-9d9c73a03b46","Type":"ContainerDied","Data":"7400678190a07e815f60ae3c25d8177294b34d7b9d552e7c6885318fa7c31f72"} Jul 17 19:23:31.178220 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:31.178157 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-59b00c87-7468697dfc-hxdk6" event={"ID":"db022d02-3aa3-43b0-80a6-9d9c73a03b46","Type":"ContainerDied","Data":"0e5ccfd48b433286dd35b98c4981e4d0efc62116843438315cf6ce50a35d63e6"} Jul 17 19:23:31.178220 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:31.178179 2561 scope.go:117] "RemoveContainer" containerID="7400678190a07e815f60ae3c25d8177294b34d7b9d552e7c6885318fa7c31f72" Jul 17 19:23:31.185827 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:31.185799 2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db022d02-3aa3-43b0-80a6-9d9c73a03b46-openshift-service-ca\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:23:31.185966 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:31.185828 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p57b\" (UniqueName: \"kubernetes.io/projected/db022d02-3aa3-43b0-80a6-9d9c73a03b46-kube-api-access-2p57b\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:23:31.185966 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:31.185846 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db022d02-3aa3-43b0-80a6-9d9c73a03b46-maas-api-tls\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:23:31.193509 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:31.193484 2561 scope.go:117] "RemoveContainer" containerID="7400678190a07e815f60ae3c25d8177294b34d7b9d552e7c6885318fa7c31f72" Jul 17 19:23:31.193837 ip-10-0-134-137 kubenswrapper[2561]: E0717 19:23:31.193820 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7400678190a07e815f60ae3c25d8177294b34d7b9d552e7c6885318fa7c31f72\": container with ID starting with 7400678190a07e815f60ae3c25d8177294b34d7b9d552e7c6885318fa7c31f72 not found: ID does not exist" containerID="7400678190a07e815f60ae3c25d8177294b34d7b9d552e7c6885318fa7c31f72" Jul 17 19:23:31.193888 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:31.193845 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7400678190a07e815f60ae3c25d8177294b34d7b9d552e7c6885318fa7c31f72"} err="failed to get container status \"7400678190a07e815f60ae3c25d8177294b34d7b9d552e7c6885318fa7c31f72\": rpc error: code = NotFound desc = could not find container \"7400678190a07e815f60ae3c25d8177294b34d7b9d552e7c6885318fa7c31f72\": container with ID starting with 7400678190a07e815f60ae3c25d8177294b34d7b9d552e7c6885318fa7c31f72 not found: ID does not exist" Jul 17 19:23:31.204330 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:31.204304 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-59b00c87-7468697dfc-hxdk6"] Jul 17 19:23:31.207595 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:31.207572 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-59b00c87-7468697dfc-hxdk6"] Jul 17 19:23:32.098772 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:32.098737 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db022d02-3aa3-43b0-80a6-9d9c73a03b46" path="/var/lib/kubelet/pods/db022d02-3aa3-43b0-80a6-9d9c73a03b46/volumes" Jul 17 19:23:33.486287 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:33.486248 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:23:38.068111 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:38.068076 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-690b208d-59969788ff-xc9hh"] Jul 17 19:23:38.068489 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:38.068359 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65b04be2-813a-4ed7-a766-04c3d8a42a8f" containerName="revoke-keys" Jul 17 19:23:38.068489 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:38.068369 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="65b04be2-813a-4ed7-a766-04c3d8a42a8f" containerName="revoke-keys" Jul 17 19:23:38.068489 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:38.068383 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db022d02-3aa3-43b0-80a6-9d9c73a03b46" containerName="maas-api" Jul 17 19:23:38.068489 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:38.068389 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="db022d02-3aa3-43b0-80a6-9d9c73a03b46" containerName="maas-api" Jul 17 19:23:38.068489 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:38.068396 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcf0e9f0-0268-41a5-8efe-d025c18d49f1" containerName="revoke-keys" Jul 17 19:23:38.068489 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:38.068402 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcf0e9f0-0268-41a5-8efe-d025c18d49f1" containerName="revoke-keys" Jul 17 19:23:38.068489 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:38.068456 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="db022d02-3aa3-43b0-80a6-9d9c73a03b46" containerName="maas-api" Jul 17 19:23:38.068489 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:38.068463 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="65b04be2-813a-4ed7-a766-04c3d8a42a8f" containerName="revoke-keys" Jul 17 19:23:38.068489 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:38.068468 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcf0e9f0-0268-41a5-8efe-d025c18d49f1" containerName="revoke-keys" Jul 17 19:23:38.071637 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:38.071618 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-690b208d-59969788ff-xc9hh" Jul 17 19:23:38.074471 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:38.074446 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-690b208d-serving-cert\"" Jul 17 19:23:38.074748 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:38.074727 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-mhzxx\"" Jul 17 19:23:38.075432 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:38.075411 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 19:23:38.075562 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:38.075440 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 17 19:23:38.084149 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:38.084118 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-690b208d-59969788ff-xc9hh"] Jul 17 19:23:38.244902 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:38.244868 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d700672-f26f-4791-a6e2-dbb71140742c-openshift-service-ca\") pod \"maas-api-e2e-ait-690b208d-59969788ff-xc9hh\" (UID: \"1d700672-f26f-4791-a6e2-dbb71140742c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-690b208d-59969788ff-xc9hh" Jul 17 19:23:38.245097 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:38.244939 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fngmm\" (UniqueName: \"kubernetes.io/projected/1d700672-f26f-4791-a6e2-dbb71140742c-kube-api-access-fngmm\") pod \"maas-api-e2e-ait-690b208d-59969788ff-xc9hh\" (UID: \"1d700672-f26f-4791-a6e2-dbb71140742c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-690b208d-59969788ff-xc9hh" Jul 17 19:23:38.245097 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:38.245039 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d700672-f26f-4791-a6e2-dbb71140742c-maas-api-tls\") pod \"maas-api-e2e-ait-690b208d-59969788ff-xc9hh\" (UID: \"1d700672-f26f-4791-a6e2-dbb71140742c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-690b208d-59969788ff-xc9hh" Jul 17 19:23:38.346397 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:38.346288 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fngmm\" (UniqueName: \"kubernetes.io/projected/1d700672-f26f-4791-a6e2-dbb71140742c-kube-api-access-fngmm\") pod \"maas-api-e2e-ait-690b208d-59969788ff-xc9hh\" (UID: \"1d700672-f26f-4791-a6e2-dbb71140742c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-690b208d-59969788ff-xc9hh" Jul 17 19:23:38.346397 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:38.346346 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d700672-f26f-4791-a6e2-dbb71140742c-maas-api-tls\") pod \"maas-api-e2e-ait-690b208d-59969788ff-xc9hh\" (UID: \"1d700672-f26f-4791-a6e2-dbb71140742c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-690b208d-59969788ff-xc9hh" Jul 17 19:23:38.346680 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:38.346406 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d700672-f26f-4791-a6e2-dbb71140742c-openshift-service-ca\") pod \"maas-api-e2e-ait-690b208d-59969788ff-xc9hh\" (UID: \"1d700672-f26f-4791-a6e2-dbb71140742c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-690b208d-59969788ff-xc9hh" Jul 17 19:23:38.347076 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:38.347050 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d700672-f26f-4791-a6e2-dbb71140742c-openshift-service-ca\") pod \"maas-api-e2e-ait-690b208d-59969788ff-xc9hh\" (UID: \"1d700672-f26f-4791-a6e2-dbb71140742c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-690b208d-59969788ff-xc9hh" Jul 17 19:23:38.348961 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:38.348939 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d700672-f26f-4791-a6e2-dbb71140742c-maas-api-tls\") pod \"maas-api-e2e-ait-690b208d-59969788ff-xc9hh\" (UID: \"1d700672-f26f-4791-a6e2-dbb71140742c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-690b208d-59969788ff-xc9hh" Jul 17 19:23:38.369405 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:38.369370 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fngmm\" (UniqueName: \"kubernetes.io/projected/1d700672-f26f-4791-a6e2-dbb71140742c-kube-api-access-fngmm\") pod \"maas-api-e2e-ait-690b208d-59969788ff-xc9hh\" (UID: \"1d700672-f26f-4791-a6e2-dbb71140742c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-690b208d-59969788ff-xc9hh" Jul 17 19:23:38.384218 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:38.384188 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-690b208d-59969788ff-xc9hh" Jul 17 19:23:38.532078 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:38.532045 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-690b208d-59969788ff-xc9hh"] Jul 17 19:23:38.533417 ip-10-0-134-137 kubenswrapper[2561]: W0717 19:23:38.533388 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d700672_f26f_4791_a6e2_dbb71140742c.slice/crio-756f5ed971d2bcaa0bfaff0ba7093063a31c80a9f5b68cf3bc9a688f1ddec7ec WatchSource:0}: Error finding container 756f5ed971d2bcaa0bfaff0ba7093063a31c80a9f5b68cf3bc9a688f1ddec7ec: Status 404 returned error can't find the container with id 756f5ed971d2bcaa0bfaff0ba7093063a31c80a9f5b68cf3bc9a688f1ddec7ec Jul 17 19:23:39.225295 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:39.225201 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-690b208d-59969788ff-xc9hh" event={"ID":"1d700672-f26f-4791-a6e2-dbb71140742c","Type":"ContainerStarted","Data":"b55427a706be77f1405d30d66fb0649a6f8984c13621d09d4cca503057707ebb"} Jul 17 19:23:39.225295 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:39.225253 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-690b208d-59969788ff-xc9hh" event={"ID":"1d700672-f26f-4791-a6e2-dbb71140742c","Type":"ContainerStarted","Data":"756f5ed971d2bcaa0bfaff0ba7093063a31c80a9f5b68cf3bc9a688f1ddec7ec"} Jul 17 19:23:39.225295 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:39.225280 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-690b208d-59969788ff-xc9hh" Jul 17 19:23:39.247293 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:39.247241 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-690b208d-59969788ff-xc9hh" podStartSLOduration=0.81785705 podStartE2EDuration="1.247224024s" totalImagesPullingTime="429.366974ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:23:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:23:39.245205678 +0000 UTC m=+1717.575021888" watchObservedRunningTime="2026-07-17 19:23:39.247224024 +0000 UTC m=+1717.577040231" Jul 17 19:23:43.053505 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:43.053469 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-690b208d-6k6r5"] Jul 17 19:23:43.056847 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:43.056831 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-690b208d-6k6r5" Jul 17 19:23:43.059307 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:43.059282 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lbvd4\"" Jul 17 19:23:43.064841 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:43.064819 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-690b208d-6k6r5"] Jul 17 19:23:43.186137 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:43.186094 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7872da82-bffa-4b41-8280-159dfd30b41d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-690b208d-6k6r5\" (UID: \"7872da82-bffa-4b41-8280-159dfd30b41d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-690b208d-6k6r5" Jul 17 19:23:43.287397 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:43.287359 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7872da82-bffa-4b41-8280-159dfd30b41d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-690b208d-6k6r5\" (UID: \"7872da82-bffa-4b41-8280-159dfd30b41d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-690b208d-6k6r5" Jul 17 19:23:43.287979 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:43.287961 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7872da82-bffa-4b41-8280-159dfd30b41d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-690b208d-6k6r5\" (UID: \"7872da82-bffa-4b41-8280-159dfd30b41d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-690b208d-6k6r5" Jul 17 19:23:43.368642 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:43.368567 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-690b208d-6k6r5" Jul 17 19:23:43.507088 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:43.507060 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-690b208d-6k6r5"] Jul 17 19:23:43.509438 ip-10-0-134-137 kubenswrapper[2561]: W0717 19:23:43.509410 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7872da82_bffa_4b41_8280_159dfd30b41d.slice/crio-17288103133d4bbaaea18cf3391c0ca113fb6bcaa72f215a49e12a90465f9c9c WatchSource:0}: Error finding container 17288103133d4bbaaea18cf3391c0ca113fb6bcaa72f215a49e12a90465f9c9c: Status 404 returned error can't find the container with id 17288103133d4bbaaea18cf3391c0ca113fb6bcaa72f215a49e12a90465f9c9c Jul 17 19:23:44.251791 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:44.251751 2561 generic.go:356] "Generic (PLEG): container finished" podID="7872da82-bffa-4b41-8280-159dfd30b41d" containerID="33e37733474a2b08ef8756b0d03fcddc0e7b38e82a347d5ab4442ad4bc2777e4" exitCode=7 Jul 17 19:23:44.252191 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:44.251836 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-690b208d-6k6r5" event={"ID":"7872da82-bffa-4b41-8280-159dfd30b41d","Type":"ContainerDied","Data":"33e37733474a2b08ef8756b0d03fcddc0e7b38e82a347d5ab4442ad4bc2777e4"} Jul 17 19:23:44.252191 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:44.251884 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-690b208d-6k6r5" event={"ID":"7872da82-bffa-4b41-8280-159dfd30b41d","Type":"ContainerStarted","Data":"17288103133d4bbaaea18cf3391c0ca113fb6bcaa72f215a49e12a90465f9c9c"} Jul 17 19:23:44.252191 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:44.252029 2561 scope.go:117] "RemoveContainer" containerID="33e37733474a2b08ef8756b0d03fcddc0e7b38e82a347d5ab4442ad4bc2777e4" Jul 17 19:23:44.579311 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:44.579272 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:23:45.236288 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:45.236262 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-690b208d-59969788ff-xc9hh" Jul 17 19:23:45.259650 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:45.259601 2561 generic.go:356] "Generic (PLEG): container finished" podID="7872da82-bffa-4b41-8280-159dfd30b41d" containerID="66178bc71d7b190f99066cb8719ca576856e59b5e7749b29661827de59331ad1" exitCode=7 Jul 17 19:23:45.260113 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:45.259675 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-690b208d-6k6r5" event={"ID":"7872da82-bffa-4b41-8280-159dfd30b41d","Type":"ContainerDied","Data":"66178bc71d7b190f99066cb8719ca576856e59b5e7749b29661827de59331ad1"} Jul 17 19:23:45.260113 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:45.259729 2561 scope.go:117] "RemoveContainer" containerID="33e37733474a2b08ef8756b0d03fcddc0e7b38e82a347d5ab4442ad4bc2777e4" Jul 17 19:23:45.260113 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:45.260033 2561 scope.go:117] "RemoveContainer" containerID="66178bc71d7b190f99066cb8719ca576856e59b5e7749b29661827de59331ad1" Jul 17 19:23:45.260322 ip-10-0-134-137 kubenswrapper[2561]: E0717 19:23:45.260297 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-690b208d-6k6r5_odh-ai-gateway-infra(7872da82-bffa-4b41-8280-159dfd30b41d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-690b208d-6k6r5" podUID="7872da82-bffa-4b41-8280-159dfd30b41d" Jul 17 19:23:46.265861 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:46.265832 2561 scope.go:117] "RemoveContainer" containerID="66178bc71d7b190f99066cb8719ca576856e59b5e7749b29661827de59331ad1" Jul 17 19:23:46.266223 ip-10-0-134-137 kubenswrapper[2561]: E0717 19:23:46.266024 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-690b208d-6k6r5_odh-ai-gateway-infra(7872da82-bffa-4b41-8280-159dfd30b41d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-690b208d-6k6r5" podUID="7872da82-bffa-4b41-8280-159dfd30b41d" Jul 17 19:23:59.093346 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:59.093312 2561 scope.go:117] "RemoveContainer" containerID="66178bc71d7b190f99066cb8719ca576856e59b5e7749b29661827de59331ad1" Jul 17 19:23:59.332091 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:59.332059 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-690b208d-6k6r5" event={"ID":"7872da82-bffa-4b41-8280-159dfd30b41d","Type":"ContainerStarted","Data":"95a7757ee3fbc95994abf860a4edebd62222e913087cd4aa241d1fbac70dcc37"} Jul 17 19:23:59.349327 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:23:59.349238 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-690b208d-6k6r5" podStartSLOduration=16.349220938 podStartE2EDuration="16.349220938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:23:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:23:59.347266429 +0000 UTC m=+1737.677082635" watchObservedRunningTime="2026-07-17 19:23:59.349220938 +0000 UTC m=+1737.679037192" Jul 17 19:24:00.338983 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:00.338948 2561 generic.go:356] "Generic (PLEG): container finished" podID="7872da82-bffa-4b41-8280-159dfd30b41d" containerID="95a7757ee3fbc95994abf860a4edebd62222e913087cd4aa241d1fbac70dcc37" exitCode=0 Jul 17 19:24:00.339416 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:00.339019 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-690b208d-6k6r5" event={"ID":"7872da82-bffa-4b41-8280-159dfd30b41d","Type":"ContainerDied","Data":"95a7757ee3fbc95994abf860a4edebd62222e913087cd4aa241d1fbac70dcc37"} Jul 17 19:24:00.339416 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:00.339082 2561 scope.go:117] "RemoveContainer" containerID="66178bc71d7b190f99066cb8719ca576856e59b5e7749b29661827de59331ad1" Jul 17 19:24:00.365865 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:00.365834 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-690b208d-6k6r5"] Jul 17 19:24:01.483498 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:01.483471 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-690b208d-6k6r5" Jul 17 19:24:01.534155 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:01.534118 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7872da82-bffa-4b41-8280-159dfd30b41d-maas-api-service-ca\") pod \"7872da82-bffa-4b41-8280-159dfd30b41d\" (UID: \"7872da82-bffa-4b41-8280-159dfd30b41d\") " Jul 17 19:24:01.534512 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:01.534484 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7872da82-bffa-4b41-8280-159dfd30b41d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7872da82-bffa-4b41-8280-159dfd30b41d" (UID: "7872da82-bffa-4b41-8280-159dfd30b41d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 19:24:01.635139 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:01.635038 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7872da82-bffa-4b41-8280-159dfd30b41d-maas-api-service-ca\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:24:02.352888 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:02.352850 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-690b208d-6k6r5" event={"ID":"7872da82-bffa-4b41-8280-159dfd30b41d","Type":"ContainerDied","Data":"17288103133d4bbaaea18cf3391c0ca113fb6bcaa72f215a49e12a90465f9c9c"} Jul 17 19:24:02.352888 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:02.352893 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-690b208d-6k6r5" Jul 17 19:24:02.353137 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:02.352905 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17288103133d4bbaaea18cf3391c0ca113fb6bcaa72f215a49e12a90465f9c9c" Jul 17 19:24:02.386401 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:02.386319 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-690b208d-6k6r5"] Jul 17 19:24:02.395407 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:02.395373 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-690b208d-6k6r5"] Jul 17 19:24:04.098914 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:04.098875 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7872da82-bffa-4b41-8280-159dfd30b41d" path="/var/lib/kubelet/pods/7872da82-bffa-4b41-8280-159dfd30b41d/volumes" Jul 17 19:24:33.110635 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:33.110599 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-690b208d-gkqtt"] Jul 17 19:24:33.111090 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:33.110920 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7872da82-bffa-4b41-8280-159dfd30b41d" containerName="revoke-keys" Jul 17 19:24:33.111090 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:33.110933 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7872da82-bffa-4b41-8280-159dfd30b41d" containerName="revoke-keys" Jul 17 19:24:33.111090 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:33.110941 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7872da82-bffa-4b41-8280-159dfd30b41d" containerName="revoke-keys" Jul 17 19:24:33.111090 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:33.110946 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7872da82-bffa-4b41-8280-159dfd30b41d" containerName="revoke-keys" Jul 17 19:24:33.111090 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:33.111011 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7872da82-bffa-4b41-8280-159dfd30b41d" containerName="revoke-keys" Jul 17 19:24:33.111090 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:33.111018 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7872da82-bffa-4b41-8280-159dfd30b41d" containerName="revoke-keys" Jul 17 19:24:33.113896 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:33.113880 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-690b208d-gkqtt" Jul 17 19:24:33.116893 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:33.116869 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lbvd4\"" Jul 17 19:24:33.123267 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:33.123244 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-690b208d-gkqtt"] Jul 17 19:24:33.199056 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:33.199022 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c835b376-b02f-4cc4-9f51-3f421d6531fc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-690b208d-gkqtt\" (UID: \"c835b376-b02f-4cc4-9f51-3f421d6531fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-690b208d-gkqtt" Jul 17 19:24:33.300407 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:33.300368 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c835b376-b02f-4cc4-9f51-3f421d6531fc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-690b208d-gkqtt\" (UID: \"c835b376-b02f-4cc4-9f51-3f421d6531fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-690b208d-gkqtt" Jul 17 19:24:33.301001 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:33.300945 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c835b376-b02f-4cc4-9f51-3f421d6531fc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-690b208d-gkqtt\" (UID: \"c835b376-b02f-4cc4-9f51-3f421d6531fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-690b208d-gkqtt" Jul 17 19:24:33.424823 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:33.424746 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-690b208d-gkqtt" Jul 17 19:24:33.566470 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:33.566436 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-690b208d-gkqtt"] Jul 17 19:24:33.567270 ip-10-0-134-137 kubenswrapper[2561]: W0717 19:24:33.567243 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc835b376_b02f_4cc4_9f51_3f421d6531fc.slice/crio-6649261ef7ad8c7a1d70790e856bb709cdfc1d51fbe37130bab45f5354c48f0a WatchSource:0}: Error finding container 6649261ef7ad8c7a1d70790e856bb709cdfc1d51fbe37130bab45f5354c48f0a: Status 404 returned error can't find the container with id 6649261ef7ad8c7a1d70790e856bb709cdfc1d51fbe37130bab45f5354c48f0a Jul 17 19:24:34.516696 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:34.516666 2561 generic.go:356] "Generic (PLEG): container finished" podID="c835b376-b02f-4cc4-9f51-3f421d6531fc" containerID="be813cbfc723380a4357ca357b8153b4f2c5d784d45b0f7fd8abc4b8777ce7ec" exitCode=0 Jul 17 19:24:34.517087 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:34.516733 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-690b208d-gkqtt" event={"ID":"c835b376-b02f-4cc4-9f51-3f421d6531fc","Type":"ContainerDied","Data":"be813cbfc723380a4357ca357b8153b4f2c5d784d45b0f7fd8abc4b8777ce7ec"} Jul 17 19:24:34.517087 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:34.516770 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-690b208d-gkqtt" event={"ID":"c835b376-b02f-4cc4-9f51-3f421d6531fc","Type":"ContainerStarted","Data":"6649261ef7ad8c7a1d70790e856bb709cdfc1d51fbe37130bab45f5354c48f0a"} Jul 17 19:24:35.648113 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:35.648089 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-690b208d-gkqtt" Jul 17 19:24:35.824147 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:35.824055 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c835b376-b02f-4cc4-9f51-3f421d6531fc-maas-api-service-ca\") pod \"c835b376-b02f-4cc4-9f51-3f421d6531fc\" (UID: \"c835b376-b02f-4cc4-9f51-3f421d6531fc\") " Jul 17 19:24:35.824455 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:35.824431 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c835b376-b02f-4cc4-9f51-3f421d6531fc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c835b376-b02f-4cc4-9f51-3f421d6531fc" (UID: "c835b376-b02f-4cc4-9f51-3f421d6531fc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 19:24:35.924944 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:35.924909 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c835b376-b02f-4cc4-9f51-3f421d6531fc-maas-api-service-ca\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:24:36.528167 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:36.528137 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-690b208d-gkqtt" Jul 17 19:24:36.528348 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:36.528170 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-690b208d-gkqtt" event={"ID":"c835b376-b02f-4cc4-9f51-3f421d6531fc","Type":"ContainerDied","Data":"6649261ef7ad8c7a1d70790e856bb709cdfc1d51fbe37130bab45f5354c48f0a"} Jul 17 19:24:36.528348 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:36.528223 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6649261ef7ad8c7a1d70790e856bb709cdfc1d51fbe37130bab45f5354c48f0a" Jul 17 19:24:43.827685 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:43.827650 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-690b208d-59969788ff-xc9hh"] Jul 17 19:24:43.828215 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:43.827881 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-690b208d-59969788ff-xc9hh" podUID="1d700672-f26f-4791-a6e2-dbb71140742c" containerName="maas-api" containerID="cri-o://b55427a706be77f1405d30d66fb0649a6f8984c13621d09d4cca503057707ebb" gracePeriod=30 Jul 17 19:24:44.080256 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:44.080193 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-690b208d-59969788ff-xc9hh" Jul 17 19:24:44.191850 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:44.191799 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d700672-f26f-4791-a6e2-dbb71140742c-maas-api-tls\") pod \"1d700672-f26f-4791-a6e2-dbb71140742c\" (UID: \"1d700672-f26f-4791-a6e2-dbb71140742c\") " Jul 17 19:24:44.191850 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:44.191859 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d700672-f26f-4791-a6e2-dbb71140742c-openshift-service-ca\") pod \"1d700672-f26f-4791-a6e2-dbb71140742c\" (UID: \"1d700672-f26f-4791-a6e2-dbb71140742c\") " Jul 17 19:24:44.192116 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:44.191931 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fngmm\" (UniqueName: \"kubernetes.io/projected/1d700672-f26f-4791-a6e2-dbb71140742c-kube-api-access-fngmm\") pod \"1d700672-f26f-4791-a6e2-dbb71140742c\" (UID: \"1d700672-f26f-4791-a6e2-dbb71140742c\") " Jul 17 19:24:44.192275 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:44.192242 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d700672-f26f-4791-a6e2-dbb71140742c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1d700672-f26f-4791-a6e2-dbb71140742c" (UID: "1d700672-f26f-4791-a6e2-dbb71140742c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 19:24:44.194138 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:44.194106 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d700672-f26f-4791-a6e2-dbb71140742c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1d700672-f26f-4791-a6e2-dbb71140742c" (UID: "1d700672-f26f-4791-a6e2-dbb71140742c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 19:24:44.194230 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:44.194148 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d700672-f26f-4791-a6e2-dbb71140742c-kube-api-access-fngmm" (OuterVolumeSpecName: "kube-api-access-fngmm") pod "1d700672-f26f-4791-a6e2-dbb71140742c" (UID: "1d700672-f26f-4791-a6e2-dbb71140742c"). InnerVolumeSpecName "kube-api-access-fngmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 19:24:44.293337 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:44.293278 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d700672-f26f-4791-a6e2-dbb71140742c-maas-api-tls\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:24:44.293337 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:44.293329 2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d700672-f26f-4791-a6e2-dbb71140742c-openshift-service-ca\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:24:44.293337 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:44.293344 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fngmm\" (UniqueName: \"kubernetes.io/projected/1d700672-f26f-4791-a6e2-dbb71140742c-kube-api-access-fngmm\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:24:44.570824 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:44.570789 2561 generic.go:356] "Generic (PLEG): container finished" podID="1d700672-f26f-4791-a6e2-dbb71140742c" containerID="b55427a706be77f1405d30d66fb0649a6f8984c13621d09d4cca503057707ebb" exitCode=0 Jul 17 19:24:44.571026 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:44.570836 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-690b208d-59969788ff-xc9hh" Jul 17 19:24:44.571026 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:44.570885 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-690b208d-59969788ff-xc9hh" event={"ID":"1d700672-f26f-4791-a6e2-dbb71140742c","Type":"ContainerDied","Data":"b55427a706be77f1405d30d66fb0649a6f8984c13621d09d4cca503057707ebb"} Jul 17 19:24:44.571026 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:44.570943 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-690b208d-59969788ff-xc9hh" event={"ID":"1d700672-f26f-4791-a6e2-dbb71140742c","Type":"ContainerDied","Data":"756f5ed971d2bcaa0bfaff0ba7093063a31c80a9f5b68cf3bc9a688f1ddec7ec"} Jul 17 19:24:44.571026 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:44.570972 2561 scope.go:117] "RemoveContainer" containerID="b55427a706be77f1405d30d66fb0649a6f8984c13621d09d4cca503057707ebb" Jul 17 19:24:44.582130 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:44.582098 2561 scope.go:117] "RemoveContainer" containerID="b55427a706be77f1405d30d66fb0649a6f8984c13621d09d4cca503057707ebb" Jul 17 19:24:44.582868 ip-10-0-134-137 kubenswrapper[2561]: E0717 19:24:44.582845 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b55427a706be77f1405d30d66fb0649a6f8984c13621d09d4cca503057707ebb\": container with ID starting with b55427a706be77f1405d30d66fb0649a6f8984c13621d09d4cca503057707ebb not found: ID does not exist" containerID="b55427a706be77f1405d30d66fb0649a6f8984c13621d09d4cca503057707ebb" Jul 17 19:24:44.582965 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:44.582873 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b55427a706be77f1405d30d66fb0649a6f8984c13621d09d4cca503057707ebb"} err="failed to get container status \"b55427a706be77f1405d30d66fb0649a6f8984c13621d09d4cca503057707ebb\": rpc error: code = NotFound desc = could not find container \"b55427a706be77f1405d30d66fb0649a6f8984c13621d09d4cca503057707ebb\": container with ID starting with b55427a706be77f1405d30d66fb0649a6f8984c13621d09d4cca503057707ebb not found: ID does not exist" Jul 17 19:24:44.597252 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:44.597219 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-690b208d-59969788ff-xc9hh"] Jul 17 19:24:44.603374 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:44.603352 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-690b208d-59969788ff-xc9hh"] Jul 17 19:24:46.099158 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:46.099127 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d700672-f26f-4791-a6e2-dbb71140742c" path="/var/lib/kubelet/pods/1d700672-f26f-4791-a6e2-dbb71140742c/volumes" Jul 17 19:24:46.889094 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:24:46.889054 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:25:00.707417 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:00.707381 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3c4aca95-6cd4cc6876-vxfrl"] Jul 17 19:25:00.707876 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:00.707687 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d700672-f26f-4791-a6e2-dbb71140742c" containerName="maas-api" Jul 17 19:25:00.707876 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:00.707699 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d700672-f26f-4791-a6e2-dbb71140742c" containerName="maas-api" Jul 17 19:25:00.707876 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:00.707710 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7872da82-bffa-4b41-8280-159dfd30b41d" containerName="revoke-keys" Jul 17 19:25:00.707876 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:00.707716 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7872da82-bffa-4b41-8280-159dfd30b41d" containerName="revoke-keys" Jul 17 19:25:00.707876 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:00.707730 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c835b376-b02f-4cc4-9f51-3f421d6531fc" containerName="revoke-keys" Jul 17 19:25:00.707876 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:00.707737 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c835b376-b02f-4cc4-9f51-3f421d6531fc" containerName="revoke-keys" Jul 17 19:25:00.707876 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:00.707786 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d700672-f26f-4791-a6e2-dbb71140742c" containerName="maas-api" Jul 17 19:25:00.707876 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:00.707793 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7872da82-bffa-4b41-8280-159dfd30b41d" containerName="revoke-keys" Jul 17 19:25:00.707876 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:00.707801 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c835b376-b02f-4cc4-9f51-3f421d6531fc" containerName="revoke-keys" Jul 17 19:25:00.715787 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:00.715760 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c4aca95-6cd4cc6876-vxfrl" Jul 17 19:25:00.718491 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:00.718054 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 19:25:00.718491 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:00.718102 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-mhzxx\"" Jul 17 19:25:00.718491 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:00.718203 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 17 19:25:00.718491 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:00.718236 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-3c4aca95-serving-cert\"" Jul 17 19:25:00.720461 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:00.720437 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3c4aca95-6cd4cc6876-vxfrl"] Jul 17 19:25:00.813924 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:00.813893 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87940f75-dfda-4576-a5dd-4e225105ff14-maas-api-tls\") pod \"maas-api-e2e-derive-3c4aca95-6cd4cc6876-vxfrl\" (UID: \"87940f75-dfda-4576-a5dd-4e225105ff14\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c4aca95-6cd4cc6876-vxfrl" Jul 17 19:25:00.814117 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:00.813949 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87940f75-dfda-4576-a5dd-4e225105ff14-openshift-service-ca\") pod \"maas-api-e2e-derive-3c4aca95-6cd4cc6876-vxfrl\" (UID: \"87940f75-dfda-4576-a5dd-4e225105ff14\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c4aca95-6cd4cc6876-vxfrl" Jul 17 19:25:00.814117 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:00.814021 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h777b\" (UniqueName: \"kubernetes.io/projected/87940f75-dfda-4576-a5dd-4e225105ff14-kube-api-access-h777b\") pod \"maas-api-e2e-derive-3c4aca95-6cd4cc6876-vxfrl\" (UID: \"87940f75-dfda-4576-a5dd-4e225105ff14\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c4aca95-6cd4cc6876-vxfrl" Jul 17 19:25:00.914975 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:00.914938 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87940f75-dfda-4576-a5dd-4e225105ff14-maas-api-tls\") pod \"maas-api-e2e-derive-3c4aca95-6cd4cc6876-vxfrl\" (UID: \"87940f75-dfda-4576-a5dd-4e225105ff14\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c4aca95-6cd4cc6876-vxfrl" Jul 17 19:25:00.914975 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:00.914990 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87940f75-dfda-4576-a5dd-4e225105ff14-openshift-service-ca\") pod \"maas-api-e2e-derive-3c4aca95-6cd4cc6876-vxfrl\" (UID: \"87940f75-dfda-4576-a5dd-4e225105ff14\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c4aca95-6cd4cc6876-vxfrl" Jul 17 19:25:00.915253 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:00.915037 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h777b\" (UniqueName: \"kubernetes.io/projected/87940f75-dfda-4576-a5dd-4e225105ff14-kube-api-access-h777b\") pod \"maas-api-e2e-derive-3c4aca95-6cd4cc6876-vxfrl\" (UID: \"87940f75-dfda-4576-a5dd-4e225105ff14\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c4aca95-6cd4cc6876-vxfrl" Jul 17 19:25:00.915253 ip-10-0-134-137 kubenswrapper[2561]: E0717 19:25:00.915095 2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-3c4aca95-serving-cert: secret "maas-api-e2e-derive-3c4aca95-serving-cert" not found Jul 17 19:25:00.915253 ip-10-0-134-137 kubenswrapper[2561]: E0717 19:25:00.915169 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87940f75-dfda-4576-a5dd-4e225105ff14-maas-api-tls podName:87940f75-dfda-4576-a5dd-4e225105ff14 nodeName:}" failed. No retries permitted until 2026-07-17 19:25:01.41514908 +0000 UTC m=+1799.744965287 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/87940f75-dfda-4576-a5dd-4e225105ff14-maas-api-tls") pod "maas-api-e2e-derive-3c4aca95-6cd4cc6876-vxfrl" (UID: "87940f75-dfda-4576-a5dd-4e225105ff14") : secret "maas-api-e2e-derive-3c4aca95-serving-cert" not found Jul 17 19:25:00.915717 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:00.915698 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87940f75-dfda-4576-a5dd-4e225105ff14-openshift-service-ca\") pod \"maas-api-e2e-derive-3c4aca95-6cd4cc6876-vxfrl\" (UID: \"87940f75-dfda-4576-a5dd-4e225105ff14\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c4aca95-6cd4cc6876-vxfrl" Jul 17 19:25:00.924161 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:00.924133 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h777b\" (UniqueName: \"kubernetes.io/projected/87940f75-dfda-4576-a5dd-4e225105ff14-kube-api-access-h777b\") pod \"maas-api-e2e-derive-3c4aca95-6cd4cc6876-vxfrl\" (UID: \"87940f75-dfda-4576-a5dd-4e225105ff14\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c4aca95-6cd4cc6876-vxfrl" Jul 17 19:25:01.420075 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:01.420041 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87940f75-dfda-4576-a5dd-4e225105ff14-maas-api-tls\") pod \"maas-api-e2e-derive-3c4aca95-6cd4cc6876-vxfrl\" (UID: \"87940f75-dfda-4576-a5dd-4e225105ff14\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c4aca95-6cd4cc6876-vxfrl" Jul 17 19:25:01.422590 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:01.422569 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87940f75-dfda-4576-a5dd-4e225105ff14-maas-api-tls\") pod \"maas-api-e2e-derive-3c4aca95-6cd4cc6876-vxfrl\" (UID: \"87940f75-dfda-4576-a5dd-4e225105ff14\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c4aca95-6cd4cc6876-vxfrl" Jul 17 19:25:01.629849 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:01.629801 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c4aca95-6cd4cc6876-vxfrl" Jul 17 19:25:01.771871 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:01.771844 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3c4aca95-6cd4cc6876-vxfrl"] Jul 17 19:25:01.773663 ip-10-0-134-137 kubenswrapper[2561]: W0717 19:25:01.773638 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87940f75_dfda_4576_a5dd_4e225105ff14.slice/crio-c2d2ca6e328fd8d69ba995abd7c978711cba9f3570666350c21089cbcab88a47 WatchSource:0}: Error finding container c2d2ca6e328fd8d69ba995abd7c978711cba9f3570666350c21089cbcab88a47: Status 404 returned error can't find the container with id c2d2ca6e328fd8d69ba995abd7c978711cba9f3570666350c21089cbcab88a47 Jul 17 19:25:02.152213 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:02.152184 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpjzq_4b162535-066a-471a-aee4-99c07c6e1bb4/ovn-acl-logging/0.log" Jul 17 19:25:02.158367 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:02.158345 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpjzq_4b162535-066a-471a-aee4-99c07c6e1bb4/ovn-acl-logging/0.log" Jul 17 19:25:02.665062 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:02.665010 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c4aca95-6cd4cc6876-vxfrl" event={"ID":"87940f75-dfda-4576-a5dd-4e225105ff14","Type":"ContainerStarted","Data":"7a80f3f536bb149d9e5067cf901d9014a9a8860b4c12d75c1c5cce6c136863e8"} Jul 17 19:25:02.665278 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:02.665115 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c4aca95-6cd4cc6876-vxfrl" Jul 17 19:25:02.665278 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:02.665135 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c4aca95-6cd4cc6876-vxfrl" event={"ID":"87940f75-dfda-4576-a5dd-4e225105ff14","Type":"ContainerStarted","Data":"c2d2ca6e328fd8d69ba995abd7c978711cba9f3570666350c21089cbcab88a47"} Jul 17 19:25:02.684898 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:02.684845 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c4aca95-6cd4cc6876-vxfrl" podStartSLOduration=2.136419808 podStartE2EDuration="2.684828564s" totalImagesPullingTime="548.408756ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:25:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:25:02.682855558 +0000 UTC m=+1801.012671769" watchObservedRunningTime="2026-07-17 19:25:02.684828564 +0000 UTC m=+1801.014644772" Jul 17 19:25:02.983087 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:02.982983 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:25:05.030931 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:05.030889 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c4aca95-b7fp9"] Jul 17 19:25:05.038555 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:05.035710 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c4aca95-b7fp9" Jul 17 19:25:05.038897 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:05.038867 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lbvd4\"" Jul 17 19:25:05.044595 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:05.044564 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c4aca95-b7fp9"] Jul 17 19:25:05.148092 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:05.148049 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54cf0d7f-50a6-4694-8125-fa0c9c13208b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-3c4aca95-b7fp9\" (UID: \"54cf0d7f-50a6-4694-8125-fa0c9c13208b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c4aca95-b7fp9" Jul 17 19:25:05.248555 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:05.248487 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54cf0d7f-50a6-4694-8125-fa0c9c13208b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-3c4aca95-b7fp9\" (UID: \"54cf0d7f-50a6-4694-8125-fa0c9c13208b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c4aca95-b7fp9" Jul 17 19:25:05.249133 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:05.249109 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54cf0d7f-50a6-4694-8125-fa0c9c13208b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-3c4aca95-b7fp9\" (UID: \"54cf0d7f-50a6-4694-8125-fa0c9c13208b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c4aca95-b7fp9" Jul 17 19:25:05.351844 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:05.351745 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c4aca95-b7fp9" Jul 17 19:25:05.486189 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:05.486161 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c4aca95-b7fp9"] Jul 17 19:25:05.487907 ip-10-0-134-137 kubenswrapper[2561]: W0717 19:25:05.487874 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54cf0d7f_50a6_4694_8125_fa0c9c13208b.slice/crio-78b6b6a844530c7115b4b6e9df19ce576b040861951352c0b75d765a7f19e031 WatchSource:0}: Error finding container 78b6b6a844530c7115b4b6e9df19ce576b040861951352c0b75d765a7f19e031: Status 404 returned error can't find the container with id 78b6b6a844530c7115b4b6e9df19ce576b040861951352c0b75d765a7f19e031 Jul 17 19:25:05.680741 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:05.680651 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c4aca95-b7fp9" event={"ID":"54cf0d7f-50a6-4694-8125-fa0c9c13208b","Type":"ContainerStarted","Data":"78b6b6a844530c7115b4b6e9df19ce576b040861951352c0b75d765a7f19e031"} Jul 17 19:25:06.694244 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:06.694205 2561 generic.go:356] "Generic (PLEG): container finished" podID="54cf0d7f-50a6-4694-8125-fa0c9c13208b" containerID="c91f838fba32e48e19a438540da5b37cd3d53cd26b0f87e9592e3ce31ec96221" exitCode=7 Jul 17 19:25:06.694730 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:06.694260 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c4aca95-b7fp9" event={"ID":"54cf0d7f-50a6-4694-8125-fa0c9c13208b","Type":"ContainerDied","Data":"c91f838fba32e48e19a438540da5b37cd3d53cd26b0f87e9592e3ce31ec96221"} Jul 17 19:25:06.694730 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:06.694565 2561 scope.go:117] "RemoveContainer" containerID="c91f838fba32e48e19a438540da5b37cd3d53cd26b0f87e9592e3ce31ec96221" Jul 17 19:25:07.702143 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:07.702047 2561 generic.go:356] "Generic (PLEG): container finished" podID="54cf0d7f-50a6-4694-8125-fa0c9c13208b" containerID="c4e1f4cdb6fd6f806ba8285cea3b01172f6aaf4739d844d7899ffe329d844333" exitCode=7 Jul 17 19:25:07.702143 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:07.702131 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c4aca95-b7fp9" event={"ID":"54cf0d7f-50a6-4694-8125-fa0c9c13208b","Type":"ContainerDied","Data":"c4e1f4cdb6fd6f806ba8285cea3b01172f6aaf4739d844d7899ffe329d844333"} Jul 17 19:25:07.702643 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:07.702198 2561 scope.go:117] "RemoveContainer" containerID="c91f838fba32e48e19a438540da5b37cd3d53cd26b0f87e9592e3ce31ec96221" Jul 17 19:25:07.702643 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:07.702378 2561 scope.go:117] "RemoveContainer" containerID="c4e1f4cdb6fd6f806ba8285cea3b01172f6aaf4739d844d7899ffe329d844333" Jul 17 19:25:07.702643 ip-10-0-134-137 kubenswrapper[2561]: E0717 19:25:07.702605 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-3c4aca95-b7fp9_odh-ai-gateway-infra(54cf0d7f-50a6-4694-8125-fa0c9c13208b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c4aca95-b7fp9" podUID="54cf0d7f-50a6-4694-8125-fa0c9c13208b" Jul 17 19:25:08.674769 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:08.674742 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c4aca95-6cd4cc6876-vxfrl" Jul 17 19:25:08.709974 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:08.709949 2561 scope.go:117] "RemoveContainer" containerID="c4e1f4cdb6fd6f806ba8285cea3b01172f6aaf4739d844d7899ffe329d844333" Jul 17 19:25:08.710450 ip-10-0-134-137 kubenswrapper[2561]: E0717 19:25:08.710127 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-3c4aca95-b7fp9_odh-ai-gateway-infra(54cf0d7f-50a6-4694-8125-fa0c9c13208b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c4aca95-b7fp9" podUID="54cf0d7f-50a6-4694-8125-fa0c9c13208b" Jul 17 19:25:20.093982 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:20.093943 2561 scope.go:117] "RemoveContainer" containerID="c4e1f4cdb6fd6f806ba8285cea3b01172f6aaf4739d844d7899ffe329d844333" Jul 17 19:25:20.774338 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:20.774304 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c4aca95-b7fp9" event={"ID":"54cf0d7f-50a6-4694-8125-fa0c9c13208b","Type":"ContainerStarted","Data":"966b967be400a9aa34e538690d3cae13ab1602fd59c8cf62232b2bccb86e6377"} Jul 17 19:25:20.794904 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:20.794820 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c4aca95-b7fp9" podStartSLOduration=15.794800083 podStartE2EDuration="15.794800083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:25:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:25:20.79442934 +0000 UTC m=+1819.124245588" watchObservedRunningTime="2026-07-17 19:25:20.794800083 +0000 UTC m=+1819.124616298" Jul 17 19:25:21.781499 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:21.781465 2561 generic.go:356] "Generic (PLEG): container finished" podID="54cf0d7f-50a6-4694-8125-fa0c9c13208b" containerID="966b967be400a9aa34e538690d3cae13ab1602fd59c8cf62232b2bccb86e6377" exitCode=0 Jul 17 19:25:21.781946 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:21.781513 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c4aca95-b7fp9" event={"ID":"54cf0d7f-50a6-4694-8125-fa0c9c13208b","Type":"ContainerDied","Data":"966b967be400a9aa34e538690d3cae13ab1602fd59c8cf62232b2bccb86e6377"} Jul 17 19:25:21.781946 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:21.781589 2561 scope.go:117] "RemoveContainer" containerID="c4e1f4cdb6fd6f806ba8285cea3b01172f6aaf4739d844d7899ffe329d844333" Jul 17 19:25:21.805407 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:21.805380 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c4aca95-b7fp9"] Jul 17 19:25:22.925341 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:22.925313 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c4aca95-b7fp9" Jul 17 19:25:23.002837 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:23.002800 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54cf0d7f-50a6-4694-8125-fa0c9c13208b-maas-api-service-ca\") pod \"54cf0d7f-50a6-4694-8125-fa0c9c13208b\" (UID: \"54cf0d7f-50a6-4694-8125-fa0c9c13208b\") " Jul 17 19:25:23.003150 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:23.003126 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54cf0d7f-50a6-4694-8125-fa0c9c13208b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "54cf0d7f-50a6-4694-8125-fa0c9c13208b" (UID: "54cf0d7f-50a6-4694-8125-fa0c9c13208b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 19:25:23.103735 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:23.103655 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54cf0d7f-50a6-4694-8125-fa0c9c13208b-maas-api-service-ca\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:25:23.799808 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:23.799767 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c4aca95-b7fp9" event={"ID":"54cf0d7f-50a6-4694-8125-fa0c9c13208b","Type":"ContainerDied","Data":"78b6b6a844530c7115b4b6e9df19ce576b040861951352c0b75d765a7f19e031"} Jul 17 19:25:23.800017 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:23.799831 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78b6b6a844530c7115b4b6e9df19ce576b040861951352c0b75d765a7f19e031" Jul 17 19:25:23.800017 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:23.799787 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c4aca95-b7fp9" Jul 17 19:25:23.831383 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:23.831353 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c4aca95-b7fp9"] Jul 17 19:25:23.833237 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:23.833217 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c4aca95-b7fp9"] Jul 17 19:25:24.099256 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:24.099178 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54cf0d7f-50a6-4694-8125-fa0c9c13208b" path="/var/lib/kubelet/pods/54cf0d7f-50a6-4694-8125-fa0c9c13208b/volumes" Jul 17 19:25:55.073582 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:55.073535 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c4aca95-q4hjp"] Jul 17 19:25:55.074058 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:55.073854 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54cf0d7f-50a6-4694-8125-fa0c9c13208b" containerName="revoke-keys" Jul 17 19:25:55.074058 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:55.073865 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="54cf0d7f-50a6-4694-8125-fa0c9c13208b" containerName="revoke-keys" Jul 17 19:25:55.074058 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:55.073873 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54cf0d7f-50a6-4694-8125-fa0c9c13208b" containerName="revoke-keys" Jul 17 19:25:55.074058 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:55.073878 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="54cf0d7f-50a6-4694-8125-fa0c9c13208b" containerName="revoke-keys" Jul 17 19:25:55.074058 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:55.073885 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54cf0d7f-50a6-4694-8125-fa0c9c13208b" containerName="revoke-keys" Jul 17 19:25:55.074058 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:55.073891 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="54cf0d7f-50a6-4694-8125-fa0c9c13208b" containerName="revoke-keys" Jul 17 19:25:55.074058 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:55.073937 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="54cf0d7f-50a6-4694-8125-fa0c9c13208b" containerName="revoke-keys" Jul 17 19:25:55.074058 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:55.073945 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="54cf0d7f-50a6-4694-8125-fa0c9c13208b" containerName="revoke-keys" Jul 17 19:25:55.076991 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:55.076970 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c4aca95-q4hjp" Jul 17 19:25:55.079243 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:55.079222 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lbvd4\"" Jul 17 19:25:55.084661 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:55.084621 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c4aca95-q4hjp"] Jul 17 19:25:55.172374 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:55.172333 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a1eb175-2e51-498f-b3cf-3970da570089-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-3c4aca95-q4hjp\" (UID: \"3a1eb175-2e51-498f-b3cf-3970da570089\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c4aca95-q4hjp" Jul 17 19:25:55.273695 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:55.273652 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a1eb175-2e51-498f-b3cf-3970da570089-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-3c4aca95-q4hjp\" (UID: \"3a1eb175-2e51-498f-b3cf-3970da570089\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c4aca95-q4hjp" Jul 17 19:25:55.274287 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:55.274267 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a1eb175-2e51-498f-b3cf-3970da570089-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-3c4aca95-q4hjp\" (UID: \"3a1eb175-2e51-498f-b3cf-3970da570089\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c4aca95-q4hjp" Jul 17 19:25:55.388462 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:55.388374 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c4aca95-q4hjp" Jul 17 19:25:55.526500 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:55.526475 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c4aca95-q4hjp"] Jul 17 19:25:55.528331 ip-10-0-134-137 kubenswrapper[2561]: W0717 19:25:55.528290 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a1eb175_2e51_498f_b3cf_3970da570089.slice/crio-0cc52536e644382725dd38fc43c44e815ee7c0d1d14b06e02db654192d9cedaa WatchSource:0}: Error finding container 0cc52536e644382725dd38fc43c44e815ee7c0d1d14b06e02db654192d9cedaa: Status 404 returned error can't find the container with id 0cc52536e644382725dd38fc43c44e815ee7c0d1d14b06e02db654192d9cedaa Jul 17 19:25:55.965944 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:55.965847 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c4aca95-q4hjp" event={"ID":"3a1eb175-2e51-498f-b3cf-3970da570089","Type":"ContainerStarted","Data":"3f07216803a85bcadd3baaed3fc49c5076a2425ae7f94a176c9476106ac0c3d6"} Jul 17 19:25:55.966088 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:55.965942 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c4aca95-q4hjp" event={"ID":"3a1eb175-2e51-498f-b3cf-3970da570089","Type":"ContainerStarted","Data":"0cc52536e644382725dd38fc43c44e815ee7c0d1d14b06e02db654192d9cedaa"} Jul 17 19:25:55.983201 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:55.983137 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c4aca95-q4hjp" podStartSLOduration=0.983121405 podStartE2EDuration="983.121405ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:25:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:25:55.981451903 +0000 UTC m=+1854.311268111" watchObservedRunningTime="2026-07-17 19:25:55.983121405 +0000 UTC m=+1854.312937614" Jul 17 19:25:56.972603 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:56.972572 2561 generic.go:356] "Generic (PLEG): container finished" podID="3a1eb175-2e51-498f-b3cf-3970da570089" containerID="3f07216803a85bcadd3baaed3fc49c5076a2425ae7f94a176c9476106ac0c3d6" exitCode=0 Jul 17 19:25:56.972993 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:56.972665 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c4aca95-q4hjp" event={"ID":"3a1eb175-2e51-498f-b3cf-3970da570089","Type":"ContainerDied","Data":"3f07216803a85bcadd3baaed3fc49c5076a2425ae7f94a176c9476106ac0c3d6"} Jul 17 19:25:58.108785 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:58.108760 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c4aca95-q4hjp" Jul 17 19:25:58.200993 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:58.200958 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a1eb175-2e51-498f-b3cf-3970da570089-maas-api-service-ca\") pod \"3a1eb175-2e51-498f-b3cf-3970da570089\" (UID: \"3a1eb175-2e51-498f-b3cf-3970da570089\") " Jul 17 19:25:58.201335 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:58.201312 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a1eb175-2e51-498f-b3cf-3970da570089-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3a1eb175-2e51-498f-b3cf-3970da570089" (UID: "3a1eb175-2e51-498f-b3cf-3970da570089"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 19:25:58.301903 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:58.301871 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a1eb175-2e51-498f-b3cf-3970da570089-maas-api-service-ca\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:25:58.983623 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:58.983582 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c4aca95-q4hjp" event={"ID":"3a1eb175-2e51-498f-b3cf-3970da570089","Type":"ContainerDied","Data":"0cc52536e644382725dd38fc43c44e815ee7c0d1d14b06e02db654192d9cedaa"} Jul 17 19:25:58.983776 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:58.983639 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cc52536e644382725dd38fc43c44e815ee7c0d1d14b06e02db654192d9cedaa" Jul 17 19:25:58.983776 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:25:58.983643 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3c4aca95-q4hjp" Jul 17 19:26:05.515859 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:26:05.515812 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3c4aca95-6cd4cc6876-vxfrl"] Jul 17 19:26:05.516285 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:26:05.516087 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c4aca95-6cd4cc6876-vxfrl" podUID="87940f75-dfda-4576-a5dd-4e225105ff14" containerName="maas-api" containerID="cri-o://7a80f3f536bb149d9e5067cf901d9014a9a8860b4c12d75c1c5cce6c136863e8" gracePeriod=30 Jul 17 19:26:05.760808 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:26:05.760782 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c4aca95-6cd4cc6876-vxfrl" Jul 17 19:26:05.863318 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:26:05.863232 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87940f75-dfda-4576-a5dd-4e225105ff14-maas-api-tls\") pod \"87940f75-dfda-4576-a5dd-4e225105ff14\" (UID: \"87940f75-dfda-4576-a5dd-4e225105ff14\") " Jul 17 19:26:05.863318 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:26:05.863308 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87940f75-dfda-4576-a5dd-4e225105ff14-openshift-service-ca\") pod \"87940f75-dfda-4576-a5dd-4e225105ff14\" (UID: \"87940f75-dfda-4576-a5dd-4e225105ff14\") " Jul 17 19:26:05.863549 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:26:05.863344 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h777b\" (UniqueName: \"kubernetes.io/projected/87940f75-dfda-4576-a5dd-4e225105ff14-kube-api-access-h777b\") pod \"87940f75-dfda-4576-a5dd-4e225105ff14\" (UID: \"87940f75-dfda-4576-a5dd-4e225105ff14\") " Jul 17 19:26:05.863695 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:26:05.863668 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87940f75-dfda-4576-a5dd-4e225105ff14-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "87940f75-dfda-4576-a5dd-4e225105ff14" (UID: "87940f75-dfda-4576-a5dd-4e225105ff14"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 19:26:05.865661 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:26:05.865633 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87940f75-dfda-4576-a5dd-4e225105ff14-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "87940f75-dfda-4576-a5dd-4e225105ff14" (UID: "87940f75-dfda-4576-a5dd-4e225105ff14"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 19:26:05.865784 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:26:05.865676 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87940f75-dfda-4576-a5dd-4e225105ff14-kube-api-access-h777b" (OuterVolumeSpecName: "kube-api-access-h777b") pod "87940f75-dfda-4576-a5dd-4e225105ff14" (UID: "87940f75-dfda-4576-a5dd-4e225105ff14"). InnerVolumeSpecName "kube-api-access-h777b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 19:26:05.964021 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:26:05.963984 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h777b\" (UniqueName: \"kubernetes.io/projected/87940f75-dfda-4576-a5dd-4e225105ff14-kube-api-access-h777b\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:26:05.964021 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:26:05.964015 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87940f75-dfda-4576-a5dd-4e225105ff14-maas-api-tls\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:26:05.964021 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:26:05.964024 2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87940f75-dfda-4576-a5dd-4e225105ff14-openshift-service-ca\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:26:06.019315 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:26:06.019283 2561 generic.go:356] "Generic (PLEG): container finished" podID="87940f75-dfda-4576-a5dd-4e225105ff14" containerID="7a80f3f536bb149d9e5067cf901d9014a9a8860b4c12d75c1c5cce6c136863e8" exitCode=0 Jul 17 19:26:06.019518 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:26:06.019328 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c4aca95-6cd4cc6876-vxfrl" Jul 17 19:26:06.019518 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:26:06.019381 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c4aca95-6cd4cc6876-vxfrl" event={"ID":"87940f75-dfda-4576-a5dd-4e225105ff14","Type":"ContainerDied","Data":"7a80f3f536bb149d9e5067cf901d9014a9a8860b4c12d75c1c5cce6c136863e8"} Jul 17 19:26:06.019518 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:26:06.019440 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3c4aca95-6cd4cc6876-vxfrl" event={"ID":"87940f75-dfda-4576-a5dd-4e225105ff14","Type":"ContainerDied","Data":"c2d2ca6e328fd8d69ba995abd7c978711cba9f3570666350c21089cbcab88a47"} Jul 17 19:26:06.019518 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:26:06.019470 2561 scope.go:117] "RemoveContainer" containerID="7a80f3f536bb149d9e5067cf901d9014a9a8860b4c12d75c1c5cce6c136863e8" Jul 17 19:26:06.028921 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:26:06.028902 2561 scope.go:117] "RemoveContainer" containerID="7a80f3f536bb149d9e5067cf901d9014a9a8860b4c12d75c1c5cce6c136863e8" Jul 17 19:26:06.029182 ip-10-0-134-137 kubenswrapper[2561]: E0717 19:26:06.029164 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a80f3f536bb149d9e5067cf901d9014a9a8860b4c12d75c1c5cce6c136863e8\": container with ID starting with 7a80f3f536bb149d9e5067cf901d9014a9a8860b4c12d75c1c5cce6c136863e8 not found: ID does not exist" containerID="7a80f3f536bb149d9e5067cf901d9014a9a8860b4c12d75c1c5cce6c136863e8" Jul 17 19:26:06.029226 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:26:06.029191 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a80f3f536bb149d9e5067cf901d9014a9a8860b4c12d75c1c5cce6c136863e8"} err="failed to get container status \"7a80f3f536bb149d9e5067cf901d9014a9a8860b4c12d75c1c5cce6c136863e8\": rpc error: code = NotFound desc = could not find container \"7a80f3f536bb149d9e5067cf901d9014a9a8860b4c12d75c1c5cce6c136863e8\": container with ID starting with 7a80f3f536bb149d9e5067cf901d9014a9a8860b4c12d75c1c5cce6c136863e8 not found: ID does not exist" Jul 17 19:26:06.040759 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:26:06.040733 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3c4aca95-6cd4cc6876-vxfrl"] Jul 17 19:26:06.044437 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:26:06.044415 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3c4aca95-6cd4cc6876-vxfrl"] Jul 17 19:26:06.099164 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:26:06.099135 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87940f75-dfda-4576-a5dd-4e225105ff14" path="/var/lib/kubelet/pods/87940f75-dfda-4576-a5dd-4e225105ff14/volumes" Jul 17 19:26:08.576964 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:26:08.576927 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:27:44.384368 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:27:44.384329 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:27:52.188687 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:27:52.188646 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:28:01.077540 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:01.077440 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:28:01.479447 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:01.479376 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6d51d6be-675c66c7d6-4qp2w"] Jul 17 19:28:01.479710 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:01.479697 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87940f75-dfda-4576-a5dd-4e225105ff14" containerName="maas-api" Jul 17 19:28:01.479756 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:01.479711 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="87940f75-dfda-4576-a5dd-4e225105ff14" containerName="maas-api" Jul 17 19:28:01.479756 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:01.479725 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a1eb175-2e51-498f-b3cf-3970da570089" containerName="revoke-keys" Jul 17 19:28:01.479756 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:01.479731 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a1eb175-2e51-498f-b3cf-3970da570089" containerName="revoke-keys" Jul 17 19:28:01.479852 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:01.479778 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="54cf0d7f-50a6-4694-8125-fa0c9c13208b" containerName="revoke-keys" Jul 17 19:28:01.479852 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:01.479787 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a1eb175-2e51-498f-b3cf-3970da570089" containerName="revoke-keys" Jul 17 19:28:01.479852 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:01.479793 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="87940f75-dfda-4576-a5dd-4e225105ff14" containerName="maas-api" Jul 17 19:28:01.483039 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:01.483019 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d51d6be-675c66c7d6-4qp2w" Jul 17 19:28:01.485405 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:01.485373 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-6d51d6be-serving-cert\"" Jul 17 19:28:01.485581 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:01.485492 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 17 19:28:01.486262 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:01.486244 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 19:28:01.486262 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:01.486255 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-mhzxx\"" Jul 17 19:28:01.494127 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:01.494100 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6d51d6be-675c66c7d6-4qp2w"] Jul 17 19:28:01.588812 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:01.588775 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ca5ae5f-0d82-4f61-8895-3eaa9a54db72-openshift-service-ca\") pod \"maas-api-e2e-mt-6d51d6be-675c66c7d6-4qp2w\" (UID: \"6ca5ae5f-0d82-4f61-8895-3eaa9a54db72\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d51d6be-675c66c7d6-4qp2w" Jul 17 19:28:01.588998 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:01.588819 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ca5ae5f-0d82-4f61-8895-3eaa9a54db72-maas-api-tls\") pod \"maas-api-e2e-mt-6d51d6be-675c66c7d6-4qp2w\" (UID: \"6ca5ae5f-0d82-4f61-8895-3eaa9a54db72\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d51d6be-675c66c7d6-4qp2w" Jul 17 19:28:01.588998 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:01.588880 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8ts2\" (UniqueName: \"kubernetes.io/projected/6ca5ae5f-0d82-4f61-8895-3eaa9a54db72-kube-api-access-f8ts2\") pod \"maas-api-e2e-mt-6d51d6be-675c66c7d6-4qp2w\" (UID: \"6ca5ae5f-0d82-4f61-8895-3eaa9a54db72\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d51d6be-675c66c7d6-4qp2w" Jul 17 19:28:01.689695 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:01.689661 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8ts2\" (UniqueName: \"kubernetes.io/projected/6ca5ae5f-0d82-4f61-8895-3eaa9a54db72-kube-api-access-f8ts2\") pod \"maas-api-e2e-mt-6d51d6be-675c66c7d6-4qp2w\" (UID: \"6ca5ae5f-0d82-4f61-8895-3eaa9a54db72\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d51d6be-675c66c7d6-4qp2w" Jul 17 19:28:01.689874 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:01.689733 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ca5ae5f-0d82-4f61-8895-3eaa9a54db72-openshift-service-ca\") pod \"maas-api-e2e-mt-6d51d6be-675c66c7d6-4qp2w\" (UID: \"6ca5ae5f-0d82-4f61-8895-3eaa9a54db72\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d51d6be-675c66c7d6-4qp2w" Jul 17 19:28:01.689874 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:01.689762 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ca5ae5f-0d82-4f61-8895-3eaa9a54db72-maas-api-tls\") pod \"maas-api-e2e-mt-6d51d6be-675c66c7d6-4qp2w\" (UID: \"6ca5ae5f-0d82-4f61-8895-3eaa9a54db72\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d51d6be-675c66c7d6-4qp2w" Jul 17 19:28:01.689956 ip-10-0-134-137 kubenswrapper[2561]: E0717 19:28:01.689879 2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-6d51d6be-serving-cert: secret "maas-api-e2e-mt-6d51d6be-serving-cert" not found Jul 17 19:28:01.690003 ip-10-0-134-137 kubenswrapper[2561]: E0717 19:28:01.689956 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ca5ae5f-0d82-4f61-8895-3eaa9a54db72-maas-api-tls podName:6ca5ae5f-0d82-4f61-8895-3eaa9a54db72 nodeName:}" failed. No retries permitted until 2026-07-17 19:28:02.189935287 +0000 UTC m=+1980.519751473 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6ca5ae5f-0d82-4f61-8895-3eaa9a54db72-maas-api-tls") pod "maas-api-e2e-mt-6d51d6be-675c66c7d6-4qp2w" (UID: "6ca5ae5f-0d82-4f61-8895-3eaa9a54db72") : secret "maas-api-e2e-mt-6d51d6be-serving-cert" not found Jul 17 19:28:01.690472 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:01.690452 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ca5ae5f-0d82-4f61-8895-3eaa9a54db72-openshift-service-ca\") pod \"maas-api-e2e-mt-6d51d6be-675c66c7d6-4qp2w\" (UID: \"6ca5ae5f-0d82-4f61-8895-3eaa9a54db72\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d51d6be-675c66c7d6-4qp2w" Jul 17 19:28:01.704180 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:01.704151 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8ts2\" (UniqueName: \"kubernetes.io/projected/6ca5ae5f-0d82-4f61-8895-3eaa9a54db72-kube-api-access-f8ts2\") pod \"maas-api-e2e-mt-6d51d6be-675c66c7d6-4qp2w\" (UID: \"6ca5ae5f-0d82-4f61-8895-3eaa9a54db72\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d51d6be-675c66c7d6-4qp2w" Jul 17 19:28:02.195031 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:02.194998 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ca5ae5f-0d82-4f61-8895-3eaa9a54db72-maas-api-tls\") pod \"maas-api-e2e-mt-6d51d6be-675c66c7d6-4qp2w\" (UID: \"6ca5ae5f-0d82-4f61-8895-3eaa9a54db72\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d51d6be-675c66c7d6-4qp2w" Jul 17 19:28:02.197567 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:02.197518 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ca5ae5f-0d82-4f61-8895-3eaa9a54db72-maas-api-tls\") pod \"maas-api-e2e-mt-6d51d6be-675c66c7d6-4qp2w\" (UID: \"6ca5ae5f-0d82-4f61-8895-3eaa9a54db72\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d51d6be-675c66c7d6-4qp2w" Jul 17 19:28:02.397772 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:02.397742 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-mhzxx\"" Jul 17 19:28:02.405633 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:02.405601 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d51d6be-675c66c7d6-4qp2w" Jul 17 19:28:02.542843 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:02.542818 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6d51d6be-675c66c7d6-4qp2w"] Jul 17 19:28:02.544970 ip-10-0-134-137 kubenswrapper[2561]: W0717 19:28:02.544940 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ca5ae5f_0d82_4f61_8895_3eaa9a54db72.slice/crio-2ea22cb1210d7a1e3c105fb6cb208bbf2cfaa2b5ac6326c861e8eb8b390449bc WatchSource:0}: Error finding container 2ea22cb1210d7a1e3c105fb6cb208bbf2cfaa2b5ac6326c861e8eb8b390449bc: Status 404 returned error can't find the container with id 2ea22cb1210d7a1e3c105fb6cb208bbf2cfaa2b5ac6326c861e8eb8b390449bc Jul 17 19:28:02.546786 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:02.546767 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 19:28:02.607604 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:02.607563 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d51d6be-675c66c7d6-4qp2w" event={"ID":"6ca5ae5f-0d82-4f61-8895-3eaa9a54db72","Type":"ContainerStarted","Data":"2ea22cb1210d7a1e3c105fb6cb208bbf2cfaa2b5ac6326c861e8eb8b390449bc"} Jul 17 19:28:03.344117 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:03.344082 2561 scope.go:117] "RemoveContainer" containerID="5c245ccfe4e43d144ac44c525b1e446877cd723fd6da4ff0758513a47c9f8ab9" Jul 17 19:28:03.614499 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:03.614413 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d51d6be-675c66c7d6-4qp2w" event={"ID":"6ca5ae5f-0d82-4f61-8895-3eaa9a54db72","Type":"ContainerStarted","Data":"3913ecbefc43491c2c059b78abb55ac0bf30176a744090df80e29fc8d4a6102e"} Jul 17 19:28:03.614669 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:03.614553 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d51d6be-675c66c7d6-4qp2w" Jul 17 19:28:03.633012 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:03.632954 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d51d6be-675c66c7d6-4qp2w" podStartSLOduration=2.245805382 podStartE2EDuration="2.632936728s" totalImagesPullingTime="387.131346ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:28:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:28:03.631573935 +0000 UTC m=+1981.961390143" watchObservedRunningTime="2026-07-17 19:28:03.632936728 +0000 UTC m=+1981.962752936" Jul 17 19:28:09.626317 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:09.626288 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d51d6be-675c66c7d6-4qp2w" Jul 17 19:28:12.477264 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:12.477225 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:28:12.992268 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:12.992231 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d51d6be-crfmm"] Jul 17 19:28:12.995378 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:12.995362 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d51d6be-crfmm" Jul 17 19:28:12.998131 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:12.998112 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lbvd4\"" Jul 17 19:28:13.003714 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:13.003691 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d51d6be-crfmm"] Jul 17 19:28:13.087675 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:13.087639 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e35649af-2c79-4ee2-a990-0c67a18f60ad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6d51d6be-crfmm\" (UID: \"e35649af-2c79-4ee2-a990-0c67a18f60ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d51d6be-crfmm" Jul 17 19:28:13.188883 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:13.188833 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e35649af-2c79-4ee2-a990-0c67a18f60ad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6d51d6be-crfmm\" (UID: \"e35649af-2c79-4ee2-a990-0c67a18f60ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d51d6be-crfmm" Jul 17 19:28:13.189590 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:13.189567 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e35649af-2c79-4ee2-a990-0c67a18f60ad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6d51d6be-crfmm\" (UID: \"e35649af-2c79-4ee2-a990-0c67a18f60ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d51d6be-crfmm" Jul 17 19:28:13.306568 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:13.306443 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d51d6be-crfmm" Jul 17 19:28:13.441390 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:13.441367 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d51d6be-crfmm"] Jul 17 19:28:13.443460 ip-10-0-134-137 kubenswrapper[2561]: W0717 19:28:13.443432 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode35649af_2c79_4ee2_a990_0c67a18f60ad.slice/crio-92070b0532700faa49af25c7def722dde41b7f65566494f7ec9af9d821dca967 WatchSource:0}: Error finding container 92070b0532700faa49af25c7def722dde41b7f65566494f7ec9af9d821dca967: Status 404 returned error can't find the container with id 92070b0532700faa49af25c7def722dde41b7f65566494f7ec9af9d821dca967 Jul 17 19:28:13.665536 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:13.665499 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d51d6be-crfmm" event={"ID":"e35649af-2c79-4ee2-a990-0c67a18f60ad","Type":"ContainerStarted","Data":"1881bf785ffce411362ea0cc6aff5da89dc11382e1b97bfe82962552f31edbed"} Jul 17 19:28:13.665998 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:13.665586 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d51d6be-crfmm" event={"ID":"e35649af-2c79-4ee2-a990-0c67a18f60ad","Type":"ContainerStarted","Data":"92070b0532700faa49af25c7def722dde41b7f65566494f7ec9af9d821dca967"} Jul 17 19:28:13.680498 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:13.680436 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d51d6be-crfmm" podStartSLOduration=1.680422538 podStartE2EDuration="1.680422538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:28:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:28:13.679976788 +0000 UTC m=+1992.009792997" watchObservedRunningTime="2026-07-17 19:28:13.680422538 +0000 UTC m=+1992.010238776" Jul 17 19:28:14.671564 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:14.671506 2561 generic.go:356] "Generic (PLEG): container finished" podID="e35649af-2c79-4ee2-a990-0c67a18f60ad" containerID="1881bf785ffce411362ea0cc6aff5da89dc11382e1b97bfe82962552f31edbed" exitCode=0 Jul 17 19:28:14.671972 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:14.671597 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d51d6be-crfmm" event={"ID":"e35649af-2c79-4ee2-a990-0c67a18f60ad","Type":"ContainerDied","Data":"1881bf785ffce411362ea0cc6aff5da89dc11382e1b97bfe82962552f31edbed"} Jul 17 19:28:15.803487 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:15.803464 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d51d6be-crfmm" Jul 17 19:28:15.910948 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:15.910914 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e35649af-2c79-4ee2-a990-0c67a18f60ad-maas-api-service-ca\") pod \"e35649af-2c79-4ee2-a990-0c67a18f60ad\" (UID: \"e35649af-2c79-4ee2-a990-0c67a18f60ad\") " Jul 17 19:28:15.911277 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:15.911251 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e35649af-2c79-4ee2-a990-0c67a18f60ad-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e35649af-2c79-4ee2-a990-0c67a18f60ad" (UID: "e35649af-2c79-4ee2-a990-0c67a18f60ad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 19:28:16.012094 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:16.012063 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e35649af-2c79-4ee2-a990-0c67a18f60ad-maas-api-service-ca\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:28:16.683270 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:16.683239 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d51d6be-crfmm" Jul 17 19:28:16.683443 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:16.683281 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6d51d6be-crfmm" event={"ID":"e35649af-2c79-4ee2-a990-0c67a18f60ad","Type":"ContainerDied","Data":"92070b0532700faa49af25c7def722dde41b7f65566494f7ec9af9d821dca967"} Jul 17 19:28:16.683443 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:16.683328 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92070b0532700faa49af25c7def722dde41b7f65566494f7ec9af9d821dca967" Jul 17 19:28:23.412645 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:23.412606 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6d51d6be-675c66c7d6-4qp2w"] Jul 17 19:28:23.413100 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:23.412830 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d51d6be-675c66c7d6-4qp2w" podUID="6ca5ae5f-0d82-4f61-8895-3eaa9a54db72" containerName="maas-api" containerID="cri-o://3913ecbefc43491c2c059b78abb55ac0bf30176a744090df80e29fc8d4a6102e" gracePeriod=30 Jul 17 19:28:23.676502 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:23.676444 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d51d6be-675c66c7d6-4qp2w" Jul 17 19:28:23.720285 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:23.720251 2561 generic.go:356] "Generic (PLEG): container finished" podID="6ca5ae5f-0d82-4f61-8895-3eaa9a54db72" containerID="3913ecbefc43491c2c059b78abb55ac0bf30176a744090df80e29fc8d4a6102e" exitCode=0 Jul 17 19:28:23.720285 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:23.720288 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d51d6be-675c66c7d6-4qp2w" Jul 17 19:28:23.720507 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:23.720301 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d51d6be-675c66c7d6-4qp2w" event={"ID":"6ca5ae5f-0d82-4f61-8895-3eaa9a54db72","Type":"ContainerDied","Data":"3913ecbefc43491c2c059b78abb55ac0bf30176a744090df80e29fc8d4a6102e"} Jul 17 19:28:23.720507 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:23.720370 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d51d6be-675c66c7d6-4qp2w" event={"ID":"6ca5ae5f-0d82-4f61-8895-3eaa9a54db72","Type":"ContainerDied","Data":"2ea22cb1210d7a1e3c105fb6cb208bbf2cfaa2b5ac6326c861e8eb8b390449bc"} Jul 17 19:28:23.720507 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:23.720405 2561 scope.go:117] "RemoveContainer" containerID="3913ecbefc43491c2c059b78abb55ac0bf30176a744090df80e29fc8d4a6102e" Jul 17 19:28:23.729345 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:23.729326 2561 scope.go:117] "RemoveContainer" containerID="3913ecbefc43491c2c059b78abb55ac0bf30176a744090df80e29fc8d4a6102e" Jul 17 19:28:23.729637 ip-10-0-134-137 kubenswrapper[2561]: E0717 19:28:23.729618 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3913ecbefc43491c2c059b78abb55ac0bf30176a744090df80e29fc8d4a6102e\": container with ID starting with 3913ecbefc43491c2c059b78abb55ac0bf30176a744090df80e29fc8d4a6102e not found: ID does not exist" containerID="3913ecbefc43491c2c059b78abb55ac0bf30176a744090df80e29fc8d4a6102e" Jul 17 19:28:23.729693 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:23.729648 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3913ecbefc43491c2c059b78abb55ac0bf30176a744090df80e29fc8d4a6102e"} err="failed to get container status \"3913ecbefc43491c2c059b78abb55ac0bf30176a744090df80e29fc8d4a6102e\": rpc error: code = NotFound desc = could not find container \"3913ecbefc43491c2c059b78abb55ac0bf30176a744090df80e29fc8d4a6102e\": container with ID starting with 3913ecbefc43491c2c059b78abb55ac0bf30176a744090df80e29fc8d4a6102e not found: ID does not exist" Jul 17 19:28:23.774992 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:23.774959 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ca5ae5f-0d82-4f61-8895-3eaa9a54db72-maas-api-tls\") pod \"6ca5ae5f-0d82-4f61-8895-3eaa9a54db72\" (UID: \"6ca5ae5f-0d82-4f61-8895-3eaa9a54db72\") " Jul 17 19:28:23.775149 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:23.775016 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8ts2\" (UniqueName: \"kubernetes.io/projected/6ca5ae5f-0d82-4f61-8895-3eaa9a54db72-kube-api-access-f8ts2\") pod \"6ca5ae5f-0d82-4f61-8895-3eaa9a54db72\" (UID: \"6ca5ae5f-0d82-4f61-8895-3eaa9a54db72\") " Jul 17 19:28:23.775149 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:23.775043 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ca5ae5f-0d82-4f61-8895-3eaa9a54db72-openshift-service-ca\") pod \"6ca5ae5f-0d82-4f61-8895-3eaa9a54db72\" (UID: \"6ca5ae5f-0d82-4f61-8895-3eaa9a54db72\") " Jul 17 19:28:23.775441 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:23.775421 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ca5ae5f-0d82-4f61-8895-3eaa9a54db72-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6ca5ae5f-0d82-4f61-8895-3eaa9a54db72" (UID: "6ca5ae5f-0d82-4f61-8895-3eaa9a54db72"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 19:28:23.777268 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:23.777240 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ca5ae5f-0d82-4f61-8895-3eaa9a54db72-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6ca5ae5f-0d82-4f61-8895-3eaa9a54db72" (UID: "6ca5ae5f-0d82-4f61-8895-3eaa9a54db72"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 19:28:23.777376 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:23.777294 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ca5ae5f-0d82-4f61-8895-3eaa9a54db72-kube-api-access-f8ts2" (OuterVolumeSpecName: "kube-api-access-f8ts2") pod "6ca5ae5f-0d82-4f61-8895-3eaa9a54db72" (UID: "6ca5ae5f-0d82-4f61-8895-3eaa9a54db72"). InnerVolumeSpecName "kube-api-access-f8ts2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 19:28:23.875863 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:23.875823 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ca5ae5f-0d82-4f61-8895-3eaa9a54db72-maas-api-tls\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:28:23.875863 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:23.875857 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8ts2\" (UniqueName: \"kubernetes.io/projected/6ca5ae5f-0d82-4f61-8895-3eaa9a54db72-kube-api-access-f8ts2\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:28:23.875863 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:23.875866 2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ca5ae5f-0d82-4f61-8895-3eaa9a54db72-openshift-service-ca\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:28:24.042266 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:24.042234 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6d51d6be-675c66c7d6-4qp2w"] Jul 17 19:28:24.045886 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:24.045860 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6d51d6be-675c66c7d6-4qp2w"] Jul 17 19:28:24.099783 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:24.099750 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ca5ae5f-0d82-4f61-8895-3eaa9a54db72" path="/var/lib/kubelet/pods/6ca5ae5f-0d82-4f61-8895-3eaa9a54db72/volumes" Jul 17 19:28:26.390902 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:28:26.390869 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:29:01.617647 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:01.617614 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-45e4d318-846965479-l5xlf"] Jul 17 19:29:01.618104 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:01.617932 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ca5ae5f-0d82-4f61-8895-3eaa9a54db72" containerName="maas-api" Jul 17 19:29:01.618104 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:01.617948 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ca5ae5f-0d82-4f61-8895-3eaa9a54db72" containerName="maas-api" Jul 17 19:29:01.618104 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:01.617956 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e35649af-2c79-4ee2-a990-0c67a18f60ad" containerName="revoke-keys" Jul 17 19:29:01.618104 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:01.617961 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e35649af-2c79-4ee2-a990-0c67a18f60ad" containerName="revoke-keys" Jul 17 19:29:01.618104 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:01.618018 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ca5ae5f-0d82-4f61-8895-3eaa9a54db72" containerName="maas-api" Jul 17 19:29:01.618104 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:01.618029 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e35649af-2c79-4ee2-a990-0c67a18f60ad" containerName="revoke-keys" Jul 17 19:29:01.621240 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:01.621218 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-45e4d318-846965479-l5xlf" Jul 17 19:29:01.623875 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:01.623852 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-45e4d318-serving-cert\"" Jul 17 19:29:01.624011 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:01.623879 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 17 19:29:01.624011 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:01.623879 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-mhzxx\"" Jul 17 19:29:01.624683 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:01.624660 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 19:29:01.631049 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:01.630501 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-45e4d318-846965479-l5xlf"] Jul 17 19:29:01.688296 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:01.688260 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b02ab5c-f713-481c-babe-0d4d91f87d3b-openshift-service-ca\") pod \"maas-api-e2e-mt-45e4d318-846965479-l5xlf\" (UID: \"0b02ab5c-f713-481c-babe-0d4d91f87d3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-45e4d318-846965479-l5xlf" Jul 17 19:29:01.688521 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:01.688323 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7sxf\" (UniqueName: \"kubernetes.io/projected/0b02ab5c-f713-481c-babe-0d4d91f87d3b-kube-api-access-k7sxf\") pod \"maas-api-e2e-mt-45e4d318-846965479-l5xlf\" (UID: \"0b02ab5c-f713-481c-babe-0d4d91f87d3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-45e4d318-846965479-l5xlf" Jul 17 19:29:01.688521 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:01.688394 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b02ab5c-f713-481c-babe-0d4d91f87d3b-maas-api-tls\") pod \"maas-api-e2e-mt-45e4d318-846965479-l5xlf\" (UID: \"0b02ab5c-f713-481c-babe-0d4d91f87d3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-45e4d318-846965479-l5xlf" Jul 17 19:29:01.789311 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:01.789273 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b02ab5c-f713-481c-babe-0d4d91f87d3b-maas-api-tls\") pod \"maas-api-e2e-mt-45e4d318-846965479-l5xlf\" (UID: \"0b02ab5c-f713-481c-babe-0d4d91f87d3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-45e4d318-846965479-l5xlf" Jul 17 19:29:01.789511 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:01.789337 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b02ab5c-f713-481c-babe-0d4d91f87d3b-openshift-service-ca\") pod \"maas-api-e2e-mt-45e4d318-846965479-l5xlf\" (UID: \"0b02ab5c-f713-481c-babe-0d4d91f87d3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-45e4d318-846965479-l5xlf" Jul 17 19:29:01.789511 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:01.789394 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7sxf\" (UniqueName: \"kubernetes.io/projected/0b02ab5c-f713-481c-babe-0d4d91f87d3b-kube-api-access-k7sxf\") pod \"maas-api-e2e-mt-45e4d318-846965479-l5xlf\" (UID: \"0b02ab5c-f713-481c-babe-0d4d91f87d3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-45e4d318-846965479-l5xlf" Jul 17 19:29:01.790076 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:01.790052 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b02ab5c-f713-481c-babe-0d4d91f87d3b-openshift-service-ca\") pod \"maas-api-e2e-mt-45e4d318-846965479-l5xlf\" (UID: \"0b02ab5c-f713-481c-babe-0d4d91f87d3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-45e4d318-846965479-l5xlf" Jul 17 19:29:01.791954 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:01.791932 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b02ab5c-f713-481c-babe-0d4d91f87d3b-maas-api-tls\") pod \"maas-api-e2e-mt-45e4d318-846965479-l5xlf\" (UID: \"0b02ab5c-f713-481c-babe-0d4d91f87d3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-45e4d318-846965479-l5xlf" Jul 17 19:29:01.799493 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:01.799468 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7sxf\" (UniqueName: \"kubernetes.io/projected/0b02ab5c-f713-481c-babe-0d4d91f87d3b-kube-api-access-k7sxf\") pod \"maas-api-e2e-mt-45e4d318-846965479-l5xlf\" (UID: \"0b02ab5c-f713-481c-babe-0d4d91f87d3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-45e4d318-846965479-l5xlf" Jul 17 19:29:01.934890 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:01.934798 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-45e4d318-846965479-l5xlf" Jul 17 19:29:02.076099 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:02.076070 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-45e4d318-846965479-l5xlf"] Jul 17 19:29:02.078196 ip-10-0-134-137 kubenswrapper[2561]: W0717 19:29:02.078167 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b02ab5c_f713_481c_babe_0d4d91f87d3b.slice/crio-2ff40bf5e0dcc3358a3efde2cae0479657bc0c41746ed604dff9e1a5e4b2d19d WatchSource:0}: Error finding container 2ff40bf5e0dcc3358a3efde2cae0479657bc0c41746ed604dff9e1a5e4b2d19d: Status 404 returned error can't find the container with id 2ff40bf5e0dcc3358a3efde2cae0479657bc0c41746ed604dff9e1a5e4b2d19d Jul 17 19:29:02.928220 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:02.928109 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-45e4d318-846965479-l5xlf" event={"ID":"0b02ab5c-f713-481c-babe-0d4d91f87d3b","Type":"ContainerStarted","Data":"40f003a9765014f3ffacbf6e1cb078abdd67947f5c700d5ef6c865e3fbb5e591"} Jul 17 19:29:02.928220 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:02.928164 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-45e4d318-846965479-l5xlf" event={"ID":"0b02ab5c-f713-481c-babe-0d4d91f87d3b","Type":"ContainerStarted","Data":"2ff40bf5e0dcc3358a3efde2cae0479657bc0c41746ed604dff9e1a5e4b2d19d"} Jul 17 19:29:02.928220 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:02.928184 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-45e4d318-846965479-l5xlf" Jul 17 19:29:02.946276 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:02.946198 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-45e4d318-846965479-l5xlf" podStartSLOduration=1.496518113 podStartE2EDuration="1.946181479s" totalImagesPullingTime="449.663366ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:29:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:29:02.944955465 +0000 UTC m=+2041.274771674" watchObservedRunningTime="2026-07-17 19:29:02.946181479 +0000 UTC m=+2041.275997687" Jul 17 19:29:03.365169 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:03.365137 2561 scope.go:117] "RemoveContainer" containerID="b499094f2f4cf8ce599c20e1764dfea6700ae3b5b31123f07dc5612a06a34cd5" Jul 17 19:29:06.578890 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:06.578854 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:29:08.938711 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:08.938681 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-45e4d318-846965479-l5xlf" Jul 17 19:29:18.886395 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:18.886354 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:29:30.273932 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:30.273848 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:29:34.037351 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:34.037314 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-45e4d318-cph9z"] Jul 17 19:29:34.040731 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:34.040714 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-45e4d318-cph9z" Jul 17 19:29:34.043499 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:34.043476 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lbvd4\"" Jul 17 19:29:34.049587 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:34.049560 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-45e4d318-cph9z"] Jul 17 19:29:34.167716 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:34.167676 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed770d13-78d5-475a-8b32-5dba94c73fe8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-45e4d318-cph9z\" (UID: \"ed770d13-78d5-475a-8b32-5dba94c73fe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-45e4d318-cph9z" Jul 17 19:29:34.268732 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:34.268691 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed770d13-78d5-475a-8b32-5dba94c73fe8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-45e4d318-cph9z\" (UID: \"ed770d13-78d5-475a-8b32-5dba94c73fe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-45e4d318-cph9z" Jul 17 19:29:34.269316 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:34.269294 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed770d13-78d5-475a-8b32-5dba94c73fe8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-45e4d318-cph9z\" (UID: \"ed770d13-78d5-475a-8b32-5dba94c73fe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-45e4d318-cph9z" Jul 17 19:29:34.352873 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:34.352774 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-45e4d318-cph9z" Jul 17 19:29:34.491578 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:34.491552 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-45e4d318-cph9z"] Jul 17 19:29:34.493210 ip-10-0-134-137 kubenswrapper[2561]: W0717 19:29:34.493184 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded770d13_78d5_475a_8b32_5dba94c73fe8.slice/crio-4ddf175cee3605cc608b45a7cf5d4ac5b1f0c5b7e3432dadcce49334bafc52a2 WatchSource:0}: Error finding container 4ddf175cee3605cc608b45a7cf5d4ac5b1f0c5b7e3432dadcce49334bafc52a2: Status 404 returned error can't find the container with id 4ddf175cee3605cc608b45a7cf5d4ac5b1f0c5b7e3432dadcce49334bafc52a2 Jul 17 19:29:35.090083 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:35.090036 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-45e4d318-cph9z" event={"ID":"ed770d13-78d5-475a-8b32-5dba94c73fe8","Type":"ContainerStarted","Data":"dcae3f3839b1ea03a6cbc1e854d20a2aa19ab718764a9469cf8ac9fad096b954"} Jul 17 19:29:35.090559 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:35.090097 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-45e4d318-cph9z" event={"ID":"ed770d13-78d5-475a-8b32-5dba94c73fe8","Type":"ContainerStarted","Data":"4ddf175cee3605cc608b45a7cf5d4ac5b1f0c5b7e3432dadcce49334bafc52a2"} Jul 17 19:29:35.107427 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:35.107373 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-45e4d318-cph9z" podStartSLOduration=1.107357653 podStartE2EDuration="1.107357653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:29:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:29:35.105082672 +0000 UTC m=+2073.434898883" watchObservedRunningTime="2026-07-17 19:29:35.107357653 +0000 UTC m=+2073.437173860" Jul 17 19:29:36.097207 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:36.097163 2561 generic.go:356] "Generic (PLEG): container finished" podID="ed770d13-78d5-475a-8b32-5dba94c73fe8" containerID="dcae3f3839b1ea03a6cbc1e854d20a2aa19ab718764a9469cf8ac9fad096b954" exitCode=0 Jul 17 19:29:36.099134 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:36.099105 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-45e4d318-cph9z" event={"ID":"ed770d13-78d5-475a-8b32-5dba94c73fe8","Type":"ContainerDied","Data":"dcae3f3839b1ea03a6cbc1e854d20a2aa19ab718764a9469cf8ac9fad096b954"} Jul 17 19:29:37.231473 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:37.231447 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-45e4d318-cph9z" Jul 17 19:29:37.397832 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:37.397738 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed770d13-78d5-475a-8b32-5dba94c73fe8-maas-api-service-ca\") pod \"ed770d13-78d5-475a-8b32-5dba94c73fe8\" (UID: \"ed770d13-78d5-475a-8b32-5dba94c73fe8\") " Jul 17 19:29:37.398114 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:37.398089 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed770d13-78d5-475a-8b32-5dba94c73fe8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ed770d13-78d5-475a-8b32-5dba94c73fe8" (UID: "ed770d13-78d5-475a-8b32-5dba94c73fe8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 19:29:37.499031 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:37.498992 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed770d13-78d5-475a-8b32-5dba94c73fe8-maas-api-service-ca\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:29:38.108617 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:38.108590 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-45e4d318-cph9z" Jul 17 19:29:38.108745 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:38.108593 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-45e4d318-cph9z" event={"ID":"ed770d13-78d5-475a-8b32-5dba94c73fe8","Type":"ContainerDied","Data":"4ddf175cee3605cc608b45a7cf5d4ac5b1f0c5b7e3432dadcce49334bafc52a2"} Jul 17 19:29:38.108745 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:38.108713 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ddf175cee3605cc608b45a7cf5d4ac5b1f0c5b7e3432dadcce49334bafc52a2" Jul 17 19:29:44.500001 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:44.499962 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-45e4d318-846965479-l5xlf"] Jul 17 19:29:44.500462 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:44.500189 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-45e4d318-846965479-l5xlf" podUID="0b02ab5c-f713-481c-babe-0d4d91f87d3b" containerName="maas-api" containerID="cri-o://40f003a9765014f3ffacbf6e1cb078abdd67947f5c700d5ef6c865e3fbb5e591" gracePeriod=30 Jul 17 19:29:44.740949 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:44.740917 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-45e4d318-846965479-l5xlf" Jul 17 19:29:44.755642 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:44.755568 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b02ab5c-f713-481c-babe-0d4d91f87d3b-openshift-service-ca\") pod \"0b02ab5c-f713-481c-babe-0d4d91f87d3b\" (UID: \"0b02ab5c-f713-481c-babe-0d4d91f87d3b\") " Jul 17 19:29:44.755642 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:44.755638 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b02ab5c-f713-481c-babe-0d4d91f87d3b-maas-api-tls\") pod \"0b02ab5c-f713-481c-babe-0d4d91f87d3b\" (UID: \"0b02ab5c-f713-481c-babe-0d4d91f87d3b\") " Jul 17 19:29:44.755833 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:44.755702 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7sxf\" (UniqueName: \"kubernetes.io/projected/0b02ab5c-f713-481c-babe-0d4d91f87d3b-kube-api-access-k7sxf\") pod \"0b02ab5c-f713-481c-babe-0d4d91f87d3b\" (UID: \"0b02ab5c-f713-481c-babe-0d4d91f87d3b\") " Jul 17 19:29:44.755956 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:44.755932 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b02ab5c-f713-481c-babe-0d4d91f87d3b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0b02ab5c-f713-481c-babe-0d4d91f87d3b" (UID: "0b02ab5c-f713-481c-babe-0d4d91f87d3b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 19:29:44.757909 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:44.757888 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b02ab5c-f713-481c-babe-0d4d91f87d3b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0b02ab5c-f713-481c-babe-0d4d91f87d3b" (UID: "0b02ab5c-f713-481c-babe-0d4d91f87d3b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 19:29:44.758066 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:44.757992 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b02ab5c-f713-481c-babe-0d4d91f87d3b-kube-api-access-k7sxf" (OuterVolumeSpecName: "kube-api-access-k7sxf") pod "0b02ab5c-f713-481c-babe-0d4d91f87d3b" (UID: "0b02ab5c-f713-481c-babe-0d4d91f87d3b"). InnerVolumeSpecName "kube-api-access-k7sxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 19:29:44.856268 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:44.856232 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7sxf\" (UniqueName: \"kubernetes.io/projected/0b02ab5c-f713-481c-babe-0d4d91f87d3b-kube-api-access-k7sxf\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:29:44.856268 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:44.856264 2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b02ab5c-f713-481c-babe-0d4d91f87d3b-openshift-service-ca\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:29:44.856268 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:44.856275 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b02ab5c-f713-481c-babe-0d4d91f87d3b-maas-api-tls\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:29:45.152351 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:45.152260 2561 generic.go:356] "Generic (PLEG): container finished" podID="0b02ab5c-f713-481c-babe-0d4d91f87d3b" containerID="40f003a9765014f3ffacbf6e1cb078abdd67947f5c700d5ef6c865e3fbb5e591" exitCode=0 Jul 17 19:29:45.152351 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:45.152328 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-45e4d318-846965479-l5xlf" Jul 17 19:29:45.152632 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:45.152345 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-45e4d318-846965479-l5xlf" event={"ID":"0b02ab5c-f713-481c-babe-0d4d91f87d3b","Type":"ContainerDied","Data":"40f003a9765014f3ffacbf6e1cb078abdd67947f5c700d5ef6c865e3fbb5e591"} Jul 17 19:29:45.152632 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:45.152437 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-45e4d318-846965479-l5xlf" event={"ID":"0b02ab5c-f713-481c-babe-0d4d91f87d3b","Type":"ContainerDied","Data":"2ff40bf5e0dcc3358a3efde2cae0479657bc0c41746ed604dff9e1a5e4b2d19d"} Jul 17 19:29:45.152632 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:45.152470 2561 scope.go:117] "RemoveContainer" containerID="40f003a9765014f3ffacbf6e1cb078abdd67947f5c700d5ef6c865e3fbb5e591" Jul 17 19:29:45.161851 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:45.161833 2561 scope.go:117] "RemoveContainer" containerID="40f003a9765014f3ffacbf6e1cb078abdd67947f5c700d5ef6c865e3fbb5e591" Jul 17 19:29:45.162119 ip-10-0-134-137 kubenswrapper[2561]: E0717 19:29:45.162096 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40f003a9765014f3ffacbf6e1cb078abdd67947f5c700d5ef6c865e3fbb5e591\": container with ID starting with 40f003a9765014f3ffacbf6e1cb078abdd67947f5c700d5ef6c865e3fbb5e591 not found: ID does not exist" containerID="40f003a9765014f3ffacbf6e1cb078abdd67947f5c700d5ef6c865e3fbb5e591" Jul 17 19:29:45.162219 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:45.162127 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40f003a9765014f3ffacbf6e1cb078abdd67947f5c700d5ef6c865e3fbb5e591"} err="failed to get container status \"40f003a9765014f3ffacbf6e1cb078abdd67947f5c700d5ef6c865e3fbb5e591\": rpc error: code = NotFound desc = could not find container \"40f003a9765014f3ffacbf6e1cb078abdd67947f5c700d5ef6c865e3fbb5e591\": container with ID starting with 40f003a9765014f3ffacbf6e1cb078abdd67947f5c700d5ef6c865e3fbb5e591 not found: ID does not exist" Jul 17 19:29:45.173994 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:45.173963 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-45e4d318-846965479-l5xlf"] Jul 17 19:29:45.176872 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:45.176847 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-45e4d318-846965479-l5xlf"] Jul 17 19:29:46.099019 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:46.098984 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b02ab5c-f713-481c-babe-0d4d91f87d3b" path="/var/lib/kubelet/pods/0b02ab5c-f713-481c-babe-0d4d91f87d3b/volumes" Jul 17 19:29:48.172762 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:29:48.172728 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:30:00.134062 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:30:00.134020 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738610-m26bt"] Jul 17 19:30:00.134454 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:30:00.134319 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed770d13-78d5-475a-8b32-5dba94c73fe8" containerName="revoke-keys" Jul 17 19:30:00.134454 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:30:00.134330 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed770d13-78d5-475a-8b32-5dba94c73fe8" containerName="revoke-keys" Jul 17 19:30:00.134454 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:30:00.134355 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b02ab5c-f713-481c-babe-0d4d91f87d3b" containerName="maas-api" Jul 17 19:30:00.134454 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:30:00.134361 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b02ab5c-f713-481c-babe-0d4d91f87d3b" containerName="maas-api" Jul 17 19:30:00.134454 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:30:00.134410 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed770d13-78d5-475a-8b32-5dba94c73fe8" containerName="revoke-keys" Jul 17 19:30:00.134454 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:30:00.134419 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b02ab5c-f713-481c-babe-0d4d91f87d3b" containerName="maas-api" Jul 17 19:30:00.138534 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:30:00.138503 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738610-m26bt" Jul 17 19:30:00.140955 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:30:00.140928 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lbvd4\"" Jul 17 19:30:00.144363 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:30:00.144338 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738610-m26bt"] Jul 17 19:30:00.154110 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:30:00.154083 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738610-m26bt" Jul 17 19:30:00.287843 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:30:00.287814 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738610-m26bt"] Jul 17 19:30:00.289755 ip-10-0-134-137 kubenswrapper[2561]: W0717 19:30:00.289724 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7bfccf5_7572_4af5_b346_4361b49b5fc9.slice/crio-c48dc8040d51e9ff2b2455938f8146d3d208f33f747669965534bb663fe27d13 WatchSource:0}: Error finding container c48dc8040d51e9ff2b2455938f8146d3d208f33f747669965534bb663fe27d13: Status 404 returned error can't find the container with id c48dc8040d51e9ff2b2455938f8146d3d208f33f747669965534bb663fe27d13 Jul 17 19:30:01.238268 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:30:01.238230 2561 generic.go:356] "Generic (PLEG): container finished" podID="e7bfccf5-7572-4af5-b346-4361b49b5fc9" containerID="54caccdd4cf87bd45cf6bf06ac923088ebeeb8eaf19208af8bae0aa35da4fa20" exitCode=0 Jul 17 19:30:01.238661 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:30:01.238307 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738610-m26bt" event={"ID":"e7bfccf5-7572-4af5-b346-4361b49b5fc9","Type":"ContainerDied","Data":"54caccdd4cf87bd45cf6bf06ac923088ebeeb8eaf19208af8bae0aa35da4fa20"} Jul 17 19:30:01.238661 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:30:01.238357 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738610-m26bt" event={"ID":"e7bfccf5-7572-4af5-b346-4361b49b5fc9","Type":"ContainerStarted","Data":"c48dc8040d51e9ff2b2455938f8146d3d208f33f747669965534bb663fe27d13"} Jul 17 19:30:02.185466 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:30:02.185442 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpjzq_4b162535-066a-471a-aee4-99c07c6e1bb4/ovn-acl-logging/0.log" Jul 17 19:30:02.192385 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:30:02.192362 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpjzq_4b162535-066a-471a-aee4-99c07c6e1bb4/ovn-acl-logging/0.log" Jul 17 19:30:02.370293 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:30:02.370269 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738610-m26bt" Jul 17 19:30:03.249926 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:30:03.249897 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738610-m26bt" Jul 17 19:30:03.249926 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:30:03.249919 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738610-m26bt" event={"ID":"e7bfccf5-7572-4af5-b346-4361b49b5fc9","Type":"ContainerDied","Data":"c48dc8040d51e9ff2b2455938f8146d3d208f33f747669965534bb663fe27d13"} Jul 17 19:30:03.250145 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:30:03.249960 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c48dc8040d51e9ff2b2455938f8146d3d208f33f747669965534bb663fe27d13" Jul 17 19:30:03.391514 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:30:03.391485 2561 scope.go:117] "RemoveContainer" containerID="95a7757ee3fbc95994abf860a4edebd62222e913087cd4aa241d1fbac70dcc37" Jul 17 19:30:47.083220 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:30:47.083180 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:30:58.184157 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:30:58.184070 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:31:39.591364 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:31:39.591330 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5606-6c464b5697-2d58k"] Jul 17 19:31:39.591972 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:31:39.591789 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7bfccf5-7572-4af5-b346-4361b49b5fc9" containerName="cleanup" Jul 17 19:31:39.591972 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:31:39.591809 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7bfccf5-7572-4af5-b346-4361b49b5fc9" containerName="cleanup" Jul 17 19:31:39.591972 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:31:39.591879 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7bfccf5-7572-4af5-b346-4361b49b5fc9" containerName="cleanup" Jul 17 19:31:39.594802 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:31:39.594781 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5606-6c464b5697-2d58k" Jul 17 19:31:39.597040 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:31:39.597016 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-1f5606-serving-cert\"" Jul 17 19:31:39.597040 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:31:39.597039 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-mhzxx\"" Jul 17 19:31:39.597227 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:31:39.597071 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 19:31:39.597865 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:31:39.597847 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 17 19:31:39.605963 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:31:39.605938 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5606-6c464b5697-2d58k"] Jul 17 19:31:39.687784 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:31:39.687744 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smxfb\" (UniqueName: \"kubernetes.io/projected/0c813b71-a420-4ebd-9053-2e45ce1ac455-kube-api-access-smxfb\") pod \"maas-api-e2e-inf-a-1f5606-6c464b5697-2d58k\" (UID: \"0c813b71-a420-4ebd-9053-2e45ce1ac455\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5606-6c464b5697-2d58k" Jul 17 19:31:39.688032 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:31:39.687831 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c813b71-a420-4ebd-9053-2e45ce1ac455-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1f5606-6c464b5697-2d58k\" (UID: \"0c813b71-a420-4ebd-9053-2e45ce1ac455\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5606-6c464b5697-2d58k" Jul 17 19:31:39.688032 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:31:39.687863 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c813b71-a420-4ebd-9053-2e45ce1ac455-maas-api-tls\") pod \"maas-api-e2e-inf-a-1f5606-6c464b5697-2d58k\" (UID: \"0c813b71-a420-4ebd-9053-2e45ce1ac455\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5606-6c464b5697-2d58k" Jul 17 19:31:39.788361 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:31:39.788325 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c813b71-a420-4ebd-9053-2e45ce1ac455-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1f5606-6c464b5697-2d58k\" (UID: \"0c813b71-a420-4ebd-9053-2e45ce1ac455\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5606-6c464b5697-2d58k" Jul 17 19:31:39.788361 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:31:39.788367 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c813b71-a420-4ebd-9053-2e45ce1ac455-maas-api-tls\") pod \"maas-api-e2e-inf-a-1f5606-6c464b5697-2d58k\" (UID: \"0c813b71-a420-4ebd-9053-2e45ce1ac455\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5606-6c464b5697-2d58k" Jul 17 19:31:39.788653 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:31:39.788450 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smxfb\" (UniqueName: \"kubernetes.io/projected/0c813b71-a420-4ebd-9053-2e45ce1ac455-kube-api-access-smxfb\") pod \"maas-api-e2e-inf-a-1f5606-6c464b5697-2d58k\" (UID: \"0c813b71-a420-4ebd-9053-2e45ce1ac455\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5606-6c464b5697-2d58k" Jul 17 19:31:39.789146 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:31:39.789127 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c813b71-a420-4ebd-9053-2e45ce1ac455-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1f5606-6c464b5697-2d58k\" (UID: \"0c813b71-a420-4ebd-9053-2e45ce1ac455\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5606-6c464b5697-2d58k" Jul 17 19:31:39.790975 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:31:39.790958 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c813b71-a420-4ebd-9053-2e45ce1ac455-maas-api-tls\") pod \"maas-api-e2e-inf-a-1f5606-6c464b5697-2d58k\" (UID: \"0c813b71-a420-4ebd-9053-2e45ce1ac455\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5606-6c464b5697-2d58k" Jul 17 19:31:39.797920 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:31:39.797893 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smxfb\" (UniqueName: \"kubernetes.io/projected/0c813b71-a420-4ebd-9053-2e45ce1ac455-kube-api-access-smxfb\") pod \"maas-api-e2e-inf-a-1f5606-6c464b5697-2d58k\" (UID: \"0c813b71-a420-4ebd-9053-2e45ce1ac455\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5606-6c464b5697-2d58k" Jul 17 19:31:39.906242 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:31:39.906150 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5606-6c464b5697-2d58k" Jul 17 19:31:40.040679 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:31:40.040652 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5606-6c464b5697-2d58k"] Jul 17 19:31:40.042367 ip-10-0-134-137 kubenswrapper[2561]: W0717 19:31:40.042337 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c813b71_a420_4ebd_9053_2e45ce1ac455.slice/crio-2c2326af411e0fc4ff189e99b69c4f7b7374ceb7a4d7166518250a6716433018 WatchSource:0}: Error finding container 2c2326af411e0fc4ff189e99b69c4f7b7374ceb7a4d7166518250a6716433018: Status 404 returned error can't find the container with id 2c2326af411e0fc4ff189e99b69c4f7b7374ceb7a4d7166518250a6716433018 Jul 17 19:31:40.737289 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:31:40.737243 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5606-6c464b5697-2d58k" event={"ID":"0c813b71-a420-4ebd-9053-2e45ce1ac455","Type":"ContainerStarted","Data":"d5fb4c2bb8e2a0173bc06d64ea9f79a06c6abc64c58f58a925100dcd85d7df83"} Jul 17 19:31:40.737749 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:31:40.737305 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5606-6c464b5697-2d58k" event={"ID":"0c813b71-a420-4ebd-9053-2e45ce1ac455","Type":"ContainerStarted","Data":"2c2326af411e0fc4ff189e99b69c4f7b7374ceb7a4d7166518250a6716433018"} Jul 17 19:31:40.737749 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:31:40.737333 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5606-6c464b5697-2d58k" Jul 17 19:31:40.754906 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:31:40.754858 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5606-6c464b5697-2d58k" podStartSLOduration=1.374145463 podStartE2EDuration="1.754841746s" totalImagesPullingTime="380.696283ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:31:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:31:40.753564897 +0000 UTC m=+2199.083381104" watchObservedRunningTime="2026-07-17 19:31:40.754841746 +0000 UTC m=+2199.084657952" Jul 17 19:31:46.748856 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:31:46.748824 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5606-6c464b5697-2d58k" Jul 17 19:31:47.082895 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:31:47.082808 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:31:47.667220 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:31:47.667191 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-eaec93-7f8469666b-7rdz5"] Jul 17 19:31:47.670727 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:31:47.670699 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eaec93-7f8469666b-7rdz5" Jul 17 19:31:47.672832 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:31:47.672809 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-eaec93-serving-cert\"" Jul 17 19:31:47.681571 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:31:47.681545 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-eaec93-7f8469666b-7rdz5"] Jul 17 19:31:47.761002 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:31:47.760966 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2dph\" (UniqueName: \"kubernetes.io/projected/6e6ba068-a540-4fee-9117-93dee1cc2ff3-kube-api-access-z2dph\") pod \"maas-api-e2e-inf-b-eaec93-7f8469666b-7rdz5\" (UID: \"6e6ba068-a540-4fee-9117-93dee1cc2ff3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eaec93-7f8469666b-7rdz5" Jul 17 19:31:47.761385 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:31:47.761017 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e6ba068-a540-4fee-9117-93dee1cc2ff3-openshift-service-ca\") pod \"maas-api-e2e-inf-b-eaec93-7f8469666b-7rdz5\" (UID: \"6e6ba068-a540-4fee-9117-93dee1cc2ff3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eaec93-7f8469666b-7rdz5" Jul 17 19:31:47.761385 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:31:47.761112 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e6ba068-a540-4fee-9117-93dee1cc2ff3-maas-api-tls\") pod \"maas-api-e2e-inf-b-eaec93-7f8469666b-7rdz5\" (UID: \"6e6ba068-a540-4fee-9117-93dee1cc2ff3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eaec93-7f8469666b-7rdz5" Jul 17 19:31:47.862552 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:31:47.862497 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e6ba068-a540-4fee-9117-93dee1cc2ff3-openshift-service-ca\") pod \"maas-api-e2e-inf-b-eaec93-7f8469666b-7rdz5\" (UID: \"6e6ba068-a540-4fee-9117-93dee1cc2ff3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eaec93-7f8469666b-7rdz5" Jul 17 19:31:47.862739 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:31:47.862571 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e6ba068-a540-4fee-9117-93dee1cc2ff3-maas-api-tls\") pod \"maas-api-e2e-inf-b-eaec93-7f8469666b-7rdz5\" (UID: \"6e6ba068-a540-4fee-9117-93dee1cc2ff3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eaec93-7f8469666b-7rdz5" Jul 17 19:31:47.862739 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:31:47.862628 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2dph\" (UniqueName: \"kubernetes.io/projected/6e6ba068-a540-4fee-9117-93dee1cc2ff3-kube-api-access-z2dph\") pod \"maas-api-e2e-inf-b-eaec93-7f8469666b-7rdz5\" (UID: \"6e6ba068-a540-4fee-9117-93dee1cc2ff3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eaec93-7f8469666b-7rdz5" Jul 17 19:31:47.862831 ip-10-0-134-137 kubenswrapper[2561]: E0717 19:31:47.862739 2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-eaec93-serving-cert: secret "maas-api-e2e-inf-b-eaec93-serving-cert" not found Jul 17 19:31:47.862831 ip-10-0-134-137 kubenswrapper[2561]: E0717 19:31:47.862811 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e6ba068-a540-4fee-9117-93dee1cc2ff3-maas-api-tls podName:6e6ba068-a540-4fee-9117-93dee1cc2ff3 nodeName:}" failed. No retries permitted until 2026-07-17 19:31:48.362790703 +0000 UTC m=+2206.692606894 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6e6ba068-a540-4fee-9117-93dee1cc2ff3-maas-api-tls") pod "maas-api-e2e-inf-b-eaec93-7f8469666b-7rdz5" (UID: "6e6ba068-a540-4fee-9117-93dee1cc2ff3") : secret "maas-api-e2e-inf-b-eaec93-serving-cert" not found Jul 17 19:31:47.863130 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:31:47.863111 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e6ba068-a540-4fee-9117-93dee1cc2ff3-openshift-service-ca\") pod \"maas-api-e2e-inf-b-eaec93-7f8469666b-7rdz5\" (UID: \"6e6ba068-a540-4fee-9117-93dee1cc2ff3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eaec93-7f8469666b-7rdz5" Jul 17 19:31:47.871217 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:31:47.871194 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2dph\" (UniqueName: \"kubernetes.io/projected/6e6ba068-a540-4fee-9117-93dee1cc2ff3-kube-api-access-z2dph\") pod \"maas-api-e2e-inf-b-eaec93-7f8469666b-7rdz5\" (UID: \"6e6ba068-a540-4fee-9117-93dee1cc2ff3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eaec93-7f8469666b-7rdz5" Jul 17 19:31:48.365907 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:31:48.365869 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e6ba068-a540-4fee-9117-93dee1cc2ff3-maas-api-tls\") pod \"maas-api-e2e-inf-b-eaec93-7f8469666b-7rdz5\" (UID: \"6e6ba068-a540-4fee-9117-93dee1cc2ff3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eaec93-7f8469666b-7rdz5" Jul 17 19:31:48.368416 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:31:48.368386 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e6ba068-a540-4fee-9117-93dee1cc2ff3-maas-api-tls\") pod \"maas-api-e2e-inf-b-eaec93-7f8469666b-7rdz5\" (UID: \"6e6ba068-a540-4fee-9117-93dee1cc2ff3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eaec93-7f8469666b-7rdz5" Jul 17 19:31:48.584067 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:31:48.584031 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eaec93-7f8469666b-7rdz5" Jul 17 19:31:48.715494 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:31:48.715465 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-eaec93-7f8469666b-7rdz5"] Jul 17 19:31:48.717701 ip-10-0-134-137 kubenswrapper[2561]: W0717 19:31:48.717664 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e6ba068_a540_4fee_9117_93dee1cc2ff3.slice/crio-b489263fb0bb4a2b2fc6ea43ea705ed85624bc3740da5c7b465a070c94fb128c WatchSource:0}: Error finding container b489263fb0bb4a2b2fc6ea43ea705ed85624bc3740da5c7b465a070c94fb128c: Status 404 returned error can't find the container with id b489263fb0bb4a2b2fc6ea43ea705ed85624bc3740da5c7b465a070c94fb128c Jul 17 19:31:48.781951 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:31:48.781922 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eaec93-7f8469666b-7rdz5" event={"ID":"6e6ba068-a540-4fee-9117-93dee1cc2ff3","Type":"ContainerStarted","Data":"b489263fb0bb4a2b2fc6ea43ea705ed85624bc3740da5c7b465a070c94fb128c"} Jul 17 19:31:49.788929 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:31:49.788891 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eaec93-7f8469666b-7rdz5" event={"ID":"6e6ba068-a540-4fee-9117-93dee1cc2ff3","Type":"ContainerStarted","Data":"d04d35409411ee4b74bb4cdd294024c457506d5bc5e732df68aaaa408acf393a"} Jul 17 19:31:49.789397 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:31:49.788987 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eaec93-7f8469666b-7rdz5" Jul 17 19:31:49.809870 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:31:49.809816 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eaec93-7f8469666b-7rdz5" podStartSLOduration=2.417375827 podStartE2EDuration="2.809800331s" totalImagesPullingTime="392.424504ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:31:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:31:49.808115217 +0000 UTC m=+2208.137931427" watchObservedRunningTime="2026-07-17 19:31:49.809800331 +0000 UTC m=+2208.139616538" Jul 17 19:31:55.800085 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:31:55.800053 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eaec93-7f8469666b-7rdz5" Jul 17 19:32:01.286295 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:01.286255 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:32:03.426632 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:03.426589 2561 scope.go:117] "RemoveContainer" containerID="966b967be400a9aa34e538690d3cae13ab1602fd59c8cf62232b2bccb86e6377" Jul 17 19:32:07.014919 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:07.014876 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-1f5606/test-model-1f5606-kserve-b7bd6f8fc-66jx2"] Jul 17 19:32:07.018310 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:07.018287 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1f5606/test-model-1f5606-kserve-b7bd6f8fc-66jx2" Jul 17 19:32:07.021756 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:07.021727 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1f5606\"/\"openshift-service-ca.crt\"" Jul 17 19:32:07.021756 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:07.021749 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1f5606\"/\"test-model-1f5606-kserve-self-signed-certs\"" Jul 17 19:32:07.021920 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:07.021765 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1f5606\"/\"kube-root-ca.crt\"" Jul 17 19:32:07.021920 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:07.021727 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1f5606\"/\"default-dockercfg-vcfxw\"" Jul 17 19:32:07.027828 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:07.027809 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1f5606/test-model-1f5606-kserve-b7bd6f8fc-66jx2"] Jul 17 19:32:07.124380 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:07.124337 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5db1488e-bb78-4632-8770-130d971b82c6-model-cache\") pod \"test-model-1f5606-kserve-b7bd6f8fc-66jx2\" (UID: \"5db1488e-bb78-4632-8770-130d971b82c6\") " pod="ai-tenant-e2e-inf-a-1f5606/test-model-1f5606-kserve-b7bd6f8fc-66jx2" Jul 17 19:32:07.124380 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:07.124379 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5db1488e-bb78-4632-8770-130d971b82c6-dshm\") pod \"test-model-1f5606-kserve-b7bd6f8fc-66jx2\" (UID: \"5db1488e-bb78-4632-8770-130d971b82c6\") " pod="ai-tenant-e2e-inf-a-1f5606/test-model-1f5606-kserve-b7bd6f8fc-66jx2" Jul 17 19:32:07.124602 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:07.124407 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5db1488e-bb78-4632-8770-130d971b82c6-tmp-dir\") pod \"test-model-1f5606-kserve-b7bd6f8fc-66jx2\" (UID: \"5db1488e-bb78-4632-8770-130d971b82c6\") " pod="ai-tenant-e2e-inf-a-1f5606/test-model-1f5606-kserve-b7bd6f8fc-66jx2" Jul 17 19:32:07.124602 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:07.124483 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5db1488e-bb78-4632-8770-130d971b82c6-home\") pod \"test-model-1f5606-kserve-b7bd6f8fc-66jx2\" (UID: \"5db1488e-bb78-4632-8770-130d971b82c6\") " pod="ai-tenant-e2e-inf-a-1f5606/test-model-1f5606-kserve-b7bd6f8fc-66jx2" Jul 17 19:32:07.124602 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:07.124572 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5db1488e-bb78-4632-8770-130d971b82c6-tls-certs\") pod \"test-model-1f5606-kserve-b7bd6f8fc-66jx2\" (UID: \"5db1488e-bb78-4632-8770-130d971b82c6\") " pod="ai-tenant-e2e-inf-a-1f5606/test-model-1f5606-kserve-b7bd6f8fc-66jx2" Jul 17 19:32:07.124602 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:07.124599 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxxnd\" (UniqueName: \"kubernetes.io/projected/5db1488e-bb78-4632-8770-130d971b82c6-kube-api-access-pxxnd\") pod \"test-model-1f5606-kserve-b7bd6f8fc-66jx2\" (UID: \"5db1488e-bb78-4632-8770-130d971b82c6\") " pod="ai-tenant-e2e-inf-a-1f5606/test-model-1f5606-kserve-b7bd6f8fc-66jx2" Jul 17 19:32:07.225215 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:07.225178 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5db1488e-bb78-4632-8770-130d971b82c6-tmp-dir\") pod \"test-model-1f5606-kserve-b7bd6f8fc-66jx2\" (UID: \"5db1488e-bb78-4632-8770-130d971b82c6\") " pod="ai-tenant-e2e-inf-a-1f5606/test-model-1f5606-kserve-b7bd6f8fc-66jx2" Jul 17 19:32:07.225215 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:07.225217 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5db1488e-bb78-4632-8770-130d971b82c6-home\") pod \"test-model-1f5606-kserve-b7bd6f8fc-66jx2\" (UID: \"5db1488e-bb78-4632-8770-130d971b82c6\") " pod="ai-tenant-e2e-inf-a-1f5606/test-model-1f5606-kserve-b7bd6f8fc-66jx2" Jul 17 19:32:07.225467 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:07.225251 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5db1488e-bb78-4632-8770-130d971b82c6-tls-certs\") pod \"test-model-1f5606-kserve-b7bd6f8fc-66jx2\" (UID: \"5db1488e-bb78-4632-8770-130d971b82c6\") " pod="ai-tenant-e2e-inf-a-1f5606/test-model-1f5606-kserve-b7bd6f8fc-66jx2" Jul 17 19:32:07.225467 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:07.225420 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxxnd\" (UniqueName: \"kubernetes.io/projected/5db1488e-bb78-4632-8770-130d971b82c6-kube-api-access-pxxnd\") pod \"test-model-1f5606-kserve-b7bd6f8fc-66jx2\" (UID: \"5db1488e-bb78-4632-8770-130d971b82c6\") " pod="ai-tenant-e2e-inf-a-1f5606/test-model-1f5606-kserve-b7bd6f8fc-66jx2" Jul 17 19:32:07.225467 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:07.225452 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5db1488e-bb78-4632-8770-130d971b82c6-model-cache\") pod \"test-model-1f5606-kserve-b7bd6f8fc-66jx2\" (UID: \"5db1488e-bb78-4632-8770-130d971b82c6\") " pod="ai-tenant-e2e-inf-a-1f5606/test-model-1f5606-kserve-b7bd6f8fc-66jx2" Jul 17 19:32:07.225467 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:07.225469 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5db1488e-bb78-4632-8770-130d971b82c6-dshm\") pod \"test-model-1f5606-kserve-b7bd6f8fc-66jx2\" (UID: \"5db1488e-bb78-4632-8770-130d971b82c6\") " pod="ai-tenant-e2e-inf-a-1f5606/test-model-1f5606-kserve-b7bd6f8fc-66jx2" Jul 17 19:32:07.225749 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:07.225722 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5db1488e-bb78-4632-8770-130d971b82c6-home\") pod \"test-model-1f5606-kserve-b7bd6f8fc-66jx2\" (UID: \"5db1488e-bb78-4632-8770-130d971b82c6\") " pod="ai-tenant-e2e-inf-a-1f5606/test-model-1f5606-kserve-b7bd6f8fc-66jx2" Jul 17 19:32:07.225798 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:07.225745 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5db1488e-bb78-4632-8770-130d971b82c6-tmp-dir\") pod \"test-model-1f5606-kserve-b7bd6f8fc-66jx2\" (UID: \"5db1488e-bb78-4632-8770-130d971b82c6\") " pod="ai-tenant-e2e-inf-a-1f5606/test-model-1f5606-kserve-b7bd6f8fc-66jx2" Jul 17 19:32:07.225860 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:07.225836 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5db1488e-bb78-4632-8770-130d971b82c6-model-cache\") pod \"test-model-1f5606-kserve-b7bd6f8fc-66jx2\" (UID: \"5db1488e-bb78-4632-8770-130d971b82c6\") " pod="ai-tenant-e2e-inf-a-1f5606/test-model-1f5606-kserve-b7bd6f8fc-66jx2" Jul 17 19:32:07.227686 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:07.227668 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5db1488e-bb78-4632-8770-130d971b82c6-dshm\") pod \"test-model-1f5606-kserve-b7bd6f8fc-66jx2\" (UID: \"5db1488e-bb78-4632-8770-130d971b82c6\") " pod="ai-tenant-e2e-inf-a-1f5606/test-model-1f5606-kserve-b7bd6f8fc-66jx2" Jul 17 19:32:07.227858 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:07.227841 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5db1488e-bb78-4632-8770-130d971b82c6-tls-certs\") pod \"test-model-1f5606-kserve-b7bd6f8fc-66jx2\" (UID: \"5db1488e-bb78-4632-8770-130d971b82c6\") " pod="ai-tenant-e2e-inf-a-1f5606/test-model-1f5606-kserve-b7bd6f8fc-66jx2" Jul 17 19:32:07.233213 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:07.233192 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxxnd\" (UniqueName: \"kubernetes.io/projected/5db1488e-bb78-4632-8770-130d971b82c6-kube-api-access-pxxnd\") pod \"test-model-1f5606-kserve-b7bd6f8fc-66jx2\" (UID: \"5db1488e-bb78-4632-8770-130d971b82c6\") " pod="ai-tenant-e2e-inf-a-1f5606/test-model-1f5606-kserve-b7bd6f8fc-66jx2" Jul 17 19:32:07.327832 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:07.327734 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1f5606/test-model-1f5606-kserve-b7bd6f8fc-66jx2" Jul 17 19:32:07.465147 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:07.465124 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1f5606/test-model-1f5606-kserve-b7bd6f8fc-66jx2"] Jul 17 19:32:07.466512 ip-10-0-134-137 kubenswrapper[2561]: W0717 19:32:07.466479 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5db1488e_bb78_4632_8770_130d971b82c6.slice/crio-a6f534703db5ed3c7f7d176bcd11bfdbc8626b25d1a1cbb99f776ee3badc9612 WatchSource:0}: Error finding container a6f534703db5ed3c7f7d176bcd11bfdbc8626b25d1a1cbb99f776ee3badc9612: Status 404 returned error can't find the container with id a6f534703db5ed3c7f7d176bcd11bfdbc8626b25d1a1cbb99f776ee3badc9612 Jul 17 19:32:07.891367 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:07.891329 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1f5606/test-model-1f5606-kserve-b7bd6f8fc-66jx2" event={"ID":"5db1488e-bb78-4632-8770-130d971b82c6","Type":"ContainerStarted","Data":"20e4c69aeb0f1c3698ee1e8a51fe8fa9ab7eb1125a9064e6c78f080e94635c9f"} Jul 17 19:32:07.891569 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:07.891383 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1f5606/test-model-1f5606-kserve-b7bd6f8fc-66jx2" event={"ID":"5db1488e-bb78-4632-8770-130d971b82c6","Type":"ContainerStarted","Data":"a6f534703db5ed3c7f7d176bcd11bfdbc8626b25d1a1cbb99f776ee3badc9612"} Jul 17 19:32:07.914228 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:07.914165 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-1f5606/test-model-1f5606-kserve-b7bd6f8fc-66jx2" podStartSLOduration=1.9141488469999999 podStartE2EDuration="1.914148847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:32:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:32:07.912312578 +0000 UTC m=+2226.242128786" watchObservedRunningTime="2026-07-17 19:32:07.914148847 +0000 UTC m=+2226.243965053" Jul 17 19:32:10.977898 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:10.977864 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:32:17.327853 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:17.327811 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-1f5606/test-model-1f5606-kserve-b7bd6f8fc-66jx2" Jul 17 19:32:17.328287 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:17.327867 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-1f5606/test-model-1f5606-kserve-b7bd6f8fc-66jx2" Jul 17 19:32:17.340082 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:17.340057 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-1f5606/test-model-1f5606-kserve-b7bd6f8fc-66jx2" Jul 17 19:32:17.952419 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:17.952378 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-1f5606/test-model-1f5606-kserve-b7bd6f8fc-66jx2" Jul 17 19:32:25.577693 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:25.577606 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:32:39.993713 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:39.993675 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1f5606/test-model-1f5606-kserve-b7bd6f8fc-66jx2"] Jul 17 19:32:40.576430 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:40.576396 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:32:41.057125 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:41.057082 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f5606-858mx"] Jul 17 19:32:41.060603 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:41.060573 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f5606-858mx" Jul 17 19:32:41.063151 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:41.063125 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lbvd4\"" Jul 17 19:32:41.068668 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:41.068638 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f5606-858mx"] Jul 17 19:32:41.221707 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:41.221658 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/277ebff1-06c2-403d-bd5c-0384c531988b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1f5606-858mx\" (UID: \"277ebff1-06c2-403d-bd5c-0384c531988b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f5606-858mx" Jul 17 19:32:41.323141 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:41.323036 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/277ebff1-06c2-403d-bd5c-0384c531988b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1f5606-858mx\" (UID: \"277ebff1-06c2-403d-bd5c-0384c531988b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f5606-858mx" Jul 17 19:32:41.323726 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:41.323707 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/277ebff1-06c2-403d-bd5c-0384c531988b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1f5606-858mx\" (UID: \"277ebff1-06c2-403d-bd5c-0384c531988b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f5606-858mx" Jul 17 19:32:41.372901 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:41.372868 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f5606-858mx" Jul 17 19:32:41.509555 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:41.509504 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f5606-858mx"] Jul 17 19:32:41.511135 ip-10-0-134-137 kubenswrapper[2561]: W0717 19:32:41.511108 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod277ebff1_06c2_403d_bd5c_0384c531988b.slice/crio-d530d8d5579e071663de1d521bc918015a1612d0d046ef91e04643f9c093939c WatchSource:0}: Error finding container d530d8d5579e071663de1d521bc918015a1612d0d046ef91e04643f9c093939c: Status 404 returned error can't find the container with id d530d8d5579e071663de1d521bc918015a1612d0d046ef91e04643f9c093939c Jul 17 19:32:42.072386 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:42.072350 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f5606-858mx" event={"ID":"277ebff1-06c2-403d-bd5c-0384c531988b","Type":"ContainerStarted","Data":"dd70db4b6c1936c97e82e28e8dc1fa4309a03d2824b2446ef894e919d8e23c2e"} Jul 17 19:32:42.072921 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:42.072404 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f5606-858mx" event={"ID":"277ebff1-06c2-403d-bd5c-0384c531988b","Type":"ContainerStarted","Data":"d530d8d5579e071663de1d521bc918015a1612d0d046ef91e04643f9c093939c"} Jul 17 19:32:42.090562 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:42.090491 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f5606-858mx" podStartSLOduration=1.090475971 podStartE2EDuration="1.090475971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:32:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:32:42.088045557 +0000 UTC m=+2260.417861768" watchObservedRunningTime="2026-07-17 19:32:42.090475971 +0000 UTC m=+2260.420292178" Jul 17 19:32:43.079490 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:43.079454 2561 generic.go:356] "Generic (PLEG): container finished" podID="277ebff1-06c2-403d-bd5c-0384c531988b" containerID="dd70db4b6c1936c97e82e28e8dc1fa4309a03d2824b2446ef894e919d8e23c2e" exitCode=0 Jul 17 19:32:43.079915 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:43.079564 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f5606-858mx" event={"ID":"277ebff1-06c2-403d-bd5c-0384c531988b","Type":"ContainerDied","Data":"dd70db4b6c1936c97e82e28e8dc1fa4309a03d2824b2446ef894e919d8e23c2e"} Jul 17 19:32:44.212577 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:44.212551 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f5606-858mx" Jul 17 19:32:44.350391 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:44.350295 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/277ebff1-06c2-403d-bd5c-0384c531988b-maas-api-service-ca\") pod \"277ebff1-06c2-403d-bd5c-0384c531988b\" (UID: \"277ebff1-06c2-403d-bd5c-0384c531988b\") " Jul 17 19:32:44.350695 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:44.350673 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/277ebff1-06c2-403d-bd5c-0384c531988b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "277ebff1-06c2-403d-bd5c-0384c531988b" (UID: "277ebff1-06c2-403d-bd5c-0384c531988b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 19:32:44.451846 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:44.451807 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/277ebff1-06c2-403d-bd5c-0384c531988b-maas-api-service-ca\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:32:45.091497 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:45.091462 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f5606-858mx" event={"ID":"277ebff1-06c2-403d-bd5c-0384c531988b","Type":"ContainerDied","Data":"d530d8d5579e071663de1d521bc918015a1612d0d046ef91e04643f9c093939c"} Jul 17 19:32:45.091700 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:45.091501 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f5606-858mx" Jul 17 19:32:45.091700 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:45.091510 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d530d8d5579e071663de1d521bc918015a1612d0d046ef91e04643f9c093939c" Jul 17 19:32:51.568113 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:51.568079 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5606-6c464b5697-2d58k"] Jul 17 19:32:51.568574 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:51.568314 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5606-6c464b5697-2d58k" podUID="0c813b71-a420-4ebd-9053-2e45ce1ac455" containerName="maas-api" containerID="cri-o://d5fb4c2bb8e2a0173bc06d64ea9f79a06c6abc64c58f58a925100dcd85d7df83" gracePeriod=30 Jul 17 19:32:51.819678 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:51.819612 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5606-6c464b5697-2d58k" Jul 17 19:32:52.017015 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:52.016979 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c813b71-a420-4ebd-9053-2e45ce1ac455-maas-api-tls\") pod \"0c813b71-a420-4ebd-9053-2e45ce1ac455\" (UID: \"0c813b71-a420-4ebd-9053-2e45ce1ac455\") " Jul 17 19:32:52.017015 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:52.017017 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c813b71-a420-4ebd-9053-2e45ce1ac455-openshift-service-ca\") pod \"0c813b71-a420-4ebd-9053-2e45ce1ac455\" (UID: \"0c813b71-a420-4ebd-9053-2e45ce1ac455\") " Jul 17 19:32:52.017229 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:52.017099 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smxfb\" (UniqueName: \"kubernetes.io/projected/0c813b71-a420-4ebd-9053-2e45ce1ac455-kube-api-access-smxfb\") pod \"0c813b71-a420-4ebd-9053-2e45ce1ac455\" (UID: \"0c813b71-a420-4ebd-9053-2e45ce1ac455\") " Jul 17 19:32:52.017428 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:52.017405 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c813b71-a420-4ebd-9053-2e45ce1ac455-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0c813b71-a420-4ebd-9053-2e45ce1ac455" (UID: "0c813b71-a420-4ebd-9053-2e45ce1ac455"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 19:32:52.019356 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:52.019332 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c813b71-a420-4ebd-9053-2e45ce1ac455-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0c813b71-a420-4ebd-9053-2e45ce1ac455" (UID: "0c813b71-a420-4ebd-9053-2e45ce1ac455"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 19:32:52.019519 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:52.019493 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c813b71-a420-4ebd-9053-2e45ce1ac455-kube-api-access-smxfb" (OuterVolumeSpecName: "kube-api-access-smxfb") pod "0c813b71-a420-4ebd-9053-2e45ce1ac455" (UID: "0c813b71-a420-4ebd-9053-2e45ce1ac455"). InnerVolumeSpecName "kube-api-access-smxfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 19:32:52.118265 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:52.118223 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smxfb\" (UniqueName: \"kubernetes.io/projected/0c813b71-a420-4ebd-9053-2e45ce1ac455-kube-api-access-smxfb\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:32:52.118265 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:52.118258 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c813b71-a420-4ebd-9053-2e45ce1ac455-maas-api-tls\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:32:52.118265 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:52.118267 2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c813b71-a420-4ebd-9053-2e45ce1ac455-openshift-service-ca\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:32:52.128286 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:52.128256 2561 generic.go:356] "Generic (PLEG): container finished" podID="0c813b71-a420-4ebd-9053-2e45ce1ac455" containerID="d5fb4c2bb8e2a0173bc06d64ea9f79a06c6abc64c58f58a925100dcd85d7df83" exitCode=0 Jul 17 19:32:52.128482 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:52.128316 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5606-6c464b5697-2d58k" event={"ID":"0c813b71-a420-4ebd-9053-2e45ce1ac455","Type":"ContainerDied","Data":"d5fb4c2bb8e2a0173bc06d64ea9f79a06c6abc64c58f58a925100dcd85d7df83"} Jul 17 19:32:52.128482 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:52.128336 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5606-6c464b5697-2d58k" Jul 17 19:32:52.128482 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:52.128369 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5606-6c464b5697-2d58k" event={"ID":"0c813b71-a420-4ebd-9053-2e45ce1ac455","Type":"ContainerDied","Data":"2c2326af411e0fc4ff189e99b69c4f7b7374ceb7a4d7166518250a6716433018"} Jul 17 19:32:52.128482 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:52.128397 2561 scope.go:117] "RemoveContainer" containerID="d5fb4c2bb8e2a0173bc06d64ea9f79a06c6abc64c58f58a925100dcd85d7df83" Jul 17 19:32:52.137676 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:52.137657 2561 scope.go:117] "RemoveContainer" containerID="d5fb4c2bb8e2a0173bc06d64ea9f79a06c6abc64c58f58a925100dcd85d7df83" Jul 17 19:32:52.138001 ip-10-0-134-137 kubenswrapper[2561]: E0717 19:32:52.137976 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5fb4c2bb8e2a0173bc06d64ea9f79a06c6abc64c58f58a925100dcd85d7df83\": container with ID starting with d5fb4c2bb8e2a0173bc06d64ea9f79a06c6abc64c58f58a925100dcd85d7df83 not found: ID does not exist" containerID="d5fb4c2bb8e2a0173bc06d64ea9f79a06c6abc64c58f58a925100dcd85d7df83" Jul 17 19:32:52.138055 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:52.138012 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5fb4c2bb8e2a0173bc06d64ea9f79a06c6abc64c58f58a925100dcd85d7df83"} err="failed to get container status \"d5fb4c2bb8e2a0173bc06d64ea9f79a06c6abc64c58f58a925100dcd85d7df83\": rpc error: code = NotFound desc = could not find container \"d5fb4c2bb8e2a0173bc06d64ea9f79a06c6abc64c58f58a925100dcd85d7df83\": container with ID starting with d5fb4c2bb8e2a0173bc06d64ea9f79a06c6abc64c58f58a925100dcd85d7df83 not found: ID does not exist" Jul 17 19:32:52.146310 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:52.146283 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5606-6c464b5697-2d58k"] Jul 17 19:32:52.149826 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:52.149801 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5606-6c464b5697-2d58k"] Jul 17 19:32:54.080278 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:54.080242 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:32:54.099065 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:54.099026 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c813b71-a420-4ebd-9053-2e45ce1ac455" path="/var/lib/kubelet/pods/0c813b71-a420-4ebd-9053-2e45ce1ac455/volumes" Jul 17 19:32:54.743576 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:54.743504 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f5606-6c464b5697-2d58k" podUID="0c813b71-a420-4ebd-9053-2e45ce1ac455" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.53:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 17 19:32:55.014710 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:55.014612 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-1f5606/test-model-1f5606-kserve-b7bd6f8fc-66jx2" podUID="5db1488e-bb78-4632-8770-130d971b82c6" containerName="main" containerID="cri-o://20e4c69aeb0f1c3698ee1e8a51fe8fa9ab7eb1125a9064e6c78f080e94635c9f" gracePeriod=45 Jul 17 19:32:55.148243 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:55.148211 2561 generic.go:356] "Generic (PLEG): container finished" podID="5db1488e-bb78-4632-8770-130d971b82c6" containerID="20e4c69aeb0f1c3698ee1e8a51fe8fa9ab7eb1125a9064e6c78f080e94635c9f" exitCode=0 Jul 17 19:32:55.148632 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:55.148289 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1f5606/test-model-1f5606-kserve-b7bd6f8fc-66jx2" event={"ID":"5db1488e-bb78-4632-8770-130d971b82c6","Type":"ContainerDied","Data":"20e4c69aeb0f1c3698ee1e8a51fe8fa9ab7eb1125a9064e6c78f080e94635c9f"} Jul 17 19:32:55.263628 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:55.263604 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1f5606/test-model-1f5606-kserve-b7bd6f8fc-66jx2" Jul 17 19:32:55.345084 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:55.344989 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5db1488e-bb78-4632-8770-130d971b82c6-tmp-dir\") pod \"5db1488e-bb78-4632-8770-130d971b82c6\" (UID: \"5db1488e-bb78-4632-8770-130d971b82c6\") " Jul 17 19:32:55.345084 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:55.345032 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5db1488e-bb78-4632-8770-130d971b82c6-model-cache\") pod \"5db1488e-bb78-4632-8770-130d971b82c6\" (UID: \"5db1488e-bb78-4632-8770-130d971b82c6\") " Jul 17 19:32:55.345084 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:55.345051 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5db1488e-bb78-4632-8770-130d971b82c6-dshm\") pod \"5db1488e-bb78-4632-8770-130d971b82c6\" (UID: \"5db1488e-bb78-4632-8770-130d971b82c6\") " Jul 17 19:32:55.345084 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:55.345079 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5db1488e-bb78-4632-8770-130d971b82c6-home\") pod \"5db1488e-bb78-4632-8770-130d971b82c6\" (UID: \"5db1488e-bb78-4632-8770-130d971b82c6\") " Jul 17 19:32:55.345415 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:55.345123 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxxnd\" (UniqueName: \"kubernetes.io/projected/5db1488e-bb78-4632-8770-130d971b82c6-kube-api-access-pxxnd\") pod \"5db1488e-bb78-4632-8770-130d971b82c6\" (UID: \"5db1488e-bb78-4632-8770-130d971b82c6\") " Jul 17 19:32:55.345415 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:55.345171 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5db1488e-bb78-4632-8770-130d971b82c6-tls-certs\") pod \"5db1488e-bb78-4632-8770-130d971b82c6\" (UID: \"5db1488e-bb78-4632-8770-130d971b82c6\") " Jul 17 19:32:55.345415 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:55.345350 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db1488e-bb78-4632-8770-130d971b82c6-model-cache" (OuterVolumeSpecName: "model-cache") pod "5db1488e-bb78-4632-8770-130d971b82c6" (UID: "5db1488e-bb78-4632-8770-130d971b82c6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 19:32:55.345415 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:55.345364 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db1488e-bb78-4632-8770-130d971b82c6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5db1488e-bb78-4632-8770-130d971b82c6" (UID: "5db1488e-bb78-4632-8770-130d971b82c6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 19:32:55.345415 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:55.345396 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db1488e-bb78-4632-8770-130d971b82c6-home" (OuterVolumeSpecName: "home") pod "5db1488e-bb78-4632-8770-130d971b82c6" (UID: "5db1488e-bb78-4632-8770-130d971b82c6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 19:32:55.347558 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:55.347507 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5db1488e-bb78-4632-8770-130d971b82c6-kube-api-access-pxxnd" (OuterVolumeSpecName: "kube-api-access-pxxnd") pod "5db1488e-bb78-4632-8770-130d971b82c6" (UID: "5db1488e-bb78-4632-8770-130d971b82c6"). InnerVolumeSpecName "kube-api-access-pxxnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 19:32:55.347558 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:55.347547 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db1488e-bb78-4632-8770-130d971b82c6-dshm" (OuterVolumeSpecName: "dshm") pod "5db1488e-bb78-4632-8770-130d971b82c6" (UID: "5db1488e-bb78-4632-8770-130d971b82c6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 19:32:55.347702 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:55.347554 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5db1488e-bb78-4632-8770-130d971b82c6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5db1488e-bb78-4632-8770-130d971b82c6" (UID: "5db1488e-bb78-4632-8770-130d971b82c6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 19:32:55.446596 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:55.446560 2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5db1488e-bb78-4632-8770-130d971b82c6-tls-certs\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:32:55.446596 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:55.446586 2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5db1488e-bb78-4632-8770-130d971b82c6-tmp-dir\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:32:55.446596 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:55.446595 2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5db1488e-bb78-4632-8770-130d971b82c6-model-cache\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:32:55.446596 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:55.446603 2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5db1488e-bb78-4632-8770-130d971b82c6-dshm\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:32:55.446596 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:55.446610 2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5db1488e-bb78-4632-8770-130d971b82c6-home\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:32:55.446882 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:55.446618 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxxnd\" (UniqueName: \"kubernetes.io/projected/5db1488e-bb78-4632-8770-130d971b82c6-kube-api-access-pxxnd\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:32:56.154802 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:56.154766 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1f5606/test-model-1f5606-kserve-b7bd6f8fc-66jx2" Jul 17 19:32:56.155270 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:56.154811 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1f5606/test-model-1f5606-kserve-b7bd6f8fc-66jx2" event={"ID":"5db1488e-bb78-4632-8770-130d971b82c6","Type":"ContainerDied","Data":"a6f534703db5ed3c7f7d176bcd11bfdbc8626b25d1a1cbb99f776ee3badc9612"} Jul 17 19:32:56.155270 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:56.154872 2561 scope.go:117] "RemoveContainer" containerID="20e4c69aeb0f1c3698ee1e8a51fe8fa9ab7eb1125a9064e6c78f080e94635c9f" Jul 17 19:32:56.171649 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:56.171623 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1f5606/test-model-1f5606-kserve-b7bd6f8fc-66jx2"] Jul 17 19:32:56.175211 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:56.175186 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-1f5606/test-model-1f5606-kserve-b7bd6f8fc-66jx2"] Jul 17 19:32:58.098748 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:32:58.098713 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5db1488e-bb78-4632-8770-130d971b82c6" path="/var/lib/kubelet/pods/5db1488e-bb78-4632-8770-130d971b82c6/volumes" Jul 17 19:33:07.178607 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:07.178570 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eaec93-qkt8h"] Jul 17 19:33:07.179005 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:07.178885 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c813b71-a420-4ebd-9053-2e45ce1ac455" containerName="maas-api" Jul 17 19:33:07.179005 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:07.178896 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c813b71-a420-4ebd-9053-2e45ce1ac455" containerName="maas-api" Jul 17 19:33:07.179005 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:07.178903 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5db1488e-bb78-4632-8770-130d971b82c6" containerName="main" Jul 17 19:33:07.179005 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:07.178909 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5db1488e-bb78-4632-8770-130d971b82c6" containerName="main" Jul 17 19:33:07.179005 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:07.178931 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="277ebff1-06c2-403d-bd5c-0384c531988b" containerName="revoke-keys" Jul 17 19:33:07.179005 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:07.178937 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="277ebff1-06c2-403d-bd5c-0384c531988b" containerName="revoke-keys" Jul 17 19:33:07.179005 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:07.178992 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="277ebff1-06c2-403d-bd5c-0384c531988b" containerName="revoke-keys" Jul 17 19:33:07.179005 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:07.179001 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c813b71-a420-4ebd-9053-2e45ce1ac455" containerName="maas-api" Jul 17 19:33:07.179005 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:07.179009 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5db1488e-bb78-4632-8770-130d971b82c6" containerName="main" Jul 17 19:33:07.181962 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:07.181944 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eaec93-qkt8h" Jul 17 19:33:07.187210 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:07.187183 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lbvd4\"" Jul 17 19:33:07.202886 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:07.202851 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eaec93-qkt8h"] Jul 17 19:33:07.240028 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:07.239986 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/361e3eed-dcb6-4e5e-8743-def8ac513d70-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-eaec93-qkt8h\" (UID: \"361e3eed-dcb6-4e5e-8743-def8ac513d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eaec93-qkt8h" Jul 17 19:33:07.341140 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:07.341099 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/361e3eed-dcb6-4e5e-8743-def8ac513d70-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-eaec93-qkt8h\" (UID: \"361e3eed-dcb6-4e5e-8743-def8ac513d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eaec93-qkt8h" Jul 17 19:33:07.341863 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:07.341841 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/361e3eed-dcb6-4e5e-8743-def8ac513d70-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-eaec93-qkt8h\" (UID: \"361e3eed-dcb6-4e5e-8743-def8ac513d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eaec93-qkt8h" Jul 17 19:33:07.492294 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:07.492253 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eaec93-qkt8h" Jul 17 19:33:07.629613 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:07.629588 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eaec93-qkt8h"] Jul 17 19:33:07.631261 ip-10-0-134-137 kubenswrapper[2561]: W0717 19:33:07.631227 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod361e3eed_dcb6_4e5e_8743_def8ac513d70.slice/crio-d86bf6a07b33a8fa2188cf5e569faaa8301f27761224804d1a5a3bd1bff69329 WatchSource:0}: Error finding container d86bf6a07b33a8fa2188cf5e569faaa8301f27761224804d1a5a3bd1bff69329: Status 404 returned error can't find the container with id d86bf6a07b33a8fa2188cf5e569faaa8301f27761224804d1a5a3bd1bff69329 Jul 17 19:33:08.221309 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:08.221271 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eaec93-qkt8h" event={"ID":"361e3eed-dcb6-4e5e-8743-def8ac513d70","Type":"ContainerStarted","Data":"9e0c53d5205bffbb02db6b288f58ca0878862f8985b9cf4eb899a8b0170e681b"} Jul 17 19:33:08.221785 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:08.221329 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eaec93-qkt8h" event={"ID":"361e3eed-dcb6-4e5e-8743-def8ac513d70","Type":"ContainerStarted","Data":"d86bf6a07b33a8fa2188cf5e569faaa8301f27761224804d1a5a3bd1bff69329"} Jul 17 19:33:08.237641 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:08.237585 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eaec93-qkt8h" podStartSLOduration=1.237567206 podStartE2EDuration="1.237567206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:33:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:33:08.236403386 +0000 UTC m=+2286.566219598" watchObservedRunningTime="2026-07-17 19:33:08.237567206 +0000 UTC m=+2286.567383411" Jul 17 19:33:09.228382 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:09.228347 2561 generic.go:356] "Generic (PLEG): container finished" podID="361e3eed-dcb6-4e5e-8743-def8ac513d70" containerID="9e0c53d5205bffbb02db6b288f58ca0878862f8985b9cf4eb899a8b0170e681b" exitCode=0 Jul 17 19:33:09.228829 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:09.228439 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eaec93-qkt8h" event={"ID":"361e3eed-dcb6-4e5e-8743-def8ac513d70","Type":"ContainerDied","Data":"9e0c53d5205bffbb02db6b288f58ca0878862f8985b9cf4eb899a8b0170e681b"} Jul 17 19:33:10.365867 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:10.365838 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eaec93-qkt8h" Jul 17 19:33:10.466999 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:10.466965 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/361e3eed-dcb6-4e5e-8743-def8ac513d70-maas-api-service-ca\") pod \"361e3eed-dcb6-4e5e-8743-def8ac513d70\" (UID: \"361e3eed-dcb6-4e5e-8743-def8ac513d70\") " Jul 17 19:33:10.467320 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:10.467297 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/361e3eed-dcb6-4e5e-8743-def8ac513d70-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "361e3eed-dcb6-4e5e-8743-def8ac513d70" (UID: "361e3eed-dcb6-4e5e-8743-def8ac513d70"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 19:33:10.567543 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:10.567495 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/361e3eed-dcb6-4e5e-8743-def8ac513d70-maas-api-service-ca\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:33:11.239417 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:11.239385 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eaec93-qkt8h" Jul 17 19:33:11.239674 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:11.239382 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-eaec93-qkt8h" event={"ID":"361e3eed-dcb6-4e5e-8743-def8ac513d70","Type":"ContainerDied","Data":"d86bf6a07b33a8fa2188cf5e569faaa8301f27761224804d1a5a3bd1bff69329"} Jul 17 19:33:11.239674 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:11.239514 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d86bf6a07b33a8fa2188cf5e569faaa8301f27761224804d1a5a3bd1bff69329" Jul 17 19:33:17.707298 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:17.707263 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-eaec93-7f8469666b-7rdz5"] Jul 17 19:33:17.707948 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:17.707485 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eaec93-7f8469666b-7rdz5" podUID="6e6ba068-a540-4fee-9117-93dee1cc2ff3" containerName="maas-api" containerID="cri-o://d04d35409411ee4b74bb4cdd294024c457506d5bc5e732df68aaaa408acf393a" gracePeriod=30 Jul 17 19:33:17.963172 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:17.963103 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eaec93-7f8469666b-7rdz5" Jul 17 19:33:18.034201 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:18.034159 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z2dph\" (UniqueName: \"kubernetes.io/projected/6e6ba068-a540-4fee-9117-93dee1cc2ff3-kube-api-access-z2dph\") pod \"6e6ba068-a540-4fee-9117-93dee1cc2ff3\" (UID: \"6e6ba068-a540-4fee-9117-93dee1cc2ff3\") " Jul 17 19:33:18.034388 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:18.034288 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e6ba068-a540-4fee-9117-93dee1cc2ff3-maas-api-tls\") pod \"6e6ba068-a540-4fee-9117-93dee1cc2ff3\" (UID: \"6e6ba068-a540-4fee-9117-93dee1cc2ff3\") " Jul 17 19:33:18.034388 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:18.034329 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e6ba068-a540-4fee-9117-93dee1cc2ff3-openshift-service-ca\") pod \"6e6ba068-a540-4fee-9117-93dee1cc2ff3\" (UID: \"6e6ba068-a540-4fee-9117-93dee1cc2ff3\") " Jul 17 19:33:18.034753 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:18.034729 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e6ba068-a540-4fee-9117-93dee1cc2ff3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6e6ba068-a540-4fee-9117-93dee1cc2ff3" (UID: "6e6ba068-a540-4fee-9117-93dee1cc2ff3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 19:33:18.036430 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:18.036409 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e6ba068-a540-4fee-9117-93dee1cc2ff3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6e6ba068-a540-4fee-9117-93dee1cc2ff3" (UID: "6e6ba068-a540-4fee-9117-93dee1cc2ff3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 19:33:18.036482 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:18.036463 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e6ba068-a540-4fee-9117-93dee1cc2ff3-kube-api-access-z2dph" (OuterVolumeSpecName: "kube-api-access-z2dph") pod "6e6ba068-a540-4fee-9117-93dee1cc2ff3" (UID: "6e6ba068-a540-4fee-9117-93dee1cc2ff3"). InnerVolumeSpecName "kube-api-access-z2dph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 19:33:18.135506 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:18.135467 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e6ba068-a540-4fee-9117-93dee1cc2ff3-maas-api-tls\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:33:18.135506 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:18.135500 2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e6ba068-a540-4fee-9117-93dee1cc2ff3-openshift-service-ca\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:33:18.135506 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:18.135510 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2dph\" (UniqueName: \"kubernetes.io/projected/6e6ba068-a540-4fee-9117-93dee1cc2ff3-kube-api-access-z2dph\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:33:18.277483 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:18.277443 2561 generic.go:356] "Generic (PLEG): container finished" podID="6e6ba068-a540-4fee-9117-93dee1cc2ff3" containerID="d04d35409411ee4b74bb4cdd294024c457506d5bc5e732df68aaaa408acf393a" exitCode=0 Jul 17 19:33:18.277681 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:18.277499 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eaec93-7f8469666b-7rdz5" Jul 17 19:33:18.277681 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:18.277519 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eaec93-7f8469666b-7rdz5" event={"ID":"6e6ba068-a540-4fee-9117-93dee1cc2ff3","Type":"ContainerDied","Data":"d04d35409411ee4b74bb4cdd294024c457506d5bc5e732df68aaaa408acf393a"} Jul 17 19:33:18.277681 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:18.277605 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-eaec93-7f8469666b-7rdz5" event={"ID":"6e6ba068-a540-4fee-9117-93dee1cc2ff3","Type":"ContainerDied","Data":"b489263fb0bb4a2b2fc6ea43ea705ed85624bc3740da5c7b465a070c94fb128c"} Jul 17 19:33:18.277681 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:18.277630 2561 scope.go:117] "RemoveContainer" containerID="d04d35409411ee4b74bb4cdd294024c457506d5bc5e732df68aaaa408acf393a" Jul 17 19:33:18.286669 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:18.286642 2561 scope.go:117] "RemoveContainer" containerID="d04d35409411ee4b74bb4cdd294024c457506d5bc5e732df68aaaa408acf393a" Jul 17 19:33:18.286988 ip-10-0-134-137 kubenswrapper[2561]: E0717 19:33:18.286966 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d04d35409411ee4b74bb4cdd294024c457506d5bc5e732df68aaaa408acf393a\": container with ID starting with d04d35409411ee4b74bb4cdd294024c457506d5bc5e732df68aaaa408acf393a not found: ID does not exist" containerID="d04d35409411ee4b74bb4cdd294024c457506d5bc5e732df68aaaa408acf393a" Jul 17 19:33:18.287054 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:18.286997 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d04d35409411ee4b74bb4cdd294024c457506d5bc5e732df68aaaa408acf393a"} err="failed to get container status \"d04d35409411ee4b74bb4cdd294024c457506d5bc5e732df68aaaa408acf393a\": rpc error: code = NotFound desc = could not find container \"d04d35409411ee4b74bb4cdd294024c457506d5bc5e732df68aaaa408acf393a\": container with ID starting with d04d35409411ee4b74bb4cdd294024c457506d5bc5e732df68aaaa408acf393a not found: ID does not exist" Jul 17 19:33:18.294093 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:18.294061 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-eaec93-7f8469666b-7rdz5"] Jul 17 19:33:18.298003 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:18.297975 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-eaec93-7f8469666b-7rdz5"] Jul 17 19:33:20.098574 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:20.098517 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e6ba068-a540-4fee-9117-93dee1cc2ff3" path="/var/lib/kubelet/pods/6e6ba068-a540-4fee-9117-93dee1cc2ff3/volumes" Jul 17 19:33:20.879452 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:20.879414 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:33:37.102737 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:37.102704 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0bfe89-7cd778b566-dgmhm"] Jul 17 19:33:37.103125 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:37.102993 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="361e3eed-dcb6-4e5e-8743-def8ac513d70" containerName="revoke-keys" Jul 17 19:33:37.103125 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:37.103004 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="361e3eed-dcb6-4e5e-8743-def8ac513d70" containerName="revoke-keys" Jul 17 19:33:37.103125 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:37.103025 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e6ba068-a540-4fee-9117-93dee1cc2ff3" containerName="maas-api" Jul 17 19:33:37.103125 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:37.103030 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e6ba068-a540-4fee-9117-93dee1cc2ff3" containerName="maas-api" Jul 17 19:33:37.103125 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:37.103096 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="361e3eed-dcb6-4e5e-8743-def8ac513d70" containerName="revoke-keys" Jul 17 19:33:37.103125 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:37.103106 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e6ba068-a540-4fee-9117-93dee1cc2ff3" containerName="maas-api" Jul 17 19:33:37.105919 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:37.105900 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0bfe89-7cd778b566-dgmhm" Jul 17 19:33:37.108787 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:37.108756 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-mhzxx\"" Jul 17 19:33:37.108787 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:37.108756 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 17 19:33:37.108989 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:37.108763 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-0bfe89-serving-cert\"" Jul 17 19:33:37.109756 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:37.109741 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 19:33:37.115714 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:37.115685 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0bfe89-7cd778b566-dgmhm"] Jul 17 19:33:37.203277 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:37.203229 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfdv8\" (UniqueName: \"kubernetes.io/projected/97d625b8-0c99-4d56-b47f-cc8a45e0744a-kube-api-access-rfdv8\") pod \"maas-api-e2e-api-a-0bfe89-7cd778b566-dgmhm\" (UID: \"97d625b8-0c99-4d56-b47f-cc8a45e0744a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0bfe89-7cd778b566-dgmhm" Jul 17 19:33:37.203277 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:37.203282 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97d625b8-0c99-4d56-b47f-cc8a45e0744a-maas-api-tls\") pod \"maas-api-e2e-api-a-0bfe89-7cd778b566-dgmhm\" (UID: \"97d625b8-0c99-4d56-b47f-cc8a45e0744a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0bfe89-7cd778b566-dgmhm" Jul 17 19:33:37.203569 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:37.203300 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97d625b8-0c99-4d56-b47f-cc8a45e0744a-openshift-service-ca\") pod \"maas-api-e2e-api-a-0bfe89-7cd778b566-dgmhm\" (UID: \"97d625b8-0c99-4d56-b47f-cc8a45e0744a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0bfe89-7cd778b566-dgmhm" Jul 17 19:33:37.304658 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:37.304609 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97d625b8-0c99-4d56-b47f-cc8a45e0744a-maas-api-tls\") pod \"maas-api-e2e-api-a-0bfe89-7cd778b566-dgmhm\" (UID: \"97d625b8-0c99-4d56-b47f-cc8a45e0744a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0bfe89-7cd778b566-dgmhm" Jul 17 19:33:37.304658 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:37.304655 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97d625b8-0c99-4d56-b47f-cc8a45e0744a-openshift-service-ca\") pod \"maas-api-e2e-api-a-0bfe89-7cd778b566-dgmhm\" (UID: \"97d625b8-0c99-4d56-b47f-cc8a45e0744a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0bfe89-7cd778b566-dgmhm" Jul 17 19:33:37.304951 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:37.304728 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfdv8\" (UniqueName: \"kubernetes.io/projected/97d625b8-0c99-4d56-b47f-cc8a45e0744a-kube-api-access-rfdv8\") pod \"maas-api-e2e-api-a-0bfe89-7cd778b566-dgmhm\" (UID: \"97d625b8-0c99-4d56-b47f-cc8a45e0744a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0bfe89-7cd778b566-dgmhm" Jul 17 19:33:37.304951 ip-10-0-134-137 kubenswrapper[2561]: E0717 19:33:37.304768 2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-0bfe89-serving-cert: secret "maas-api-e2e-api-a-0bfe89-serving-cert" not found Jul 17 19:33:37.304951 ip-10-0-134-137 kubenswrapper[2561]: E0717 19:33:37.304858 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97d625b8-0c99-4d56-b47f-cc8a45e0744a-maas-api-tls podName:97d625b8-0c99-4d56-b47f-cc8a45e0744a nodeName:}" failed. No retries permitted until 2026-07-17 19:33:37.804838527 +0000 UTC m=+2316.134654727 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/97d625b8-0c99-4d56-b47f-cc8a45e0744a-maas-api-tls") pod "maas-api-e2e-api-a-0bfe89-7cd778b566-dgmhm" (UID: "97d625b8-0c99-4d56-b47f-cc8a45e0744a") : secret "maas-api-e2e-api-a-0bfe89-serving-cert" not found Jul 17 19:33:37.305435 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:37.305415 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97d625b8-0c99-4d56-b47f-cc8a45e0744a-openshift-service-ca\") pod \"maas-api-e2e-api-a-0bfe89-7cd778b566-dgmhm\" (UID: \"97d625b8-0c99-4d56-b47f-cc8a45e0744a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0bfe89-7cd778b566-dgmhm" Jul 17 19:33:37.315808 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:37.315779 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfdv8\" (UniqueName: \"kubernetes.io/projected/97d625b8-0c99-4d56-b47f-cc8a45e0744a-kube-api-access-rfdv8\") pod \"maas-api-e2e-api-a-0bfe89-7cd778b566-dgmhm\" (UID: \"97d625b8-0c99-4d56-b47f-cc8a45e0744a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0bfe89-7cd778b566-dgmhm" Jul 17 19:33:37.808363 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:37.808316 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97d625b8-0c99-4d56-b47f-cc8a45e0744a-maas-api-tls\") pod \"maas-api-e2e-api-a-0bfe89-7cd778b566-dgmhm\" (UID: \"97d625b8-0c99-4d56-b47f-cc8a45e0744a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0bfe89-7cd778b566-dgmhm" Jul 17 19:33:37.810980 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:37.810958 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97d625b8-0c99-4d56-b47f-cc8a45e0744a-maas-api-tls\") pod \"maas-api-e2e-api-a-0bfe89-7cd778b566-dgmhm\" (UID: \"97d625b8-0c99-4d56-b47f-cc8a45e0744a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0bfe89-7cd778b566-dgmhm" Jul 17 19:33:38.019229 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:38.019184 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0bfe89-7cd778b566-dgmhm" Jul 17 19:33:38.169941 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:38.169910 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0bfe89-7cd778b566-dgmhm"] Jul 17 19:33:38.170835 ip-10-0-134-137 kubenswrapper[2561]: W0717 19:33:38.170809 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97d625b8_0c99_4d56_b47f_cc8a45e0744a.slice/crio-9c896f6b877e9924097ab102acc04c172220dc697646e07e3347abdaa9c5e910 WatchSource:0}: Error finding container 9c896f6b877e9924097ab102acc04c172220dc697646e07e3347abdaa9c5e910: Status 404 returned error can't find the container with id 9c896f6b877e9924097ab102acc04c172220dc697646e07e3347abdaa9c5e910 Jul 17 19:33:38.172128 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:38.172107 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 19:33:38.386089 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:38.385997 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0bfe89-7cd778b566-dgmhm" event={"ID":"97d625b8-0c99-4d56-b47f-cc8a45e0744a","Type":"ContainerStarted","Data":"9c896f6b877e9924097ab102acc04c172220dc697646e07e3347abdaa9c5e910"} Jul 17 19:33:39.392864 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:39.392824 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0bfe89-7cd778b566-dgmhm" event={"ID":"97d625b8-0c99-4d56-b47f-cc8a45e0744a","Type":"ContainerStarted","Data":"16534aa1bfe4d521544afd090af7ab4e8192d1703e98aeeb2087a7244eadd845"} Jul 17 19:33:39.393277 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:39.392895 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0bfe89-7cd778b566-dgmhm" Jul 17 19:33:39.411044 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:39.410986 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0bfe89-7cd778b566-dgmhm" podStartSLOduration=1.885609724 podStartE2EDuration="2.410968089s" totalImagesPullingTime="525.358365ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:33:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:33:39.409928653 +0000 UTC m=+2317.739744867" watchObservedRunningTime="2026-07-17 19:33:39.410968089 +0000 UTC m=+2317.740784296" Jul 17 19:33:41.681636 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:41.681591 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:33:45.403914 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:45.403884 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0bfe89-7cd778b566-dgmhm" Jul 17 19:33:45.584097 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:45.584063 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b6c908-66bf6c6f49-f9b2n"] Jul 17 19:33:45.587701 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:45.587669 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6c908-66bf6c6f49-f9b2n" Jul 17 19:33:45.590877 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:45.590843 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-b6c908-serving-cert\"" Jul 17 19:33:45.601325 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:45.601296 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b6c908-66bf6c6f49-f9b2n"] Jul 17 19:33:45.772060 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:45.772012 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cztbs\" (UniqueName: \"kubernetes.io/projected/e22f48bb-c652-4b30-8cfa-55deca1ba669-kube-api-access-cztbs\") pod \"maas-api-e2e-api-b-b6c908-66bf6c6f49-f9b2n\" (UID: \"e22f48bb-c652-4b30-8cfa-55deca1ba669\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6c908-66bf6c6f49-f9b2n" Jul 17 19:33:45.772274 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:45.772075 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e22f48bb-c652-4b30-8cfa-55deca1ba669-maas-api-tls\") pod \"maas-api-e2e-api-b-b6c908-66bf6c6f49-f9b2n\" (UID: \"e22f48bb-c652-4b30-8cfa-55deca1ba669\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6c908-66bf6c6f49-f9b2n" Jul 17 19:33:45.772274 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:45.772144 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e22f48bb-c652-4b30-8cfa-55deca1ba669-openshift-service-ca\") pod \"maas-api-e2e-api-b-b6c908-66bf6c6f49-f9b2n\" (UID: \"e22f48bb-c652-4b30-8cfa-55deca1ba669\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6c908-66bf6c6f49-f9b2n" Jul 17 19:33:45.872990 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:45.872946 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e22f48bb-c652-4b30-8cfa-55deca1ba669-openshift-service-ca\") pod \"maas-api-e2e-api-b-b6c908-66bf6c6f49-f9b2n\" (UID: \"e22f48bb-c652-4b30-8cfa-55deca1ba669\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6c908-66bf6c6f49-f9b2n" Jul 17 19:33:45.873167 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:45.873024 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cztbs\" (UniqueName: \"kubernetes.io/projected/e22f48bb-c652-4b30-8cfa-55deca1ba669-kube-api-access-cztbs\") pod \"maas-api-e2e-api-b-b6c908-66bf6c6f49-f9b2n\" (UID: \"e22f48bb-c652-4b30-8cfa-55deca1ba669\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6c908-66bf6c6f49-f9b2n" Jul 17 19:33:45.873167 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:45.873056 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e22f48bb-c652-4b30-8cfa-55deca1ba669-maas-api-tls\") pod \"maas-api-e2e-api-b-b6c908-66bf6c6f49-f9b2n\" (UID: \"e22f48bb-c652-4b30-8cfa-55deca1ba669\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6c908-66bf6c6f49-f9b2n" Jul 17 19:33:45.873829 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:45.873762 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e22f48bb-c652-4b30-8cfa-55deca1ba669-openshift-service-ca\") pod \"maas-api-e2e-api-b-b6c908-66bf6c6f49-f9b2n\" (UID: \"e22f48bb-c652-4b30-8cfa-55deca1ba669\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6c908-66bf6c6f49-f9b2n" Jul 17 19:33:45.875630 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:45.875608 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e22f48bb-c652-4b30-8cfa-55deca1ba669-maas-api-tls\") pod \"maas-api-e2e-api-b-b6c908-66bf6c6f49-f9b2n\" (UID: \"e22f48bb-c652-4b30-8cfa-55deca1ba669\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6c908-66bf6c6f49-f9b2n" Jul 17 19:33:45.897842 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:45.897812 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cztbs\" (UniqueName: \"kubernetes.io/projected/e22f48bb-c652-4b30-8cfa-55deca1ba669-kube-api-access-cztbs\") pod \"maas-api-e2e-api-b-b6c908-66bf6c6f49-f9b2n\" (UID: \"e22f48bb-c652-4b30-8cfa-55deca1ba669\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6c908-66bf6c6f49-f9b2n" Jul 17 19:33:45.899690 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:45.899667 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6c908-66bf6c6f49-f9b2n" Jul 17 19:33:46.053166 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:46.053141 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b6c908-66bf6c6f49-f9b2n"] Jul 17 19:33:46.054774 ip-10-0-134-137 kubenswrapper[2561]: W0717 19:33:46.054741 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode22f48bb_c652_4b30_8cfa_55deca1ba669.slice/crio-89cee981c9060ab0f491f858e3993c8f7cf2d8d2e562b7cf579fc22f02be8f10 WatchSource:0}: Error finding container 89cee981c9060ab0f491f858e3993c8f7cf2d8d2e562b7cf579fc22f02be8f10: Status 404 returned error can't find the container with id 89cee981c9060ab0f491f858e3993c8f7cf2d8d2e562b7cf579fc22f02be8f10 Jul 17 19:33:46.432139 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:46.432113 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6c908-66bf6c6f49-f9b2n" event={"ID":"e22f48bb-c652-4b30-8cfa-55deca1ba669","Type":"ContainerStarted","Data":"89cee981c9060ab0f491f858e3993c8f7cf2d8d2e562b7cf579fc22f02be8f10"} Jul 17 19:33:47.441180 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:47.441135 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6c908-66bf6c6f49-f9b2n" event={"ID":"e22f48bb-c652-4b30-8cfa-55deca1ba669","Type":"ContainerStarted","Data":"142fac22711cb29be8d0452702bbf666b474b0a44a202ad34e0941fb61989548"} Jul 17 19:33:47.442221 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:47.442189 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6c908-66bf6c6f49-f9b2n" Jul 17 19:33:47.464343 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:47.464256 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6c908-66bf6c6f49-f9b2n" podStartSLOduration=2.091657962 podStartE2EDuration="2.464233085s" totalImagesPullingTime="372.575123ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:33:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:33:47.461504457 +0000 UTC m=+2325.791320680" watchObservedRunningTime="2026-07-17 19:33:47.464233085 +0000 UTC m=+2325.794049293" Jul 17 19:33:54.457317 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:54.457229 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6c908-66bf6c6f49-f9b2n" Jul 17 19:33:56.579675 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:56.579637 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:33:57.876412 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:57.876372 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0bfe89-ch87b"] Jul 17 19:33:57.880100 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:57.880079 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0bfe89-ch87b" Jul 17 19:33:57.882356 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:57.882334 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lbvd4\"" Jul 17 19:33:57.886839 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:57.886806 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0bfe89-ch87b"] Jul 17 19:33:57.970227 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:57.970187 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b21294b4-c87b-43ed-919c-073604b296b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0bfe89-ch87b\" (UID: \"b21294b4-c87b-43ed-919c-073604b296b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0bfe89-ch87b" Jul 17 19:33:58.071747 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:58.071707 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b21294b4-c87b-43ed-919c-073604b296b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0bfe89-ch87b\" (UID: \"b21294b4-c87b-43ed-919c-073604b296b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0bfe89-ch87b" Jul 17 19:33:58.072330 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:58.072309 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b21294b4-c87b-43ed-919c-073604b296b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0bfe89-ch87b\" (UID: \"b21294b4-c87b-43ed-919c-073604b296b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0bfe89-ch87b" Jul 17 19:33:58.192183 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:58.192094 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0bfe89-ch87b" Jul 17 19:33:58.333594 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:58.333558 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0bfe89-ch87b"] Jul 17 19:33:58.336051 ip-10-0-134-137 kubenswrapper[2561]: W0717 19:33:58.336018 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb21294b4_c87b_43ed_919c_073604b296b0.slice/crio-c8c35696e1e8450162e75561fc253a5904c6abed20d4b6c721a9503e03df4b2f WatchSource:0}: Error finding container c8c35696e1e8450162e75561fc253a5904c6abed20d4b6c721a9503e03df4b2f: Status 404 returned error can't find the container with id c8c35696e1e8450162e75561fc253a5904c6abed20d4b6c721a9503e03df4b2f Jul 17 19:33:58.499667 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:58.499554 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0bfe89-ch87b" event={"ID":"b21294b4-c87b-43ed-919c-073604b296b0","Type":"ContainerStarted","Data":"c8c35696e1e8450162e75561fc253a5904c6abed20d4b6c721a9503e03df4b2f"} Jul 17 19:33:59.506088 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:59.506050 2561 generic.go:356] "Generic (PLEG): container finished" podID="b21294b4-c87b-43ed-919c-073604b296b0" containerID="ed23a1b540f06fa437994b9ac549d48454da654760cbe4b4d9a1a2b02186b9a7" exitCode=0 Jul 17 19:33:59.506499 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:33:59.506125 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0bfe89-ch87b" event={"ID":"b21294b4-c87b-43ed-919c-073604b296b0","Type":"ContainerDied","Data":"ed23a1b540f06fa437994b9ac549d48454da654760cbe4b4d9a1a2b02186b9a7"} Jul 17 19:34:00.645404 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:00.645377 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0bfe89-ch87b" Jul 17 19:34:00.692374 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:00.692335 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b21294b4-c87b-43ed-919c-073604b296b0-maas-api-service-ca\") pod \"b21294b4-c87b-43ed-919c-073604b296b0\" (UID: \"b21294b4-c87b-43ed-919c-073604b296b0\") " Jul 17 19:34:00.692816 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:00.692788 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b21294b4-c87b-43ed-919c-073604b296b0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b21294b4-c87b-43ed-919c-073604b296b0" (UID: "b21294b4-c87b-43ed-919c-073604b296b0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 19:34:00.793332 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:00.793231 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b21294b4-c87b-43ed-919c-073604b296b0-maas-api-service-ca\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:34:01.518349 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:01.518314 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0bfe89-ch87b" Jul 17 19:34:01.518349 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:01.518338 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0bfe89-ch87b" event={"ID":"b21294b4-c87b-43ed-919c-073604b296b0","Type":"ContainerDied","Data":"c8c35696e1e8450162e75561fc253a5904c6abed20d4b6c721a9503e03df4b2f"} Jul 17 19:34:01.518662 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:01.518384 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8c35696e1e8450162e75561fc253a5904c6abed20d4b6c721a9503e03df4b2f" Jul 17 19:34:08.674304 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:08.674272 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0bfe89-7cd778b566-dgmhm"] Jul 17 19:34:08.674784 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:08.674470 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0bfe89-7cd778b566-dgmhm" podUID="97d625b8-0c99-4d56-b47f-cc8a45e0744a" containerName="maas-api" containerID="cri-o://16534aa1bfe4d521544afd090af7ab4e8192d1703e98aeeb2087a7244eadd845" gracePeriod=30 Jul 17 19:34:08.926917 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:08.926834 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0bfe89-7cd778b566-dgmhm" Jul 17 19:34:08.965747 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:08.965713 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rfdv8\" (UniqueName: \"kubernetes.io/projected/97d625b8-0c99-4d56-b47f-cc8a45e0744a-kube-api-access-rfdv8\") pod \"97d625b8-0c99-4d56-b47f-cc8a45e0744a\" (UID: \"97d625b8-0c99-4d56-b47f-cc8a45e0744a\") " Jul 17 19:34:08.965976 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:08.965773 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97d625b8-0c99-4d56-b47f-cc8a45e0744a-openshift-service-ca\") pod \"97d625b8-0c99-4d56-b47f-cc8a45e0744a\" (UID: \"97d625b8-0c99-4d56-b47f-cc8a45e0744a\") " Jul 17 19:34:08.965976 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:08.965833 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97d625b8-0c99-4d56-b47f-cc8a45e0744a-maas-api-tls\") pod \"97d625b8-0c99-4d56-b47f-cc8a45e0744a\" (UID: \"97d625b8-0c99-4d56-b47f-cc8a45e0744a\") " Jul 17 19:34:08.966128 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:08.966100 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97d625b8-0c99-4d56-b47f-cc8a45e0744a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "97d625b8-0c99-4d56-b47f-cc8a45e0744a" (UID: "97d625b8-0c99-4d56-b47f-cc8a45e0744a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 19:34:08.968183 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:08.968147 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97d625b8-0c99-4d56-b47f-cc8a45e0744a-kube-api-access-rfdv8" (OuterVolumeSpecName: "kube-api-access-rfdv8") pod "97d625b8-0c99-4d56-b47f-cc8a45e0744a" (UID: "97d625b8-0c99-4d56-b47f-cc8a45e0744a"). InnerVolumeSpecName "kube-api-access-rfdv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 19:34:08.968303 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:08.968230 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97d625b8-0c99-4d56-b47f-cc8a45e0744a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "97d625b8-0c99-4d56-b47f-cc8a45e0744a" (UID: "97d625b8-0c99-4d56-b47f-cc8a45e0744a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 19:34:09.066580 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:09.066515 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfdv8\" (UniqueName: \"kubernetes.io/projected/97d625b8-0c99-4d56-b47f-cc8a45e0744a-kube-api-access-rfdv8\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:34:09.066580 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:09.066572 2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97d625b8-0c99-4d56-b47f-cc8a45e0744a-openshift-service-ca\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:34:09.066580 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:09.066582 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97d625b8-0c99-4d56-b47f-cc8a45e0744a-maas-api-tls\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:34:09.561046 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:09.561013 2561 generic.go:356] "Generic (PLEG): container finished" podID="97d625b8-0c99-4d56-b47f-cc8a45e0744a" containerID="16534aa1bfe4d521544afd090af7ab4e8192d1703e98aeeb2087a7244eadd845" exitCode=0 Jul 17 19:34:09.561225 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:09.561060 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0bfe89-7cd778b566-dgmhm" Jul 17 19:34:09.561225 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:09.561103 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0bfe89-7cd778b566-dgmhm" event={"ID":"97d625b8-0c99-4d56-b47f-cc8a45e0744a","Type":"ContainerDied","Data":"16534aa1bfe4d521544afd090af7ab4e8192d1703e98aeeb2087a7244eadd845"} Jul 17 19:34:09.561225 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:09.561164 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0bfe89-7cd778b566-dgmhm" event={"ID":"97d625b8-0c99-4d56-b47f-cc8a45e0744a","Type":"ContainerDied","Data":"9c896f6b877e9924097ab102acc04c172220dc697646e07e3347abdaa9c5e910"} Jul 17 19:34:09.561225 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:09.561192 2561 scope.go:117] "RemoveContainer" containerID="16534aa1bfe4d521544afd090af7ab4e8192d1703e98aeeb2087a7244eadd845" Jul 17 19:34:09.570829 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:09.570805 2561 scope.go:117] "RemoveContainer" containerID="16534aa1bfe4d521544afd090af7ab4e8192d1703e98aeeb2087a7244eadd845" Jul 17 19:34:09.571147 ip-10-0-134-137 kubenswrapper[2561]: E0717 19:34:09.571123 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16534aa1bfe4d521544afd090af7ab4e8192d1703e98aeeb2087a7244eadd845\": container with ID starting with 16534aa1bfe4d521544afd090af7ab4e8192d1703e98aeeb2087a7244eadd845 not found: ID does not exist" containerID="16534aa1bfe4d521544afd090af7ab4e8192d1703e98aeeb2087a7244eadd845" Jul 17 19:34:09.571213 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:09.571157 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16534aa1bfe4d521544afd090af7ab4e8192d1703e98aeeb2087a7244eadd845"} err="failed to get container status \"16534aa1bfe4d521544afd090af7ab4e8192d1703e98aeeb2087a7244eadd845\": rpc error: code = NotFound desc = could not find container \"16534aa1bfe4d521544afd090af7ab4e8192d1703e98aeeb2087a7244eadd845\": container with ID starting with 16534aa1bfe4d521544afd090af7ab4e8192d1703e98aeeb2087a7244eadd845 not found: ID does not exist" Jul 17 19:34:09.582792 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:09.582754 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0bfe89-7cd778b566-dgmhm"] Jul 17 19:34:09.586138 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:09.586106 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0bfe89-7cd778b566-dgmhm"] Jul 17 19:34:10.098889 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:10.098853 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97d625b8-0c99-4d56-b47f-cc8a45e0744a" path="/var/lib/kubelet/pods/97d625b8-0c99-4d56-b47f-cc8a45e0744a/volumes" Jul 17 19:34:11.980876 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:11.980835 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:34:23.994359 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:23.994322 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b6c908-4t2f2"] Jul 17 19:34:23.994767 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:23.994680 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b21294b4-c87b-43ed-919c-073604b296b0" containerName="revoke-keys" Jul 17 19:34:23.994767 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:23.994692 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b21294b4-c87b-43ed-919c-073604b296b0" containerName="revoke-keys" Jul 17 19:34:23.994767 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:23.994714 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97d625b8-0c99-4d56-b47f-cc8a45e0744a" containerName="maas-api" Jul 17 19:34:23.994767 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:23.994723 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="97d625b8-0c99-4d56-b47f-cc8a45e0744a" containerName="maas-api" Jul 17 19:34:23.994912 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:23.994774 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="97d625b8-0c99-4d56-b47f-cc8a45e0744a" containerName="maas-api" Jul 17 19:34:23.994912 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:23.994788 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b21294b4-c87b-43ed-919c-073604b296b0" containerName="revoke-keys" Jul 17 19:34:23.999202 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:23.999184 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b6c908-4t2f2" Jul 17 19:34:24.001728 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:24.001711 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lbvd4\"" Jul 17 19:34:24.008539 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:24.007416 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b6c908-4t2f2"] Jul 17 19:34:24.093648 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:24.093594 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5be5c445-dd2b-46e1-abcc-6becd189ad3a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b6c908-4t2f2\" (UID: \"5be5c445-dd2b-46e1-abcc-6becd189ad3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b6c908-4t2f2" Jul 17 19:34:24.194791 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:24.194752 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5be5c445-dd2b-46e1-abcc-6becd189ad3a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b6c908-4t2f2\" (UID: \"5be5c445-dd2b-46e1-abcc-6becd189ad3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b6c908-4t2f2" Jul 17 19:34:24.195377 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:24.195356 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5be5c445-dd2b-46e1-abcc-6becd189ad3a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b6c908-4t2f2\" (UID: \"5be5c445-dd2b-46e1-abcc-6becd189ad3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b6c908-4t2f2" Jul 17 19:34:24.317048 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:24.316941 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b6c908-4t2f2" Jul 17 19:34:24.464229 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:24.464204 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b6c908-4t2f2"] Jul 17 19:34:24.466337 ip-10-0-134-137 kubenswrapper[2561]: W0717 19:34:24.466303 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5be5c445_dd2b_46e1_abcc_6becd189ad3a.slice/crio-d961985ff3c2a7bd07209951a0a86767ee402a5ceae1711532629cc7624824f1 WatchSource:0}: Error finding container d961985ff3c2a7bd07209951a0a86767ee402a5ceae1711532629cc7624824f1: Status 404 returned error can't find the container with id d961985ff3c2a7bd07209951a0a86767ee402a5ceae1711532629cc7624824f1 Jul 17 19:34:24.647820 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:24.647730 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b6c908-4t2f2" event={"ID":"5be5c445-dd2b-46e1-abcc-6becd189ad3a","Type":"ContainerStarted","Data":"d961985ff3c2a7bd07209951a0a86767ee402a5ceae1711532629cc7624824f1"} Jul 17 19:34:25.655087 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:25.655052 2561 generic.go:356] "Generic (PLEG): container finished" podID="5be5c445-dd2b-46e1-abcc-6becd189ad3a" containerID="d22ddc04a6a66829d75ae4ee1ed1cf09aff33e819b571e4c8a6e498ba67bd339" exitCode=0 Jul 17 19:34:25.655572 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:25.655134 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b6c908-4t2f2" event={"ID":"5be5c445-dd2b-46e1-abcc-6becd189ad3a","Type":"ContainerDied","Data":"d22ddc04a6a66829d75ae4ee1ed1cf09aff33e819b571e4c8a6e498ba67bd339"} Jul 17 19:34:26.796725 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:26.796696 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b6c908-4t2f2" Jul 17 19:34:26.917878 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:26.917781 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5be5c445-dd2b-46e1-abcc-6becd189ad3a-maas-api-service-ca\") pod \"5be5c445-dd2b-46e1-abcc-6becd189ad3a\" (UID: \"5be5c445-dd2b-46e1-abcc-6becd189ad3a\") " Jul 17 19:34:26.918140 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:26.918115 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5be5c445-dd2b-46e1-abcc-6becd189ad3a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5be5c445-dd2b-46e1-abcc-6becd189ad3a" (UID: "5be5c445-dd2b-46e1-abcc-6becd189ad3a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 19:34:27.018449 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:27.018407 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5be5c445-dd2b-46e1-abcc-6becd189ad3a-maas-api-service-ca\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:34:27.669215 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:27.669182 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b6c908-4t2f2" Jul 17 19:34:27.669394 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:27.669219 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b6c908-4t2f2" event={"ID":"5be5c445-dd2b-46e1-abcc-6becd189ad3a","Type":"ContainerDied","Data":"d961985ff3c2a7bd07209951a0a86767ee402a5ceae1711532629cc7624824f1"} Jul 17 19:34:27.669394 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:27.669265 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d961985ff3c2a7bd07209951a0a86767ee402a5ceae1711532629cc7624824f1" Jul 17 19:34:34.506540 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:34.506482 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b6c908-66bf6c6f49-f9b2n"] Jul 17 19:34:34.506996 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:34.506743 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6c908-66bf6c6f49-f9b2n" podUID="e22f48bb-c652-4b30-8cfa-55deca1ba669" containerName="maas-api" containerID="cri-o://142fac22711cb29be8d0452702bbf666b474b0a44a202ad34e0941fb61989548" gracePeriod=30 Jul 17 19:34:34.708408 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:34.708376 2561 generic.go:356] "Generic (PLEG): container finished" podID="e22f48bb-c652-4b30-8cfa-55deca1ba669" containerID="142fac22711cb29be8d0452702bbf666b474b0a44a202ad34e0941fb61989548" exitCode=0 Jul 17 19:34:34.708611 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:34.708451 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6c908-66bf6c6f49-f9b2n" event={"ID":"e22f48bb-c652-4b30-8cfa-55deca1ba669","Type":"ContainerDied","Data":"142fac22711cb29be8d0452702bbf666b474b0a44a202ad34e0941fb61989548"} Jul 17 19:34:34.763850 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:34.763779 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6c908-66bf6c6f49-f9b2n" Jul 17 19:34:34.885355 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:34.885319 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e22f48bb-c652-4b30-8cfa-55deca1ba669-openshift-service-ca\") pod \"e22f48bb-c652-4b30-8cfa-55deca1ba669\" (UID: \"e22f48bb-c652-4b30-8cfa-55deca1ba669\") " Jul 17 19:34:34.885561 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:34.885384 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cztbs\" (UniqueName: \"kubernetes.io/projected/e22f48bb-c652-4b30-8cfa-55deca1ba669-kube-api-access-cztbs\") pod \"e22f48bb-c652-4b30-8cfa-55deca1ba669\" (UID: \"e22f48bb-c652-4b30-8cfa-55deca1ba669\") " Jul 17 19:34:34.885561 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:34.885419 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e22f48bb-c652-4b30-8cfa-55deca1ba669-maas-api-tls\") pod \"e22f48bb-c652-4b30-8cfa-55deca1ba669\" (UID: \"e22f48bb-c652-4b30-8cfa-55deca1ba669\") " Jul 17 19:34:34.885833 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:34.885803 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e22f48bb-c652-4b30-8cfa-55deca1ba669-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e22f48bb-c652-4b30-8cfa-55deca1ba669" (UID: "e22f48bb-c652-4b30-8cfa-55deca1ba669"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 19:34:34.887780 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:34.887755 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e22f48bb-c652-4b30-8cfa-55deca1ba669-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e22f48bb-c652-4b30-8cfa-55deca1ba669" (UID: "e22f48bb-c652-4b30-8cfa-55deca1ba669"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 19:34:34.887897 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:34.887816 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e22f48bb-c652-4b30-8cfa-55deca1ba669-kube-api-access-cztbs" (OuterVolumeSpecName: "kube-api-access-cztbs") pod "e22f48bb-c652-4b30-8cfa-55deca1ba669" (UID: "e22f48bb-c652-4b30-8cfa-55deca1ba669"). InnerVolumeSpecName "kube-api-access-cztbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 19:34:34.986410 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:34.986359 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cztbs\" (UniqueName: \"kubernetes.io/projected/e22f48bb-c652-4b30-8cfa-55deca1ba669-kube-api-access-cztbs\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:34:34.986410 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:34.986399 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e22f48bb-c652-4b30-8cfa-55deca1ba669-maas-api-tls\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:34:34.986410 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:34.986409 2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e22f48bb-c652-4b30-8cfa-55deca1ba669-openshift-service-ca\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:34:35.715481 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:35.715449 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6c908-66bf6c6f49-f9b2n" Jul 17 19:34:35.715934 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:35.715491 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6c908-66bf6c6f49-f9b2n" event={"ID":"e22f48bb-c652-4b30-8cfa-55deca1ba669","Type":"ContainerDied","Data":"89cee981c9060ab0f491f858e3993c8f7cf2d8d2e562b7cf579fc22f02be8f10"} Jul 17 19:34:35.715934 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:35.715571 2561 scope.go:117] "RemoveContainer" containerID="142fac22711cb29be8d0452702bbf666b474b0a44a202ad34e0941fb61989548" Jul 17 19:34:35.736796 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:35.736756 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b6c908-66bf6c6f49-f9b2n"] Jul 17 19:34:35.742592 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:35.742562 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b6c908-66bf6c6f49-f9b2n"] Jul 17 19:34:36.103880 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:36.103845 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e22f48bb-c652-4b30-8cfa-55deca1ba669" path="/var/lib/kubelet/pods/e22f48bb-c652-4b30-8cfa-55deca1ba669/volumes" Jul 17 19:34:38.076860 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:38.076826 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:34:53.488252 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:53.488211 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-b3dd7f-95f8b99c7-bdb5k"] Jul 17 19:34:53.488674 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:53.488561 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e22f48bb-c652-4b30-8cfa-55deca1ba669" containerName="maas-api" Jul 17 19:34:53.488674 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:53.488577 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e22f48bb-c652-4b30-8cfa-55deca1ba669" containerName="maas-api" Jul 17 19:34:53.488674 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:53.488592 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5be5c445-dd2b-46e1-abcc-6becd189ad3a" containerName="revoke-keys" Jul 17 19:34:53.488674 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:53.488600 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5be5c445-dd2b-46e1-abcc-6becd189ad3a" containerName="revoke-keys" Jul 17 19:34:53.488674 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:53.488661 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e22f48bb-c652-4b30-8cfa-55deca1ba669" containerName="maas-api" Jul 17 19:34:53.488674 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:53.488672 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5be5c445-dd2b-46e1-abcc-6becd189ad3a" containerName="revoke-keys" Jul 17 19:34:53.491801 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:53.491779 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b3dd7f-95f8b99c7-bdb5k" Jul 17 19:34:53.494279 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:53.494254 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-b3dd7f-serving-cert\"" Jul 17 19:34:53.494410 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:53.494294 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-mhzxx\"" Jul 17 19:34:53.494760 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:53.494742 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 19:34:53.495127 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:53.495111 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 17 19:34:53.503891 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:53.503868 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-b3dd7f-95f8b99c7-bdb5k"] Jul 17 19:34:53.541841 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:53.541812 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c8d502e-7547-44f9-b69e-5fbc6f21acf9-maas-api-tls\") pod \"maas-api-e2e-shared-a-b3dd7f-95f8b99c7-bdb5k\" (UID: \"6c8d502e-7547-44f9-b69e-5fbc6f21acf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b3dd7f-95f8b99c7-bdb5k" Jul 17 19:34:53.542016 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:53.541846 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-542r6\" (UniqueName: \"kubernetes.io/projected/6c8d502e-7547-44f9-b69e-5fbc6f21acf9-kube-api-access-542r6\") pod \"maas-api-e2e-shared-a-b3dd7f-95f8b99c7-bdb5k\" (UID: \"6c8d502e-7547-44f9-b69e-5fbc6f21acf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b3dd7f-95f8b99c7-bdb5k" Jul 17 19:34:53.542016 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:53.541885 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c8d502e-7547-44f9-b69e-5fbc6f21acf9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-b3dd7f-95f8b99c7-bdb5k\" (UID: \"6c8d502e-7547-44f9-b69e-5fbc6f21acf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b3dd7f-95f8b99c7-bdb5k" Jul 17 19:34:53.642635 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:53.642593 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c8d502e-7547-44f9-b69e-5fbc6f21acf9-maas-api-tls\") pod \"maas-api-e2e-shared-a-b3dd7f-95f8b99c7-bdb5k\" (UID: \"6c8d502e-7547-44f9-b69e-5fbc6f21acf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b3dd7f-95f8b99c7-bdb5k" Jul 17 19:34:53.642635 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:53.642635 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-542r6\" (UniqueName: \"kubernetes.io/projected/6c8d502e-7547-44f9-b69e-5fbc6f21acf9-kube-api-access-542r6\") pod \"maas-api-e2e-shared-a-b3dd7f-95f8b99c7-bdb5k\" (UID: \"6c8d502e-7547-44f9-b69e-5fbc6f21acf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b3dd7f-95f8b99c7-bdb5k" Jul 17 19:34:53.642940 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:53.642759 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c8d502e-7547-44f9-b69e-5fbc6f21acf9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-b3dd7f-95f8b99c7-bdb5k\" (UID: \"6c8d502e-7547-44f9-b69e-5fbc6f21acf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b3dd7f-95f8b99c7-bdb5k" Jul 17 19:34:53.643385 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:53.643366 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c8d502e-7547-44f9-b69e-5fbc6f21acf9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-b3dd7f-95f8b99c7-bdb5k\" (UID: \"6c8d502e-7547-44f9-b69e-5fbc6f21acf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b3dd7f-95f8b99c7-bdb5k" Jul 17 19:34:53.645304 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:53.645280 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c8d502e-7547-44f9-b69e-5fbc6f21acf9-maas-api-tls\") pod \"maas-api-e2e-shared-a-b3dd7f-95f8b99c7-bdb5k\" (UID: \"6c8d502e-7547-44f9-b69e-5fbc6f21acf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b3dd7f-95f8b99c7-bdb5k" Jul 17 19:34:53.650191 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:53.650169 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-542r6\" (UniqueName: \"kubernetes.io/projected/6c8d502e-7547-44f9-b69e-5fbc6f21acf9-kube-api-access-542r6\") pod \"maas-api-e2e-shared-a-b3dd7f-95f8b99c7-bdb5k\" (UID: \"6c8d502e-7547-44f9-b69e-5fbc6f21acf9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b3dd7f-95f8b99c7-bdb5k" Jul 17 19:34:53.803389 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:53.803300 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b3dd7f-95f8b99c7-bdb5k" Jul 17 19:34:53.942028 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:53.941999 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-b3dd7f-95f8b99c7-bdb5k"] Jul 17 19:34:53.943327 ip-10-0-134-137 kubenswrapper[2561]: W0717 19:34:53.943302 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c8d502e_7547_44f9_b69e_5fbc6f21acf9.slice/crio-de2513595724340e5355a13edcbdeac7e1101fc435dc4d3dead01b3b2ccd4fbd WatchSource:0}: Error finding container de2513595724340e5355a13edcbdeac7e1101fc435dc4d3dead01b3b2ccd4fbd: Status 404 returned error can't find the container with id de2513595724340e5355a13edcbdeac7e1101fc435dc4d3dead01b3b2ccd4fbd Jul 17 19:34:54.818346 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:54.818314 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b3dd7f-95f8b99c7-bdb5k" event={"ID":"6c8d502e-7547-44f9-b69e-5fbc6f21acf9","Type":"ContainerStarted","Data":"3812109a92c1663c505680ae1102c029ada4222b9af5e736f912ea46f84b4a31"} Jul 17 19:34:54.818738 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:54.818363 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b3dd7f-95f8b99c7-bdb5k" event={"ID":"6c8d502e-7547-44f9-b69e-5fbc6f21acf9","Type":"ContainerStarted","Data":"de2513595724340e5355a13edcbdeac7e1101fc435dc4d3dead01b3b2ccd4fbd"} Jul 17 19:34:54.818738 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:54.818433 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b3dd7f-95f8b99c7-bdb5k" Jul 17 19:34:54.846895 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:54.846849 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b3dd7f-95f8b99c7-bdb5k" podStartSLOduration=1.486378695 podStartE2EDuration="1.846831564s" totalImagesPullingTime="360.452869ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:34:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:34:54.846094865 +0000 UTC m=+2393.175911077" watchObservedRunningTime="2026-07-17 19:34:54.846831564 +0000 UTC m=+2393.176647815" Jul 17 19:34:58.674602 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:34:58.674562 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:35:00.828768 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:00.828738 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b3dd7f-95f8b99c7-bdb5k" Jul 17 19:35:01.583611 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:01.583568 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-79d51b-55cdf87db8-kz7n9"] Jul 17 19:35:01.587118 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:01.587093 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-79d51b-55cdf87db8-kz7n9" Jul 17 19:35:01.589156 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:01.589136 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-79d51b-serving-cert\"" Jul 17 19:35:01.597587 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:01.597555 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-79d51b-55cdf87db8-kz7n9"] Jul 17 19:35:01.710003 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:01.709968 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89da65ea-9550-48de-8a63-33809021a506-maas-api-tls\") pod \"maas-api-e2e-shared-b-79d51b-55cdf87db8-kz7n9\" (UID: \"89da65ea-9550-48de-8a63-33809021a506\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-79d51b-55cdf87db8-kz7n9" Jul 17 19:35:01.710187 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:01.710027 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89da65ea-9550-48de-8a63-33809021a506-openshift-service-ca\") pod \"maas-api-e2e-shared-b-79d51b-55cdf87db8-kz7n9\" (UID: \"89da65ea-9550-48de-8a63-33809021a506\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-79d51b-55cdf87db8-kz7n9" Jul 17 19:35:01.710187 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:01.710151 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dk9m\" (UniqueName: \"kubernetes.io/projected/89da65ea-9550-48de-8a63-33809021a506-kube-api-access-2dk9m\") pod \"maas-api-e2e-shared-b-79d51b-55cdf87db8-kz7n9\" (UID: \"89da65ea-9550-48de-8a63-33809021a506\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-79d51b-55cdf87db8-kz7n9" Jul 17 19:35:01.811373 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:01.811342 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dk9m\" (UniqueName: \"kubernetes.io/projected/89da65ea-9550-48de-8a63-33809021a506-kube-api-access-2dk9m\") pod \"maas-api-e2e-shared-b-79d51b-55cdf87db8-kz7n9\" (UID: \"89da65ea-9550-48de-8a63-33809021a506\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-79d51b-55cdf87db8-kz7n9" Jul 17 19:35:01.811567 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:01.811382 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89da65ea-9550-48de-8a63-33809021a506-maas-api-tls\") pod \"maas-api-e2e-shared-b-79d51b-55cdf87db8-kz7n9\" (UID: \"89da65ea-9550-48de-8a63-33809021a506\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-79d51b-55cdf87db8-kz7n9" Jul 17 19:35:01.811567 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:01.811422 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89da65ea-9550-48de-8a63-33809021a506-openshift-service-ca\") pod \"maas-api-e2e-shared-b-79d51b-55cdf87db8-kz7n9\" (UID: \"89da65ea-9550-48de-8a63-33809021a506\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-79d51b-55cdf87db8-kz7n9" Jul 17 19:35:01.812061 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:01.812041 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89da65ea-9550-48de-8a63-33809021a506-openshift-service-ca\") pod \"maas-api-e2e-shared-b-79d51b-55cdf87db8-kz7n9\" (UID: \"89da65ea-9550-48de-8a63-33809021a506\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-79d51b-55cdf87db8-kz7n9" Jul 17 19:35:01.814018 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:01.813999 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89da65ea-9550-48de-8a63-33809021a506-maas-api-tls\") pod \"maas-api-e2e-shared-b-79d51b-55cdf87db8-kz7n9\" (UID: \"89da65ea-9550-48de-8a63-33809021a506\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-79d51b-55cdf87db8-kz7n9" Jul 17 19:35:01.818900 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:01.818877 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dk9m\" (UniqueName: \"kubernetes.io/projected/89da65ea-9550-48de-8a63-33809021a506-kube-api-access-2dk9m\") pod \"maas-api-e2e-shared-b-79d51b-55cdf87db8-kz7n9\" (UID: \"89da65ea-9550-48de-8a63-33809021a506\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-79d51b-55cdf87db8-kz7n9" Jul 17 19:35:01.898591 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:01.898464 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-79d51b-55cdf87db8-kz7n9" Jul 17 19:35:02.040703 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:02.040675 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-79d51b-55cdf87db8-kz7n9"] Jul 17 19:35:02.041719 ip-10-0-134-137 kubenswrapper[2561]: W0717 19:35:02.041682 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89da65ea_9550_48de_8a63_33809021a506.slice/crio-058d1dafd0e80ad204fd264a1b2141b3971851da37c69ad6577a0eab355a7558 WatchSource:0}: Error finding container 058d1dafd0e80ad204fd264a1b2141b3971851da37c69ad6577a0eab355a7558: Status 404 returned error can't find the container with id 058d1dafd0e80ad204fd264a1b2141b3971851da37c69ad6577a0eab355a7558 Jul 17 19:35:02.221896 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:02.221814 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpjzq_4b162535-066a-471a-aee4-99c07c6e1bb4/ovn-acl-logging/0.log" Jul 17 19:35:02.226686 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:02.226661 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpjzq_4b162535-066a-471a-aee4-99c07c6e1bb4/ovn-acl-logging/0.log" Jul 17 19:35:02.861775 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:02.861738 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-79d51b-55cdf87db8-kz7n9" event={"ID":"89da65ea-9550-48de-8a63-33809021a506","Type":"ContainerStarted","Data":"7bb2de0b9ee7e1cd7c93be85fd782ed254c3406edb77693a94939b695293780d"} Jul 17 19:35:02.862004 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:02.861799 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-79d51b-55cdf87db8-kz7n9" event={"ID":"89da65ea-9550-48de-8a63-33809021a506","Type":"ContainerStarted","Data":"058d1dafd0e80ad204fd264a1b2141b3971851da37c69ad6577a0eab355a7558"} Jul 17 19:35:02.862004 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:02.861859 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-79d51b-55cdf87db8-kz7n9" Jul 17 19:35:02.881349 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:02.881291 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-79d51b-55cdf87db8-kz7n9" podStartSLOduration=1.442802315 podStartE2EDuration="1.881273345s" totalImagesPullingTime="438.47103ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:35:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:35:02.878840204 +0000 UTC m=+2401.208656413" watchObservedRunningTime="2026-07-17 19:35:02.881273345 +0000 UTC m=+2401.211089552" Jul 17 19:35:08.872107 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:08.872074 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-79d51b-55cdf87db8-kz7n9" Jul 17 19:35:14.683141 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:14.683094 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:35:25.972718 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:25.972633 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:35:36.205840 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:36.205799 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-79d51b/auth-test-model-79d51b-kserve-5f976787bc-xsdhk"] Jul 17 19:35:36.210603 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:36.210583 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-79d51b/auth-test-model-79d51b-kserve-5f976787bc-xsdhk" Jul 17 19:35:36.214311 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:36.214285 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-79d51b\"/\"openshift-service-ca.crt\"" Jul 17 19:35:36.214474 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:36.214312 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-79d51b\"/\"default-dockercfg-sx4s7\"" Jul 17 19:35:36.214474 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:36.214346 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-79d51b\"/\"kube-root-ca.crt\"" Jul 17 19:35:36.214474 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:36.214285 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-79d51b\"/\"auth-test-model-79d51b-kserve-self-signed-certs\"" Jul 17 19:35:36.219118 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:36.219093 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-79d51b/auth-test-model-79d51b-kserve-5f976787bc-xsdhk"] Jul 17 19:35:36.305903 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:36.305856 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/662e0488-6f70-4417-bd71-a7e9000d0856-home\") pod \"auth-test-model-79d51b-kserve-5f976787bc-xsdhk\" (UID: \"662e0488-6f70-4417-bd71-a7e9000d0856\") " pod="ai-tenant-e2e-shared-b-79d51b/auth-test-model-79d51b-kserve-5f976787bc-xsdhk" Jul 17 19:35:36.305903 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:36.305896 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/662e0488-6f70-4417-bd71-a7e9000d0856-model-cache\") pod \"auth-test-model-79d51b-kserve-5f976787bc-xsdhk\" (UID: \"662e0488-6f70-4417-bd71-a7e9000d0856\") " pod="ai-tenant-e2e-shared-b-79d51b/auth-test-model-79d51b-kserve-5f976787bc-xsdhk" Jul 17 19:35:36.306154 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:36.305951 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/662e0488-6f70-4417-bd71-a7e9000d0856-tmp-dir\") pod \"auth-test-model-79d51b-kserve-5f976787bc-xsdhk\" (UID: \"662e0488-6f70-4417-bd71-a7e9000d0856\") " pod="ai-tenant-e2e-shared-b-79d51b/auth-test-model-79d51b-kserve-5f976787bc-xsdhk" Jul 17 19:35:36.306154 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:36.306007 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/662e0488-6f70-4417-bd71-a7e9000d0856-tls-certs\") pod \"auth-test-model-79d51b-kserve-5f976787bc-xsdhk\" (UID: \"662e0488-6f70-4417-bd71-a7e9000d0856\") " pod="ai-tenant-e2e-shared-b-79d51b/auth-test-model-79d51b-kserve-5f976787bc-xsdhk" Jul 17 19:35:36.306154 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:36.306098 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/662e0488-6f70-4417-bd71-a7e9000d0856-dshm\") pod \"auth-test-model-79d51b-kserve-5f976787bc-xsdhk\" (UID: \"662e0488-6f70-4417-bd71-a7e9000d0856\") " pod="ai-tenant-e2e-shared-b-79d51b/auth-test-model-79d51b-kserve-5f976787bc-xsdhk" Jul 17 19:35:36.306154 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:36.306130 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cx6j\" (UniqueName: \"kubernetes.io/projected/662e0488-6f70-4417-bd71-a7e9000d0856-kube-api-access-6cx6j\") pod \"auth-test-model-79d51b-kserve-5f976787bc-xsdhk\" (UID: \"662e0488-6f70-4417-bd71-a7e9000d0856\") " pod="ai-tenant-e2e-shared-b-79d51b/auth-test-model-79d51b-kserve-5f976787bc-xsdhk" Jul 17 19:35:36.407233 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:36.407194 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/662e0488-6f70-4417-bd71-a7e9000d0856-tmp-dir\") pod \"auth-test-model-79d51b-kserve-5f976787bc-xsdhk\" (UID: \"662e0488-6f70-4417-bd71-a7e9000d0856\") " pod="ai-tenant-e2e-shared-b-79d51b/auth-test-model-79d51b-kserve-5f976787bc-xsdhk" Jul 17 19:35:36.407233 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:36.407240 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/662e0488-6f70-4417-bd71-a7e9000d0856-tls-certs\") pod \"auth-test-model-79d51b-kserve-5f976787bc-xsdhk\" (UID: \"662e0488-6f70-4417-bd71-a7e9000d0856\") " pod="ai-tenant-e2e-shared-b-79d51b/auth-test-model-79d51b-kserve-5f976787bc-xsdhk" Jul 17 19:35:36.407488 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:36.407282 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/662e0488-6f70-4417-bd71-a7e9000d0856-dshm\") pod \"auth-test-model-79d51b-kserve-5f976787bc-xsdhk\" (UID: \"662e0488-6f70-4417-bd71-a7e9000d0856\") " pod="ai-tenant-e2e-shared-b-79d51b/auth-test-model-79d51b-kserve-5f976787bc-xsdhk" Jul 17 19:35:36.407488 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:36.407395 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cx6j\" (UniqueName: \"kubernetes.io/projected/662e0488-6f70-4417-bd71-a7e9000d0856-kube-api-access-6cx6j\") pod \"auth-test-model-79d51b-kserve-5f976787bc-xsdhk\" (UID: \"662e0488-6f70-4417-bd71-a7e9000d0856\") " pod="ai-tenant-e2e-shared-b-79d51b/auth-test-model-79d51b-kserve-5f976787bc-xsdhk" Jul 17 19:35:36.407488 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:36.407447 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/662e0488-6f70-4417-bd71-a7e9000d0856-home\") pod \"auth-test-model-79d51b-kserve-5f976787bc-xsdhk\" (UID: \"662e0488-6f70-4417-bd71-a7e9000d0856\") " pod="ai-tenant-e2e-shared-b-79d51b/auth-test-model-79d51b-kserve-5f976787bc-xsdhk" Jul 17 19:35:36.407488 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:36.407476 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/662e0488-6f70-4417-bd71-a7e9000d0856-model-cache\") pod \"auth-test-model-79d51b-kserve-5f976787bc-xsdhk\" (UID: \"662e0488-6f70-4417-bd71-a7e9000d0856\") " pod="ai-tenant-e2e-shared-b-79d51b/auth-test-model-79d51b-kserve-5f976787bc-xsdhk" Jul 17 19:35:36.407743 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:36.407697 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/662e0488-6f70-4417-bd71-a7e9000d0856-tmp-dir\") pod \"auth-test-model-79d51b-kserve-5f976787bc-xsdhk\" (UID: \"662e0488-6f70-4417-bd71-a7e9000d0856\") " pod="ai-tenant-e2e-shared-b-79d51b/auth-test-model-79d51b-kserve-5f976787bc-xsdhk" Jul 17 19:35:36.407789 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:36.407745 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/662e0488-6f70-4417-bd71-a7e9000d0856-home\") pod \"auth-test-model-79d51b-kserve-5f976787bc-xsdhk\" (UID: \"662e0488-6f70-4417-bd71-a7e9000d0856\") " pod="ai-tenant-e2e-shared-b-79d51b/auth-test-model-79d51b-kserve-5f976787bc-xsdhk" Jul 17 19:35:36.407860 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:36.407843 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/662e0488-6f70-4417-bd71-a7e9000d0856-model-cache\") pod \"auth-test-model-79d51b-kserve-5f976787bc-xsdhk\" (UID: \"662e0488-6f70-4417-bd71-a7e9000d0856\") " pod="ai-tenant-e2e-shared-b-79d51b/auth-test-model-79d51b-kserve-5f976787bc-xsdhk" Jul 17 19:35:36.409679 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:36.409657 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/662e0488-6f70-4417-bd71-a7e9000d0856-dshm\") pod \"auth-test-model-79d51b-kserve-5f976787bc-xsdhk\" (UID: \"662e0488-6f70-4417-bd71-a7e9000d0856\") " pod="ai-tenant-e2e-shared-b-79d51b/auth-test-model-79d51b-kserve-5f976787bc-xsdhk" Jul 17 19:35:36.409957 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:36.409937 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/662e0488-6f70-4417-bd71-a7e9000d0856-tls-certs\") pod \"auth-test-model-79d51b-kserve-5f976787bc-xsdhk\" (UID: \"662e0488-6f70-4417-bd71-a7e9000d0856\") " pod="ai-tenant-e2e-shared-b-79d51b/auth-test-model-79d51b-kserve-5f976787bc-xsdhk" Jul 17 19:35:36.416394 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:36.416371 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cx6j\" (UniqueName: \"kubernetes.io/projected/662e0488-6f70-4417-bd71-a7e9000d0856-kube-api-access-6cx6j\") pod \"auth-test-model-79d51b-kserve-5f976787bc-xsdhk\" (UID: \"662e0488-6f70-4417-bd71-a7e9000d0856\") " pod="ai-tenant-e2e-shared-b-79d51b/auth-test-model-79d51b-kserve-5f976787bc-xsdhk" Jul 17 19:35:36.521842 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:36.521809 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-79d51b/auth-test-model-79d51b-kserve-5f976787bc-xsdhk" Jul 17 19:35:36.660939 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:36.660911 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-79d51b/auth-test-model-79d51b-kserve-5f976787bc-xsdhk"] Jul 17 19:35:36.662695 ip-10-0-134-137 kubenswrapper[2561]: W0717 19:35:36.662667 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod662e0488_6f70_4417_bd71_a7e9000d0856.slice/crio-0d9c7a557570a81d8418ecf6b18377c0ca5426280a0c062a952d39ab52cc476d WatchSource:0}: Error finding container 0d9c7a557570a81d8418ecf6b18377c0ca5426280a0c062a952d39ab52cc476d: Status 404 returned error can't find the container with id 0d9c7a557570a81d8418ecf6b18377c0ca5426280a0c062a952d39ab52cc476d Jul 17 19:35:37.047071 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:37.047034 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-79d51b/auth-test-model-79d51b-kserve-5f976787bc-xsdhk" event={"ID":"662e0488-6f70-4417-bd71-a7e9000d0856","Type":"ContainerStarted","Data":"e4e551e384f3d70d0832700a6679203f3703f3b1898211eee19a7434577de1d2"} Jul 17 19:35:37.047265 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:37.047082 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-79d51b/auth-test-model-79d51b-kserve-5f976787bc-xsdhk" event={"ID":"662e0488-6f70-4417-bd71-a7e9000d0856","Type":"ContainerStarted","Data":"0d9c7a557570a81d8418ecf6b18377c0ca5426280a0c062a952d39ab52cc476d"} Jul 17 19:35:37.065434 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:37.065380 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-79d51b/auth-test-model-79d51b-kserve-5f976787bc-xsdhk" podStartSLOduration=1.065363853 podStartE2EDuration="1.065363853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:35:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:35:37.063495886 +0000 UTC m=+2435.393312094" watchObservedRunningTime="2026-07-17 19:35:37.065363853 +0000 UTC m=+2435.395180060" Jul 17 19:35:39.979380 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:39.979337 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:35:46.522053 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:46.522006 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-79d51b/auth-test-model-79d51b-kserve-5f976787bc-xsdhk" Jul 17 19:35:46.522053 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:46.522061 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-79d51b/auth-test-model-79d51b-kserve-5f976787bc-xsdhk" Jul 17 19:35:46.534993 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:46.534966 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-79d51b/auth-test-model-79d51b-kserve-5f976787bc-xsdhk" Jul 17 19:35:47.109121 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:47.109014 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-79d51b/auth-test-model-79d51b-kserve-5f976787bc-xsdhk" Jul 17 19:35:57.177269 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:35:57.177227 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:36:16.678771 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:36:16.678733 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:36:32.179909 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:36:32.179868 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:36:50.976603 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:36:50.976562 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:37:07.876279 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:07.876192 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:37:25.773243 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:25.773211 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:37:42.124971 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:42.124933 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-79d51b/auth-test-model-79d51b-kserve-5f976787bc-xsdhk"] Jul 17 19:37:44.772994 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:44.772956 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:37:50.114017 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:50.113983 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-b3dd7f/sub-test-model-b3dd7f-kserve-77557c556c-slq6c"] Jul 17 19:37:50.117106 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:50.117089 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b3dd7f/sub-test-model-b3dd7f-kserve-77557c556c-slq6c" Jul 17 19:37:50.120324 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:50.120297 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-b3dd7f\"/\"kube-root-ca.crt\"" Jul 17 19:37:50.120469 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:50.120407 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-b3dd7f\"/\"openshift-service-ca.crt\"" Jul 17 19:37:50.120469 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:50.120410 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-b3dd7f\"/\"sub-test-model-b3dd7f-kserve-self-signed-certs\"" Jul 17 19:37:50.120469 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:50.120441 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-b3dd7f\"/\"default-dockercfg-gv6zz\"" Jul 17 19:37:50.126622 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:50.126597 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-b3dd7f/sub-test-model-b3dd7f-kserve-77557c556c-slq6c"] Jul 17 19:37:50.153823 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:50.153793 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6crr\" (UniqueName: \"kubernetes.io/projected/1250420d-ab82-47db-9d9a-5370d442f99e-kube-api-access-s6crr\") pod \"sub-test-model-b3dd7f-kserve-77557c556c-slq6c\" (UID: \"1250420d-ab82-47db-9d9a-5370d442f99e\") " pod="ai-tenant-e2e-shared-a-b3dd7f/sub-test-model-b3dd7f-kserve-77557c556c-slq6c" Jul 17 19:37:50.153823 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:50.153829 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1250420d-ab82-47db-9d9a-5370d442f99e-tmp-dir\") pod \"sub-test-model-b3dd7f-kserve-77557c556c-slq6c\" (UID: \"1250420d-ab82-47db-9d9a-5370d442f99e\") " pod="ai-tenant-e2e-shared-a-b3dd7f/sub-test-model-b3dd7f-kserve-77557c556c-slq6c" Jul 17 19:37:50.154104 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:50.153853 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1250420d-ab82-47db-9d9a-5370d442f99e-dshm\") pod \"sub-test-model-b3dd7f-kserve-77557c556c-slq6c\" (UID: \"1250420d-ab82-47db-9d9a-5370d442f99e\") " pod="ai-tenant-e2e-shared-a-b3dd7f/sub-test-model-b3dd7f-kserve-77557c556c-slq6c" Jul 17 19:37:50.154104 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:50.153915 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1250420d-ab82-47db-9d9a-5370d442f99e-home\") pod \"sub-test-model-b3dd7f-kserve-77557c556c-slq6c\" (UID: \"1250420d-ab82-47db-9d9a-5370d442f99e\") " pod="ai-tenant-e2e-shared-a-b3dd7f/sub-test-model-b3dd7f-kserve-77557c556c-slq6c" Jul 17 19:37:50.154104 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:50.153977 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1250420d-ab82-47db-9d9a-5370d442f99e-tls-certs\") pod \"sub-test-model-b3dd7f-kserve-77557c556c-slq6c\" (UID: \"1250420d-ab82-47db-9d9a-5370d442f99e\") " pod="ai-tenant-e2e-shared-a-b3dd7f/sub-test-model-b3dd7f-kserve-77557c556c-slq6c" Jul 17 19:37:50.154104 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:50.154019 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1250420d-ab82-47db-9d9a-5370d442f99e-model-cache\") pod \"sub-test-model-b3dd7f-kserve-77557c556c-slq6c\" (UID: \"1250420d-ab82-47db-9d9a-5370d442f99e\") " pod="ai-tenant-e2e-shared-a-b3dd7f/sub-test-model-b3dd7f-kserve-77557c556c-slq6c" Jul 17 19:37:50.255130 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:50.255093 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6crr\" (UniqueName: \"kubernetes.io/projected/1250420d-ab82-47db-9d9a-5370d442f99e-kube-api-access-s6crr\") pod \"sub-test-model-b3dd7f-kserve-77557c556c-slq6c\" (UID: \"1250420d-ab82-47db-9d9a-5370d442f99e\") " pod="ai-tenant-e2e-shared-a-b3dd7f/sub-test-model-b3dd7f-kserve-77557c556c-slq6c" Jul 17 19:37:50.255130 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:50.255131 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1250420d-ab82-47db-9d9a-5370d442f99e-tmp-dir\") pod \"sub-test-model-b3dd7f-kserve-77557c556c-slq6c\" (UID: \"1250420d-ab82-47db-9d9a-5370d442f99e\") " pod="ai-tenant-e2e-shared-a-b3dd7f/sub-test-model-b3dd7f-kserve-77557c556c-slq6c" Jul 17 19:37:50.255402 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:50.255158 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1250420d-ab82-47db-9d9a-5370d442f99e-dshm\") pod \"sub-test-model-b3dd7f-kserve-77557c556c-slq6c\" (UID: \"1250420d-ab82-47db-9d9a-5370d442f99e\") " pod="ai-tenant-e2e-shared-a-b3dd7f/sub-test-model-b3dd7f-kserve-77557c556c-slq6c" Jul 17 19:37:50.255402 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:50.255270 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1250420d-ab82-47db-9d9a-5370d442f99e-home\") pod \"sub-test-model-b3dd7f-kserve-77557c556c-slq6c\" (UID: \"1250420d-ab82-47db-9d9a-5370d442f99e\") " pod="ai-tenant-e2e-shared-a-b3dd7f/sub-test-model-b3dd7f-kserve-77557c556c-slq6c" Jul 17 19:37:50.255402 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:50.255329 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1250420d-ab82-47db-9d9a-5370d442f99e-tls-certs\") pod \"sub-test-model-b3dd7f-kserve-77557c556c-slq6c\" (UID: \"1250420d-ab82-47db-9d9a-5370d442f99e\") " pod="ai-tenant-e2e-shared-a-b3dd7f/sub-test-model-b3dd7f-kserve-77557c556c-slq6c" Jul 17 19:37:50.255402 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:50.255368 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1250420d-ab82-47db-9d9a-5370d442f99e-model-cache\") pod \"sub-test-model-b3dd7f-kserve-77557c556c-slq6c\" (UID: \"1250420d-ab82-47db-9d9a-5370d442f99e\") " pod="ai-tenant-e2e-shared-a-b3dd7f/sub-test-model-b3dd7f-kserve-77557c556c-slq6c" Jul 17 19:37:50.255639 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:50.255618 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1250420d-ab82-47db-9d9a-5370d442f99e-tmp-dir\") pod \"sub-test-model-b3dd7f-kserve-77557c556c-slq6c\" (UID: \"1250420d-ab82-47db-9d9a-5370d442f99e\") " pod="ai-tenant-e2e-shared-a-b3dd7f/sub-test-model-b3dd7f-kserve-77557c556c-slq6c" Jul 17 19:37:50.255703 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:50.255659 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1250420d-ab82-47db-9d9a-5370d442f99e-home\") pod \"sub-test-model-b3dd7f-kserve-77557c556c-slq6c\" (UID: \"1250420d-ab82-47db-9d9a-5370d442f99e\") " pod="ai-tenant-e2e-shared-a-b3dd7f/sub-test-model-b3dd7f-kserve-77557c556c-slq6c" Jul 17 19:37:50.255755 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:50.255742 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1250420d-ab82-47db-9d9a-5370d442f99e-model-cache\") pod \"sub-test-model-b3dd7f-kserve-77557c556c-slq6c\" (UID: \"1250420d-ab82-47db-9d9a-5370d442f99e\") " pod="ai-tenant-e2e-shared-a-b3dd7f/sub-test-model-b3dd7f-kserve-77557c556c-slq6c" Jul 17 19:37:50.257726 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:50.257706 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1250420d-ab82-47db-9d9a-5370d442f99e-dshm\") pod \"sub-test-model-b3dd7f-kserve-77557c556c-slq6c\" (UID: \"1250420d-ab82-47db-9d9a-5370d442f99e\") " pod="ai-tenant-e2e-shared-a-b3dd7f/sub-test-model-b3dd7f-kserve-77557c556c-slq6c" Jul 17 19:37:50.257912 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:50.257894 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1250420d-ab82-47db-9d9a-5370d442f99e-tls-certs\") pod \"sub-test-model-b3dd7f-kserve-77557c556c-slq6c\" (UID: \"1250420d-ab82-47db-9d9a-5370d442f99e\") " pod="ai-tenant-e2e-shared-a-b3dd7f/sub-test-model-b3dd7f-kserve-77557c556c-slq6c" Jul 17 19:37:50.265090 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:50.265062 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6crr\" (UniqueName: \"kubernetes.io/projected/1250420d-ab82-47db-9d9a-5370d442f99e-kube-api-access-s6crr\") pod \"sub-test-model-b3dd7f-kserve-77557c556c-slq6c\" (UID: \"1250420d-ab82-47db-9d9a-5370d442f99e\") " pod="ai-tenant-e2e-shared-a-b3dd7f/sub-test-model-b3dd7f-kserve-77557c556c-slq6c" Jul 17 19:37:50.426932 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:50.426838 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b3dd7f/sub-test-model-b3dd7f-kserve-77557c556c-slq6c" Jul 17 19:37:50.568568 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:50.567516 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-b3dd7f/sub-test-model-b3dd7f-kserve-77557c556c-slq6c"] Jul 17 19:37:50.745443 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:50.745403 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b3dd7f/sub-test-model-b3dd7f-kserve-77557c556c-slq6c" event={"ID":"1250420d-ab82-47db-9d9a-5370d442f99e","Type":"ContainerStarted","Data":"21d7f31e3408909f359b24e4ac96dd747f76bdd0b89618ffb4852e50cf20afa3"} Jul 17 19:37:50.745658 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:50.745456 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b3dd7f/sub-test-model-b3dd7f-kserve-77557c556c-slq6c" event={"ID":"1250420d-ab82-47db-9d9a-5370d442f99e","Type":"ContainerStarted","Data":"b89e18433000bec55d6ebdbea9d2e1b8ae629cdcfb03ab19620a3d568f40ca1f"} Jul 17 19:37:50.765053 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:50.764998 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-b3dd7f/sub-test-model-b3dd7f-kserve-77557c556c-slq6c" podStartSLOduration=0.76498113 podStartE2EDuration="764.98113ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:37:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:37:50.763892054 +0000 UTC m=+2569.093708262" watchObservedRunningTime="2026-07-17 19:37:50.76498113 +0000 UTC m=+2569.094797337" Jul 17 19:37:57.146859 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:57.146788 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-79d51b/auth-test-model-79d51b-kserve-5f976787bc-xsdhk" podUID="662e0488-6f70-4417-bd71-a7e9000d0856" containerName="main" containerID="cri-o://e4e551e384f3d70d0832700a6679203f3703f3b1898211eee19a7434577de1d2" gracePeriod=45 Jul 17 19:37:57.406509 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:57.406437 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-79d51b/auth-test-model-79d51b-kserve-5f976787bc-xsdhk" Jul 17 19:37:57.523604 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:57.523567 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/662e0488-6f70-4417-bd71-a7e9000d0856-dshm\") pod \"662e0488-6f70-4417-bd71-a7e9000d0856\" (UID: \"662e0488-6f70-4417-bd71-a7e9000d0856\") " Jul 17 19:37:57.523790 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:57.523637 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/662e0488-6f70-4417-bd71-a7e9000d0856-model-cache\") pod \"662e0488-6f70-4417-bd71-a7e9000d0856\" (UID: \"662e0488-6f70-4417-bd71-a7e9000d0856\") " Jul 17 19:37:57.523790 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:57.523695 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/662e0488-6f70-4417-bd71-a7e9000d0856-home\") pod \"662e0488-6f70-4417-bd71-a7e9000d0856\" (UID: \"662e0488-6f70-4417-bd71-a7e9000d0856\") " Jul 17 19:37:57.523790 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:57.523725 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6cx6j\" (UniqueName: \"kubernetes.io/projected/662e0488-6f70-4417-bd71-a7e9000d0856-kube-api-access-6cx6j\") pod \"662e0488-6f70-4417-bd71-a7e9000d0856\" (UID: \"662e0488-6f70-4417-bd71-a7e9000d0856\") " Jul 17 19:37:57.523790 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:57.523763 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/662e0488-6f70-4417-bd71-a7e9000d0856-tmp-dir\") pod \"662e0488-6f70-4417-bd71-a7e9000d0856\" (UID: \"662e0488-6f70-4417-bd71-a7e9000d0856\") " Jul 17 19:37:57.523790 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:57.523787 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/662e0488-6f70-4417-bd71-a7e9000d0856-tls-certs\") pod \"662e0488-6f70-4417-bd71-a7e9000d0856\" (UID: \"662e0488-6f70-4417-bd71-a7e9000d0856\") " Jul 17 19:37:57.524054 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:57.523883 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/662e0488-6f70-4417-bd71-a7e9000d0856-model-cache" (OuterVolumeSpecName: "model-cache") pod "662e0488-6f70-4417-bd71-a7e9000d0856" (UID: "662e0488-6f70-4417-bd71-a7e9000d0856"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 19:37:57.524054 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:57.524000 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/662e0488-6f70-4417-bd71-a7e9000d0856-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "662e0488-6f70-4417-bd71-a7e9000d0856" (UID: "662e0488-6f70-4417-bd71-a7e9000d0856"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 19:37:57.524054 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:57.524014 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/662e0488-6f70-4417-bd71-a7e9000d0856-home" (OuterVolumeSpecName: "home") pod "662e0488-6f70-4417-bd71-a7e9000d0856" (UID: "662e0488-6f70-4417-bd71-a7e9000d0856"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 19:37:57.524181 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:57.524138 2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/662e0488-6f70-4417-bd71-a7e9000d0856-model-cache\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:37:57.524181 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:57.524154 2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/662e0488-6f70-4417-bd71-a7e9000d0856-home\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:37:57.524181 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:57.524163 2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/662e0488-6f70-4417-bd71-a7e9000d0856-tmp-dir\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:37:57.526026 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:57.526000 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/662e0488-6f70-4417-bd71-a7e9000d0856-dshm" (OuterVolumeSpecName: "dshm") pod "662e0488-6f70-4417-bd71-a7e9000d0856" (UID: "662e0488-6f70-4417-bd71-a7e9000d0856"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 19:37:57.526026 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:57.526014 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/662e0488-6f70-4417-bd71-a7e9000d0856-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "662e0488-6f70-4417-bd71-a7e9000d0856" (UID: "662e0488-6f70-4417-bd71-a7e9000d0856"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 19:37:57.526171 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:57.526055 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/662e0488-6f70-4417-bd71-a7e9000d0856-kube-api-access-6cx6j" (OuterVolumeSpecName: "kube-api-access-6cx6j") pod "662e0488-6f70-4417-bd71-a7e9000d0856" (UID: "662e0488-6f70-4417-bd71-a7e9000d0856"). InnerVolumeSpecName "kube-api-access-6cx6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 19:37:57.625113 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:57.625068 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cx6j\" (UniqueName: \"kubernetes.io/projected/662e0488-6f70-4417-bd71-a7e9000d0856-kube-api-access-6cx6j\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:37:57.625113 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:57.625106 2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/662e0488-6f70-4417-bd71-a7e9000d0856-tls-certs\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:37:57.625113 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:57.625116 2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/662e0488-6f70-4417-bd71-a7e9000d0856-dshm\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:37:57.787936 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:57.787903 2561 generic.go:356] "Generic (PLEG): container finished" podID="662e0488-6f70-4417-bd71-a7e9000d0856" containerID="e4e551e384f3d70d0832700a6679203f3703f3b1898211eee19a7434577de1d2" exitCode=0 Jul 17 19:37:57.788108 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:57.787944 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-79d51b/auth-test-model-79d51b-kserve-5f976787bc-xsdhk" Jul 17 19:37:57.788108 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:57.787989 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-79d51b/auth-test-model-79d51b-kserve-5f976787bc-xsdhk" event={"ID":"662e0488-6f70-4417-bd71-a7e9000d0856","Type":"ContainerDied","Data":"e4e551e384f3d70d0832700a6679203f3703f3b1898211eee19a7434577de1d2"} Jul 17 19:37:57.788108 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:57.788033 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-79d51b/auth-test-model-79d51b-kserve-5f976787bc-xsdhk" event={"ID":"662e0488-6f70-4417-bd71-a7e9000d0856","Type":"ContainerDied","Data":"0d9c7a557570a81d8418ecf6b18377c0ca5426280a0c062a952d39ab52cc476d"} Jul 17 19:37:57.788108 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:57.788050 2561 scope.go:117] "RemoveContainer" containerID="e4e551e384f3d70d0832700a6679203f3703f3b1898211eee19a7434577de1d2" Jul 17 19:37:57.797848 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:57.797831 2561 scope.go:117] "RemoveContainer" containerID="e4e551e384f3d70d0832700a6679203f3703f3b1898211eee19a7434577de1d2" Jul 17 19:37:57.798132 ip-10-0-134-137 kubenswrapper[2561]: E0717 19:37:57.798114 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4e551e384f3d70d0832700a6679203f3703f3b1898211eee19a7434577de1d2\": container with ID starting with e4e551e384f3d70d0832700a6679203f3703f3b1898211eee19a7434577de1d2 not found: ID does not exist" containerID="e4e551e384f3d70d0832700a6679203f3703f3b1898211eee19a7434577de1d2" Jul 17 19:37:57.798180 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:57.798142 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4e551e384f3d70d0832700a6679203f3703f3b1898211eee19a7434577de1d2"} err="failed to get container status \"e4e551e384f3d70d0832700a6679203f3703f3b1898211eee19a7434577de1d2\": rpc error: code = NotFound desc = could not find container \"e4e551e384f3d70d0832700a6679203f3703f3b1898211eee19a7434577de1d2\": container with ID starting with e4e551e384f3d70d0832700a6679203f3703f3b1898211eee19a7434577de1d2 not found: ID does not exist" Jul 17 19:37:57.810036 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:57.810008 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-79d51b/auth-test-model-79d51b-kserve-5f976787bc-xsdhk"] Jul 17 19:37:57.812230 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:57.812209 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-79d51b/auth-test-model-79d51b-kserve-5f976787bc-xsdhk"] Jul 17 19:37:58.099989 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:37:58.099903 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="662e0488-6f70-4417-bd71-a7e9000d0856" path="/var/lib/kubelet/pods/662e0488-6f70-4417-bd71-a7e9000d0856/volumes" Jul 17 19:38:00.427603 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:38:00.427554 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-b3dd7f/sub-test-model-b3dd7f-kserve-77557c556c-slq6c" Jul 17 19:38:00.427603 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:38:00.427609 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-b3dd7f/sub-test-model-b3dd7f-kserve-77557c556c-slq6c" Jul 17 19:38:00.440019 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:38:00.439987 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-b3dd7f/sub-test-model-b3dd7f-kserve-77557c556c-slq6c" Jul 17 19:38:00.576090 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:38:00.576052 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:38:00.816503 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:38:00.816462 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-b3dd7f/sub-test-model-b3dd7f-kserve-77557c556c-slq6c" Jul 17 19:38:23.382337 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:38:23.382242 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:38:44.169450 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:38:44.169413 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:39:00.392670 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:00.392628 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:39:05.901013 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:05.900980 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-b3dd7f/sub-test-model-b3dd7f-kserve-77557c556c-slq6c"] Jul 17 19:39:19.984710 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:19.984670 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:39:20.923016 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:20.922979 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-b3dd7f/sub-test-model-b3dd7f-kserve-77557c556c-slq6c" podUID="1250420d-ab82-47db-9d9a-5370d442f99e" containerName="main" containerID="cri-o://21d7f31e3408909f359b24e4ac96dd747f76bdd0b89618ffb4852e50cf20afa3" gracePeriod=45 Jul 17 19:39:21.181238 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:21.181179 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b3dd7f/sub-test-model-b3dd7f-kserve-77557c556c-slq6c" Jul 17 19:39:21.233963 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:21.233926 2561 generic.go:356] "Generic (PLEG): container finished" podID="1250420d-ab82-47db-9d9a-5370d442f99e" containerID="21d7f31e3408909f359b24e4ac96dd747f76bdd0b89618ffb4852e50cf20afa3" exitCode=0 Jul 17 19:39:21.234142 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:21.233977 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b3dd7f/sub-test-model-b3dd7f-kserve-77557c556c-slq6c" Jul 17 19:39:21.234142 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:21.234028 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b3dd7f/sub-test-model-b3dd7f-kserve-77557c556c-slq6c" event={"ID":"1250420d-ab82-47db-9d9a-5370d442f99e","Type":"ContainerDied","Data":"21d7f31e3408909f359b24e4ac96dd747f76bdd0b89618ffb4852e50cf20afa3"} Jul 17 19:39:21.234142 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:21.234078 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b3dd7f/sub-test-model-b3dd7f-kserve-77557c556c-slq6c" event={"ID":"1250420d-ab82-47db-9d9a-5370d442f99e","Type":"ContainerDied","Data":"b89e18433000bec55d6ebdbea9d2e1b8ae629cdcfb03ab19620a3d568f40ca1f"} Jul 17 19:39:21.234142 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:21.234103 2561 scope.go:117] "RemoveContainer" containerID="21d7f31e3408909f359b24e4ac96dd747f76bdd0b89618ffb4852e50cf20afa3" Jul 17 19:39:21.236415 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:21.236384 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1250420d-ab82-47db-9d9a-5370d442f99e-tmp-dir\") pod \"1250420d-ab82-47db-9d9a-5370d442f99e\" (UID: \"1250420d-ab82-47db-9d9a-5370d442f99e\") " Jul 17 19:39:21.236565 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:21.236437 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1250420d-ab82-47db-9d9a-5370d442f99e-dshm\") pod \"1250420d-ab82-47db-9d9a-5370d442f99e\" (UID: \"1250420d-ab82-47db-9d9a-5370d442f99e\") " Jul 17 19:39:21.236565 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:21.236511 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6crr\" (UniqueName: \"kubernetes.io/projected/1250420d-ab82-47db-9d9a-5370d442f99e-kube-api-access-s6crr\") pod \"1250420d-ab82-47db-9d9a-5370d442f99e\" (UID: \"1250420d-ab82-47db-9d9a-5370d442f99e\") " Jul 17 19:39:21.236565 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:21.236561 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1250420d-ab82-47db-9d9a-5370d442f99e-home\") pod \"1250420d-ab82-47db-9d9a-5370d442f99e\" (UID: \"1250420d-ab82-47db-9d9a-5370d442f99e\") " Jul 17 19:39:21.236753 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:21.236605 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1250420d-ab82-47db-9d9a-5370d442f99e-tls-certs\") pod \"1250420d-ab82-47db-9d9a-5370d442f99e\" (UID: \"1250420d-ab82-47db-9d9a-5370d442f99e\") " Jul 17 19:39:21.236753 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:21.236641 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1250420d-ab82-47db-9d9a-5370d442f99e-model-cache\") pod \"1250420d-ab82-47db-9d9a-5370d442f99e\" (UID: \"1250420d-ab82-47db-9d9a-5370d442f99e\") " Jul 17 19:39:21.236753 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:21.236700 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1250420d-ab82-47db-9d9a-5370d442f99e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1250420d-ab82-47db-9d9a-5370d442f99e" (UID: "1250420d-ab82-47db-9d9a-5370d442f99e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 19:39:21.236922 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:21.236856 2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1250420d-ab82-47db-9d9a-5370d442f99e-tmp-dir\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:39:21.236922 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:21.236868 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1250420d-ab82-47db-9d9a-5370d442f99e-home" (OuterVolumeSpecName: "home") pod "1250420d-ab82-47db-9d9a-5370d442f99e" (UID: "1250420d-ab82-47db-9d9a-5370d442f99e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 19:39:21.236922 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:21.236903 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1250420d-ab82-47db-9d9a-5370d442f99e-model-cache" (OuterVolumeSpecName: "model-cache") pod "1250420d-ab82-47db-9d9a-5370d442f99e" (UID: "1250420d-ab82-47db-9d9a-5370d442f99e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 19:39:21.239094 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:21.239068 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1250420d-ab82-47db-9d9a-5370d442f99e-dshm" (OuterVolumeSpecName: "dshm") pod "1250420d-ab82-47db-9d9a-5370d442f99e" (UID: "1250420d-ab82-47db-9d9a-5370d442f99e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 19:39:21.239249 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:21.239094 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1250420d-ab82-47db-9d9a-5370d442f99e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1250420d-ab82-47db-9d9a-5370d442f99e" (UID: "1250420d-ab82-47db-9d9a-5370d442f99e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 19:39:21.239249 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:21.239121 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1250420d-ab82-47db-9d9a-5370d442f99e-kube-api-access-s6crr" (OuterVolumeSpecName: "kube-api-access-s6crr") pod "1250420d-ab82-47db-9d9a-5370d442f99e" (UID: "1250420d-ab82-47db-9d9a-5370d442f99e"). InnerVolumeSpecName "kube-api-access-s6crr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 19:39:21.243587 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:21.243563 2561 scope.go:117] "RemoveContainer" containerID="21d7f31e3408909f359b24e4ac96dd747f76bdd0b89618ffb4852e50cf20afa3" Jul 17 19:39:21.243918 ip-10-0-134-137 kubenswrapper[2561]: E0717 19:39:21.243895 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21d7f31e3408909f359b24e4ac96dd747f76bdd0b89618ffb4852e50cf20afa3\": container with ID starting with 21d7f31e3408909f359b24e4ac96dd747f76bdd0b89618ffb4852e50cf20afa3 not found: ID does not exist" containerID="21d7f31e3408909f359b24e4ac96dd747f76bdd0b89618ffb4852e50cf20afa3" Jul 17 19:39:21.243999 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:21.243925 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21d7f31e3408909f359b24e4ac96dd747f76bdd0b89618ffb4852e50cf20afa3"} err="failed to get container status \"21d7f31e3408909f359b24e4ac96dd747f76bdd0b89618ffb4852e50cf20afa3\": rpc error: code = NotFound desc = could not find container \"21d7f31e3408909f359b24e4ac96dd747f76bdd0b89618ffb4852e50cf20afa3\": container with ID starting with 21d7f31e3408909f359b24e4ac96dd747f76bdd0b89618ffb4852e50cf20afa3 not found: ID does not exist" Jul 17 19:39:21.337469 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:21.337429 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6crr\" (UniqueName: \"kubernetes.io/projected/1250420d-ab82-47db-9d9a-5370d442f99e-kube-api-access-s6crr\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:39:21.337469 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:21.337463 2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1250420d-ab82-47db-9d9a-5370d442f99e-home\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:39:21.337469 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:21.337472 2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1250420d-ab82-47db-9d9a-5370d442f99e-tls-certs\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:39:21.337746 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:21.337483 2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1250420d-ab82-47db-9d9a-5370d442f99e-model-cache\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:39:21.337746 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:21.337490 2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1250420d-ab82-47db-9d9a-5370d442f99e-dshm\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:39:21.559437 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:21.559400 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-b3dd7f/sub-test-model-b3dd7f-kserve-77557c556c-slq6c"] Jul 17 19:39:21.561926 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:21.561902 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-b3dd7f/sub-test-model-b3dd7f-kserve-77557c556c-slq6c"] Jul 17 19:39:22.098919 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:22.098874 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1250420d-ab82-47db-9d9a-5370d442f99e" path="/var/lib/kubelet/pods/1250420d-ab82-47db-9d9a-5370d442f99e/volumes" Jul 17 19:39:46.625122 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:46.625083 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-79d51b/rate-test-model-79d51b-kserve-86596d6f8c-gqnmh"] Jul 17 19:39:46.625681 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:46.625414 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="662e0488-6f70-4417-bd71-a7e9000d0856" containerName="main" Jul 17 19:39:46.625681 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:46.625426 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="662e0488-6f70-4417-bd71-a7e9000d0856" containerName="main" Jul 17 19:39:46.625681 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:46.625446 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1250420d-ab82-47db-9d9a-5370d442f99e" containerName="main" Jul 17 19:39:46.625681 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:46.625455 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1250420d-ab82-47db-9d9a-5370d442f99e" containerName="main" Jul 17 19:39:46.625681 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:46.625510 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="662e0488-6f70-4417-bd71-a7e9000d0856" containerName="main" Jul 17 19:39:46.625681 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:46.625519 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1250420d-ab82-47db-9d9a-5370d442f99e" containerName="main" Jul 17 19:39:46.628625 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:46.628609 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-79d51b/rate-test-model-79d51b-kserve-86596d6f8c-gqnmh" Jul 17 19:39:46.631884 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:46.631853 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-79d51b\"/\"openshift-service-ca.crt\"" Jul 17 19:39:46.632039 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:46.631899 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-79d51b\"/\"kube-root-ca.crt\"" Jul 17 19:39:46.632039 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:46.631898 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-79d51b\"/\"default-dockercfg-sx4s7\"" Jul 17 19:39:46.632039 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:46.631907 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-79d51b\"/\"rate-test-model-79d51b-kserve-self-signed-certs\"" Jul 17 19:39:46.638490 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:46.638464 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-79d51b/rate-test-model-79d51b-kserve-86596d6f8c-gqnmh"] Jul 17 19:39:46.732866 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:46.732828 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd48739b-5f0e-4627-9611-e01e652dd352-tls-certs\") pod \"rate-test-model-79d51b-kserve-86596d6f8c-gqnmh\" (UID: \"cd48739b-5f0e-4627-9611-e01e652dd352\") " pod="ai-tenant-e2e-shared-b-79d51b/rate-test-model-79d51b-kserve-86596d6f8c-gqnmh" Jul 17 19:39:46.733044 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:46.732878 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd48739b-5f0e-4627-9611-e01e652dd352-dshm\") pod \"rate-test-model-79d51b-kserve-86596d6f8c-gqnmh\" (UID: \"cd48739b-5f0e-4627-9611-e01e652dd352\") " pod="ai-tenant-e2e-shared-b-79d51b/rate-test-model-79d51b-kserve-86596d6f8c-gqnmh" Jul 17 19:39:46.733044 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:46.732905 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd6xt\" (UniqueName: \"kubernetes.io/projected/cd48739b-5f0e-4627-9611-e01e652dd352-kube-api-access-gd6xt\") pod \"rate-test-model-79d51b-kserve-86596d6f8c-gqnmh\" (UID: \"cd48739b-5f0e-4627-9611-e01e652dd352\") " pod="ai-tenant-e2e-shared-b-79d51b/rate-test-model-79d51b-kserve-86596d6f8c-gqnmh" Jul 17 19:39:46.733044 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:46.732921 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd48739b-5f0e-4627-9611-e01e652dd352-tmp-dir\") pod \"rate-test-model-79d51b-kserve-86596d6f8c-gqnmh\" (UID: \"cd48739b-5f0e-4627-9611-e01e652dd352\") " pod="ai-tenant-e2e-shared-b-79d51b/rate-test-model-79d51b-kserve-86596d6f8c-gqnmh" Jul 17 19:39:46.733044 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:46.732980 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd48739b-5f0e-4627-9611-e01e652dd352-model-cache\") pod \"rate-test-model-79d51b-kserve-86596d6f8c-gqnmh\" (UID: \"cd48739b-5f0e-4627-9611-e01e652dd352\") " pod="ai-tenant-e2e-shared-b-79d51b/rate-test-model-79d51b-kserve-86596d6f8c-gqnmh" Jul 17 19:39:46.733044 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:46.733022 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd48739b-5f0e-4627-9611-e01e652dd352-home\") pod \"rate-test-model-79d51b-kserve-86596d6f8c-gqnmh\" (UID: \"cd48739b-5f0e-4627-9611-e01e652dd352\") " pod="ai-tenant-e2e-shared-b-79d51b/rate-test-model-79d51b-kserve-86596d6f8c-gqnmh" Jul 17 19:39:46.834400 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:46.834365 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd48739b-5f0e-4627-9611-e01e652dd352-tls-certs\") pod \"rate-test-model-79d51b-kserve-86596d6f8c-gqnmh\" (UID: \"cd48739b-5f0e-4627-9611-e01e652dd352\") " pod="ai-tenant-e2e-shared-b-79d51b/rate-test-model-79d51b-kserve-86596d6f8c-gqnmh" Jul 17 19:39:46.834640 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:46.834413 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd48739b-5f0e-4627-9611-e01e652dd352-dshm\") pod \"rate-test-model-79d51b-kserve-86596d6f8c-gqnmh\" (UID: \"cd48739b-5f0e-4627-9611-e01e652dd352\") " pod="ai-tenant-e2e-shared-b-79d51b/rate-test-model-79d51b-kserve-86596d6f8c-gqnmh" Jul 17 19:39:46.834640 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:46.834441 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd6xt\" (UniqueName: \"kubernetes.io/projected/cd48739b-5f0e-4627-9611-e01e652dd352-kube-api-access-gd6xt\") pod \"rate-test-model-79d51b-kserve-86596d6f8c-gqnmh\" (UID: \"cd48739b-5f0e-4627-9611-e01e652dd352\") " pod="ai-tenant-e2e-shared-b-79d51b/rate-test-model-79d51b-kserve-86596d6f8c-gqnmh" Jul 17 19:39:46.834640 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:46.834464 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd48739b-5f0e-4627-9611-e01e652dd352-tmp-dir\") pod \"rate-test-model-79d51b-kserve-86596d6f8c-gqnmh\" (UID: \"cd48739b-5f0e-4627-9611-e01e652dd352\") " pod="ai-tenant-e2e-shared-b-79d51b/rate-test-model-79d51b-kserve-86596d6f8c-gqnmh" Jul 17 19:39:46.834640 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:46.834492 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd48739b-5f0e-4627-9611-e01e652dd352-model-cache\") pod \"rate-test-model-79d51b-kserve-86596d6f8c-gqnmh\" (UID: \"cd48739b-5f0e-4627-9611-e01e652dd352\") " pod="ai-tenant-e2e-shared-b-79d51b/rate-test-model-79d51b-kserve-86596d6f8c-gqnmh" Jul 17 19:39:46.834640 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:46.834511 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd48739b-5f0e-4627-9611-e01e652dd352-home\") pod \"rate-test-model-79d51b-kserve-86596d6f8c-gqnmh\" (UID: \"cd48739b-5f0e-4627-9611-e01e652dd352\") " pod="ai-tenant-e2e-shared-b-79d51b/rate-test-model-79d51b-kserve-86596d6f8c-gqnmh" Jul 17 19:39:46.834966 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:46.834927 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd48739b-5f0e-4627-9611-e01e652dd352-tmp-dir\") pod \"rate-test-model-79d51b-kserve-86596d6f8c-gqnmh\" (UID: \"cd48739b-5f0e-4627-9611-e01e652dd352\") " pod="ai-tenant-e2e-shared-b-79d51b/rate-test-model-79d51b-kserve-86596d6f8c-gqnmh" Jul 17 19:39:46.835022 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:46.834963 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd48739b-5f0e-4627-9611-e01e652dd352-home\") pod \"rate-test-model-79d51b-kserve-86596d6f8c-gqnmh\" (UID: \"cd48739b-5f0e-4627-9611-e01e652dd352\") " pod="ai-tenant-e2e-shared-b-79d51b/rate-test-model-79d51b-kserve-86596d6f8c-gqnmh" Jul 17 19:39:46.835022 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:46.835008 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd48739b-5f0e-4627-9611-e01e652dd352-model-cache\") pod \"rate-test-model-79d51b-kserve-86596d6f8c-gqnmh\" (UID: \"cd48739b-5f0e-4627-9611-e01e652dd352\") " pod="ai-tenant-e2e-shared-b-79d51b/rate-test-model-79d51b-kserve-86596d6f8c-gqnmh" Jul 17 19:39:46.837313 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:46.837290 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd48739b-5f0e-4627-9611-e01e652dd352-dshm\") pod \"rate-test-model-79d51b-kserve-86596d6f8c-gqnmh\" (UID: \"cd48739b-5f0e-4627-9611-e01e652dd352\") " pod="ai-tenant-e2e-shared-b-79d51b/rate-test-model-79d51b-kserve-86596d6f8c-gqnmh" Jul 17 19:39:46.837577 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:46.837553 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd48739b-5f0e-4627-9611-e01e652dd352-tls-certs\") pod \"rate-test-model-79d51b-kserve-86596d6f8c-gqnmh\" (UID: \"cd48739b-5f0e-4627-9611-e01e652dd352\") " pod="ai-tenant-e2e-shared-b-79d51b/rate-test-model-79d51b-kserve-86596d6f8c-gqnmh" Jul 17 19:39:46.843647 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:46.843620 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd6xt\" (UniqueName: \"kubernetes.io/projected/cd48739b-5f0e-4627-9611-e01e652dd352-kube-api-access-gd6xt\") pod \"rate-test-model-79d51b-kserve-86596d6f8c-gqnmh\" (UID: \"cd48739b-5f0e-4627-9611-e01e652dd352\") " pod="ai-tenant-e2e-shared-b-79d51b/rate-test-model-79d51b-kserve-86596d6f8c-gqnmh" Jul 17 19:39:46.938784 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:46.938685 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-79d51b/rate-test-model-79d51b-kserve-86596d6f8c-gqnmh" Jul 17 19:39:47.074058 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:47.074032 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-79d51b/rate-test-model-79d51b-kserve-86596d6f8c-gqnmh"] Jul 17 19:39:47.371582 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:47.371547 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-79d51b/rate-test-model-79d51b-kserve-86596d6f8c-gqnmh" event={"ID":"cd48739b-5f0e-4627-9611-e01e652dd352","Type":"ContainerStarted","Data":"e54e265b57e895a97b4001ec31765335e1d13862599d3be3f1528b05fab25591"} Jul 17 19:39:47.371582 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:47.371591 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-79d51b/rate-test-model-79d51b-kserve-86596d6f8c-gqnmh" event={"ID":"cd48739b-5f0e-4627-9611-e01e652dd352","Type":"ContainerStarted","Data":"777ee5cfe9b0606fd3fb55c44041f7933894c77533f7a60e9ee98613e9b96c4a"} Jul 17 19:39:47.391838 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:47.391775 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-79d51b/rate-test-model-79d51b-kserve-86596d6f8c-gqnmh" podStartSLOduration=1.39175659 podStartE2EDuration="1.39175659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:39:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:39:47.390590274 +0000 UTC m=+2685.720406482" watchObservedRunningTime="2026-07-17 19:39:47.39175659 +0000 UTC m=+2685.721572799" Jul 17 19:39:51.187475 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:51.187437 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:39:56.939456 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:56.939366 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-79d51b/rate-test-model-79d51b-kserve-86596d6f8c-gqnmh" Jul 17 19:39:56.939456 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:56.939418 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-79d51b/rate-test-model-79d51b-kserve-86596d6f8c-gqnmh" Jul 17 19:39:56.952137 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:56.952111 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-79d51b/rate-test-model-79d51b-kserve-86596d6f8c-gqnmh" Jul 17 19:39:57.437049 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:39:57.437020 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-79d51b/rate-test-model-79d51b-kserve-86596d6f8c-gqnmh" Jul 17 19:40:02.255538 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:02.255493 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpjzq_4b162535-066a-471a-aee4-99c07c6e1bb4/ovn-acl-logging/0.log" Jul 17 19:40:02.261241 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:02.261219 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpjzq_4b162535-066a-471a-aee4-99c07c6e1bb4/ovn-acl-logging/0.log" Jul 17 19:40:06.680398 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:06.680360 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:40:22.679666 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:22.679633 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:40:30.895267 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:30.895231 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-79d51b/rate-test-model-79d51b-kserve-86596d6f8c-gqnmh"] Jul 17 19:40:37.582238 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:37.582203 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:40:45.915349 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:45.915298 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-79d51b/rate-test-model-79d51b-kserve-86596d6f8c-gqnmh" podUID="cd48739b-5f0e-4627-9611-e01e652dd352" containerName="main" containerID="cri-o://e54e265b57e895a97b4001ec31765335e1d13862599d3be3f1528b05fab25591" gracePeriod=45 Jul 17 19:40:46.152786 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:46.152757 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-79d51b/rate-test-model-79d51b-kserve-86596d6f8c-gqnmh" Jul 17 19:40:46.225314 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:46.225226 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd48739b-5f0e-4627-9611-e01e652dd352-tls-certs\") pod \"cd48739b-5f0e-4627-9611-e01e652dd352\" (UID: \"cd48739b-5f0e-4627-9611-e01e652dd352\") " Jul 17 19:40:46.225314 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:46.225266 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gd6xt\" (UniqueName: \"kubernetes.io/projected/cd48739b-5f0e-4627-9611-e01e652dd352-kube-api-access-gd6xt\") pod \"cd48739b-5f0e-4627-9611-e01e652dd352\" (UID: \"cd48739b-5f0e-4627-9611-e01e652dd352\") " Jul 17 19:40:46.225314 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:46.225300 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd48739b-5f0e-4627-9611-e01e652dd352-tmp-dir\") pod \"cd48739b-5f0e-4627-9611-e01e652dd352\" (UID: \"cd48739b-5f0e-4627-9611-e01e652dd352\") " Jul 17 19:40:46.225633 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:46.225337 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd48739b-5f0e-4627-9611-e01e652dd352-dshm\") pod \"cd48739b-5f0e-4627-9611-e01e652dd352\" (UID: \"cd48739b-5f0e-4627-9611-e01e652dd352\") " Jul 17 19:40:46.225633 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:46.225366 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd48739b-5f0e-4627-9611-e01e652dd352-model-cache\") pod \"cd48739b-5f0e-4627-9611-e01e652dd352\" (UID: \"cd48739b-5f0e-4627-9611-e01e652dd352\") " Jul 17 19:40:46.225633 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:46.225418 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd48739b-5f0e-4627-9611-e01e652dd352-home\") pod \"cd48739b-5f0e-4627-9611-e01e652dd352\" (UID: \"cd48739b-5f0e-4627-9611-e01e652dd352\") " Jul 17 19:40:46.225780 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:46.225692 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd48739b-5f0e-4627-9611-e01e652dd352-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cd48739b-5f0e-4627-9611-e01e652dd352" (UID: "cd48739b-5f0e-4627-9611-e01e652dd352"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 19:40:46.225780 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:46.225709 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd48739b-5f0e-4627-9611-e01e652dd352-model-cache" (OuterVolumeSpecName: "model-cache") pod "cd48739b-5f0e-4627-9611-e01e652dd352" (UID: "cd48739b-5f0e-4627-9611-e01e652dd352"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 19:40:46.225780 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:46.225757 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd48739b-5f0e-4627-9611-e01e652dd352-home" (OuterVolumeSpecName: "home") pod "cd48739b-5f0e-4627-9611-e01e652dd352" (UID: "cd48739b-5f0e-4627-9611-e01e652dd352"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 19:40:46.227710 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:46.227679 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd48739b-5f0e-4627-9611-e01e652dd352-dshm" (OuterVolumeSpecName: "dshm") pod "cd48739b-5f0e-4627-9611-e01e652dd352" (UID: "cd48739b-5f0e-4627-9611-e01e652dd352"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 19:40:46.227820 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:46.227743 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd48739b-5f0e-4627-9611-e01e652dd352-kube-api-access-gd6xt" (OuterVolumeSpecName: "kube-api-access-gd6xt") pod "cd48739b-5f0e-4627-9611-e01e652dd352" (UID: "cd48739b-5f0e-4627-9611-e01e652dd352"). InnerVolumeSpecName "kube-api-access-gd6xt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 19:40:46.227820 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:46.227810 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd48739b-5f0e-4627-9611-e01e652dd352-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cd48739b-5f0e-4627-9611-e01e652dd352" (UID: "cd48739b-5f0e-4627-9611-e01e652dd352"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 19:40:46.326942 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:46.326901 2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd48739b-5f0e-4627-9611-e01e652dd352-tls-certs\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:40:46.326942 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:46.326933 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd6xt\" (UniqueName: \"kubernetes.io/projected/cd48739b-5f0e-4627-9611-e01e652dd352-kube-api-access-gd6xt\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:40:46.326942 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:46.326941 2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd48739b-5f0e-4627-9611-e01e652dd352-tmp-dir\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:40:46.326942 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:46.326951 2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd48739b-5f0e-4627-9611-e01e652dd352-dshm\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:40:46.327201 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:46.326958 2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd48739b-5f0e-4627-9611-e01e652dd352-model-cache\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:40:46.327201 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:46.326966 2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd48739b-5f0e-4627-9611-e01e652dd352-home\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:40:46.689840 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:46.689808 2561 generic.go:356] "Generic (PLEG): container finished" podID="cd48739b-5f0e-4627-9611-e01e652dd352" containerID="e54e265b57e895a97b4001ec31765335e1d13862599d3be3f1528b05fab25591" exitCode=0 Jul 17 19:40:46.690023 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:46.689846 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-79d51b/rate-test-model-79d51b-kserve-86596d6f8c-gqnmh" event={"ID":"cd48739b-5f0e-4627-9611-e01e652dd352","Type":"ContainerDied","Data":"e54e265b57e895a97b4001ec31765335e1d13862599d3be3f1528b05fab25591"} Jul 17 19:40:46.690023 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:46.689861 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-79d51b/rate-test-model-79d51b-kserve-86596d6f8c-gqnmh" Jul 17 19:40:46.690023 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:46.689897 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-79d51b/rate-test-model-79d51b-kserve-86596d6f8c-gqnmh" event={"ID":"cd48739b-5f0e-4627-9611-e01e652dd352","Type":"ContainerDied","Data":"777ee5cfe9b0606fd3fb55c44041f7933894c77533f7a60e9ee98613e9b96c4a"} Jul 17 19:40:46.690023 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:46.689912 2561 scope.go:117] "RemoveContainer" containerID="e54e265b57e895a97b4001ec31765335e1d13862599d3be3f1528b05fab25591" Jul 17 19:40:46.699337 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:46.699320 2561 scope.go:117] "RemoveContainer" containerID="e54e265b57e895a97b4001ec31765335e1d13862599d3be3f1528b05fab25591" Jul 17 19:40:46.699654 ip-10-0-134-137 kubenswrapper[2561]: E0717 19:40:46.699633 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e54e265b57e895a97b4001ec31765335e1d13862599d3be3f1528b05fab25591\": container with ID starting with e54e265b57e895a97b4001ec31765335e1d13862599d3be3f1528b05fab25591 not found: ID does not exist" containerID="e54e265b57e895a97b4001ec31765335e1d13862599d3be3f1528b05fab25591" Jul 17 19:40:46.699733 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:46.699666 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e54e265b57e895a97b4001ec31765335e1d13862599d3be3f1528b05fab25591"} err="failed to get container status \"e54e265b57e895a97b4001ec31765335e1d13862599d3be3f1528b05fab25591\": rpc error: code = NotFound desc = could not find container \"e54e265b57e895a97b4001ec31765335e1d13862599d3be3f1528b05fab25591\": container with ID starting with e54e265b57e895a97b4001ec31765335e1d13862599d3be3f1528b05fab25591 not found: ID does not exist" Jul 17 19:40:46.711577 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:46.711553 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-79d51b/rate-test-model-79d51b-kserve-86596d6f8c-gqnmh"] Jul 17 19:40:46.716959 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:46.716937 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-79d51b/rate-test-model-79d51b-kserve-86596d6f8c-gqnmh"] Jul 17 19:40:47.066471 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:47.066432 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7507-139919876461600"] Jul 17 19:40:47.066888 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:47.066749 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd48739b-5f0e-4627-9611-e01e652dd352" containerName="main" Jul 17 19:40:47.066888 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:47.066760 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd48739b-5f0e-4627-9611-e01e652dd352" containerName="main" Jul 17 19:40:47.066888 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:47.066836 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd48739b-5f0e-4627-9611-e01e652dd352" containerName="main" Jul 17 19:40:47.071062 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:47.071044 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7507-139919876461600" Jul 17 19:40:47.073719 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:47.073693 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-6kh8w\"" Jul 17 19:40:47.075957 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:47.075927 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7507-139919876461600"] Jul 17 19:40:47.134416 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:47.134304 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plj8l\" (UniqueName: \"kubernetes.io/projected/18e6bd4a-3ba1-4857-af36-1b6da3a2a716-kube-api-access-plj8l\") pod \"test-curl-7507-139919876461600\" (UID: \"18e6bd4a-3ba1-4857-af36-1b6da3a2a716\") " pod="opendatahub/test-curl-7507-139919876461600" Jul 17 19:40:47.235724 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:47.235679 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plj8l\" (UniqueName: \"kubernetes.io/projected/18e6bd4a-3ba1-4857-af36-1b6da3a2a716-kube-api-access-plj8l\") pod \"test-curl-7507-139919876461600\" (UID: \"18e6bd4a-3ba1-4857-af36-1b6da3a2a716\") " pod="opendatahub/test-curl-7507-139919876461600" Jul 17 19:40:47.244549 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:47.244495 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plj8l\" (UniqueName: \"kubernetes.io/projected/18e6bd4a-3ba1-4857-af36-1b6da3a2a716-kube-api-access-plj8l\") pod \"test-curl-7507-139919876461600\" (UID: \"18e6bd4a-3ba1-4857-af36-1b6da3a2a716\") " pod="opendatahub/test-curl-7507-139919876461600" Jul 17 19:40:47.381740 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:47.381647 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7507-139919876461600" Jul 17 19:40:47.518753 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:47.518718 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7507-139919876461600"] Jul 17 19:40:47.520089 ip-10-0-134-137 kubenswrapper[2561]: W0717 19:40:47.520060 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18e6bd4a_3ba1_4857_af36_1b6da3a2a716.slice/crio-5da2006868f749c647be422f839c1b2cb8d71349229b6b9557351249984095d5 WatchSource:0}: Error finding container 5da2006868f749c647be422f839c1b2cb8d71349229b6b9557351249984095d5: Status 404 returned error can't find the container with id 5da2006868f749c647be422f839c1b2cb8d71349229b6b9557351249984095d5 Jul 17 19:40:47.521396 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:47.521373 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 19:40:47.696634 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:47.696501 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7507-139919876461600" event={"ID":"18e6bd4a-3ba1-4857-af36-1b6da3a2a716","Type":"ContainerStarted","Data":"5da2006868f749c647be422f839c1b2cb8d71349229b6b9557351249984095d5"} Jul 17 19:40:48.101000 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:48.100953 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd48739b-5f0e-4627-9611-e01e652dd352" path="/var/lib/kubelet/pods/cd48739b-5f0e-4627-9611-e01e652dd352/volumes" Jul 17 19:40:48.704006 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:48.703972 2561 generic.go:356] "Generic (PLEG): container finished" podID="18e6bd4a-3ba1-4857-af36-1b6da3a2a716" containerID="9c5629dca03ce36e797572c9c8d72fb297ea7d69b22e1d96b75173934a39d8f2" exitCode=0 Jul 17 19:40:48.704194 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:48.704082 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7507-139919876461600" event={"ID":"18e6bd4a-3ba1-4857-af36-1b6da3a2a716","Type":"ContainerDied","Data":"9c5629dca03ce36e797572c9c8d72fb297ea7d69b22e1d96b75173934a39d8f2"} Jul 17 19:40:49.840595 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:49.840571 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7507-139919876461600" Jul 17 19:40:49.858085 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:49.858055 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-plj8l\" (UniqueName: \"kubernetes.io/projected/18e6bd4a-3ba1-4857-af36-1b6da3a2a716-kube-api-access-plj8l\") pod \"18e6bd4a-3ba1-4857-af36-1b6da3a2a716\" (UID: \"18e6bd4a-3ba1-4857-af36-1b6da3a2a716\") " Jul 17 19:40:49.860412 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:49.860382 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18e6bd4a-3ba1-4857-af36-1b6da3a2a716-kube-api-access-plj8l" (OuterVolumeSpecName: "kube-api-access-plj8l") pod "18e6bd4a-3ba1-4857-af36-1b6da3a2a716" (UID: "18e6bd4a-3ba1-4857-af36-1b6da3a2a716"). InnerVolumeSpecName "kube-api-access-plj8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 19:40:49.877885 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:49.877853 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7507-139919876461600_18e6bd4a-3ba1-4857-af36-1b6da3a2a716/test-curl-7507-139919876461600/0.log" Jul 17 19:40:49.925790 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:49.925761 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7507-139919876461600"] Jul 17 19:40:49.929866 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:49.929838 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7507-139919876461600"] Jul 17 19:40:49.959573 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:49.959489 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plj8l\" (UniqueName: \"kubernetes.io/projected/18e6bd4a-3ba1-4857-af36-1b6da3a2a716-kube-api-access-plj8l\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:40:50.098906 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:50.098873 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18e6bd4a-3ba1-4857-af36-1b6da3a2a716" path="/var/lib/kubelet/pods/18e6bd4a-3ba1-4857-af36-1b6da3a2a716/volumes" Jul 17 19:40:50.112005 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:50.111980 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b3dd7f-72wj9"] Jul 17 19:40:50.112285 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:50.112273 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18e6bd4a-3ba1-4857-af36-1b6da3a2a716" containerName="test-curl-7507-139919876461600" Jul 17 19:40:50.112324 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:50.112287 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="18e6bd4a-3ba1-4857-af36-1b6da3a2a716" containerName="test-curl-7507-139919876461600" Jul 17 19:40:50.112362 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:50.112333 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="18e6bd4a-3ba1-4857-af36-1b6da3a2a716" containerName="test-curl-7507-139919876461600" Jul 17 19:40:50.115348 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:50.115330 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b3dd7f-72wj9" Jul 17 19:40:50.117424 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:50.117403 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lbvd4\"" Jul 17 19:40:50.122175 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:50.122153 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b3dd7f-72wj9"] Jul 17 19:40:50.162001 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:50.161971 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f07b5617-75e8-42d6-bd14-3a010124bc5b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-b3dd7f-72wj9\" (UID: \"f07b5617-75e8-42d6-bd14-3a010124bc5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b3dd7f-72wj9" Jul 17 19:40:50.263375 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:50.263335 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f07b5617-75e8-42d6-bd14-3a010124bc5b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-b3dd7f-72wj9\" (UID: \"f07b5617-75e8-42d6-bd14-3a010124bc5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b3dd7f-72wj9" Jul 17 19:40:50.264082 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:50.264061 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f07b5617-75e8-42d6-bd14-3a010124bc5b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-b3dd7f-72wj9\" (UID: \"f07b5617-75e8-42d6-bd14-3a010124bc5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b3dd7f-72wj9" Jul 17 19:40:50.427142 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:50.427108 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b3dd7f-72wj9" Jul 17 19:40:50.555512 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:50.555483 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b3dd7f-72wj9"] Jul 17 19:40:50.557184 ip-10-0-134-137 kubenswrapper[2561]: W0717 19:40:50.557161 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf07b5617_75e8_42d6_bd14_3a010124bc5b.slice/crio-2c421a9cb8ad8854a05c721d101caadbf62750f2c75405f0080b55355b27da19 WatchSource:0}: Error finding container 2c421a9cb8ad8854a05c721d101caadbf62750f2c75405f0080b55355b27da19: Status 404 returned error can't find the container with id 2c421a9cb8ad8854a05c721d101caadbf62750f2c75405f0080b55355b27da19 Jul 17 19:40:50.715867 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:50.715827 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b3dd7f-72wj9" event={"ID":"f07b5617-75e8-42d6-bd14-3a010124bc5b","Type":"ContainerStarted","Data":"2c421a9cb8ad8854a05c721d101caadbf62750f2c75405f0080b55355b27da19"} Jul 17 19:40:50.717120 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:50.717100 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7507-139919876461600" Jul 17 19:40:50.717230 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:50.717102 2561 scope.go:117] "RemoveContainer" containerID="9c5629dca03ce36e797572c9c8d72fb297ea7d69b22e1d96b75173934a39d8f2" Jul 17 19:40:51.724610 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:51.724573 2561 generic.go:356] "Generic (PLEG): container finished" podID="f07b5617-75e8-42d6-bd14-3a010124bc5b" containerID="fc97365d4abc5bf583ecf9c8ba10f5defcbfa17c3c62201208387e7194e506e0" exitCode=0 Jul 17 19:40:51.724991 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:51.724663 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b3dd7f-72wj9" event={"ID":"f07b5617-75e8-42d6-bd14-3a010124bc5b","Type":"ContainerDied","Data":"fc97365d4abc5bf583ecf9c8ba10f5defcbfa17c3c62201208387e7194e506e0"} Jul 17 19:40:52.863864 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:52.863839 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b3dd7f-72wj9" Jul 17 19:40:52.983623 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:52.983541 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f07b5617-75e8-42d6-bd14-3a010124bc5b-maas-api-service-ca\") pod \"f07b5617-75e8-42d6-bd14-3a010124bc5b\" (UID: \"f07b5617-75e8-42d6-bd14-3a010124bc5b\") " Jul 17 19:40:52.983862 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:52.983838 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f07b5617-75e8-42d6-bd14-3a010124bc5b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f07b5617-75e8-42d6-bd14-3a010124bc5b" (UID: "f07b5617-75e8-42d6-bd14-3a010124bc5b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 19:40:53.084345 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:53.084310 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f07b5617-75e8-42d6-bd14-3a010124bc5b-maas-api-service-ca\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:40:53.736774 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:53.736739 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b3dd7f-72wj9" event={"ID":"f07b5617-75e8-42d6-bd14-3a010124bc5b","Type":"ContainerDied","Data":"2c421a9cb8ad8854a05c721d101caadbf62750f2c75405f0080b55355b27da19"} Jul 17 19:40:53.736774 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:53.736759 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b3dd7f-72wj9" Jul 17 19:40:53.737086 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:40:53.736794 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c421a9cb8ad8854a05c721d101caadbf62750f2c75405f0080b55355b27da19" Jul 17 19:41:00.578766 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:00.578730 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-b3dd7f-95f8b99c7-bdb5k"] Jul 17 19:41:00.773388 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:00.773323 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b3dd7f-95f8b99c7-bdb5k" podUID="6c8d502e-7547-44f9-b69e-5fbc6f21acf9" containerName="maas-api" containerID="cri-o://3812109a92c1663c505680ae1102c029ada4222b9af5e736f912ea46f84b4a31" gracePeriod=30 Jul 17 19:41:00.824976 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:00.824940 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b3dd7f-95f8b99c7-bdb5k" podUID="6c8d502e-7547-44f9-b69e-5fbc6f21acf9" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.62:8443/health\": dial tcp 10.133.0.62:8443: connect: connection refused" Jul 17 19:41:01.014896 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:01.014870 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b3dd7f-95f8b99c7-bdb5k" Jul 17 19:41:01.149945 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:01.149844 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c8d502e-7547-44f9-b69e-5fbc6f21acf9-maas-api-tls\") pod \"6c8d502e-7547-44f9-b69e-5fbc6f21acf9\" (UID: \"6c8d502e-7547-44f9-b69e-5fbc6f21acf9\") " Jul 17 19:41:01.149945 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:01.149897 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-542r6\" (UniqueName: \"kubernetes.io/projected/6c8d502e-7547-44f9-b69e-5fbc6f21acf9-kube-api-access-542r6\") pod \"6c8d502e-7547-44f9-b69e-5fbc6f21acf9\" (UID: \"6c8d502e-7547-44f9-b69e-5fbc6f21acf9\") " Jul 17 19:41:01.150160 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:01.150025 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c8d502e-7547-44f9-b69e-5fbc6f21acf9-openshift-service-ca\") pod \"6c8d502e-7547-44f9-b69e-5fbc6f21acf9\" (UID: \"6c8d502e-7547-44f9-b69e-5fbc6f21acf9\") " Jul 17 19:41:01.150383 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:01.150356 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c8d502e-7547-44f9-b69e-5fbc6f21acf9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6c8d502e-7547-44f9-b69e-5fbc6f21acf9" (UID: "6c8d502e-7547-44f9-b69e-5fbc6f21acf9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 19:41:01.152081 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:01.152060 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c8d502e-7547-44f9-b69e-5fbc6f21acf9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6c8d502e-7547-44f9-b69e-5fbc6f21acf9" (UID: "6c8d502e-7547-44f9-b69e-5fbc6f21acf9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 19:41:01.152168 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:01.152148 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c8d502e-7547-44f9-b69e-5fbc6f21acf9-kube-api-access-542r6" (OuterVolumeSpecName: "kube-api-access-542r6") pod "6c8d502e-7547-44f9-b69e-5fbc6f21acf9" (UID: "6c8d502e-7547-44f9-b69e-5fbc6f21acf9"). InnerVolumeSpecName "kube-api-access-542r6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 19:41:01.250740 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:01.250692 2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c8d502e-7547-44f9-b69e-5fbc6f21acf9-openshift-service-ca\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:41:01.250740 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:01.250734 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c8d502e-7547-44f9-b69e-5fbc6f21acf9-maas-api-tls\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:41:01.250740 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:01.250744 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-542r6\" (UniqueName: \"kubernetes.io/projected/6c8d502e-7547-44f9-b69e-5fbc6f21acf9-kube-api-access-542r6\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:41:01.779896 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:01.779858 2561 generic.go:356] "Generic (PLEG): container finished" podID="6c8d502e-7547-44f9-b69e-5fbc6f21acf9" containerID="3812109a92c1663c505680ae1102c029ada4222b9af5e736f912ea46f84b4a31" exitCode=0 Jul 17 19:41:01.780296 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:01.779907 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b3dd7f-95f8b99c7-bdb5k" Jul 17 19:41:01.780296 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:01.779925 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b3dd7f-95f8b99c7-bdb5k" event={"ID":"6c8d502e-7547-44f9-b69e-5fbc6f21acf9","Type":"ContainerDied","Data":"3812109a92c1663c505680ae1102c029ada4222b9af5e736f912ea46f84b4a31"} Jul 17 19:41:01.780296 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:01.779977 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b3dd7f-95f8b99c7-bdb5k" event={"ID":"6c8d502e-7547-44f9-b69e-5fbc6f21acf9","Type":"ContainerDied","Data":"de2513595724340e5355a13edcbdeac7e1101fc435dc4d3dead01b3b2ccd4fbd"} Jul 17 19:41:01.780296 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:01.780007 2561 scope.go:117] "RemoveContainer" containerID="3812109a92c1663c505680ae1102c029ada4222b9af5e736f912ea46f84b4a31" Jul 17 19:41:01.789602 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:01.789514 2561 scope.go:117] "RemoveContainer" containerID="3812109a92c1663c505680ae1102c029ada4222b9af5e736f912ea46f84b4a31" Jul 17 19:41:01.789878 ip-10-0-134-137 kubenswrapper[2561]: E0717 19:41:01.789856 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3812109a92c1663c505680ae1102c029ada4222b9af5e736f912ea46f84b4a31\": container with ID starting with 3812109a92c1663c505680ae1102c029ada4222b9af5e736f912ea46f84b4a31 not found: ID does not exist" containerID="3812109a92c1663c505680ae1102c029ada4222b9af5e736f912ea46f84b4a31" Jul 17 19:41:01.789931 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:01.789888 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3812109a92c1663c505680ae1102c029ada4222b9af5e736f912ea46f84b4a31"} err="failed to get container status \"3812109a92c1663c505680ae1102c029ada4222b9af5e736f912ea46f84b4a31\": rpc error: code = NotFound desc = could not find container \"3812109a92c1663c505680ae1102c029ada4222b9af5e736f912ea46f84b4a31\": container with ID starting with 3812109a92c1663c505680ae1102c029ada4222b9af5e736f912ea46f84b4a31 not found: ID does not exist" Jul 17 19:41:01.801358 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:01.801326 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-b3dd7f-95f8b99c7-bdb5k"] Jul 17 19:41:01.805071 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:01.805047 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-b3dd7f-95f8b99c7-bdb5k"] Jul 17 19:41:02.099034 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:02.098958 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c8d502e-7547-44f9-b69e-5fbc6f21acf9" path="/var/lib/kubelet/pods/6c8d502e-7547-44f9-b69e-5fbc6f21acf9/volumes" Jul 17 19:41:04.374172 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:04.374134 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:41:16.238590 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:16.238544 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-79d51b-s4qpn"] Jul 17 19:41:16.239089 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:16.239012 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c8d502e-7547-44f9-b69e-5fbc6f21acf9" containerName="maas-api" Jul 17 19:41:16.239089 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:16.239032 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c8d502e-7547-44f9-b69e-5fbc6f21acf9" containerName="maas-api" Jul 17 19:41:16.239089 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:16.239059 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f07b5617-75e8-42d6-bd14-3a010124bc5b" containerName="revoke-keys" Jul 17 19:41:16.239089 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:16.239068 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f07b5617-75e8-42d6-bd14-3a010124bc5b" containerName="revoke-keys" Jul 17 19:41:16.239293 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:16.239139 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f07b5617-75e8-42d6-bd14-3a010124bc5b" containerName="revoke-keys" Jul 17 19:41:16.239293 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:16.239153 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c8d502e-7547-44f9-b69e-5fbc6f21acf9" containerName="maas-api" Jul 17 19:41:16.242181 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:16.242158 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-79d51b-s4qpn" Jul 17 19:41:16.244720 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:16.244695 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lbvd4\"" Jul 17 19:41:16.250266 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:16.249735 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-79d51b-s4qpn"] Jul 17 19:41:16.380221 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:16.380175 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b969420-d618-4f91-a12b-b4851626f3ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-79d51b-s4qpn\" (UID: \"3b969420-d618-4f91-a12b-b4851626f3ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-79d51b-s4qpn" Jul 17 19:41:16.480625 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:16.480586 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b969420-d618-4f91-a12b-b4851626f3ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-79d51b-s4qpn\" (UID: \"3b969420-d618-4f91-a12b-b4851626f3ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-79d51b-s4qpn" Jul 17 19:41:16.481272 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:16.481243 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b969420-d618-4f91-a12b-b4851626f3ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-79d51b-s4qpn\" (UID: \"3b969420-d618-4f91-a12b-b4851626f3ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-79d51b-s4qpn" Jul 17 19:41:16.554276 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:16.554184 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-79d51b-s4qpn" Jul 17 19:41:16.689583 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:16.689543 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-79d51b-s4qpn"] Jul 17 19:41:16.691662 ip-10-0-134-137 kubenswrapper[2561]: W0717 19:41:16.691633 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b969420_d618_4f91_a12b_b4851626f3ff.slice/crio-c05947f895c85572a286902f2273fcf2bd4e66bdcb29c2fa19befb7af4ce3b66 WatchSource:0}: Error finding container c05947f895c85572a286902f2273fcf2bd4e66bdcb29c2fa19befb7af4ce3b66: Status 404 returned error can't find the container with id c05947f895c85572a286902f2273fcf2bd4e66bdcb29c2fa19befb7af4ce3b66 Jul 17 19:41:16.863596 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:16.863498 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-79d51b-s4qpn" event={"ID":"3b969420-d618-4f91-a12b-b4851626f3ff","Type":"ContainerStarted","Data":"c05947f895c85572a286902f2273fcf2bd4e66bdcb29c2fa19befb7af4ce3b66"} Jul 17 19:41:17.871332 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:17.871285 2561 generic.go:356] "Generic (PLEG): container finished" podID="3b969420-d618-4f91-a12b-b4851626f3ff" containerID="4850ffcf26358a8a01bbd5c424c83039605eb8532587faa582060c234822ad3f" exitCode=0 Jul 17 19:41:17.871736 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:17.871371 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-79d51b-s4qpn" event={"ID":"3b969420-d618-4f91-a12b-b4851626f3ff","Type":"ContainerDied","Data":"4850ffcf26358a8a01bbd5c424c83039605eb8532587faa582060c234822ad3f"} Jul 17 19:41:19.011855 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:19.011819 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-79d51b-s4qpn" Jul 17 19:41:19.102625 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:19.102589 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b969420-d618-4f91-a12b-b4851626f3ff-maas-api-service-ca\") pod \"3b969420-d618-4f91-a12b-b4851626f3ff\" (UID: \"3b969420-d618-4f91-a12b-b4851626f3ff\") " Jul 17 19:41:19.102979 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:19.102955 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b969420-d618-4f91-a12b-b4851626f3ff-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3b969420-d618-4f91-a12b-b4851626f3ff" (UID: "3b969420-d618-4f91-a12b-b4851626f3ff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 19:41:19.203688 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:19.203609 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b969420-d618-4f91-a12b-b4851626f3ff-maas-api-service-ca\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:41:19.884761 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:19.884732 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-79d51b-s4qpn" Jul 17 19:41:19.884915 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:19.884775 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-79d51b-s4qpn" event={"ID":"3b969420-d618-4f91-a12b-b4851626f3ff","Type":"ContainerDied","Data":"c05947f895c85572a286902f2273fcf2bd4e66bdcb29c2fa19befb7af4ce3b66"} Jul 17 19:41:19.884915 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:19.884820 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c05947f895c85572a286902f2273fcf2bd4e66bdcb29c2fa19befb7af4ce3b66" Jul 17 19:41:26.690888 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:26.690797 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-79d51b-55cdf87db8-kz7n9"] Jul 17 19:41:26.691454 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:26.691057 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-79d51b-55cdf87db8-kz7n9" podUID="89da65ea-9550-48de-8a63-33809021a506" containerName="maas-api" containerID="cri-o://7bb2de0b9ee7e1cd7c93be85fd782ed254c3406edb77693a94939b695293780d" gracePeriod=30 Jul 17 19:41:26.924040 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:26.924011 2561 generic.go:356] "Generic (PLEG): container finished" podID="89da65ea-9550-48de-8a63-33809021a506" containerID="7bb2de0b9ee7e1cd7c93be85fd782ed254c3406edb77693a94939b695293780d" exitCode=0 Jul 17 19:41:26.924164 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:26.924092 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-79d51b-55cdf87db8-kz7n9" event={"ID":"89da65ea-9550-48de-8a63-33809021a506","Type":"ContainerDied","Data":"7bb2de0b9ee7e1cd7c93be85fd782ed254c3406edb77693a94939b695293780d"} Jul 17 19:41:26.924207 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:26.924165 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-79d51b-55cdf87db8-kz7n9" event={"ID":"89da65ea-9550-48de-8a63-33809021a506","Type":"ContainerDied","Data":"058d1dafd0e80ad204fd264a1b2141b3971851da37c69ad6577a0eab355a7558"} Jul 17 19:41:26.924207 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:26.924190 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="058d1dafd0e80ad204fd264a1b2141b3971851da37c69ad6577a0eab355a7558" Jul 17 19:41:26.931851 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:26.931831 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-79d51b-55cdf87db8-kz7n9" Jul 17 19:41:27.067857 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:27.067801 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89da65ea-9550-48de-8a63-33809021a506-openshift-service-ca\") pod \"89da65ea-9550-48de-8a63-33809021a506\" (UID: \"89da65ea-9550-48de-8a63-33809021a506\") " Jul 17 19:41:27.068138 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:27.067873 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89da65ea-9550-48de-8a63-33809021a506-maas-api-tls\") pod \"89da65ea-9550-48de-8a63-33809021a506\" (UID: \"89da65ea-9550-48de-8a63-33809021a506\") " Jul 17 19:41:27.068138 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:27.067940 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dk9m\" (UniqueName: \"kubernetes.io/projected/89da65ea-9550-48de-8a63-33809021a506-kube-api-access-2dk9m\") pod \"89da65ea-9550-48de-8a63-33809021a506\" (UID: \"89da65ea-9550-48de-8a63-33809021a506\") " Jul 17 19:41:27.068232 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:27.068148 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89da65ea-9550-48de-8a63-33809021a506-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "89da65ea-9550-48de-8a63-33809021a506" (UID: "89da65ea-9550-48de-8a63-33809021a506"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 19:41:27.070198 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:27.070177 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89da65ea-9550-48de-8a63-33809021a506-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "89da65ea-9550-48de-8a63-33809021a506" (UID: "89da65ea-9550-48de-8a63-33809021a506"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 19:41:27.070266 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:27.070194 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89da65ea-9550-48de-8a63-33809021a506-kube-api-access-2dk9m" (OuterVolumeSpecName: "kube-api-access-2dk9m") pod "89da65ea-9550-48de-8a63-33809021a506" (UID: "89da65ea-9550-48de-8a63-33809021a506"). InnerVolumeSpecName "kube-api-access-2dk9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 19:41:27.168893 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:27.168855 2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89da65ea-9550-48de-8a63-33809021a506-openshift-service-ca\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:41:27.168893 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:27.168888 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89da65ea-9550-48de-8a63-33809021a506-maas-api-tls\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:41:27.168893 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:27.168897 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dk9m\" (UniqueName: \"kubernetes.io/projected/89da65ea-9550-48de-8a63-33809021a506-kube-api-access-2dk9m\") on node \"ip-10-0-134-137.ec2.internal\" DevicePath \"\"" Jul 17 19:41:27.929297 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:27.929264 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-79d51b-55cdf87db8-kz7n9" Jul 17 19:41:27.951596 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:27.951552 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-79d51b-55cdf87db8-kz7n9"] Jul 17 19:41:27.955400 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:27.955370 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-79d51b-55cdf87db8-kz7n9"] Jul 17 19:41:28.098926 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:28.098887 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89da65ea-9550-48de-8a63-33809021a506" path="/var/lib/kubelet/pods/89da65ea-9550-48de-8a63-33809021a506/volumes" Jul 17 19:41:30.174693 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:30.174656 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-cqg68"] Jul 17 19:41:48.767565 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:48.767512 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b55dc8c5f-24x2m_b186dd68-8d8e-4f7a-9565-fed517965a70/copy-manifests/0.log" Jul 17 19:41:48.779322 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:48.779294 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b55dc8c5f-24x2m_b186dd68-8d8e-4f7a-9565-fed517965a70/manager/0.log" Jul 17 19:41:49.495479 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:49.495445 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29738595-9jpgd_58028cbe-d060-46c5-82ea-5a94b55906bb/cleanup/0.log" Jul 17 19:41:49.609073 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:49.609038 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29738610-m26bt_e7bfccf5-7572-4af5-b346-4361b49b5fc9/cleanup/0.log" Jul 17 19:41:49.713723 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:49.713686 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-690b208d-gkqtt_c835b376-b02f-4cc4-9f51-3f421d6531fc/revoke-keys/0.log" Jul 17 19:41:49.821730 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:49.821641 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-a4acef77-s28rl_a2476029-0002-4134-9c1f-9c3461563a1b/revoke-keys/0.log" Jul 17 19:41:49.933634 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:49.933605 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-a-0bfe89-ch87b_b21294b4-c87b-43ed-919c-073604b296b0/revoke-keys/0.log" Jul 17 19:41:50.049059 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:50.049026 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-b6c908-4t2f2_5be5c445-dd2b-46e1-abcc-6becd189ad3a/revoke-keys/0.log" Jul 17 19:41:50.179101 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:50.179026 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-derive-3c4aca95-q4hjp_3a1eb175-2e51-498f-b3cf-3970da570089/revoke-keys/0.log" Jul 17 19:41:50.287092 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:50.287061 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-a-1f5606-858mx_277ebff1-06c2-403d-bd5c-0384c531988b/revoke-keys/0.log" Jul 17 19:41:50.403052 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:50.403024 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-b-eaec93-qkt8h_361e3eed-dcb6-4e5e-8743-def8ac513d70/revoke-keys/0.log" Jul 17 19:41:50.507438 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:50.507406 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-migrate-59b00c87-g965g_dcf0e9f0-0268-41a5-8efe-d025c18d49f1/revoke-keys/0.log" Jul 17 19:41:50.614432 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:50.614406 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-45e4d318-cph9z_ed770d13-78d5-475a-8b32-5dba94c73fe8/revoke-keys/0.log" Jul 17 19:41:50.722879 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:50.722849 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-6d51d6be-crfmm_e35649af-2c79-4ee2-a990-0c67a18f60ad/revoke-keys/0.log" Jul 17 19:41:50.827251 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:50.827177 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-a-b3dd7f-72wj9_f07b5617-75e8-42d6-bd14-3a010124bc5b/revoke-keys/0.log" Jul 17 19:41:50.932541 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:50.932504 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-b-79d51b-s4qpn_3b969420-d618-4f91-a12b-b4851626f3ff/revoke-keys/0.log" Jul 17 19:41:51.846826 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:51.846791 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nww7x_a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499/util/0.log" Jul 17 19:41:51.854205 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:51.854175 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nww7x_a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499/pull/0.log" Jul 17 19:41:51.860710 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:51.860689 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nww7x_a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499/extract/0.log" Jul 17 19:41:51.982408 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:51.982379 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzwzv_b4c09057-4e4c-464c-9985-05025e1e0e09/util/0.log" Jul 17 19:41:51.989364 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:51.989336 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzwzv_b4c09057-4e4c-464c-9985-05025e1e0e09/pull/0.log" Jul 17 19:41:51.995574 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:51.995553 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzwzv_b4c09057-4e4c-464c-9985-05025e1e0e09/extract/0.log" Jul 17 19:41:52.109292 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:52.109206 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tgnfr_3f7df721-5719-4fe3-be5c-cf1f8d92ab2c/util/0.log" Jul 17 19:41:52.115629 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:52.115602 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tgnfr_3f7df721-5719-4fe3-be5c-cf1f8d92ab2c/pull/0.log" Jul 17 19:41:52.121934 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:52.121910 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tgnfr_3f7df721-5719-4fe3-be5c-cf1f8d92ab2c/extract/0.log" Jul 17 19:41:52.236642 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:52.236604 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mfkhp_47ae49e7-0d98-49e5-beb7-dc7d972803b8/pull/0.log" Jul 17 19:41:52.242545 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:52.242505 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mfkhp_47ae49e7-0d98-49e5-beb7-dc7d972803b8/extract/0.log" Jul 17 19:41:52.250339 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:52.250309 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mfkhp_47ae49e7-0d98-49e5-beb7-dc7d972803b8/util/0.log" Jul 17 19:41:53.080996 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:53.080968 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-cqg68_f1a73fff-acd0-4416-ba1a-844a75c62607/limitador/0.log" Jul 17 19:41:53.676689 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:53.676648 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-kz8sh_724470c2-a23a-496e-8246-e1a73870190b/discovery/0.log" Jul 17 19:41:54.833886 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:41:54.833854 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-s7f4n_4ea241aa-9bb1-4b1f-9096-6db6421a11c4/main/0.log" Jul 17 19:42:03.565188 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:03.565152 2561 scope.go:117] "RemoveContainer" containerID="7bb2de0b9ee7e1cd7c93be85fd782ed254c3406edb77693a94939b695293780d" Jul 17 19:42:04.175104 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:04.175069 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-m9wzh_3591b9f8-a2ff-4adb-93ec-449225ca6390/global-pull-secret-syncer/0.log" Jul 17 19:42:04.270930 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:04.270897 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-2fgtz_0135e7e9-6294-4524-ad81-17f251822127/konnectivity-agent/0.log" Jul 17 19:42:04.358821 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:04.358788 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-137.ec2.internal_4055332a3b86c15989d82b71c24d29a9/haproxy/0.log" Jul 17 19:42:08.987116 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:08.987079 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nww7x_a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499/extract/0.log" Jul 17 19:42:09.014043 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:09.014010 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nww7x_a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499/util/0.log" Jul 17 19:42:09.048898 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:09.048869 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nww7x_a12f9ca0-8182-4b4a-b96c-d5c8a7c1e499/pull/0.log" Jul 17 19:42:09.096241 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:09.096208 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzwzv_b4c09057-4e4c-464c-9985-05025e1e0e09/extract/0.log" Jul 17 19:42:09.121731 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:09.121700 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzwzv_b4c09057-4e4c-464c-9985-05025e1e0e09/util/0.log" Jul 17 19:42:09.146714 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:09.146684 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xzwzv_b4c09057-4e4c-464c-9985-05025e1e0e09/pull/0.log" Jul 17 19:42:09.178757 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:09.178729 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tgnfr_3f7df721-5719-4fe3-be5c-cf1f8d92ab2c/extract/0.log" Jul 17 19:42:09.205141 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:09.205113 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tgnfr_3f7df721-5719-4fe3-be5c-cf1f8d92ab2c/util/0.log" Jul 17 19:42:09.230206 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:09.230179 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tgnfr_3f7df721-5719-4fe3-be5c-cf1f8d92ab2c/pull/0.log" Jul 17 19:42:09.260351 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:09.260286 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mfkhp_47ae49e7-0d98-49e5-beb7-dc7d972803b8/extract/0.log" Jul 17 19:42:09.291882 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:09.291851 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mfkhp_47ae49e7-0d98-49e5-beb7-dc7d972803b8/util/0.log" Jul 17 19:42:09.318749 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:09.318722 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mfkhp_47ae49e7-0d98-49e5-beb7-dc7d972803b8/pull/0.log" Jul 17 19:42:10.085252 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:10.085223 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-cqg68_f1a73fff-acd0-4416-ba1a-844a75c62607/limitador/0.log" Jul 17 19:42:12.283153 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:12.283121 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cfkrj_7051876f-4c82-48f1-9ab9-0e2b64818bc5/node-exporter/0.log" Jul 17 19:42:12.305020 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:12.304997 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cfkrj_7051876f-4c82-48f1-9ab9-0e2b64818bc5/kube-rbac-proxy/0.log" Jul 17 19:42:12.326724 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:12.326698 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cfkrj_7051876f-4c82-48f1-9ab9-0e2b64818bc5/init-textfile/0.log" Jul 17 19:42:12.827384 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:12.827355 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-hw2f9_24f74548-eab9-4297-acc1-03b123c45161/prometheus-operator-admission-webhook/0.log" Jul 17 19:42:14.587218 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:14.587175 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8mpmt/perf-node-gather-daemonset-d2x9z"] Jul 17 19:42:14.587643 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:14.587471 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89da65ea-9550-48de-8a63-33809021a506" containerName="maas-api" Jul 17 19:42:14.587643 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:14.587481 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="89da65ea-9550-48de-8a63-33809021a506" containerName="maas-api" Jul 17 19:42:14.587643 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:14.587495 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b969420-d618-4f91-a12b-b4851626f3ff" containerName="revoke-keys" Jul 17 19:42:14.587643 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:14.587502 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b969420-d618-4f91-a12b-b4851626f3ff" containerName="revoke-keys" Jul 17 19:42:14.587643 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:14.587570 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="89da65ea-9550-48de-8a63-33809021a506" containerName="maas-api" Jul 17 19:42:14.587643 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:14.587579 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b969420-d618-4f91-a12b-b4851626f3ff" containerName="revoke-keys" Jul 17 19:42:14.590757 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:14.590734 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8mpmt/perf-node-gather-daemonset-d2x9z" Jul 17 19:42:14.593141 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:14.593114 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8mpmt\"/\"openshift-service-ca.crt\"" Jul 17 19:42:14.593488 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:14.593469 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8mpmt\"/\"kube-root-ca.crt\"" Jul 17 19:42:14.594082 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:14.594069 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8mpmt\"/\"default-dockercfg-zcdlp\"" Jul 17 19:42:14.600754 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:14.600726 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8mpmt/perf-node-gather-daemonset-d2x9z"] Jul 17 19:42:14.642384 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:14.642337 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/30a8133c-bdd0-4e19-a3f2-cce1be31bd4a-lib-modules\") pod \"perf-node-gather-daemonset-d2x9z\" (UID: \"30a8133c-bdd0-4e19-a3f2-cce1be31bd4a\") " pod="openshift-must-gather-8mpmt/perf-node-gather-daemonset-d2x9z" Jul 17 19:42:14.642599 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:14.642396 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/30a8133c-bdd0-4e19-a3f2-cce1be31bd4a-proc\") pod \"perf-node-gather-daemonset-d2x9z\" (UID: \"30a8133c-bdd0-4e19-a3f2-cce1be31bd4a\") " pod="openshift-must-gather-8mpmt/perf-node-gather-daemonset-d2x9z" Jul 17 19:42:14.642599 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:14.642468 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30a8133c-bdd0-4e19-a3f2-cce1be31bd4a-sys\") pod \"perf-node-gather-daemonset-d2x9z\" (UID: \"30a8133c-bdd0-4e19-a3f2-cce1be31bd4a\") " pod="openshift-must-gather-8mpmt/perf-node-gather-daemonset-d2x9z" Jul 17 19:42:14.642599 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:14.642562 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nxwq\" (UniqueName: \"kubernetes.io/projected/30a8133c-bdd0-4e19-a3f2-cce1be31bd4a-kube-api-access-9nxwq\") pod \"perf-node-gather-daemonset-d2x9z\" (UID: \"30a8133c-bdd0-4e19-a3f2-cce1be31bd4a\") " pod="openshift-must-gather-8mpmt/perf-node-gather-daemonset-d2x9z" Jul 17 19:42:14.642713 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:14.642605 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/30a8133c-bdd0-4e19-a3f2-cce1be31bd4a-podres\") pod \"perf-node-gather-daemonset-d2x9z\" (UID: \"30a8133c-bdd0-4e19-a3f2-cce1be31bd4a\") " pod="openshift-must-gather-8mpmt/perf-node-gather-daemonset-d2x9z" Jul 17 19:42:14.743436 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:14.743397 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/30a8133c-bdd0-4e19-a3f2-cce1be31bd4a-lib-modules\") pod \"perf-node-gather-daemonset-d2x9z\" (UID: \"30a8133c-bdd0-4e19-a3f2-cce1be31bd4a\") " pod="openshift-must-gather-8mpmt/perf-node-gather-daemonset-d2x9z" Jul 17 19:42:14.743669 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:14.743450 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/30a8133c-bdd0-4e19-a3f2-cce1be31bd4a-proc\") pod \"perf-node-gather-daemonset-d2x9z\" (UID: \"30a8133c-bdd0-4e19-a3f2-cce1be31bd4a\") " pod="openshift-must-gather-8mpmt/perf-node-gather-daemonset-d2x9z" Jul 17 19:42:14.743669 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:14.743474 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30a8133c-bdd0-4e19-a3f2-cce1be31bd4a-sys\") pod \"perf-node-gather-daemonset-d2x9z\" (UID: \"30a8133c-bdd0-4e19-a3f2-cce1be31bd4a\") " pod="openshift-must-gather-8mpmt/perf-node-gather-daemonset-d2x9z" Jul 17 19:42:14.743669 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:14.743504 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nxwq\" (UniqueName: \"kubernetes.io/projected/30a8133c-bdd0-4e19-a3f2-cce1be31bd4a-kube-api-access-9nxwq\") pod \"perf-node-gather-daemonset-d2x9z\" (UID: \"30a8133c-bdd0-4e19-a3f2-cce1be31bd4a\") " pod="openshift-must-gather-8mpmt/perf-node-gather-daemonset-d2x9z" Jul 17 19:42:14.743669 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:14.743555 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/30a8133c-bdd0-4e19-a3f2-cce1be31bd4a-podres\") pod \"perf-node-gather-daemonset-d2x9z\" (UID: \"30a8133c-bdd0-4e19-a3f2-cce1be31bd4a\") " pod="openshift-must-gather-8mpmt/perf-node-gather-daemonset-d2x9z" Jul 17 19:42:14.743669 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:14.743567 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/30a8133c-bdd0-4e19-a3f2-cce1be31bd4a-proc\") pod \"perf-node-gather-daemonset-d2x9z\" (UID: \"30a8133c-bdd0-4e19-a3f2-cce1be31bd4a\") " pod="openshift-must-gather-8mpmt/perf-node-gather-daemonset-d2x9z" Jul 17 19:42:14.743669 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:14.743610 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30a8133c-bdd0-4e19-a3f2-cce1be31bd4a-sys\") pod \"perf-node-gather-daemonset-d2x9z\" (UID: \"30a8133c-bdd0-4e19-a3f2-cce1be31bd4a\") " pod="openshift-must-gather-8mpmt/perf-node-gather-daemonset-d2x9z" Jul 17 19:42:14.743669 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:14.743641 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/30a8133c-bdd0-4e19-a3f2-cce1be31bd4a-lib-modules\") pod \"perf-node-gather-daemonset-d2x9z\" (UID: \"30a8133c-bdd0-4e19-a3f2-cce1be31bd4a\") " pod="openshift-must-gather-8mpmt/perf-node-gather-daemonset-d2x9z" Jul 17 19:42:14.743932 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:14.743683 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/30a8133c-bdd0-4e19-a3f2-cce1be31bd4a-podres\") pod \"perf-node-gather-daemonset-d2x9z\" (UID: \"30a8133c-bdd0-4e19-a3f2-cce1be31bd4a\") " pod="openshift-must-gather-8mpmt/perf-node-gather-daemonset-d2x9z" Jul 17 19:42:14.751655 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:14.751626 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nxwq\" (UniqueName: \"kubernetes.io/projected/30a8133c-bdd0-4e19-a3f2-cce1be31bd4a-kube-api-access-9nxwq\") pod \"perf-node-gather-daemonset-d2x9z\" (UID: \"30a8133c-bdd0-4e19-a3f2-cce1be31bd4a\") " pod="openshift-must-gather-8mpmt/perf-node-gather-daemonset-d2x9z" Jul 17 19:42:14.901932 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:14.901819 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8mpmt/perf-node-gather-daemonset-d2x9z" Jul 17 19:42:15.038952 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:15.038925 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8mpmt/perf-node-gather-daemonset-d2x9z"] Jul 17 19:42:15.040562 ip-10-0-134-137 kubenswrapper[2561]: W0717 19:42:15.040512 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod30a8133c_bdd0_4e19_a3f2_cce1be31bd4a.slice/crio-1abea92fe65bf90b5cafdbcfac0dcc75229678f7e7bc0c3e0b855d9d5eedd8cf WatchSource:0}: Error finding container 1abea92fe65bf90b5cafdbcfac0dcc75229678f7e7bc0c3e0b855d9d5eedd8cf: Status 404 returned error can't find the container with id 1abea92fe65bf90b5cafdbcfac0dcc75229678f7e7bc0c3e0b855d9d5eedd8cf Jul 17 19:42:15.183915 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:15.183815 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8mpmt/perf-node-gather-daemonset-d2x9z" event={"ID":"30a8133c-bdd0-4e19-a3f2-cce1be31bd4a","Type":"ContainerStarted","Data":"20821ec10d28b899fa607fe92ca9614d289d10b3456a71d484fc82697d33806d"} Jul 17 19:42:15.183915 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:15.183870 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8mpmt/perf-node-gather-daemonset-d2x9z" event={"ID":"30a8133c-bdd0-4e19-a3f2-cce1be31bd4a","Type":"ContainerStarted","Data":"1abea92fe65bf90b5cafdbcfac0dcc75229678f7e7bc0c3e0b855d9d5eedd8cf"} Jul 17 19:42:15.184168 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:15.183944 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8mpmt/perf-node-gather-daemonset-d2x9z" Jul 17 19:42:15.201932 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:15.201879 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8mpmt/perf-node-gather-daemonset-d2x9z" podStartSLOduration=1.201863627 podStartE2EDuration="1.201863627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 19:42:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:42:15.199365283 +0000 UTC m=+2833.529181492" watchObservedRunningTime="2026-07-17 19:42:15.201863627 +0000 UTC m=+2833.531679836" Jul 17 19:42:15.529214 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:15.529184 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-xbsqk_67b653fc-1593-4bee-a23f-3f54197879c0/download-server/0.log" Jul 17 19:42:17.329481 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:17.329453 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-crvtk_71e3530f-7f67-4f34-a65a-444d1f270c93/dns/0.log" Jul 17 19:42:17.352398 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:17.352355 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-crvtk_71e3530f-7f67-4f34-a65a-444d1f270c93/kube-rbac-proxy/0.log" Jul 17 19:42:17.410170 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:17.410137 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pvpbg_728c6dad-4aae-44c2-ac11-60be3b680cca/dns-node-resolver/0.log" Jul 17 19:42:18.106630 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:18.106586 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-n2wtb_7f5ebd27-cd62-4b2b-9c30-4863ca02f422/node-ca/0.log" Jul 17 19:42:19.199330 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:19.199293 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-kz8sh_724470c2-a23a-496e-8246-e1a73870190b/discovery/0.log" Jul 17 19:42:20.197352 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:20.197323 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ps4bw_c7ab0b84-2d5c-4dd8-817a-b6baa962fab6/serve-healthcheck-canary/0.log" Jul 17 19:42:20.747818 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:20.747791 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6c6g4_96280fd2-5cf1-4adc-a0a2-a8f54aef67a9/kube-rbac-proxy/0.log" Jul 17 19:42:20.773914 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:20.773888 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6c6g4_96280fd2-5cf1-4adc-a0a2-a8f54aef67a9/exporter/0.log" Jul 17 19:42:20.796790 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:20.796764 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6c6g4_96280fd2-5cf1-4adc-a0a2-a8f54aef67a9/extractor/0.log" Jul 17 19:42:21.199701 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:21.199622 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8mpmt/perf-node-gather-daemonset-d2x9z" Jul 17 19:42:23.645125 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:23.645091 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b55dc8c5f-24x2m_b186dd68-8d8e-4f7a-9565-fed517965a70/manager/0.log" Jul 17 19:42:23.665991 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:23.665967 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b55dc8c5f-24x2m_b186dd68-8d8e-4f7a-9565-fed517965a70/copy-manifests/0.log" Jul 17 19:42:25.013187 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:25.013152 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-kz8sh_724470c2-a23a-496e-8246-e1a73870190b/discovery/0.log" Jul 17 19:42:25.247142 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:25.247117 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-64d7f9f4cf-t44gk_2748a3b1-0239-4a12-8181-4949c5b7aa84/manager/0.log" Jul 17 19:42:26.632634 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:26.632601 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-kz8sh_724470c2-a23a-496e-8246-e1a73870190b/discovery/0.log" Jul 17 19:42:30.591727 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:30.591696 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-6p5gk_18235a43-e5c1-4146-876c-55c96b2cbb5c/migrator/0.log" Jul 17 19:42:30.613242 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:30.613210 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-6p5gk_18235a43-e5c1-4146-876c-55c96b2cbb5c/graceful-termination/0.log" Jul 17 19:42:32.135999 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:32.135918 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nb8rb_08c31757-7c13-4c31-a1ad-570aad3cde70/kube-multus-additional-cni-plugins/0.log" Jul 17 19:42:32.157694 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:32.157669 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nb8rb_08c31757-7c13-4c31-a1ad-570aad3cde70/egress-router-binary-copy/0.log" Jul 17 19:42:32.182111 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:32.182085 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nb8rb_08c31757-7c13-4c31-a1ad-570aad3cde70/cni-plugins/0.log" Jul 17 19:42:32.202836 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:32.202811 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nb8rb_08c31757-7c13-4c31-a1ad-570aad3cde70/bond-cni-plugin/0.log" Jul 17 19:42:32.224478 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:32.224448 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nb8rb_08c31757-7c13-4c31-a1ad-570aad3cde70/routeoverride-cni/0.log" Jul 17 19:42:32.245141 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:32.245108 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nb8rb_08c31757-7c13-4c31-a1ad-570aad3cde70/whereabouts-cni-bincopy/0.log" Jul 17 19:42:32.265680 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:32.265652 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nb8rb_08c31757-7c13-4c31-a1ad-570aad3cde70/whereabouts-cni/0.log" Jul 17 19:42:32.471429 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:32.471351 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-nrss7_1edd2f99-7d44-4eed-8850-50ca38aa5153/kube-multus/0.log" Jul 17 19:42:32.575233 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:32.575196 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-7l6k9_16db4114-eec4-4eb4-80be-1e6238e5c67b/network-metrics-daemon/0.log" Jul 17 19:42:32.600790 ip-10-0-134-137 kubenswrapper[2561]: I0717 19:42:32.600756 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-7l6k9_16db4114-eec4-4eb4-80be-1e6238e5c67b/kube-rbac-proxy/0.log"