Jul 09 18:56:38.676274 ip-10-0-140-153 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 09 18:56:38.676296 ip-10-0-140-153 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 09 18:56:38.676305 ip-10-0-140-153 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 09 18:56:38.676632 ip-10-0-140-153 systemd[1]: Failed to start Kubernetes Kubelet. Jul 09 18:56:48.789679 ip-10-0-140-153 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 09 18:56:48.789697 ip-10-0-140-153 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 5ccb950f36f34cf0b37d74cf9443d065 -- Jul 09 18:58:49.281604 ip-10-0-140-153 systemd[1]: Starting Kubernetes Kubelet... Jul 09 18:58:49.725670 ip-10-0-140-153 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 18:58:49.725670 ip-10-0-140-153 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 09 18:58:49.725670 ip-10-0-140-153 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 18:58:49.725670 ip-10-0-140-153 kubenswrapper[2566]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 09 18:58:49.725670 ip-10-0-140-153 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 18:58:49.726664 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.726573 2566 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 09 18:58:49.731115 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731098 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 18:58:49.731115 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731115 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 18:58:49.731176 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731119 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 18:58:49.731176 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731123 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 18:58:49.731176 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731126 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 18:58:49.731176 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731129 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 18:58:49.731176 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731132 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 18:58:49.731176 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731136 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 18:58:49.731176 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731140 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 18:58:49.731176 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731145 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 18:58:49.731176 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731148 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 18:58:49.731176 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731151 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 18:58:49.731176 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731155 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 18:58:49.731176 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731159 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 18:58:49.731176 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731174 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 18:58:49.731176 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731177 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 18:58:49.731176 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731180 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 18:58:49.731176 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731184 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 18:58:49.731563 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731187 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 18:58:49.731563 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731189 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 18:58:49.731563 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731192 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 18:58:49.731563 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731194 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 18:58:49.731563 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731197 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 18:58:49.731563 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731200 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 18:58:49.731563 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731202 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 18:58:49.731563 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731205 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 18:58:49.731563 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731208 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 18:58:49.731563 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731210 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 18:58:49.731563 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731213 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 18:58:49.731563 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731215 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 18:58:49.731563 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731218 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 18:58:49.731563 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731220 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 18:58:49.731563 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731224 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 18:58:49.731563 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731227 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 09 18:58:49.731563 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731230 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 18:58:49.731563 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731232 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 18:58:49.731563 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731235 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 18:58:49.732011 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731238 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 18:58:49.732011 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731241 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 18:58:49.732011 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731244 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 18:58:49.732011 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731247 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 18:58:49.732011 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731249 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 18:58:49.732011 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731252 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 18:58:49.732011 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731254 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 18:58:49.732011 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731257 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 18:58:49.732011 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731259 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 18:58:49.732011 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731261 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 18:58:49.732011 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731264 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 18:58:49.732011 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731266 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 18:58:49.732011 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731269 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 18:58:49.732011 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731271 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 18:58:49.732011 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731276 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 18:58:49.732011 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731278 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 18:58:49.732011 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731281 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 18:58:49.732011 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731284 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 18:58:49.732011 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731286 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 18:58:49.732011 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731289 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 18:58:49.732497 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731292 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 18:58:49.732497 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731294 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 18:58:49.732497 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731297 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 18:58:49.732497 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731300 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 18:58:49.732497 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731304 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 18:58:49.732497 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731324 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 18:58:49.732497 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731328 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 18:58:49.732497 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731330 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 18:58:49.732497 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731333 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 18:58:49.732497 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731336 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 18:58:49.732497 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731338 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 18:58:49.732497 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731341 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 18:58:49.732497 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731344 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 18:58:49.732497 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731347 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 18:58:49.732497 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731357 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 18:58:49.732497 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731366 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 18:58:49.732497 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731369 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 18:58:49.732497 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731371 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 18:58:49.732497 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731375 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 18:58:49.732497 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731378 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 18:58:49.732968 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731381 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 18:58:49.732968 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731383 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 18:58:49.732968 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731386 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 18:58:49.732968 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731388 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 18:58:49.732968 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731391 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 18:58:49.732968 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731394 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 18:58:49.732968 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731397 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 18:58:49.732968 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731400 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 18:58:49.732968 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731403 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 18:58:49.732968 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731405 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 18:58:49.732968 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731408 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 18:58:49.732968 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731410 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 18:58:49.732968 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731413 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 18:58:49.732968 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731415 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 18:58:49.732968 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731418 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 18:58:49.732968 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731421 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 18:58:49.732968 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731423 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 18:58:49.732968 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731425 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 18:58:49.732968 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731428 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 18:58:49.732968 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731431 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 18:58:49.733455 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731433 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 18:58:49.733455 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.731435 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 18:58:49.733455 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732151 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 18:58:49.733455 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732157 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 18:58:49.733455 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732160 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 18:58:49.733455 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732163 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 18:58:49.733455 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732166 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 18:58:49.733455 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732169 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 18:58:49.733455 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732172 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 18:58:49.733455 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732174 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 18:58:49.733455 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732177 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 18:58:49.733455 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732180 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 18:58:49.733455 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732182 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 18:58:49.733455 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732185 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 18:58:49.733455 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732188 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 18:58:49.733455 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732191 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 18:58:49.733455 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732193 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 18:58:49.733455 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732195 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 18:58:49.733455 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732199 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 18:58:49.733455 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732202 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 18:58:49.733918 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732205 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 18:58:49.733918 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732207 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 18:58:49.733918 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732210 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 18:58:49.733918 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732213 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 18:58:49.733918 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732217 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 18:58:49.733918 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732221 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 18:58:49.733918 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732224 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 18:58:49.733918 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732227 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 18:58:49.733918 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732229 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 18:58:49.733918 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732231 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 18:58:49.733918 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732234 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 18:58:49.733918 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732237 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 18:58:49.733918 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732239 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 18:58:49.733918 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732243 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 18:58:49.733918 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732246 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 18:58:49.733918 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732250 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 18:58:49.733918 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732253 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 18:58:49.733918 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732255 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 18:58:49.733918 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732258 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 09 18:58:49.734430 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732261 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 18:58:49.734430 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732264 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 18:58:49.734430 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732266 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 18:58:49.734430 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732269 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 18:58:49.734430 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732272 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 18:58:49.734430 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732274 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 18:58:49.734430 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732276 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 18:58:49.734430 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732281 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 18:58:49.734430 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732284 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 18:58:49.734430 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732287 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 18:58:49.734430 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732291 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 18:58:49.734430 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732295 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 18:58:49.734430 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732298 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 18:58:49.734430 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732301 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 18:58:49.734430 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732304 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 18:58:49.734430 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732306 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 18:58:49.734430 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732324 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 18:58:49.734430 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732327 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 18:58:49.734430 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732330 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 18:58:49.734926 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732333 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 18:58:49.734926 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732335 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 18:58:49.734926 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732338 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 18:58:49.734926 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732341 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 18:58:49.734926 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732343 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 18:58:49.734926 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732346 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 18:58:49.734926 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732349 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 18:58:49.734926 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732352 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 18:58:49.734926 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732354 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 18:58:49.734926 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732357 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 18:58:49.734926 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732359 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 18:58:49.734926 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732362 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 18:58:49.734926 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732365 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 18:58:49.734926 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732369 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 18:58:49.734926 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732372 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 18:58:49.734926 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732375 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 18:58:49.734926 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732378 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 18:58:49.734926 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732381 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 18:58:49.734926 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732383 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 18:58:49.734926 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732386 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 18:58:49.735497 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732388 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 18:58:49.735497 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732391 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 18:58:49.735497 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732393 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 18:58:49.735497 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732396 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 18:58:49.735497 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732399 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 18:58:49.735497 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732401 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 18:58:49.735497 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732404 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 18:58:49.735497 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732406 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 18:58:49.735497 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732409 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 18:58:49.735497 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732411 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 18:58:49.735497 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732414 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 18:58:49.735497 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732416 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 18:58:49.735497 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732419 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 18:58:49.735497 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732421 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 18:58:49.735497 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732424 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 18:58:49.735497 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732427 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 18:58:49.735497 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732429 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 18:58:49.735497 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732432 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 18:58:49.735497 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732434 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 18:58:49.735497 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732437 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 18:58:49.736033 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732439 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 18:58:49.736033 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732442 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 18:58:49.736033 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732444 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 18:58:49.736033 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732817 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 18:58:49.736033 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732821 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 18:58:49.736033 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732824 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 18:58:49.736033 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732827 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 18:58:49.736033 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732832 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 18:58:49.736033 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732836 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 18:58:49.736033 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732839 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 18:58:49.736033 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732842 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 18:58:49.736033 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732845 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 18:58:49.736033 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732847 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 18:58:49.736033 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732850 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 18:58:49.736033 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732852 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 18:58:49.736033 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732855 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 18:58:49.736033 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732858 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 18:58:49.736033 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732861 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 18:58:49.736033 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732863 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 18:58:49.736033 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732866 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 18:58:49.736625 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732868 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 18:58:49.736625 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732871 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 18:58:49.736625 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732874 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 18:58:49.736625 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732876 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 18:58:49.736625 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732879 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 18:58:49.736625 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732881 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 18:58:49.736625 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732884 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 18:58:49.736625 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732886 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 18:58:49.736625 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732888 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 18:58:49.736625 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732891 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 18:58:49.736625 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732893 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 18:58:49.736625 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732896 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 18:58:49.736625 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732899 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 18:58:49.736625 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732901 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 18:58:49.736625 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732904 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 18:58:49.736625 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732906 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 18:58:49.736625 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732908 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 18:58:49.736625 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732911 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 18:58:49.736625 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732913 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 18:58:49.736625 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732916 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 18:58:49.737228 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732923 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 18:58:49.737228 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732926 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 18:58:49.737228 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732929 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 18:58:49.737228 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732931 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 18:58:49.737228 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732934 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 18:58:49.737228 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732936 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 18:58:49.737228 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732939 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 18:58:49.737228 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732941 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 18:58:49.737228 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732944 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 18:58:49.737228 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732946 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 18:58:49.737228 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732949 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 18:58:49.737228 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732952 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 18:58:49.737228 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732954 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 18:58:49.737228 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732957 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 18:58:49.737228 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732960 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 18:58:49.737228 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732962 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 18:58:49.737228 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732965 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 18:58:49.737228 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732967 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 18:58:49.737228 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732970 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 18:58:49.737766 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732973 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 18:58:49.737766 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732976 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 18:58:49.737766 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732978 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 18:58:49.737766 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732981 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 18:58:49.737766 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732984 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 18:58:49.737766 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732987 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 18:58:49.737766 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732989 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 18:58:49.737766 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732992 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 18:58:49.737766 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732995 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 18:58:49.737766 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.732997 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 18:58:49.737766 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733000 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 18:58:49.737766 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733002 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 18:58:49.737766 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733005 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 18:58:49.737766 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733008 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 18:58:49.737766 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733011 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 18:58:49.737766 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733013 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 18:58:49.737766 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733016 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 18:58:49.737766 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733018 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 18:58:49.737766 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733020 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 18:58:49.737766 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733023 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 18:58:49.738377 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733025 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 18:58:49.738377 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733029 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 18:58:49.738377 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733033 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 18:58:49.738377 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733036 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 18:58:49.738377 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733038 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 18:58:49.738377 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733041 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 18:58:49.738377 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733044 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 18:58:49.738377 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733046 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 09 18:58:49.738377 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733049 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 18:58:49.738377 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733051 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 18:58:49.738377 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733054 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 18:58:49.738377 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733057 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 18:58:49.738377 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733060 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 18:58:49.738377 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733062 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 18:58:49.738377 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733065 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 18:58:49.738377 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733068 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 18:58:49.738377 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733070 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 18:58:49.738377 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733072 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 18:58:49.738377 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733076 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 18:58:49.738890 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733080 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 18:58:49.738890 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733082 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 18:58:49.738890 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733085 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 18:58:49.738890 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733087 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 18:58:49.738890 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733166 2566 flags.go:64] FLAG: --address="0.0.0.0" Jul 09 18:58:49.738890 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733176 2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 09 18:58:49.738890 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733187 2566 flags.go:64] FLAG: --anonymous-auth="true" Jul 09 18:58:49.738890 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733194 2566 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 09 18:58:49.738890 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733199 2566 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 09 18:58:49.738890 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733202 2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 09 18:58:49.738890 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733206 2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 09 18:58:49.738890 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733210 2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 09 18:58:49.738890 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733214 2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 09 18:58:49.738890 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733217 2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 09 18:58:49.738890 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733220 2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 09 18:58:49.738890 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733223 2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 09 18:58:49.738890 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733226 2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 09 18:58:49.738890 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733229 2566 flags.go:64] FLAG: --cgroup-root="" Jul 09 18:58:49.738890 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733232 2566 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 09 18:58:49.738890 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733235 2566 flags.go:64] FLAG: --client-ca-file="" Jul 09 18:58:49.738890 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733243 2566 flags.go:64] FLAG: --cloud-provider="external" Jul 09 18:58:49.738890 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733246 2566 flags.go:64] FLAG: --cluster-dns="[]" Jul 09 18:58:49.738890 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733251 2566 flags.go:64] FLAG: --cluster-domain="" Jul 09 18:58:49.738890 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733254 2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 09 18:58:49.739511 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733257 2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 09 18:58:49.739511 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733260 2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 09 18:58:49.739511 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733263 2566 flags.go:64] FLAG: --container-log-max-files="5" Jul 09 18:58:49.739511 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733267 2566 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 09 18:58:49.739511 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733270 2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 09 18:58:49.739511 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733273 2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 09 18:58:49.739511 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733277 2566 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 09 18:58:49.739511 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733280 2566 flags.go:64] FLAG: --contention-profiling="false" Jul 09 18:58:49.739511 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733282 2566 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 09 18:58:49.739511 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733285 2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 09 18:58:49.739511 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733288 2566 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 09 18:58:49.739511 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733291 2566 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 09 18:58:49.739511 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733295 2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 09 18:58:49.739511 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733298 2566 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 09 18:58:49.739511 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733301 2566 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 09 18:58:49.739511 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733304 2566 flags.go:64] FLAG: --enable-load-reader="false" Jul 09 18:58:49.739511 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733319 2566 flags.go:64] FLAG: --enable-server="true" Jul 09 18:58:49.739511 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733323 2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 09 18:58:49.739511 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733328 2566 flags.go:64] FLAG: --event-burst="100" Jul 09 18:58:49.739511 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733331 2566 flags.go:64] FLAG: --event-qps="50" Jul 09 18:58:49.739511 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733334 2566 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 09 18:58:49.739511 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733337 2566 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 09 18:58:49.739511 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733340 2566 flags.go:64] FLAG: --eviction-hard="" Jul 09 18:58:49.739511 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733344 2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 09 18:58:49.740130 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733346 2566 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 09 18:58:49.740130 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733350 2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 09 18:58:49.740130 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733353 2566 flags.go:64] FLAG: --eviction-soft="" Jul 09 18:58:49.740130 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733356 2566 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 09 18:58:49.740130 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733360 2566 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 09 18:58:49.740130 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733363 2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 09 18:58:49.740130 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733366 2566 flags.go:64] FLAG: --experimental-mounter-path="" Jul 09 18:58:49.740130 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733369 2566 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 09 18:58:49.740130 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733372 2566 flags.go:64] FLAG: --fail-swap-on="true" Jul 09 18:58:49.740130 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733374 2566 flags.go:64] FLAG: --feature-gates="" Jul 09 18:58:49.740130 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733378 2566 flags.go:64] FLAG: --file-check-frequency="20s" Jul 09 18:58:49.740130 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733381 2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 09 18:58:49.740130 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733384 2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 09 18:58:49.740130 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733387 2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 09 18:58:49.740130 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733390 2566 flags.go:64] FLAG: --healthz-port="10248" Jul 09 18:58:49.740130 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733393 2566 flags.go:64] FLAG: --help="false" Jul 09 18:58:49.740130 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733396 2566 flags.go:64] FLAG: --hostname-override="ip-10-0-140-153.ec2.internal" Jul 09 18:58:49.740130 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733399 2566 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 09 18:58:49.740130 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733402 2566 flags.go:64] FLAG: --http-check-frequency="20s" Jul 09 18:58:49.740130 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733405 2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 09 18:58:49.740130 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733408 2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 09 18:58:49.740130 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733412 2566 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 09 18:58:49.740130 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733414 2566 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 09 18:58:49.740130 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733417 2566 flags.go:64] FLAG: --image-service-endpoint="" Jul 09 18:58:49.740737 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733420 2566 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 09 18:58:49.740737 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733423 2566 flags.go:64] FLAG: --kube-api-burst="100" Jul 09 18:58:49.740737 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733426 2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 09 18:58:49.740737 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733429 2566 flags.go:64] FLAG: --kube-api-qps="50" Jul 09 18:58:49.740737 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733432 2566 flags.go:64] FLAG: --kube-reserved="" Jul 09 18:58:49.740737 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733435 2566 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 09 18:58:49.740737 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733437 2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 09 18:58:49.740737 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733440 2566 flags.go:64] FLAG: --kubelet-cgroups="" Jul 09 18:58:49.740737 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733443 2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 09 18:58:49.740737 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733446 2566 flags.go:64] FLAG: --lock-file="" Jul 09 18:58:49.740737 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733448 2566 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 09 18:58:49.740737 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733451 2566 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 09 18:58:49.740737 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733455 2566 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 09 18:58:49.740737 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733461 2566 flags.go:64] FLAG: --log-json-split-stream="false" Jul 09 18:58:49.740737 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733464 2566 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 09 18:58:49.740737 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733467 2566 flags.go:64] FLAG: --log-text-split-stream="false" Jul 09 18:58:49.740737 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733469 2566 flags.go:64] FLAG: --logging-format="text" Jul 09 18:58:49.740737 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733472 2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 09 18:58:49.740737 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733476 2566 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 09 18:58:49.740737 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733478 2566 flags.go:64] FLAG: --manifest-url="" Jul 09 18:58:49.740737 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733481 2566 flags.go:64] FLAG: --manifest-url-header="" Jul 09 18:58:49.740737 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733485 2566 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 09 18:58:49.740737 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733488 2566 flags.go:64] FLAG: --max-open-files="1000000" Jul 09 18:58:49.740737 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733492 2566 flags.go:64] FLAG: --max-pods="110" Jul 09 18:58:49.740737 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733495 2566 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 09 18:58:49.741381 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733498 2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 09 18:58:49.741381 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733501 2566 flags.go:64] FLAG: --memory-manager-policy="None" Jul 09 18:58:49.741381 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733504 2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 09 18:58:49.741381 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733506 2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 09 18:58:49.741381 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733509 2566 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 09 18:58:49.741381 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733514 2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 09 18:58:49.741381 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733521 2566 flags.go:64] FLAG: --node-status-max-images="50" Jul 09 18:58:49.741381 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733524 2566 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 09 18:58:49.741381 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733527 2566 flags.go:64] FLAG: --oom-score-adj="-999" Jul 09 18:58:49.741381 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733530 2566 flags.go:64] FLAG: --pod-cidr="" Jul 09 18:58:49.741381 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733533 2566 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 09 18:58:49.741381 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733538 2566 flags.go:64] FLAG: --pod-manifest-path="" Jul 09 18:58:49.741381 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733541 2566 flags.go:64] FLAG: --pod-max-pids="-1" Jul 09 18:58:49.741381 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733545 2566 flags.go:64] FLAG: --pods-per-core="0" Jul 09 18:58:49.741381 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733548 2566 flags.go:64] FLAG: --port="10250" Jul 09 18:58:49.741381 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733551 2566 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 09 18:58:49.741381 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733554 2566 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a5679159ff57dd73" Jul 09 18:58:49.741381 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733557 2566 flags.go:64] FLAG: --qos-reserved="" Jul 09 18:58:49.741381 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733560 2566 flags.go:64] FLAG: --read-only-port="10255" Jul 09 18:58:49.741381 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733564 2566 flags.go:64] FLAG: --register-node="true" Jul 09 18:58:49.741381 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733566 2566 flags.go:64] FLAG: --register-with-taints="" Jul 09 18:58:49.741381 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733570 2566 flags.go:64] FLAG: --registry-burst="10" Jul 09 18:58:49.741381 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733573 2566 flags.go:64] FLAG: --registry-qps="5" Jul 09 18:58:49.741381 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733576 2566 flags.go:64] FLAG: --reserved-cpus="" Jul 09 18:58:49.741956 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733579 2566 flags.go:64] FLAG: --reserved-memory="" Jul 09 18:58:49.741956 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733582 2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 09 18:58:49.741956 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733585 2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 09 18:58:49.741956 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733588 2566 flags.go:64] FLAG: --rotate-certificates="false" Jul 09 18:58:49.741956 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733591 2566 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 09 18:58:49.741956 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733594 2566 flags.go:64] FLAG: --runonce="false" Jul 09 18:58:49.741956 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733597 2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 09 18:58:49.741956 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733600 2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 09 18:58:49.741956 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733602 2566 flags.go:64] FLAG: --seccomp-default="false" Jul 09 18:58:49.741956 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733605 2566 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 09 18:58:49.741956 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733608 2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 09 18:58:49.741956 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733611 2566 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 09 18:58:49.741956 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733614 2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 09 18:58:49.741956 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733617 2566 flags.go:64] FLAG: --storage-driver-password="root" Jul 09 18:58:49.741956 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733622 2566 flags.go:64] FLAG: --storage-driver-secure="false" Jul 09 18:58:49.741956 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733625 2566 flags.go:64] FLAG: --storage-driver-table="stats" Jul 09 18:58:49.741956 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733628 2566 flags.go:64] FLAG: --storage-driver-user="root" Jul 09 18:58:49.741956 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733631 2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 09 18:58:49.741956 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733634 2566 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 09 18:58:49.741956 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733638 2566 flags.go:64] FLAG: --system-cgroups="" Jul 09 18:58:49.741956 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733641 2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 09 18:58:49.741956 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733646 2566 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 09 18:58:49.741956 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733649 2566 flags.go:64] FLAG: --tls-cert-file="" Jul 09 18:58:49.741956 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733652 2566 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 09 18:58:49.741956 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733655 2566 flags.go:64] FLAG: --tls-min-version="" Jul 09 18:58:49.742564 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733658 2566 flags.go:64] FLAG: --tls-private-key-file="" Jul 09 18:58:49.742564 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733661 2566 flags.go:64] FLAG: --topology-manager-policy="none" Jul 09 18:58:49.742564 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733664 2566 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 09 18:58:49.742564 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733667 2566 flags.go:64] FLAG: --topology-manager-scope="container" Jul 09 18:58:49.742564 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733670 2566 flags.go:64] FLAG: --v="2" Jul 09 18:58:49.742564 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733673 2566 flags.go:64] FLAG: --version="false" Jul 09 18:58:49.742564 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733678 2566 flags.go:64] FLAG: --vmodule="" Jul 09 18:58:49.742564 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733682 2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 09 18:58:49.742564 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.733685 2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 09 18:58:49.742564 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733799 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 18:58:49.742564 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733803 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 18:58:49.742564 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733806 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 18:58:49.742564 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733809 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 18:58:49.742564 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733812 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 18:58:49.742564 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733814 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 18:58:49.742564 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733817 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 18:58:49.742564 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733819 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 18:58:49.742564 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733822 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 18:58:49.742564 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733825 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 18:58:49.742564 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733827 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 18:58:49.742564 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733830 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 18:58:49.742564 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733834 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 18:58:49.743127 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733837 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 18:58:49.743127 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733840 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 18:58:49.743127 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733843 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 18:58:49.743127 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733846 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 18:58:49.743127 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733848 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 18:58:49.743127 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733851 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 18:58:49.743127 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733854 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 18:58:49.743127 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733857 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 18:58:49.743127 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733860 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 18:58:49.743127 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733862 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 18:58:49.743127 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733865 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 18:58:49.743127 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733867 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 18:58:49.743127 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733872 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 18:58:49.743127 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733875 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 18:58:49.743127 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733879 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 18:58:49.743127 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733881 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 18:58:49.743127 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733885 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 18:58:49.743127 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733888 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 18:58:49.743127 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733890 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 18:58:49.743636 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733893 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 18:58:49.743636 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733895 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 18:58:49.743636 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733898 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 18:58:49.743636 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733901 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 18:58:49.743636 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733903 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 18:58:49.743636 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733906 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 09 18:58:49.743636 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733908 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 18:58:49.743636 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733910 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 18:58:49.743636 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733913 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 18:58:49.743636 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733915 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 18:58:49.743636 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733918 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 18:58:49.743636 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733921 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 18:58:49.743636 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733926 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 18:58:49.743636 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733931 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 18:58:49.743636 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733933 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 18:58:49.743636 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733936 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 18:58:49.743636 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733939 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 18:58:49.743636 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733941 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 18:58:49.743636 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733944 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 18:58:49.743636 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733947 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 18:58:49.744121 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733950 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 18:58:49.744121 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733953 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 18:58:49.744121 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733955 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 18:58:49.744121 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733958 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 18:58:49.744121 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733960 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 18:58:49.744121 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733963 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 18:58:49.744121 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733966 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 18:58:49.744121 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733968 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 18:58:49.744121 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733971 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 18:58:49.744121 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733974 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 18:58:49.744121 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733976 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 18:58:49.744121 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733979 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 18:58:49.744121 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733981 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 18:58:49.744121 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733983 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 18:58:49.744121 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733986 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 18:58:49.744121 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733988 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 18:58:49.744121 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733991 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 18:58:49.744121 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733993 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 18:58:49.744121 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733996 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 18:58:49.744594 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.733998 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 18:58:49.744594 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.734001 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 18:58:49.744594 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.734003 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 18:58:49.744594 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.734006 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 18:58:49.744594 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.734009 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 18:58:49.744594 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.734013 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 18:58:49.744594 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.734017 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 18:58:49.744594 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.734021 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 18:58:49.744594 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.734023 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 18:58:49.744594 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.734026 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 18:58:49.744594 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.734029 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 18:58:49.744594 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.734032 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 18:58:49.744594 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.734035 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 18:58:49.744594 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.734041 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 18:58:49.744594 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.734044 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 18:58:49.744594 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.734046 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 18:58:49.744594 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.734049 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 18:58:49.744594 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.734052 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 18:58:49.744594 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.734056 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 18:58:49.744594 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.734058 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 18:58:49.745068 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.734061 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 18:58:49.745068 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.734064 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 18:58:49.745068 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.734067 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 18:58:49.745068 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.734069 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 18:58:49.745068 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.734072 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 18:58:49.745068 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.734074 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 18:58:49.745068 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.734077 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 18:58:49.745068 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.734080 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 18:58:49.745068 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.734086 2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 18:58:49.745068 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.741152 2566 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 09 18:58:49.745068 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.741169 2566 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 09 18:58:49.745068 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.742609 2566 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 09 18:58:49.745068 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.742617 2566 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 09 18:58:49.745068 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742674 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 18:58:49.745068 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742680 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 18:58:49.745551 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742683 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 18:58:49.745551 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742686 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 18:58:49.745551 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742689 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 18:58:49.745551 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742692 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 18:58:49.745551 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742696 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 18:58:49.745551 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742700 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 18:58:49.745551 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742704 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 18:58:49.745551 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742707 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 18:58:49.745551 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742709 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 18:58:49.745551 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742712 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 18:58:49.745551 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742715 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 18:58:49.745551 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742718 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 18:58:49.745551 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742721 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 18:58:49.745551 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742733 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 18:58:49.745551 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742737 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 18:58:49.745551 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742740 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 18:58:49.745551 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742743 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 18:58:49.745551 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742746 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 18:58:49.745551 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742748 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 18:58:49.746010 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742751 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 18:58:49.746010 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742754 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 18:58:49.746010 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742756 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 18:58:49.746010 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742759 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 18:58:49.746010 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742762 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 18:58:49.746010 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742765 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 18:58:49.746010 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742768 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 18:58:49.746010 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742770 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 18:58:49.746010 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742779 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 18:58:49.746010 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742782 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 18:58:49.746010 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742784 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 18:58:49.746010 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742789 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 18:58:49.746010 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742792 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 18:58:49.746010 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742795 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 18:58:49.746010 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742798 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 18:58:49.746010 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742801 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 18:58:49.746010 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742803 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 18:58:49.746010 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742806 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 18:58:49.746010 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742808 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 18:58:49.746478 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742811 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 18:58:49.746478 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742813 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 18:58:49.746478 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742816 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 18:58:49.746478 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742818 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 18:58:49.746478 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742821 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 18:58:49.746478 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742825 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 18:58:49.746478 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742828 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 18:58:49.746478 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742830 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 18:58:49.746478 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742832 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 18:58:49.746478 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742835 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 18:58:49.746478 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742837 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 18:58:49.746478 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742840 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 18:58:49.746478 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742843 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 18:58:49.746478 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742845 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 18:58:49.746478 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742847 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 18:58:49.746478 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742850 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 18:58:49.746478 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742852 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 18:58:49.746478 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742855 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 18:58:49.746478 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742857 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 18:58:49.746478 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742860 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 18:58:49.746983 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742862 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 18:58:49.746983 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742865 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 18:58:49.746983 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742868 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 18:58:49.746983 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742871 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 18:58:49.746983 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742874 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 18:58:49.746983 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742876 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 18:58:49.746983 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742879 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 18:58:49.746983 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742882 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 18:58:49.746983 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742884 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 18:58:49.746983 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742887 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 18:58:49.746983 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742889 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 18:58:49.746983 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742892 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 18:58:49.746983 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742894 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 18:58:49.746983 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742897 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 18:58:49.746983 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742899 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 18:58:49.746983 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742902 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 18:58:49.746983 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742904 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 18:58:49.746983 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742907 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 18:58:49.746983 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742910 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 18:58:49.746983 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742912 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 18:58:49.747475 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742915 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 18:58:49.747475 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742917 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 18:58:49.747475 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742919 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 18:58:49.747475 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742922 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 18:58:49.747475 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742924 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 18:58:49.747475 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742927 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 18:58:49.747475 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742929 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 18:58:49.747475 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742931 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 18:58:49.747475 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742934 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 18:58:49.747475 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742937 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 18:58:49.747475 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742939 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 09 18:58:49.747475 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742941 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 18:58:49.747475 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742944 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 18:58:49.747475 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742947 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 18:58:49.747475 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742949 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 18:58:49.747475 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742952 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 18:58:49.747475 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742955 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 18:58:49.747475 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742959 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 18:58:49.747475 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.742962 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 18:58:49.747944 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.742968 2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 18:58:49.747944 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743076 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 18:58:49.747944 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743081 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 18:58:49.747944 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743084 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 18:58:49.747944 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743087 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 18:58:49.747944 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743089 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 18:58:49.747944 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743092 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 18:58:49.747944 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743095 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 18:58:49.747944 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743097 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 18:58:49.747944 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743100 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 18:58:49.747944 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743103 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 18:58:49.747944 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743105 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 18:58:49.747944 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743108 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 18:58:49.747944 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743110 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 18:58:49.747944 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743113 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 18:58:49.748328 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743115 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 18:58:49.748328 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743118 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 18:58:49.748328 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743120 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 18:58:49.748328 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743123 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 18:58:49.748328 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743125 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 18:58:49.748328 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743128 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 18:58:49.748328 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743130 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 18:58:49.748328 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743134 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 18:58:49.748328 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743137 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 18:58:49.748328 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743139 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 18:58:49.748328 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743142 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 18:58:49.748328 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743144 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 18:58:49.748328 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743147 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 18:58:49.748328 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743150 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 18:58:49.748328 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743153 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 18:58:49.748328 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743155 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 18:58:49.748328 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743158 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 18:58:49.748328 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743160 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 18:58:49.748328 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743163 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 18:58:49.748328 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743165 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 18:58:49.748328 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743168 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 18:58:49.748835 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743170 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 18:58:49.748835 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743173 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 18:58:49.748835 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743175 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 18:58:49.748835 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743177 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 18:58:49.748835 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743180 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 18:58:49.748835 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743183 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 09 18:58:49.748835 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743186 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 18:58:49.748835 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743188 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 18:58:49.748835 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743191 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 18:58:49.748835 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743194 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 18:58:49.748835 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743196 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 18:58:49.748835 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743199 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 18:58:49.748835 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743201 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 18:58:49.748835 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743204 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 18:58:49.748835 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743206 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 18:58:49.748835 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743209 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 18:58:49.748835 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743211 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 18:58:49.748835 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743214 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 18:58:49.748835 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743216 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 18:58:49.748835 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743218 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 18:58:49.749336 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743221 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 18:58:49.749336 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743223 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 18:58:49.749336 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743225 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 18:58:49.749336 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743228 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 18:58:49.749336 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743231 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 18:58:49.749336 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743234 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 18:58:49.749336 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743237 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 18:58:49.749336 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743239 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 18:58:49.749336 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743242 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 18:58:49.749336 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743245 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 18:58:49.749336 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743249 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 18:58:49.749336 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743253 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 18:58:49.749336 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743256 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 18:58:49.749336 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743259 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 18:58:49.749336 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743261 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 18:58:49.749336 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743264 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 18:58:49.749336 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743267 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 18:58:49.749336 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743269 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 18:58:49.749784 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743272 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 18:58:49.749784 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743274 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 18:58:49.749784 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743276 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 18:58:49.749784 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743279 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 18:58:49.749784 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743281 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 18:58:49.749784 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743284 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 18:58:49.749784 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743286 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 18:58:49.749784 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743289 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 18:58:49.749784 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743291 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 18:58:49.749784 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743293 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 18:58:49.749784 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743296 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 18:58:49.749784 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743298 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 18:58:49.749784 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743301 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 18:58:49.749784 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743303 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 18:58:49.749784 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743305 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 18:58:49.749784 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743319 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 18:58:49.749784 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743322 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 18:58:49.749784 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743324 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 18:58:49.749784 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743344 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 18:58:49.750228 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743349 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 18:58:49.750228 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743352 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 18:58:49.750228 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743356 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 18:58:49.750228 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743359 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 18:58:49.750228 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743362 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 18:58:49.750228 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743365 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 18:58:49.750228 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:49.743368 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 18:58:49.750228 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.743372 2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 18:58:49.750228 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.743475 2566 server.go:962] "Client rotation is on, will bootstrap in background" Jul 09 18:58:49.750228 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.746270 2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 09 18:58:49.750228 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.747760 2566 server.go:1019] "Starting client certificate rotation" Jul 09 18:58:49.750228 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.748235 2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 09 18:58:49.750228 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.748265 2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 09 18:58:49.753078 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.752656 2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 09 18:58:49.757171 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:49.757153 2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 09 18:58:50.837823 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.837804 2566 log.go:25] "Validated CRI v1 runtime API" Jul 09 18:58:50.840197 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.840181 2566 log.go:25] "Validated CRI v1 image API" Jul 09 18:58:50.842284 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.842257 2566 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 09 18:58:50.844810 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.844790 2566 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 cdccca8b-de69-4097-80f2-d1d22fa509c5:/dev/nvme0n1p3 f678e80d-63e2-41f7-8f4c-3577f9238f12:/dev/nvme0n1p4] Jul 09 18:58:50.844876 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.844809 2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 09 18:58:50.849373 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.849355 2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 09 18:58:50.851267 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.851165 2566 manager.go:216] Machine: {Timestamp:2026-07-09 18:58:50.849704709 +0000 UTC m=+0.357484474 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3088313 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec210a4786c9da935162eee4da5829f0 SystemUUID:ec210a47-86c9-da93-5162-eee4da5829f0 BootID:5ccb950f-36f3-4cf0-b37d-74cf9443d065 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:fb:a3:33:57:6f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:fb:a3:33:57:6f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e6:6b:66:77:83:6a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 09 18:58:50.851267 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.851265 2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 09 18:58:50.851379 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.851345 2566 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 09 18:58:50.854932 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.854904 2566 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 09 18:58:50.855077 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.854934 2566 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-153.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 09 18:58:50.856585 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.856575 2566 topology_manager.go:138] "Creating topology manager with none policy" Jul 09 18:58:50.856632 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.856586 2566 container_manager_linux.go:309] "Creating device plugin manager" Jul 09 18:58:50.856632 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.856594 2566 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 09 18:58:50.857292 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.857282 2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 09 18:58:50.858239 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.858229 2566 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 09 18:58:50.861312 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.861300 2566 state_mem.go:36] "Initialized new in-memory state store" Jul 09 18:58:50.861425 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.861416 2566 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 09 18:58:50.861457 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.861446 2566 kubelet.go:486] "Attempting to sync node with API server" Jul 09 18:58:50.861457 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.861455 2566 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 09 18:58:50.861511 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.861471 2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 09 18:58:50.861511 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.861481 2566 kubelet.go:398] "Adding apiserver pod source" Jul 09 18:58:50.861511 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.861491 2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 09 18:58:50.861866 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.861852 2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 09 18:58:50.861924 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.861874 2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 09 18:58:50.864354 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.864337 2566 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 09 18:58:50.865758 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.865744 2566 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 09 18:58:50.865819 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.865773 2566 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 09 18:58:50.866707 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.866685 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 09 18:58:50.866707 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.866710 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 09 18:58:50.866810 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.866716 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 09 18:58:50.866810 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.866722 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 09 18:58:50.866810 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.866731 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 09 18:58:50.866810 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.866736 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 09 18:58:50.866810 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.866798 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 09 18:58:50.866810 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.866804 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 09 18:58:50.866995 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.866815 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 09 18:58:50.866995 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.866821 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 09 18:58:50.866995 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.866830 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 09 18:58:50.866995 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.866837 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 09 18:58:50.868359 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.868345 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 09 18:58:50.868411 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.868361 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 09 18:58:50.871232 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.871181 2566 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-153.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 09 18:58:50.871634 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.871599 2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5gr27" Jul 09 18:58:50.872058 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:50.872029 2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 09 18:58:50.872102 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:50.872032 2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-153.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 09 18:58:50.873014 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.872999 2566 server.go:1268] "Started kubelet" Jul 09 18:58:50.873143 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.873097 2566 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 09 18:58:50.873279 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.873134 2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 09 18:58:50.873369 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.873299 2566 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 09 18:58:50.873902 ip-10-0-140-153 systemd[1]: Started Kubernetes Kubelet. Jul 09 18:58:50.875875 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.875858 2566 server.go:310] "Adding debug handlers to kubelet server" Jul 09 18:58:50.876904 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.876886 2566 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 09 18:58:50.878752 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:50.878728 2566 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 09 18:58:50.878915 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.878903 2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5gr27" Jul 09 18:58:50.879242 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:50.878145 2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-153.ec2.internal.18c0b426bab61124 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-153.ec2.internal,UID:ip-10-0-140-153.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-153.ec2.internal,},FirstTimestamp:2026-07-09 18:58:50.87296746 +0000 UTC m=+0.380747226,LastTimestamp:2026-07-09 18:58:50.87296746 +0000 UTC m=+0.380747226,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-153.ec2.internal,}" Jul 09 18:58:50.881569 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.881554 2566 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 09 18:58:50.881642 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.881557 2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 09 18:58:50.881776 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:50.881761 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-153.ec2.internal\" not found" Jul 09 18:58:50.882438 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.882418 2566 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 09 18:58:50.882438 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.882440 2566 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 09 18:58:50.882600 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.882420 2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 09 18:58:50.882600 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.882550 2566 reconstruct.go:97] "Volume reconstruction finished" Jul 09 18:58:50.882600 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.882558 2566 reconciler.go:29] "Reconciler: start to sync state" Jul 09 18:58:50.882600 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.882584 2566 factory.go:153] Registering CRI-O factory Jul 09 18:58:50.882600 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.882598 2566 factory.go:223] Registration of the crio container factory successfully Jul 09 18:58:50.882783 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.882665 2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 09 18:58:50.882783 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.882674 2566 factory.go:55] Registering systemd factory Jul 09 18:58:50.882783 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.882680 2566 factory.go:223] Registration of the systemd container factory successfully Jul 09 18:58:50.882783 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.882699 2566 factory.go:103] Registering Raw factory Jul 09 18:58:50.882783 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.882713 2566 manager.go:1196] Started watching for new ooms in manager Jul 09 18:58:50.883401 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.883389 2566 manager.go:349] Starting recovery of all containers Jul 09 18:58:50.893365 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.893254 2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 18:58:50.893977 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.893949 2566 manager.go:354] Recovery completed Jul 09 18:58:50.896751 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:50.896523 2566 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-140-153.ec2.internal\" not found" node="ip-10-0-140-153.ec2.internal" Jul 09 18:58:50.899839 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.899827 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 18:58:50.901635 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.901620 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-153.ec2.internal" event="NodeHasSufficientMemory" Jul 09 18:58:50.901694 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.901649 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-153.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 18:58:50.901694 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.901660 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-153.ec2.internal" event="NodeHasSufficientPID" Jul 09 18:58:50.902179 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.902166 2566 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 09 18:58:50.902179 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.902179 2566 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 09 18:58:50.902257 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.902197 2566 state_mem.go:36] "Initialized new in-memory state store" Jul 09 18:58:50.902257 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.902199 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 18:58:50.902856 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.902840 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-153.ec2.internal" event="NodeHasSufficientMemory" Jul 09 18:58:50.902896 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.902865 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-153.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 18:58:50.902896 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.902875 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-153.ec2.internal" event="NodeHasSufficientPID" Jul 09 18:58:50.903514 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.903503 2566 policy_none.go:49] "None policy: Start" Jul 09 18:58:50.903545 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.903517 2566 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 09 18:58:50.903545 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.903527 2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 09 18:58:50.904560 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.904549 2566 policy_none.go:47] "Start" Jul 09 18:58:50.944197 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.944174 2566 manager.go:337] "Starting Device Plugin manager" Jul 09 18:58:50.944276 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:50.944213 2566 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 09 18:58:50.944276 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.944227 2566 server.go:87] "Starting device plugin registration server" Jul 09 18:58:50.944486 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.944475 2566 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 09 18:58:50.944527 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.944496 2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 09 18:58:50.944636 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.944616 2566 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 09 18:58:50.944736 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.944715 2566 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 09 18:58:50.944736 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:50.944728 2566 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 09 18:58:50.945332 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:50.945312 2566 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 09 18:58:50.945403 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:50.945367 2566 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-153.ec2.internal\" not found" Jul 09 18:58:51.045045 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.044982 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 18:58:51.046553 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.045908 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-153.ec2.internal" event="NodeHasSufficientMemory" Jul 09 18:58:51.046553 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.045934 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-153.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 18:58:51.046553 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.045943 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-153.ec2.internal" event="NodeHasSufficientPID" Jul 09 18:58:51.046553 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.045978 2566 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-153.ec2.internal" Jul 09 18:58:51.049394 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.049370 2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 09 18:58:51.050600 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.050585 2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 09 18:58:51.050662 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.050634 2566 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 09 18:58:51.050662 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.050658 2566 kubelet.go:2448] "Starting kubelet main sync loop" Jul 09 18:58:51.050725 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:51.050693 2566 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 09 18:58:51.052633 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.052599 2566 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-153.ec2.internal" Jul 09 18:58:51.052713 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:51.052635 2566 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-153.ec2.internal\": node \"ip-10-0-140-153.ec2.internal\" not found" Jul 09 18:58:51.053635 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.053600 2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 18:58:51.111054 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:51.111026 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-153.ec2.internal\" not found" Jul 09 18:58:51.151129 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.151094 2566 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-140-153.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-153.ec2.internal"] Jul 09 18:58:51.151206 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.151182 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 18:58:51.153313 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.153295 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-153.ec2.internal" event="NodeHasSufficientMemory" Jul 09 18:58:51.153402 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.153324 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-153.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 18:58:51.153402 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.153334 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-153.ec2.internal" event="NodeHasSufficientPID" Jul 09 18:58:51.154587 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.154575 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 18:58:51.154693 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.154678 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-153.ec2.internal" Jul 09 18:58:51.154743 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.154709 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 18:58:51.155383 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.155362 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-153.ec2.internal" event="NodeHasSufficientMemory" Jul 09 18:58:51.155491 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.155389 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-153.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 18:58:51.155491 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.155398 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-153.ec2.internal" event="NodeHasSufficientPID" Jul 09 18:58:51.155491 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.155370 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-153.ec2.internal" event="NodeHasSufficientMemory" Jul 09 18:58:51.155491 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.155469 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-153.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 18:58:51.155491 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.155483 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-153.ec2.internal" event="NodeHasSufficientPID" Jul 09 18:58:51.156535 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.156512 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-153.ec2.internal" Jul 09 18:58:51.156589 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.156546 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 18:58:51.157216 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.157198 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-153.ec2.internal" event="NodeHasSufficientMemory" Jul 09 18:58:51.157300 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.157229 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-153.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 18:58:51.157300 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.157246 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-153.ec2.internal" event="NodeHasSufficientPID" Jul 09 18:58:51.183963 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.183934 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2bd4f2bfb833729e4a5ed7d4607ff8c1-config\") pod \"kube-apiserver-proxy-ip-10-0-140-153.ec2.internal\" (UID: \"2bd4f2bfb833729e4a5ed7d4607ff8c1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-153.ec2.internal" Jul 09 18:58:51.184095 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.183966 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dbfbad2bcd3a64e3008a9009f32541a5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-153.ec2.internal\" (UID: \"dbfbad2bcd3a64e3008a9009f32541a5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-153.ec2.internal" Jul 09 18:58:51.184095 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.183994 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dbfbad2bcd3a64e3008a9009f32541a5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-153.ec2.internal\" (UID: \"dbfbad2bcd3a64e3008a9009f32541a5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-153.ec2.internal" Jul 09 18:58:51.190405 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:51.190377 2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-153.ec2.internal\" not found" node="ip-10-0-140-153.ec2.internal" Jul 09 18:58:51.194714 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:51.194698 2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-153.ec2.internal\" not found" node="ip-10-0-140-153.ec2.internal" Jul 09 18:58:51.211581 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:51.211562 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-153.ec2.internal\" not found" Jul 09 18:58:51.284734 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.284711 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2bd4f2bfb833729e4a5ed7d4607ff8c1-config\") pod \"kube-apiserver-proxy-ip-10-0-140-153.ec2.internal\" (UID: \"2bd4f2bfb833729e4a5ed7d4607ff8c1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-153.ec2.internal" Jul 09 18:58:51.284871 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.284741 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dbfbad2bcd3a64e3008a9009f32541a5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-153.ec2.internal\" (UID: \"dbfbad2bcd3a64e3008a9009f32541a5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-153.ec2.internal" Jul 09 18:58:51.284871 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.284759 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dbfbad2bcd3a64e3008a9009f32541a5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-153.ec2.internal\" (UID: \"dbfbad2bcd3a64e3008a9009f32541a5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-153.ec2.internal" Jul 09 18:58:51.284871 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.284812 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dbfbad2bcd3a64e3008a9009f32541a5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-153.ec2.internal\" (UID: \"dbfbad2bcd3a64e3008a9009f32541a5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-153.ec2.internal" Jul 09 18:58:51.284871 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.284813 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2bd4f2bfb833729e4a5ed7d4607ff8c1-config\") pod \"kube-apiserver-proxy-ip-10-0-140-153.ec2.internal\" (UID: \"2bd4f2bfb833729e4a5ed7d4607ff8c1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-153.ec2.internal" Jul 09 18:58:51.284871 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.284828 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dbfbad2bcd3a64e3008a9009f32541a5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-153.ec2.internal\" (UID: \"dbfbad2bcd3a64e3008a9009f32541a5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-153.ec2.internal" Jul 09 18:58:51.311883 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:51.311821 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-153.ec2.internal\" not found" Jul 09 18:58:51.412601 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:51.412561 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-153.ec2.internal\" not found" Jul 09 18:58:51.493047 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.493025 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-153.ec2.internal" Jul 09 18:58:51.497924 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.497902 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-153.ec2.internal" Jul 09 18:58:51.513665 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:51.513643 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-153.ec2.internal\" not found" Jul 09 18:58:51.614155 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:51.614060 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-153.ec2.internal\" not found" Jul 09 18:58:51.688890 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.688865 2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 18:58:51.714206 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:51.714179 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-153.ec2.internal\" not found" Jul 09 18:58:51.790507 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.790481 2566 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 18:58:51.813996 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.813971 2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 09 18:58:51.814511 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.814095 2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 09 18:58:51.814511 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.814129 2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 09 18:58:51.814511 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.814144 2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 09 18:58:51.814511 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.814151 2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 09 18:58:51.862141 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.862117 2566 apiserver.go:52] "Watching apiserver" Jul 09 18:58:51.880776 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.880718 2566 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 09 18:58:51.881503 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.881467 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-c2bn7","openshift-cluster-node-tuning-operator/tuned-hrjdx","openshift-dns/node-resolver-x8k65","openshift-multus/multus-additional-cni-plugins-swdmr","openshift-multus/network-metrics-daemon-vtc8p","openshift-network-diagnostics/network-check-target-mtz87","openshift-network-operator/iptables-alerter-km5nl","openshift-ovn-kubernetes/ovnkube-node-6dh7g","kube-system/global-pull-secret-syncer-tqtv5","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcfcz","openshift-image-registry/node-ca-jnt6g","openshift-multus/multus-5mwmg"] Jul 09 18:58:51.882044 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.882006 2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-153.ec2.internal" Jul 09 18:58:51.882159 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.882061 2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 09 18:58:51.883223 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.883203 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-c2bn7" Jul 09 18:58:51.884010 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.883710 2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-08 18:53:50 +0000 UTC" deadline="2028-02-12 03:35:21.568184707 +0000 UTC" Jul 09 18:58:51.884010 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.883756 2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13976h36m29.684432439s" Jul 09 18:58:51.884634 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.884599 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hrjdx" Jul 09 18:58:51.885675 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.885657 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-x8k65" Jul 09 18:58:51.886256 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.886220 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 09 18:58:51.886388 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.886370 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 09 18:58:51.886474 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.886418 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-gkt44\"" Jul 09 18:58:51.886711 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.886694 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-swdmr" Jul 09 18:58:51.887119 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.887103 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 09 18:58:51.887119 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.887114 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-sj4v7\"" Jul 09 18:58:51.887761 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.887713 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/51b7fa90-97cc-4b69-80d4-d63ea5f30d6f-agent-certs\") pod \"konnectivity-agent-c2bn7\" (UID: \"51b7fa90-97cc-4b69-80d4-d63ea5f30d6f\") " pod="kube-system/konnectivity-agent-c2bn7" Jul 09 18:58:51.887761 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.887741 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/51b7fa90-97cc-4b69-80d4-d63ea5f30d6f-konnectivity-ca\") pod \"konnectivity-agent-c2bn7\" (UID: \"51b7fa90-97cc-4b69-80d4-d63ea5f30d6f\") " pod="kube-system/konnectivity-agent-c2bn7" Jul 09 18:58:51.887902 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.887719 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 09 18:58:51.887902 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.887898 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtc8p" Jul 09 18:58:51.888025 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:51.887962 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtc8p" podUID="30cf5feb-cf9a-40c0-8a6a-de8fc6c60506" Jul 09 18:58:51.888325 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.888313 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 09 18:58:51.888409 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.888393 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 09 18:58:51.888459 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.888393 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7s27s\"" Jul 09 18:58:51.889044 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.889028 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mtz87" Jul 09 18:58:51.889128 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:51.889090 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mtz87" podUID="2fe73719-af90-4093-ae73-d4e462e5e43c" Jul 09 18:58:51.889190 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.889134 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 09 18:58:51.889231 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.889193 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 09 18:58:51.889816 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.889800 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 09 18:58:51.889875 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.889854 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-65lmx\"" Jul 09 18:58:51.889920 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.889880 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 09 18:58:51.890081 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.890059 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 09 18:58:51.890156 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.890086 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-km5nl" Jul 09 18:58:51.891465 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.891450 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:51.892531 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.892515 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tqtv5" Jul 09 18:58:51.892629 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:51.892596 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tqtv5" podUID="7435737c-5954-4ef0-8b41-1889b8231561" Jul 09 18:58:51.892906 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.892889 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 09 18:58:51.892977 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.892925 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 09 18:58:51.893034 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.892984 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ftp8m\"" Jul 09 18:58:51.893205 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.893192 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 09 18:58:51.893675 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.893663 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcfcz" Jul 09 18:58:51.894833 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.894818 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jnt6g" Jul 09 18:58:51.895694 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.895680 2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 09 18:58:51.895758 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.895732 2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-153.ec2.internal" Jul 09 18:58:51.895909 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.895892 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-140-153.ec2.internal"] Jul 09 18:58:51.895961 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.895951 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5mwmg" Jul 09 18:58:51.897216 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.897203 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 09 18:58:51.898936 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.898918 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 09 18:58:51.899023 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.898942 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-htlzl\"" Jul 09 18:58:51.899023 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.898973 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 09 18:58:51.899107 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.899036 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 09 18:58:51.899183 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.899169 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 09 18:58:51.905106 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.905085 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 09 18:58:51.905238 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.905125 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 09 18:58:51.905238 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.905172 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 09 18:58:51.905424 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.905411 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 09 18:58:51.905465 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.905448 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jr259\"" Jul 09 18:58:51.905648 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.905628 2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 09 18:58:51.905747 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.905664 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 09 18:58:51.905747 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.905678 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 09 18:58:51.905747 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.905696 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 09 18:58:51.905884 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.905845 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-svrl4\"" Jul 09 18:58:51.905929 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.905891 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 09 18:58:51.906141 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.906125 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-krgmb\"" Jul 09 18:58:51.936008 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.935659 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-153.ec2.internal"] Jul 09 18:58:51.936008 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.935666 2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 09 18:58:51.983532 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.983514 2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 09 18:58:51.987886 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.987862 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47f9bbdb-bfc1-464a-a495-b310124b104b-etc-kubernetes\") pod \"tuned-hrjdx\" (UID: \"47f9bbdb-bfc1-464a-a495-b310124b104b\") " pod="openshift-cluster-node-tuning-operator/tuned-hrjdx" Jul 09 18:58:51.988027 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.987897 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/db233460-134d-4d9d-8da9-31d89366a4e0-ovnkube-script-lib\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:51.988027 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.987923 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dqrd\" (UniqueName: \"kubernetes.io/projected/b1f91f8e-9064-4538-a912-6e6354050534-kube-api-access-4dqrd\") pod \"aws-ebs-csi-driver-node-gcfcz\" (UID: \"b1f91f8e-9064-4538-a912-6e6354050534\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcfcz" Jul 09 18:58:51.988027 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.987947 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e5c65950-27b0-41de-8ee1-e2ced769d6b4-host-var-lib-cni-multus\") pod \"multus-5mwmg\" (UID: \"e5c65950-27b0-41de-8ee1-e2ced769d6b4\") " pod="openshift-multus/multus-5mwmg" Jul 09 18:58:51.988027 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.987971 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e5c65950-27b0-41de-8ee1-e2ced769d6b4-hostroot\") pod \"multus-5mwmg\" (UID: \"e5c65950-27b0-41de-8ee1-e2ced769d6b4\") " pod="openshift-multus/multus-5mwmg" Jul 09 18:58:51.988027 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.987993 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/47f9bbdb-bfc1-464a-a495-b310124b104b-etc-sysctl-d\") pod \"tuned-hrjdx\" (UID: \"47f9bbdb-bfc1-464a-a495-b310124b104b\") " pod="openshift-cluster-node-tuning-operator/tuned-hrjdx" Jul 09 18:58:51.988027 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.988015 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/db52b7ec-a201-4d65-81b3-56e37f0b6d36-tuning-conf-dir\") pod \"multus-additional-cni-plugins-swdmr\" (UID: \"db52b7ec-a201-4d65-81b3-56e37f0b6d36\") " pod="openshift-multus/multus-additional-cni-plugins-swdmr" Jul 09 18:58:51.988319 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.988037 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d40ecb0c-9c3c-4fbe-ad50-742b0f848c44-host-slash\") pod \"iptables-alerter-km5nl\" (UID: \"d40ecb0c-9c3c-4fbe-ad50-742b0f848c44\") " pod="openshift-network-operator/iptables-alerter-km5nl" Jul 09 18:58:51.988319 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.988060 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b1f91f8e-9064-4538-a912-6e6354050534-device-dir\") pod \"aws-ebs-csi-driver-node-gcfcz\" (UID: \"b1f91f8e-9064-4538-a912-6e6354050534\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcfcz" Jul 09 18:58:51.988319 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.988087 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bndfr\" (UniqueName: \"kubernetes.io/projected/0da71879-5de7-47a0-819a-0b1951b1e817-kube-api-access-bndfr\") pod \"node-resolver-x8k65\" (UID: \"0da71879-5de7-47a0-819a-0b1951b1e817\") " pod="openshift-dns/node-resolver-x8k65" Jul 09 18:58:51.988319 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.988110 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/db233460-134d-4d9d-8da9-31d89366a4e0-run-ovn\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:51.988319 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.988132 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/db233460-134d-4d9d-8da9-31d89366a4e0-host-cni-bin\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:51.988319 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.988154 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e5c65950-27b0-41de-8ee1-e2ced769d6b4-multus-socket-dir-parent\") pod \"multus-5mwmg\" (UID: \"e5c65950-27b0-41de-8ee1-e2ced769d6b4\") " pod="openshift-multus/multus-5mwmg" Jul 09 18:58:51.988319 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.988178 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e5c65950-27b0-41de-8ee1-e2ced769d6b4-host-run-netns\") pod \"multus-5mwmg\" (UID: \"e5c65950-27b0-41de-8ee1-e2ced769d6b4\") " pod="openshift-multus/multus-5mwmg" Jul 09 18:58:51.988319 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.988201 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5c65950-27b0-41de-8ee1-e2ced769d6b4-host-var-lib-kubelet\") pod \"multus-5mwmg\" (UID: \"e5c65950-27b0-41de-8ee1-e2ced769d6b4\") " pod="openshift-multus/multus-5mwmg" Jul 09 18:58:51.988319 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.988227 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/db233460-134d-4d9d-8da9-31d89366a4e0-log-socket\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:51.988319 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.988250 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7435737c-5954-4ef0-8b41-1889b8231561-original-pull-secret\") pod \"global-pull-secret-syncer-tqtv5\" (UID: \"7435737c-5954-4ef0-8b41-1889b8231561\") " pod="kube-system/global-pull-secret-syncer-tqtv5" Jul 09 18:58:51.988319 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.988275 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5c65950-27b0-41de-8ee1-e2ced769d6b4-etc-kubernetes\") pod \"multus-5mwmg\" (UID: \"e5c65950-27b0-41de-8ee1-e2ced769d6b4\") " pod="openshift-multus/multus-5mwmg" Jul 09 18:58:51.988319 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.988297 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf2vh\" (UniqueName: \"kubernetes.io/projected/47f9bbdb-bfc1-464a-a495-b310124b104b-kube-api-access-kf2vh\") pod \"tuned-hrjdx\" (UID: \"47f9bbdb-bfc1-464a-a495-b310124b104b\") " pod="openshift-cluster-node-tuning-operator/tuned-hrjdx" Jul 09 18:58:51.988801 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.988328 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db233460-134d-4d9d-8da9-31d89366a4e0-run-openvswitch\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:51.988801 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.988358 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b1f91f8e-9064-4538-a912-6e6354050534-etc-selinux\") pod \"aws-ebs-csi-driver-node-gcfcz\" (UID: \"b1f91f8e-9064-4538-a912-6e6354050534\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcfcz" Jul 09 18:58:51.988801 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.988378 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0da71879-5de7-47a0-819a-0b1951b1e817-hosts-file\") pod \"node-resolver-x8k65\" (UID: \"0da71879-5de7-47a0-819a-0b1951b1e817\") " pod="openshift-dns/node-resolver-x8k65" Jul 09 18:58:51.988801 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.988405 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/db52b7ec-a201-4d65-81b3-56e37f0b6d36-cnibin\") pod \"multus-additional-cni-plugins-swdmr\" (UID: \"db52b7ec-a201-4d65-81b3-56e37f0b6d36\") " pod="openshift-multus/multus-additional-cni-plugins-swdmr" Jul 09 18:58:51.988801 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.988428 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/51b7fa90-97cc-4b69-80d4-d63ea5f30d6f-agent-certs\") pod \"konnectivity-agent-c2bn7\" (UID: \"51b7fa90-97cc-4b69-80d4-d63ea5f30d6f\") " pod="kube-system/konnectivity-agent-c2bn7" Jul 09 18:58:51.988801 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.988444 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/47f9bbdb-bfc1-464a-a495-b310124b104b-var-lib-kubelet\") pod \"tuned-hrjdx\" (UID: \"47f9bbdb-bfc1-464a-a495-b310124b104b\") " pod="openshift-cluster-node-tuning-operator/tuned-hrjdx" Jul 09 18:58:51.988801 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.988458 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/db233460-134d-4d9d-8da9-31d89366a4e0-host-slash\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:51.988801 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.988472 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e5c65950-27b0-41de-8ee1-e2ced769d6b4-os-release\") pod \"multus-5mwmg\" (UID: \"e5c65950-27b0-41de-8ee1-e2ced769d6b4\") " pod="openshift-multus/multus-5mwmg" Jul 09 18:58:51.988801 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.988486 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/47f9bbdb-bfc1-464a-a495-b310124b104b-etc-tuned\") pod \"tuned-hrjdx\" (UID: \"47f9bbdb-bfc1-464a-a495-b310124b104b\") " pod="openshift-cluster-node-tuning-operator/tuned-hrjdx" Jul 09 18:58:51.988801 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.988500 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/47f9bbdb-bfc1-464a-a495-b310124b104b-tmp\") pod \"tuned-hrjdx\" (UID: \"47f9bbdb-bfc1-464a-a495-b310124b104b\") " pod="openshift-cluster-node-tuning-operator/tuned-hrjdx" Jul 09 18:58:51.988801 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.988515 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/db52b7ec-a201-4d65-81b3-56e37f0b6d36-cni-binary-copy\") pod \"multus-additional-cni-plugins-swdmr\" (UID: \"db52b7ec-a201-4d65-81b3-56e37f0b6d36\") " pod="openshift-multus/multus-additional-cni-plugins-swdmr" Jul 09 18:58:51.988801 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.988529 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/db233460-134d-4d9d-8da9-31d89366a4e0-host-cni-netd\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:51.988801 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.988558 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30cf5feb-cf9a-40c0-8a6a-de8fc6c60506-metrics-certs\") pod \"network-metrics-daemon-vtc8p\" (UID: \"30cf5feb-cf9a-40c0-8a6a-de8fc6c60506\") " pod="openshift-multus/network-metrics-daemon-vtc8p" Jul 09 18:58:51.988801 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.988587 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e5c65950-27b0-41de-8ee1-e2ced769d6b4-multus-daemon-config\") pod \"multus-5mwmg\" (UID: \"e5c65950-27b0-41de-8ee1-e2ced769d6b4\") " pod="openshift-multus/multus-5mwmg" Jul 09 18:58:51.988801 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.988637 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/db52b7ec-a201-4d65-81b3-56e37f0b6d36-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-swdmr\" (UID: \"db52b7ec-a201-4d65-81b3-56e37f0b6d36\") " pod="openshift-multus/multus-additional-cni-plugins-swdmr" Jul 09 18:58:51.988801 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.988664 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d40ecb0c-9c3c-4fbe-ad50-742b0f848c44-iptables-alerter-script\") pod \"iptables-alerter-km5nl\" (UID: \"d40ecb0c-9c3c-4fbe-ad50-742b0f848c44\") " pod="openshift-network-operator/iptables-alerter-km5nl" Jul 09 18:58:51.989376 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.988688 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/db233460-134d-4d9d-8da9-31d89366a4e0-systemd-units\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:51.989376 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.988731 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db233460-134d-4d9d-8da9-31d89366a4e0-var-lib-openvswitch\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:51.989376 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.988766 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db233460-134d-4d9d-8da9-31d89366a4e0-etc-openvswitch\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:51.989376 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.988773 2566 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 09 18:58:51.989376 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.988793 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db233460-134d-4d9d-8da9-31d89366a4e0-host-run-ovn-kubernetes\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:51.989376 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.988817 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/db233460-134d-4d9d-8da9-31d89366a4e0-ovn-node-metrics-cert\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:51.989376 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.988837 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pntjm\" (UniqueName: \"kubernetes.io/projected/2fe73719-af90-4093-ae73-d4e462e5e43c-kube-api-access-pntjm\") pod \"network-check-target-mtz87\" (UID: \"2fe73719-af90-4093-ae73-d4e462e5e43c\") " pod="openshift-network-diagnostics/network-check-target-mtz87" Jul 09 18:58:51.989376 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.988857 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/db52b7ec-a201-4d65-81b3-56e37f0b6d36-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-swdmr\" (UID: \"db52b7ec-a201-4d65-81b3-56e37f0b6d36\") " pod="openshift-multus/multus-additional-cni-plugins-swdmr" Jul 09 18:58:51.989376 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.988878 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhtgh\" (UniqueName: \"kubernetes.io/projected/d40ecb0c-9c3c-4fbe-ad50-742b0f848c44-kube-api-access-jhtgh\") pod \"iptables-alerter-km5nl\" (UID: \"d40ecb0c-9c3c-4fbe-ad50-742b0f848c44\") " pod="openshift-network-operator/iptables-alerter-km5nl" Jul 09 18:58:51.989376 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.988898 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/47f9bbdb-bfc1-464a-a495-b310124b104b-etc-sysctl-conf\") pod \"tuned-hrjdx\" (UID: \"47f9bbdb-bfc1-464a-a495-b310124b104b\") " pod="openshift-cluster-node-tuning-operator/tuned-hrjdx" Jul 09 18:58:51.989376 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.988925 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/51b7fa90-97cc-4b69-80d4-d63ea5f30d6f-konnectivity-ca\") pod \"konnectivity-agent-c2bn7\" (UID: \"51b7fa90-97cc-4b69-80d4-d63ea5f30d6f\") " pod="kube-system/konnectivity-agent-c2bn7" Jul 09 18:58:51.989376 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.988951 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7435737c-5954-4ef0-8b41-1889b8231561-kubelet-config\") pod \"global-pull-secret-syncer-tqtv5\" (UID: \"7435737c-5954-4ef0-8b41-1889b8231561\") " pod="kube-system/global-pull-secret-syncer-tqtv5" Jul 09 18:58:51.989376 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.988973 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e5c65950-27b0-41de-8ee1-e2ced769d6b4-host-var-lib-cni-bin\") pod \"multus-5mwmg\" (UID: \"e5c65950-27b0-41de-8ee1-e2ced769d6b4\") " pod="openshift-multus/multus-5mwmg" Jul 09 18:58:51.989376 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.988998 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/47f9bbdb-bfc1-464a-a495-b310124b104b-etc-systemd\") pod \"tuned-hrjdx\" (UID: \"47f9bbdb-bfc1-464a-a495-b310124b104b\") " pod="openshift-cluster-node-tuning-operator/tuned-hrjdx" Jul 09 18:58:51.989376 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.989015 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/47f9bbdb-bfc1-464a-a495-b310124b104b-host\") pod \"tuned-hrjdx\" (UID: \"47f9bbdb-bfc1-464a-a495-b310124b104b\") " pod="openshift-cluster-node-tuning-operator/tuned-hrjdx" Jul 09 18:58:51.989376 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.989030 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db52b7ec-a201-4d65-81b3-56e37f0b6d36-system-cni-dir\") pod \"multus-additional-cni-plugins-swdmr\" (UID: \"db52b7ec-a201-4d65-81b3-56e37f0b6d36\") " pod="openshift-multus/multus-additional-cni-plugins-swdmr" Jul 09 18:58:51.990000 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.989049 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92n9z\" (UniqueName: \"kubernetes.io/projected/db52b7ec-a201-4d65-81b3-56e37f0b6d36-kube-api-access-92n9z\") pod \"multus-additional-cni-plugins-swdmr\" (UID: \"db52b7ec-a201-4d65-81b3-56e37f0b6d36\") " pod="openshift-multus/multus-additional-cni-plugins-swdmr" Jul 09 18:58:51.990000 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.989079 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzzvp\" (UniqueName: \"kubernetes.io/projected/30cf5feb-cf9a-40c0-8a6a-de8fc6c60506-kube-api-access-gzzvp\") pod \"network-metrics-daemon-vtc8p\" (UID: \"30cf5feb-cf9a-40c0-8a6a-de8fc6c60506\") " pod="openshift-multus/network-metrics-daemon-vtc8p" Jul 09 18:58:51.990000 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.989105 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5bln\" (UniqueName: \"kubernetes.io/projected/e5c65950-27b0-41de-8ee1-e2ced769d6b4-kube-api-access-q5bln\") pod \"multus-5mwmg\" (UID: \"e5c65950-27b0-41de-8ee1-e2ced769d6b4\") " pod="openshift-multus/multus-5mwmg" Jul 09 18:58:51.990000 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.989120 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/47f9bbdb-bfc1-464a-a495-b310124b104b-etc-sysconfig\") pod \"tuned-hrjdx\" (UID: \"47f9bbdb-bfc1-464a-a495-b310124b104b\") " pod="openshift-cluster-node-tuning-operator/tuned-hrjdx" Jul 09 18:58:51.990000 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.989136 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94s46\" (UniqueName: \"kubernetes.io/projected/db233460-134d-4d9d-8da9-31d89366a4e0-kube-api-access-94s46\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:51.990000 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.989159 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e5c65950-27b0-41de-8ee1-e2ced769d6b4-host-run-k8s-cni-cncf-io\") pod \"multus-5mwmg\" (UID: \"e5c65950-27b0-41de-8ee1-e2ced769d6b4\") " pod="openshift-multus/multus-5mwmg" Jul 09 18:58:51.990000 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.989194 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/db233460-134d-4d9d-8da9-31d89366a4e0-host-kubelet\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:51.990000 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.989209 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/db233460-134d-4d9d-8da9-31d89366a4e0-run-systemd\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:51.990000 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.989226 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/db233460-134d-4d9d-8da9-31d89366a4e0-ovnkube-config\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:51.990000 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.989248 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b1f91f8e-9064-4538-a912-6e6354050534-socket-dir\") pod \"aws-ebs-csi-driver-node-gcfcz\" (UID: \"b1f91f8e-9064-4538-a912-6e6354050534\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcfcz" Jul 09 18:58:51.990000 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.989269 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b1f91f8e-9064-4538-a912-6e6354050534-registration-dir\") pod \"aws-ebs-csi-driver-node-gcfcz\" (UID: \"b1f91f8e-9064-4538-a912-6e6354050534\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcfcz" Jul 09 18:58:51.990000 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.989294 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7435737c-5954-4ef0-8b41-1889b8231561-dbus\") pod \"global-pull-secret-syncer-tqtv5\" (UID: \"7435737c-5954-4ef0-8b41-1889b8231561\") " pod="kube-system/global-pull-secret-syncer-tqtv5" Jul 09 18:58:51.990000 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.989327 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5c298798-c68a-4450-b9dc-358fa978305a-host\") pod \"node-ca-jnt6g\" (UID: \"5c298798-c68a-4450-b9dc-358fa978305a\") " pod="openshift-image-registry/node-ca-jnt6g" Jul 09 18:58:51.990000 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.989370 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5c298798-c68a-4450-b9dc-358fa978305a-serviceca\") pod \"node-ca-jnt6g\" (UID: \"5c298798-c68a-4450-b9dc-358fa978305a\") " pod="openshift-image-registry/node-ca-jnt6g" Jul 09 18:58:51.990000 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.989395 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e5c65950-27b0-41de-8ee1-e2ced769d6b4-host-run-multus-certs\") pod \"multus-5mwmg\" (UID: \"e5c65950-27b0-41de-8ee1-e2ced769d6b4\") " pod="openshift-multus/multus-5mwmg" Jul 09 18:58:51.990000 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.989420 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/47f9bbdb-bfc1-464a-a495-b310124b104b-run\") pod \"tuned-hrjdx\" (UID: \"47f9bbdb-bfc1-464a-a495-b310124b104b\") " pod="openshift-cluster-node-tuning-operator/tuned-hrjdx" Jul 09 18:58:51.990527 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.989444 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/db233460-134d-4d9d-8da9-31d89366a4e0-host-run-netns\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:51.990527 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.989466 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/db233460-134d-4d9d-8da9-31d89366a4e0-node-log\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:51.990527 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.989488 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b1f91f8e-9064-4538-a912-6e6354050534-sys-fs\") pod \"aws-ebs-csi-driver-node-gcfcz\" (UID: \"b1f91f8e-9064-4538-a912-6e6354050534\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcfcz" Jul 09 18:58:51.990527 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.989501 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/51b7fa90-97cc-4b69-80d4-d63ea5f30d6f-konnectivity-ca\") pod \"konnectivity-agent-c2bn7\" (UID: \"51b7fa90-97cc-4b69-80d4-d63ea5f30d6f\") " pod="kube-system/konnectivity-agent-c2bn7" Jul 09 18:58:51.990527 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.989514 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e5c65950-27b0-41de-8ee1-e2ced769d6b4-cnibin\") pod \"multus-5mwmg\" (UID: \"e5c65950-27b0-41de-8ee1-e2ced769d6b4\") " pod="openshift-multus/multus-5mwmg" Jul 09 18:58:51.990527 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.989549 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e5c65950-27b0-41de-8ee1-e2ced769d6b4-cni-binary-copy\") pod \"multus-5mwmg\" (UID: \"e5c65950-27b0-41de-8ee1-e2ced769d6b4\") " pod="openshift-multus/multus-5mwmg" Jul 09 18:58:51.990527 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.989574 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/db233460-134d-4d9d-8da9-31d89366a4e0-env-overrides\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:51.990527 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.989592 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b1f91f8e-9064-4538-a912-6e6354050534-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gcfcz\" (UID: \"b1f91f8e-9064-4538-a912-6e6354050534\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcfcz" Jul 09 18:58:51.990527 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.989626 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlbxl\" (UniqueName: \"kubernetes.io/projected/5c298798-c68a-4450-b9dc-358fa978305a-kube-api-access-xlbxl\") pod \"node-ca-jnt6g\" (UID: \"5c298798-c68a-4450-b9dc-358fa978305a\") " pod="openshift-image-registry/node-ca-jnt6g" Jul 09 18:58:51.990527 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.989641 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e5c65950-27b0-41de-8ee1-e2ced769d6b4-multus-conf-dir\") pod \"multus-5mwmg\" (UID: \"e5c65950-27b0-41de-8ee1-e2ced769d6b4\") " pod="openshift-multus/multus-5mwmg" Jul 09 18:58:51.990527 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.989656 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/47f9bbdb-bfc1-464a-a495-b310124b104b-sys\") pod \"tuned-hrjdx\" (UID: \"47f9bbdb-bfc1-464a-a495-b310124b104b\") " pod="openshift-cluster-node-tuning-operator/tuned-hrjdx" Jul 09 18:58:51.990527 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.989682 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/47f9bbdb-bfc1-464a-a495-b310124b104b-lib-modules\") pod \"tuned-hrjdx\" (UID: \"47f9bbdb-bfc1-464a-a495-b310124b104b\") " pod="openshift-cluster-node-tuning-operator/tuned-hrjdx" Jul 09 18:58:51.990527 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.989696 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0da71879-5de7-47a0-819a-0b1951b1e817-tmp-dir\") pod \"node-resolver-x8k65\" (UID: \"0da71879-5de7-47a0-819a-0b1951b1e817\") " pod="openshift-dns/node-resolver-x8k65" Jul 09 18:58:51.990527 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.989710 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/db52b7ec-a201-4d65-81b3-56e37f0b6d36-os-release\") pod \"multus-additional-cni-plugins-swdmr\" (UID: \"db52b7ec-a201-4d65-81b3-56e37f0b6d36\") " pod="openshift-multus/multus-additional-cni-plugins-swdmr" Jul 09 18:58:51.990527 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.989751 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db233460-134d-4d9d-8da9-31d89366a4e0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:51.990527 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.989781 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5c65950-27b0-41de-8ee1-e2ced769d6b4-system-cni-dir\") pod \"multus-5mwmg\" (UID: \"e5c65950-27b0-41de-8ee1-e2ced769d6b4\") " pod="openshift-multus/multus-5mwmg" Jul 09 18:58:51.990527 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.989795 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5c65950-27b0-41de-8ee1-e2ced769d6b4-multus-cni-dir\") pod \"multus-5mwmg\" (UID: \"e5c65950-27b0-41de-8ee1-e2ced769d6b4\") " pod="openshift-multus/multus-5mwmg" Jul 09 18:58:51.991021 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.989817 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/47f9bbdb-bfc1-464a-a495-b310124b104b-etc-modprobe-d\") pod \"tuned-hrjdx\" (UID: \"47f9bbdb-bfc1-464a-a495-b310124b104b\") " pod="openshift-cluster-node-tuning-operator/tuned-hrjdx" Jul 09 18:58:51.991885 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:51.991869 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/51b7fa90-97cc-4b69-80d4-d63ea5f30d6f-agent-certs\") pod \"konnectivity-agent-c2bn7\" (UID: \"51b7fa90-97cc-4b69-80d4-d63ea5f30d6f\") " pod="kube-system/konnectivity-agent-c2bn7" Jul 09 18:58:52.003038 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.003019 2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-s96s7" Jul 09 18:58:52.013090 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.013071 2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-s96s7" Jul 09 18:58:52.036299 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:52.036269 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbfbad2bcd3a64e3008a9009f32541a5.slice/crio-53190415a09cfa3e3db37d89b603d5d15d0943c4bb032cdc02cac0ccaf9af1c8 WatchSource:0}: Error finding container 53190415a09cfa3e3db37d89b603d5d15d0943c4bb032cdc02cac0ccaf9af1c8: Status 404 returned error can't find the container with id 53190415a09cfa3e3db37d89b603d5d15d0943c4bb032cdc02cac0ccaf9af1c8 Jul 09 18:58:52.036797 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:52.036784 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bd4f2bfb833729e4a5ed7d4607ff8c1.slice/crio-d0d2c9f84d0bc6e338319379750bc60b58779d363987fbf5881c2942fee1ac7a WatchSource:0}: Error finding container d0d2c9f84d0bc6e338319379750bc60b58779d363987fbf5881c2942fee1ac7a: Status 404 returned error can't find the container with id d0d2c9f84d0bc6e338319379750bc60b58779d363987fbf5881c2942fee1ac7a Jul 09 18:58:52.040783 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.040768 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 18:58:52.054478 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.054436 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-153.ec2.internal" event={"ID":"2bd4f2bfb833729e4a5ed7d4607ff8c1","Type":"ContainerStarted","Data":"d0d2c9f84d0bc6e338319379750bc60b58779d363987fbf5881c2942fee1ac7a"} Jul 09 18:58:52.055636 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.055596 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-153.ec2.internal" event={"ID":"dbfbad2bcd3a64e3008a9009f32541a5","Type":"ContainerStarted","Data":"53190415a09cfa3e3db37d89b603d5d15d0943c4bb032cdc02cac0ccaf9af1c8"} Jul 09 18:58:52.090556 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.090529 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7435737c-5954-4ef0-8b41-1889b8231561-original-pull-secret\") pod \"global-pull-secret-syncer-tqtv5\" (UID: \"7435737c-5954-4ef0-8b41-1889b8231561\") " pod="kube-system/global-pull-secret-syncer-tqtv5" Jul 09 18:58:52.090556 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.090557 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5c65950-27b0-41de-8ee1-e2ced769d6b4-etc-kubernetes\") pod \"multus-5mwmg\" (UID: \"e5c65950-27b0-41de-8ee1-e2ced769d6b4\") " pod="openshift-multus/multus-5mwmg" Jul 09 18:58:52.090751 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.090575 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf2vh\" (UniqueName: \"kubernetes.io/projected/47f9bbdb-bfc1-464a-a495-b310124b104b-kube-api-access-kf2vh\") pod \"tuned-hrjdx\" (UID: \"47f9bbdb-bfc1-464a-a495-b310124b104b\") " pod="openshift-cluster-node-tuning-operator/tuned-hrjdx" Jul 09 18:58:52.090751 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.090590 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db233460-134d-4d9d-8da9-31d89366a4e0-run-openvswitch\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:52.090751 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.090616 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b1f91f8e-9064-4538-a912-6e6354050534-etc-selinux\") pod \"aws-ebs-csi-driver-node-gcfcz\" (UID: \"b1f91f8e-9064-4538-a912-6e6354050534\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcfcz" Jul 09 18:58:52.090751 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.090639 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0da71879-5de7-47a0-819a-0b1951b1e817-hosts-file\") pod \"node-resolver-x8k65\" (UID: \"0da71879-5de7-47a0-819a-0b1951b1e817\") " pod="openshift-dns/node-resolver-x8k65" Jul 09 18:58:52.090751 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.090661 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/db52b7ec-a201-4d65-81b3-56e37f0b6d36-cnibin\") pod \"multus-additional-cni-plugins-swdmr\" (UID: \"db52b7ec-a201-4d65-81b3-56e37f0b6d36\") " pod="openshift-multus/multus-additional-cni-plugins-swdmr" Jul 09 18:58:52.090751 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.090662 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5c65950-27b0-41de-8ee1-e2ced769d6b4-etc-kubernetes\") pod \"multus-5mwmg\" (UID: \"e5c65950-27b0-41de-8ee1-e2ced769d6b4\") " pod="openshift-multus/multus-5mwmg" Jul 09 18:58:52.090751 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.090685 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/47f9bbdb-bfc1-464a-a495-b310124b104b-var-lib-kubelet\") pod \"tuned-hrjdx\" (UID: \"47f9bbdb-bfc1-464a-a495-b310124b104b\") " pod="openshift-cluster-node-tuning-operator/tuned-hrjdx" Jul 09 18:58:52.090751 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.090694 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db233460-134d-4d9d-8da9-31d89366a4e0-run-openvswitch\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:52.090751 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.090708 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/db233460-134d-4d9d-8da9-31d89366a4e0-host-slash\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:52.090751 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:52.090727 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 18:58:52.090751 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.090753 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/db52b7ec-a201-4d65-81b3-56e37f0b6d36-cnibin\") pod \"multus-additional-cni-plugins-swdmr\" (UID: \"db52b7ec-a201-4d65-81b3-56e37f0b6d36\") " pod="openshift-multus/multus-additional-cni-plugins-swdmr" Jul 09 18:58:52.091245 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.090762 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/db233460-134d-4d9d-8da9-31d89366a4e0-host-slash\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:52.091245 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.090768 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/47f9bbdb-bfc1-464a-a495-b310124b104b-var-lib-kubelet\") pod \"tuned-hrjdx\" (UID: \"47f9bbdb-bfc1-464a-a495-b310124b104b\") " pod="openshift-cluster-node-tuning-operator/tuned-hrjdx" Jul 09 18:58:52.091245 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.090725 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0da71879-5de7-47a0-819a-0b1951b1e817-hosts-file\") pod \"node-resolver-x8k65\" (UID: \"0da71879-5de7-47a0-819a-0b1951b1e817\") " pod="openshift-dns/node-resolver-x8k65" Jul 09 18:58:52.091245 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.090731 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e5c65950-27b0-41de-8ee1-e2ced769d6b4-os-release\") pod \"multus-5mwmg\" (UID: \"e5c65950-27b0-41de-8ee1-e2ced769d6b4\") " pod="openshift-multus/multus-5mwmg" Jul 09 18:58:52.091245 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.090790 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b1f91f8e-9064-4538-a912-6e6354050534-etc-selinux\") pod \"aws-ebs-csi-driver-node-gcfcz\" (UID: \"b1f91f8e-9064-4538-a912-6e6354050534\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcfcz" Jul 09 18:58:52.091245 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:52.090810 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7435737c-5954-4ef0-8b41-1889b8231561-original-pull-secret podName:7435737c-5954-4ef0-8b41-1889b8231561 nodeName:}" failed. No retries permitted until 2026-07-09 18:58:52.590788232 +0000 UTC m=+2.098568005 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7435737c-5954-4ef0-8b41-1889b8231561-original-pull-secret") pod "global-pull-secret-syncer-tqtv5" (UID: "7435737c-5954-4ef0-8b41-1889b8231561") : object "kube-system"/"original-pull-secret" not registered Jul 09 18:58:52.091245 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.090793 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e5c65950-27b0-41de-8ee1-e2ced769d6b4-os-release\") pod \"multus-5mwmg\" (UID: \"e5c65950-27b0-41de-8ee1-e2ced769d6b4\") " pod="openshift-multus/multus-5mwmg" Jul 09 18:58:52.091245 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.090852 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/47f9bbdb-bfc1-464a-a495-b310124b104b-etc-tuned\") pod \"tuned-hrjdx\" (UID: \"47f9bbdb-bfc1-464a-a495-b310124b104b\") " pod="openshift-cluster-node-tuning-operator/tuned-hrjdx" Jul 09 18:58:52.091245 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.090875 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/47f9bbdb-bfc1-464a-a495-b310124b104b-tmp\") pod \"tuned-hrjdx\" (UID: \"47f9bbdb-bfc1-464a-a495-b310124b104b\") " pod="openshift-cluster-node-tuning-operator/tuned-hrjdx" Jul 09 18:58:52.091245 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.090892 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/db52b7ec-a201-4d65-81b3-56e37f0b6d36-cni-binary-copy\") pod \"multus-additional-cni-plugins-swdmr\" (UID: \"db52b7ec-a201-4d65-81b3-56e37f0b6d36\") " pod="openshift-multus/multus-additional-cni-plugins-swdmr" Jul 09 18:58:52.091245 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.090907 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/db233460-134d-4d9d-8da9-31d89366a4e0-host-cni-netd\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:52.091245 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.090947 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/db233460-134d-4d9d-8da9-31d89366a4e0-host-cni-netd\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:52.091245 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.091014 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30cf5feb-cf9a-40c0-8a6a-de8fc6c60506-metrics-certs\") pod \"network-metrics-daemon-vtc8p\" (UID: \"30cf5feb-cf9a-40c0-8a6a-de8fc6c60506\") " pod="openshift-multus/network-metrics-daemon-vtc8p" Jul 09 18:58:52.091245 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.091035 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e5c65950-27b0-41de-8ee1-e2ced769d6b4-multus-daemon-config\") pod \"multus-5mwmg\" (UID: \"e5c65950-27b0-41de-8ee1-e2ced769d6b4\") " pod="openshift-multus/multus-5mwmg" Jul 09 18:58:52.091245 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.091052 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/db52b7ec-a201-4d65-81b3-56e37f0b6d36-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-swdmr\" (UID: \"db52b7ec-a201-4d65-81b3-56e37f0b6d36\") " pod="openshift-multus/multus-additional-cni-plugins-swdmr" Jul 09 18:58:52.091245 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.091068 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d40ecb0c-9c3c-4fbe-ad50-742b0f848c44-iptables-alerter-script\") pod \"iptables-alerter-km5nl\" (UID: \"d40ecb0c-9c3c-4fbe-ad50-742b0f848c44\") " pod="openshift-network-operator/iptables-alerter-km5nl" Jul 09 18:58:52.091245 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.091083 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/db233460-134d-4d9d-8da9-31d89366a4e0-systemd-units\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:52.092063 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.091098 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db233460-134d-4d9d-8da9-31d89366a4e0-var-lib-openvswitch\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:52.092063 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.091119 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db233460-134d-4d9d-8da9-31d89366a4e0-etc-openvswitch\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:52.092063 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.091152 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/db233460-134d-4d9d-8da9-31d89366a4e0-systemd-units\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:52.092063 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.091160 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db233460-134d-4d9d-8da9-31d89366a4e0-etc-openvswitch\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:52.092063 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.091187 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/db233460-134d-4d9d-8da9-31d89366a4e0-var-lib-openvswitch\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:52.092063 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:52.091214 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 18:58:52.092063 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.091255 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db233460-134d-4d9d-8da9-31d89366a4e0-host-run-ovn-kubernetes\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:52.092063 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:52.091265 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30cf5feb-cf9a-40c0-8a6a-de8fc6c60506-metrics-certs podName:30cf5feb-cf9a-40c0-8a6a-de8fc6c60506 nodeName:}" failed. No retries permitted until 2026-07-09 18:58:52.591250625 +0000 UTC m=+2.099030394 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30cf5feb-cf9a-40c0-8a6a-de8fc6c60506-metrics-certs") pod "network-metrics-daemon-vtc8p" (UID: "30cf5feb-cf9a-40c0-8a6a-de8fc6c60506") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 18:58:52.092063 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.091287 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db233460-134d-4d9d-8da9-31d89366a4e0-host-run-ovn-kubernetes\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:52.092063 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.091291 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/db233460-134d-4d9d-8da9-31d89366a4e0-ovn-node-metrics-cert\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:52.092063 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.091335 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pntjm\" (UniqueName: \"kubernetes.io/projected/2fe73719-af90-4093-ae73-d4e462e5e43c-kube-api-access-pntjm\") pod \"network-check-target-mtz87\" (UID: \"2fe73719-af90-4093-ae73-d4e462e5e43c\") " pod="openshift-network-diagnostics/network-check-target-mtz87" Jul 09 18:58:52.092063 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.091359 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/db52b7ec-a201-4d65-81b3-56e37f0b6d36-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-swdmr\" (UID: \"db52b7ec-a201-4d65-81b3-56e37f0b6d36\") " pod="openshift-multus/multus-additional-cni-plugins-swdmr" Jul 09 18:58:52.092063 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.091402 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhtgh\" (UniqueName: \"kubernetes.io/projected/d40ecb0c-9c3c-4fbe-ad50-742b0f848c44-kube-api-access-jhtgh\") pod \"iptables-alerter-km5nl\" (UID: \"d40ecb0c-9c3c-4fbe-ad50-742b0f848c44\") " pod="openshift-network-operator/iptables-alerter-km5nl" Jul 09 18:58:52.092063 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.091427 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/47f9bbdb-bfc1-464a-a495-b310124b104b-etc-sysctl-conf\") pod \"tuned-hrjdx\" (UID: \"47f9bbdb-bfc1-464a-a495-b310124b104b\") " pod="openshift-cluster-node-tuning-operator/tuned-hrjdx" Jul 09 18:58:52.092063 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.091459 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/db52b7ec-a201-4d65-81b3-56e37f0b6d36-cni-binary-copy\") pod \"multus-additional-cni-plugins-swdmr\" (UID: \"db52b7ec-a201-4d65-81b3-56e37f0b6d36\") " pod="openshift-multus/multus-additional-cni-plugins-swdmr" Jul 09 18:58:52.092063 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.091453 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7435737c-5954-4ef0-8b41-1889b8231561-kubelet-config\") pod \"global-pull-secret-syncer-tqtv5\" (UID: \"7435737c-5954-4ef0-8b41-1889b8231561\") " pod="kube-system/global-pull-secret-syncer-tqtv5" Jul 09 18:58:52.092063 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.091513 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e5c65950-27b0-41de-8ee1-e2ced769d6b4-host-var-lib-cni-bin\") pod \"multus-5mwmg\" (UID: \"e5c65950-27b0-41de-8ee1-e2ced769d6b4\") " pod="openshift-multus/multus-5mwmg" Jul 09 18:58:52.092873 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.091523 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7435737c-5954-4ef0-8b41-1889b8231561-kubelet-config\") pod \"global-pull-secret-syncer-tqtv5\" (UID: \"7435737c-5954-4ef0-8b41-1889b8231561\") " pod="kube-system/global-pull-secret-syncer-tqtv5" Jul 09 18:58:52.092873 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.091538 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/47f9bbdb-bfc1-464a-a495-b310124b104b-etc-systemd\") pod \"tuned-hrjdx\" (UID: \"47f9bbdb-bfc1-464a-a495-b310124b104b\") " pod="openshift-cluster-node-tuning-operator/tuned-hrjdx" Jul 09 18:58:52.092873 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.091563 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/47f9bbdb-bfc1-464a-a495-b310124b104b-host\") pod \"tuned-hrjdx\" (UID: \"47f9bbdb-bfc1-464a-a495-b310124b104b\") " pod="openshift-cluster-node-tuning-operator/tuned-hrjdx" Jul 09 18:58:52.092873 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.091588 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db52b7ec-a201-4d65-81b3-56e37f0b6d36-system-cni-dir\") pod \"multus-additional-cni-plugins-swdmr\" (UID: \"db52b7ec-a201-4d65-81b3-56e37f0b6d36\") " pod="openshift-multus/multus-additional-cni-plugins-swdmr" Jul 09 18:58:52.092873 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.091636 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92n9z\" (UniqueName: \"kubernetes.io/projected/db52b7ec-a201-4d65-81b3-56e37f0b6d36-kube-api-access-92n9z\") pod \"multus-additional-cni-plugins-swdmr\" (UID: \"db52b7ec-a201-4d65-81b3-56e37f0b6d36\") " pod="openshift-multus/multus-additional-cni-plugins-swdmr" Jul 09 18:58:52.092873 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.091663 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzzvp\" (UniqueName: \"kubernetes.io/projected/30cf5feb-cf9a-40c0-8a6a-de8fc6c60506-kube-api-access-gzzvp\") pod \"network-metrics-daemon-vtc8p\" (UID: \"30cf5feb-cf9a-40c0-8a6a-de8fc6c60506\") " pod="openshift-multus/network-metrics-daemon-vtc8p" Jul 09 18:58:52.092873 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.091687 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5bln\" (UniqueName: \"kubernetes.io/projected/e5c65950-27b0-41de-8ee1-e2ced769d6b4-kube-api-access-q5bln\") pod \"multus-5mwmg\" (UID: \"e5c65950-27b0-41de-8ee1-e2ced769d6b4\") " pod="openshift-multus/multus-5mwmg" Jul 09 18:58:52.092873 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.091707 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e5c65950-27b0-41de-8ee1-e2ced769d6b4-multus-daemon-config\") pod \"multus-5mwmg\" (UID: \"e5c65950-27b0-41de-8ee1-e2ced769d6b4\") " pod="openshift-multus/multus-5mwmg" Jul 09 18:58:52.092873 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.091712 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/47f9bbdb-bfc1-464a-a495-b310124b104b-etc-sysconfig\") pod \"tuned-hrjdx\" (UID: \"47f9bbdb-bfc1-464a-a495-b310124b104b\") " pod="openshift-cluster-node-tuning-operator/tuned-hrjdx" Jul 09 18:58:52.092873 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.091757 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94s46\" (UniqueName: \"kubernetes.io/projected/db233460-134d-4d9d-8da9-31d89366a4e0-kube-api-access-94s46\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:52.092873 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.091760 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db52b7ec-a201-4d65-81b3-56e37f0b6d36-system-cni-dir\") pod \"multus-additional-cni-plugins-swdmr\" (UID: \"db52b7ec-a201-4d65-81b3-56e37f0b6d36\") " pod="openshift-multus/multus-additional-cni-plugins-swdmr" Jul 09 18:58:52.092873 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.091758 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e5c65950-27b0-41de-8ee1-e2ced769d6b4-host-var-lib-cni-bin\") pod \"multus-5mwmg\" (UID: \"e5c65950-27b0-41de-8ee1-e2ced769d6b4\") " pod="openshift-multus/multus-5mwmg" Jul 09 18:58:52.092873 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.091756 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/db52b7ec-a201-4d65-81b3-56e37f0b6d36-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-swdmr\" (UID: \"db52b7ec-a201-4d65-81b3-56e37f0b6d36\") " pod="openshift-multus/multus-additional-cni-plugins-swdmr" Jul 09 18:58:52.092873 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.091805 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/47f9bbdb-bfc1-464a-a495-b310124b104b-host\") pod \"tuned-hrjdx\" (UID: \"47f9bbdb-bfc1-464a-a495-b310124b104b\") " pod="openshift-cluster-node-tuning-operator/tuned-hrjdx" Jul 09 18:58:52.092873 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.091810 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/47f9bbdb-bfc1-464a-a495-b310124b104b-etc-sysconfig\") pod \"tuned-hrjdx\" (UID: \"47f9bbdb-bfc1-464a-a495-b310124b104b\") " pod="openshift-cluster-node-tuning-operator/tuned-hrjdx" Jul 09 18:58:52.092873 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.091823 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e5c65950-27b0-41de-8ee1-e2ced769d6b4-host-run-k8s-cni-cncf-io\") pod \"multus-5mwmg\" (UID: \"e5c65950-27b0-41de-8ee1-e2ced769d6b4\") " pod="openshift-multus/multus-5mwmg" Jul 09 18:58:52.092873 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.091873 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/47f9bbdb-bfc1-464a-a495-b310124b104b-etc-systemd\") pod \"tuned-hrjdx\" (UID: \"47f9bbdb-bfc1-464a-a495-b310124b104b\") " pod="openshift-cluster-node-tuning-operator/tuned-hrjdx" Jul 09 18:58:52.093619 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.091882 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/47f9bbdb-bfc1-464a-a495-b310124b104b-etc-sysctl-conf\") pod \"tuned-hrjdx\" (UID: \"47f9bbdb-bfc1-464a-a495-b310124b104b\") " pod="openshift-cluster-node-tuning-operator/tuned-hrjdx" Jul 09 18:58:52.093619 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.091920 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e5c65950-27b0-41de-8ee1-e2ced769d6b4-host-run-k8s-cni-cncf-io\") pod \"multus-5mwmg\" (UID: \"e5c65950-27b0-41de-8ee1-e2ced769d6b4\") " pod="openshift-multus/multus-5mwmg" Jul 09 18:58:52.093619 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.091977 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/db233460-134d-4d9d-8da9-31d89366a4e0-host-kubelet\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:52.093619 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.092155 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/db233460-134d-4d9d-8da9-31d89366a4e0-run-systemd\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:52.093619 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.092158 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/db233460-134d-4d9d-8da9-31d89366a4e0-host-kubelet\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:52.093619 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.092223 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/db233460-134d-4d9d-8da9-31d89366a4e0-run-systemd\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:52.093619 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.092252 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/db233460-134d-4d9d-8da9-31d89366a4e0-ovnkube-config\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:52.093619 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.092277 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b1f91f8e-9064-4538-a912-6e6354050534-socket-dir\") pod \"aws-ebs-csi-driver-node-gcfcz\" (UID: \"b1f91f8e-9064-4538-a912-6e6354050534\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcfcz" Jul 09 18:58:52.093619 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.092292 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b1f91f8e-9064-4538-a912-6e6354050534-registration-dir\") pod \"aws-ebs-csi-driver-node-gcfcz\" (UID: \"b1f91f8e-9064-4538-a912-6e6354050534\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcfcz" Jul 09 18:58:52.093619 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.092310 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7435737c-5954-4ef0-8b41-1889b8231561-dbus\") pod \"global-pull-secret-syncer-tqtv5\" (UID: \"7435737c-5954-4ef0-8b41-1889b8231561\") " pod="kube-system/global-pull-secret-syncer-tqtv5" Jul 09 18:58:52.093619 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.092334 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5c298798-c68a-4450-b9dc-358fa978305a-host\") pod \"node-ca-jnt6g\" (UID: \"5c298798-c68a-4450-b9dc-358fa978305a\") " pod="openshift-image-registry/node-ca-jnt6g" Jul 09 18:58:52.093619 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.092357 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5c298798-c68a-4450-b9dc-358fa978305a-serviceca\") pod \"node-ca-jnt6g\" (UID: \"5c298798-c68a-4450-b9dc-358fa978305a\") " pod="openshift-image-registry/node-ca-jnt6g" Jul 09 18:58:52.093619 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.092380 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e5c65950-27b0-41de-8ee1-e2ced769d6b4-host-run-multus-certs\") pod \"multus-5mwmg\" (UID: \"e5c65950-27b0-41de-8ee1-e2ced769d6b4\") " pod="openshift-multus/multus-5mwmg" Jul 09 18:58:52.093619 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.092405 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/47f9bbdb-bfc1-464a-a495-b310124b104b-run\") pod \"tuned-hrjdx\" (UID: \"47f9bbdb-bfc1-464a-a495-b310124b104b\") " pod="openshift-cluster-node-tuning-operator/tuned-hrjdx" Jul 09 18:58:52.093619 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.092430 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/db233460-134d-4d9d-8da9-31d89366a4e0-host-run-netns\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:52.093619 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.092433 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b1f91f8e-9064-4538-a912-6e6354050534-socket-dir\") pod \"aws-ebs-csi-driver-node-gcfcz\" (UID: \"b1f91f8e-9064-4538-a912-6e6354050534\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcfcz" Jul 09 18:58:52.093619 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.092453 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/db233460-134d-4d9d-8da9-31d89366a4e0-node-log\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:52.093619 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.092494 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b1f91f8e-9064-4538-a912-6e6354050534-sys-fs\") pod \"aws-ebs-csi-driver-node-gcfcz\" (UID: \"b1f91f8e-9064-4538-a912-6e6354050534\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcfcz" Jul 09 18:58:52.094528 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.092517 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e5c65950-27b0-41de-8ee1-e2ced769d6b4-cnibin\") pod \"multus-5mwmg\" (UID: \"e5c65950-27b0-41de-8ee1-e2ced769d6b4\") " pod="openshift-multus/multus-5mwmg" Jul 09 18:58:52.094528 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.092536 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d40ecb0c-9c3c-4fbe-ad50-742b0f848c44-iptables-alerter-script\") pod \"iptables-alerter-km5nl\" (UID: \"d40ecb0c-9c3c-4fbe-ad50-742b0f848c44\") " pod="openshift-network-operator/iptables-alerter-km5nl" Jul 09 18:58:52.094528 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.092546 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e5c65950-27b0-41de-8ee1-e2ced769d6b4-cni-binary-copy\") pod \"multus-5mwmg\" (UID: \"e5c65950-27b0-41de-8ee1-e2ced769d6b4\") " pod="openshift-multus/multus-5mwmg" Jul 09 18:58:52.094528 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.092572 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/db233460-134d-4d9d-8da9-31d89366a4e0-env-overrides\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:52.094528 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.092594 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b1f91f8e-9064-4538-a912-6e6354050534-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gcfcz\" (UID: \"b1f91f8e-9064-4538-a912-6e6354050534\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcfcz" Jul 09 18:58:52.094528 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.092639 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlbxl\" (UniqueName: \"kubernetes.io/projected/5c298798-c68a-4450-b9dc-358fa978305a-kube-api-access-xlbxl\") pod \"node-ca-jnt6g\" (UID: \"5c298798-c68a-4450-b9dc-358fa978305a\") " pod="openshift-image-registry/node-ca-jnt6g" Jul 09 18:58:52.094528 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.092663 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e5c65950-27b0-41de-8ee1-e2ced769d6b4-multus-conf-dir\") pod \"multus-5mwmg\" (UID: \"e5c65950-27b0-41de-8ee1-e2ced769d6b4\") " pod="openshift-multus/multus-5mwmg" Jul 09 18:58:52.094528 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.092710 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/47f9bbdb-bfc1-464a-a495-b310124b104b-sys\") pod \"tuned-hrjdx\" (UID: \"47f9bbdb-bfc1-464a-a495-b310124b104b\") " pod="openshift-cluster-node-tuning-operator/tuned-hrjdx" Jul 09 18:58:52.094528 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.092710 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7435737c-5954-4ef0-8b41-1889b8231561-dbus\") pod \"global-pull-secret-syncer-tqtv5\" (UID: \"7435737c-5954-4ef0-8b41-1889b8231561\") " pod="kube-system/global-pull-secret-syncer-tqtv5" Jul 09 18:58:52.094528 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.092768 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/47f9bbdb-bfc1-464a-a495-b310124b104b-sys\") pod \"tuned-hrjdx\" (UID: \"47f9bbdb-bfc1-464a-a495-b310124b104b\") " pod="openshift-cluster-node-tuning-operator/tuned-hrjdx" Jul 09 18:58:52.094528 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.092776 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b1f91f8e-9064-4538-a912-6e6354050534-registration-dir\") pod \"aws-ebs-csi-driver-node-gcfcz\" (UID: \"b1f91f8e-9064-4538-a912-6e6354050534\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcfcz" Jul 09 18:58:52.094528 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.092799 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5c298798-c68a-4450-b9dc-358fa978305a-host\") pod \"node-ca-jnt6g\" (UID: \"5c298798-c68a-4450-b9dc-358fa978305a\") " pod="openshift-image-registry/node-ca-jnt6g" Jul 09 18:58:52.094528 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.092801 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/db233460-134d-4d9d-8da9-31d89366a4e0-ovnkube-config\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:52.094528 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.092823 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e5c65950-27b0-41de-8ee1-e2ced769d6b4-cnibin\") pod \"multus-5mwmg\" (UID: \"e5c65950-27b0-41de-8ee1-e2ced769d6b4\") " pod="openshift-multus/multus-5mwmg" Jul 09 18:58:52.094528 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.092863 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e5c65950-27b0-41de-8ee1-e2ced769d6b4-multus-conf-dir\") pod \"multus-5mwmg\" (UID: \"e5c65950-27b0-41de-8ee1-e2ced769d6b4\") " pod="openshift-multus/multus-5mwmg" Jul 09 18:58:52.094528 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.092868 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e5c65950-27b0-41de-8ee1-e2ced769d6b4-host-run-multus-certs\") pod \"multus-5mwmg\" (UID: \"e5c65950-27b0-41de-8ee1-e2ced769d6b4\") " pod="openshift-multus/multus-5mwmg" Jul 09 18:58:52.094528 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.092891 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/47f9bbdb-bfc1-464a-a495-b310124b104b-lib-modules\") pod \"tuned-hrjdx\" (UID: \"47f9bbdb-bfc1-464a-a495-b310124b104b\") " pod="openshift-cluster-node-tuning-operator/tuned-hrjdx" Jul 09 18:58:52.094528 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.092916 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/47f9bbdb-bfc1-464a-a495-b310124b104b-run\") pod \"tuned-hrjdx\" (UID: \"47f9bbdb-bfc1-464a-a495-b310124b104b\") " pod="openshift-cluster-node-tuning-operator/tuned-hrjdx" Jul 09 18:58:52.095375 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.092917 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0da71879-5de7-47a0-819a-0b1951b1e817-tmp-dir\") pod \"node-resolver-x8k65\" (UID: \"0da71879-5de7-47a0-819a-0b1951b1e817\") " pod="openshift-dns/node-resolver-x8k65" Jul 09 18:58:52.095375 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.092940 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5c298798-c68a-4450-b9dc-358fa978305a-serviceca\") pod \"node-ca-jnt6g\" (UID: \"5c298798-c68a-4450-b9dc-358fa978305a\") " pod="openshift-image-registry/node-ca-jnt6g" Jul 09 18:58:52.095375 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.092954 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/db52b7ec-a201-4d65-81b3-56e37f0b6d36-os-release\") pod \"multus-additional-cni-plugins-swdmr\" (UID: \"db52b7ec-a201-4d65-81b3-56e37f0b6d36\") " pod="openshift-multus/multus-additional-cni-plugins-swdmr" Jul 09 18:58:52.095375 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.092980 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db233460-134d-4d9d-8da9-31d89366a4e0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:52.095375 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.093005 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5c65950-27b0-41de-8ee1-e2ced769d6b4-system-cni-dir\") pod \"multus-5mwmg\" (UID: \"e5c65950-27b0-41de-8ee1-e2ced769d6b4\") " pod="openshift-multus/multus-5mwmg" Jul 09 18:58:52.095375 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.093024 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b1f91f8e-9064-4538-a912-6e6354050534-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gcfcz\" (UID: \"b1f91f8e-9064-4538-a912-6e6354050534\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcfcz" Jul 09 18:58:52.095375 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.093029 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5c65950-27b0-41de-8ee1-e2ced769d6b4-multus-cni-dir\") pod \"multus-5mwmg\" (UID: \"e5c65950-27b0-41de-8ee1-e2ced769d6b4\") " pod="openshift-multus/multus-5mwmg" Jul 09 18:58:52.095375 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.093060 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/47f9bbdb-bfc1-464a-a495-b310124b104b-etc-modprobe-d\") pod \"tuned-hrjdx\" (UID: \"47f9bbdb-bfc1-464a-a495-b310124b104b\") " pod="openshift-cluster-node-tuning-operator/tuned-hrjdx" Jul 09 18:58:52.095375 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.093086 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47f9bbdb-bfc1-464a-a495-b310124b104b-etc-kubernetes\") pod \"tuned-hrjdx\" (UID: \"47f9bbdb-bfc1-464a-a495-b310124b104b\") " pod="openshift-cluster-node-tuning-operator/tuned-hrjdx" Jul 09 18:58:52.095375 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.093101 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5c65950-27b0-41de-8ee1-e2ced769d6b4-multus-cni-dir\") pod \"multus-5mwmg\" (UID: \"e5c65950-27b0-41de-8ee1-e2ced769d6b4\") " pod="openshift-multus/multus-5mwmg" Jul 09 18:58:52.095375 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.093111 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/db233460-134d-4d9d-8da9-31d89366a4e0-ovnkube-script-lib\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:52.095375 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.093138 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dqrd\" (UniqueName: \"kubernetes.io/projected/b1f91f8e-9064-4538-a912-6e6354050534-kube-api-access-4dqrd\") pod \"aws-ebs-csi-driver-node-gcfcz\" (UID: \"b1f91f8e-9064-4538-a912-6e6354050534\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcfcz" Jul 09 18:58:52.095375 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.093144 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/db233460-134d-4d9d-8da9-31d89366a4e0-host-run-netns\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:52.095375 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.093163 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e5c65950-27b0-41de-8ee1-e2ced769d6b4-host-var-lib-cni-multus\") pod \"multus-5mwmg\" (UID: \"e5c65950-27b0-41de-8ee1-e2ced769d6b4\") " pod="openshift-multus/multus-5mwmg" Jul 09 18:58:52.095375 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.093186 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/db233460-134d-4d9d-8da9-31d89366a4e0-node-log\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:52.095375 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.093186 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0da71879-5de7-47a0-819a-0b1951b1e817-tmp-dir\") pod \"node-resolver-x8k65\" (UID: \"0da71879-5de7-47a0-819a-0b1951b1e817\") " pod="openshift-dns/node-resolver-x8k65" Jul 09 18:58:52.095375 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.093187 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e5c65950-27b0-41de-8ee1-e2ced769d6b4-hostroot\") pod \"multus-5mwmg\" (UID: \"e5c65950-27b0-41de-8ee1-e2ced769d6b4\") " pod="openshift-multus/multus-5mwmg" Jul 09 18:58:52.095375 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.093270 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/47f9bbdb-bfc1-464a-a495-b310124b104b-etc-modprobe-d\") pod \"tuned-hrjdx\" (UID: \"47f9bbdb-bfc1-464a-a495-b310124b104b\") " pod="openshift-cluster-node-tuning-operator/tuned-hrjdx" Jul 09 18:58:52.095915 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.093280 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/47f9bbdb-bfc1-464a-a495-b310124b104b-etc-sysctl-d\") pod \"tuned-hrjdx\" (UID: \"47f9bbdb-bfc1-464a-a495-b310124b104b\") " pod="openshift-cluster-node-tuning-operator/tuned-hrjdx" Jul 09 18:58:52.095915 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.093280 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e5c65950-27b0-41de-8ee1-e2ced769d6b4-cni-binary-copy\") pod \"multus-5mwmg\" (UID: \"e5c65950-27b0-41de-8ee1-e2ced769d6b4\") " pod="openshift-multus/multus-5mwmg" Jul 09 18:58:52.095915 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.093305 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/db52b7ec-a201-4d65-81b3-56e37f0b6d36-tuning-conf-dir\") pod \"multus-additional-cni-plugins-swdmr\" (UID: \"db52b7ec-a201-4d65-81b3-56e37f0b6d36\") " pod="openshift-multus/multus-additional-cni-plugins-swdmr" Jul 09 18:58:52.095915 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.093295 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/47f9bbdb-bfc1-464a-a495-b310124b104b-lib-modules\") pod \"tuned-hrjdx\" (UID: \"47f9bbdb-bfc1-464a-a495-b310124b104b\") " pod="openshift-cluster-node-tuning-operator/tuned-hrjdx" Jul 09 18:58:52.095915 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.093354 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/db233460-134d-4d9d-8da9-31d89366a4e0-env-overrides\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:52.095915 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.093378 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5c65950-27b0-41de-8ee1-e2ced769d6b4-system-cni-dir\") pod \"multus-5mwmg\" (UID: \"e5c65950-27b0-41de-8ee1-e2ced769d6b4\") " pod="openshift-multus/multus-5mwmg" Jul 09 18:58:52.095915 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.093382 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b1f91f8e-9064-4538-a912-6e6354050534-sys-fs\") pod \"aws-ebs-csi-driver-node-gcfcz\" (UID: \"b1f91f8e-9064-4538-a912-6e6354050534\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcfcz" Jul 09 18:58:52.095915 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.093417 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db233460-134d-4d9d-8da9-31d89366a4e0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:52.095915 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.093437 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/db52b7ec-a201-4d65-81b3-56e37f0b6d36-os-release\") pod \"multus-additional-cni-plugins-swdmr\" (UID: \"db52b7ec-a201-4d65-81b3-56e37f0b6d36\") " pod="openshift-multus/multus-additional-cni-plugins-swdmr" Jul 09 18:58:52.095915 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.093468 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47f9bbdb-bfc1-464a-a495-b310124b104b-etc-kubernetes\") pod \"tuned-hrjdx\" (UID: \"47f9bbdb-bfc1-464a-a495-b310124b104b\") " pod="openshift-cluster-node-tuning-operator/tuned-hrjdx" Jul 09 18:58:52.095915 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.093490 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d40ecb0c-9c3c-4fbe-ad50-742b0f848c44-host-slash\") pod \"iptables-alerter-km5nl\" (UID: \"d40ecb0c-9c3c-4fbe-ad50-742b0f848c44\") " pod="openshift-network-operator/iptables-alerter-km5nl" Jul 09 18:58:52.095915 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.093470 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/47f9bbdb-bfc1-464a-a495-b310124b104b-etc-sysctl-d\") pod \"tuned-hrjdx\" (UID: \"47f9bbdb-bfc1-464a-a495-b310124b104b\") " pod="openshift-cluster-node-tuning-operator/tuned-hrjdx" Jul 09 18:58:52.095915 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.093501 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e5c65950-27b0-41de-8ee1-e2ced769d6b4-host-var-lib-cni-multus\") pod \"multus-5mwmg\" (UID: \"e5c65950-27b0-41de-8ee1-e2ced769d6b4\") " pod="openshift-multus/multus-5mwmg" Jul 09 18:58:52.095915 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.093507 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e5c65950-27b0-41de-8ee1-e2ced769d6b4-hostroot\") pod \"multus-5mwmg\" (UID: \"e5c65950-27b0-41de-8ee1-e2ced769d6b4\") " pod="openshift-multus/multus-5mwmg" Jul 09 18:58:52.095915 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.093533 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b1f91f8e-9064-4538-a912-6e6354050534-device-dir\") pod \"aws-ebs-csi-driver-node-gcfcz\" (UID: \"b1f91f8e-9064-4538-a912-6e6354050534\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcfcz" Jul 09 18:58:52.095915 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.093554 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d40ecb0c-9c3c-4fbe-ad50-742b0f848c44-host-slash\") pod \"iptables-alerter-km5nl\" (UID: \"d40ecb0c-9c3c-4fbe-ad50-742b0f848c44\") " pod="openshift-network-operator/iptables-alerter-km5nl" Jul 09 18:58:52.095915 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.093561 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bndfr\" (UniqueName: \"kubernetes.io/projected/0da71879-5de7-47a0-819a-0b1951b1e817-kube-api-access-bndfr\") pod \"node-resolver-x8k65\" (UID: \"0da71879-5de7-47a0-819a-0b1951b1e817\") " pod="openshift-dns/node-resolver-x8k65" Jul 09 18:58:52.095915 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.093619 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/db233460-134d-4d9d-8da9-31d89366a4e0-run-ovn\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:52.096398 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.093632 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b1f91f8e-9064-4538-a912-6e6354050534-device-dir\") pod \"aws-ebs-csi-driver-node-gcfcz\" (UID: \"b1f91f8e-9064-4538-a912-6e6354050534\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcfcz" Jul 09 18:58:52.096398 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.093645 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/db233460-134d-4d9d-8da9-31d89366a4e0-host-cni-bin\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:52.096398 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.093669 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e5c65950-27b0-41de-8ee1-e2ced769d6b4-multus-socket-dir-parent\") pod \"multus-5mwmg\" (UID: \"e5c65950-27b0-41de-8ee1-e2ced769d6b4\") " pod="openshift-multus/multus-5mwmg" Jul 09 18:58:52.096398 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.093682 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/db233460-134d-4d9d-8da9-31d89366a4e0-run-ovn\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:52.096398 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.093694 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e5c65950-27b0-41de-8ee1-e2ced769d6b4-host-run-netns\") pod \"multus-5mwmg\" (UID: \"e5c65950-27b0-41de-8ee1-e2ced769d6b4\") " pod="openshift-multus/multus-5mwmg" Jul 09 18:58:52.096398 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.093716 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5c65950-27b0-41de-8ee1-e2ced769d6b4-host-var-lib-kubelet\") pod \"multus-5mwmg\" (UID: \"e5c65950-27b0-41de-8ee1-e2ced769d6b4\") " pod="openshift-multus/multus-5mwmg" Jul 09 18:58:52.096398 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.093740 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/db233460-134d-4d9d-8da9-31d89366a4e0-log-socket\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:52.096398 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.093759 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e5c65950-27b0-41de-8ee1-e2ced769d6b4-multus-socket-dir-parent\") pod \"multus-5mwmg\" (UID: \"e5c65950-27b0-41de-8ee1-e2ced769d6b4\") " pod="openshift-multus/multus-5mwmg" Jul 09 18:58:52.096398 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.093764 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/db233460-134d-4d9d-8da9-31d89366a4e0-ovnkube-script-lib\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:52.096398 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.093739 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/db233460-134d-4d9d-8da9-31d89366a4e0-host-cni-bin\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:52.096398 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.093710 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/db52b7ec-a201-4d65-81b3-56e37f0b6d36-tuning-conf-dir\") pod \"multus-additional-cni-plugins-swdmr\" (UID: \"db52b7ec-a201-4d65-81b3-56e37f0b6d36\") " pod="openshift-multus/multus-additional-cni-plugins-swdmr" Jul 09 18:58:52.096398 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.093802 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/db233460-134d-4d9d-8da9-31d89366a4e0-log-socket\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:52.096398 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.093771 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5c65950-27b0-41de-8ee1-e2ced769d6b4-host-var-lib-kubelet\") pod \"multus-5mwmg\" (UID: \"e5c65950-27b0-41de-8ee1-e2ced769d6b4\") " pod="openshift-multus/multus-5mwmg" Jul 09 18:58:52.096398 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.093810 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e5c65950-27b0-41de-8ee1-e2ced769d6b4-host-run-netns\") pod \"multus-5mwmg\" (UID: \"e5c65950-27b0-41de-8ee1-e2ced769d6b4\") " pod="openshift-multus/multus-5mwmg" Jul 09 18:58:52.096398 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.093959 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/47f9bbdb-bfc1-464a-a495-b310124b104b-etc-tuned\") pod \"tuned-hrjdx\" (UID: \"47f9bbdb-bfc1-464a-a495-b310124b104b\") " pod="openshift-cluster-node-tuning-operator/tuned-hrjdx" Jul 09 18:58:52.096398 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.093991 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/47f9bbdb-bfc1-464a-a495-b310124b104b-tmp\") pod \"tuned-hrjdx\" (UID: \"47f9bbdb-bfc1-464a-a495-b310124b104b\") " pod="openshift-cluster-node-tuning-operator/tuned-hrjdx" Jul 09 18:58:52.096398 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.094093 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/db52b7ec-a201-4d65-81b3-56e37f0b6d36-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-swdmr\" (UID: \"db52b7ec-a201-4d65-81b3-56e37f0b6d36\") " pod="openshift-multus/multus-additional-cni-plugins-swdmr" Jul 09 18:58:52.096398 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.095384 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/db233460-134d-4d9d-8da9-31d89366a4e0-ovn-node-metrics-cert\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:52.102642 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:52.102623 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 18:58:52.102642 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:52.102645 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 18:58:52.102798 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:52.102655 2566 projected.go:196] Error preparing data for projected volume kube-api-access-pntjm for pod openshift-network-diagnostics/network-check-target-mtz87: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 18:58:52.102798 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:52.102698 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fe73719-af90-4093-ae73-d4e462e5e43c-kube-api-access-pntjm podName:2fe73719-af90-4093-ae73-d4e462e5e43c nodeName:}" failed. No retries permitted until 2026-07-09 18:58:52.602685846 +0000 UTC m=+2.110465599 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-pntjm" (UniqueName: "kubernetes.io/projected/2fe73719-af90-4093-ae73-d4e462e5e43c-kube-api-access-pntjm") pod "network-check-target-mtz87" (UID: "2fe73719-af90-4093-ae73-d4e462e5e43c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 18:58:52.104057 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.104042 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf2vh\" (UniqueName: \"kubernetes.io/projected/47f9bbdb-bfc1-464a-a495-b310124b104b-kube-api-access-kf2vh\") pod \"tuned-hrjdx\" (UID: \"47f9bbdb-bfc1-464a-a495-b310124b104b\") " pod="openshift-cluster-node-tuning-operator/tuned-hrjdx" Jul 09 18:58:52.106892 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.106867 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bndfr\" (UniqueName: \"kubernetes.io/projected/0da71879-5de7-47a0-819a-0b1951b1e817-kube-api-access-bndfr\") pod \"node-resolver-x8k65\" (UID: \"0da71879-5de7-47a0-819a-0b1951b1e817\") " pod="openshift-dns/node-resolver-x8k65" Jul 09 18:58:52.107326 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.107301 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92n9z\" (UniqueName: \"kubernetes.io/projected/db52b7ec-a201-4d65-81b3-56e37f0b6d36-kube-api-access-92n9z\") pod \"multus-additional-cni-plugins-swdmr\" (UID: \"db52b7ec-a201-4d65-81b3-56e37f0b6d36\") " pod="openshift-multus/multus-additional-cni-plugins-swdmr" Jul 09 18:58:52.107430 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.107324 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5bln\" (UniqueName: \"kubernetes.io/projected/e5c65950-27b0-41de-8ee1-e2ced769d6b4-kube-api-access-q5bln\") pod \"multus-5mwmg\" (UID: \"e5c65950-27b0-41de-8ee1-e2ced769d6b4\") " pod="openshift-multus/multus-5mwmg" Jul 09 18:58:52.107430 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.107373 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlbxl\" (UniqueName: \"kubernetes.io/projected/5c298798-c68a-4450-b9dc-358fa978305a-kube-api-access-xlbxl\") pod \"node-ca-jnt6g\" (UID: \"5c298798-c68a-4450-b9dc-358fa978305a\") " pod="openshift-image-registry/node-ca-jnt6g" Jul 09 18:58:52.107823 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.107799 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzzvp\" (UniqueName: \"kubernetes.io/projected/30cf5feb-cf9a-40c0-8a6a-de8fc6c60506-kube-api-access-gzzvp\") pod \"network-metrics-daemon-vtc8p\" (UID: \"30cf5feb-cf9a-40c0-8a6a-de8fc6c60506\") " pod="openshift-multus/network-metrics-daemon-vtc8p" Jul 09 18:58:52.107900 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.107867 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhtgh\" (UniqueName: \"kubernetes.io/projected/d40ecb0c-9c3c-4fbe-ad50-742b0f848c44-kube-api-access-jhtgh\") pod \"iptables-alerter-km5nl\" (UID: \"d40ecb0c-9c3c-4fbe-ad50-742b0f848c44\") " pod="openshift-network-operator/iptables-alerter-km5nl" Jul 09 18:58:52.108313 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.108293 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94s46\" (UniqueName: \"kubernetes.io/projected/db233460-134d-4d9d-8da9-31d89366a4e0-kube-api-access-94s46\") pod \"ovnkube-node-6dh7g\" (UID: \"db233460-134d-4d9d-8da9-31d89366a4e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:52.108773 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.108759 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dqrd\" (UniqueName: \"kubernetes.io/projected/b1f91f8e-9064-4538-a912-6e6354050534-kube-api-access-4dqrd\") pod \"aws-ebs-csi-driver-node-gcfcz\" (UID: \"b1f91f8e-9064-4538-a912-6e6354050534\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcfcz" Jul 09 18:58:52.216036 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.216010 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-c2bn7" Jul 09 18:58:52.224696 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:52.224663 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b7fa90_97cc_4b69_80d4_d63ea5f30d6f.slice/crio-7254f5aca2a39332f9110077e04d8d5d444085fd2e479346bc14b90cc1afbc06 WatchSource:0}: Error finding container 7254f5aca2a39332f9110077e04d8d5d444085fd2e479346bc14b90cc1afbc06: Status 404 returned error can't find the container with id 7254f5aca2a39332f9110077e04d8d5d444085fd2e479346bc14b90cc1afbc06 Jul 09 18:58:52.226721 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.226699 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hrjdx" Jul 09 18:58:52.233809 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:52.233786 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47f9bbdb_bfc1_464a_a495_b310124b104b.slice/crio-aab3882568fadb6f870ec629e9417cd55772888ebbe4f05c1123edf35a9eb8b3 WatchSource:0}: Error finding container aab3882568fadb6f870ec629e9417cd55772888ebbe4f05c1123edf35a9eb8b3: Status 404 returned error can't find the container with id aab3882568fadb6f870ec629e9417cd55772888ebbe4f05c1123edf35a9eb8b3 Jul 09 18:58:52.246413 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.246392 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-x8k65" Jul 09 18:58:52.253297 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:52.253264 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0da71879_5de7_47a0_819a_0b1951b1e817.slice/crio-1728a7d8238d86ea0ec425545c424935d49346ab7a6204f067a7bc3113df1b66 WatchSource:0}: Error finding container 1728a7d8238d86ea0ec425545c424935d49346ab7a6204f067a7bc3113df1b66: Status 404 returned error can't find the container with id 1728a7d8238d86ea0ec425545c424935d49346ab7a6204f067a7bc3113df1b66 Jul 09 18:58:52.260279 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.260261 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-swdmr" Jul 09 18:58:52.267730 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:52.267707 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb52b7ec_a201_4d65_81b3_56e37f0b6d36.slice/crio-0cac66ba0904d2904e06cc557a764726a1a4844e6c280aa94ce718a966fbbfc7 WatchSource:0}: Error finding container 0cac66ba0904d2904e06cc557a764726a1a4844e6c280aa94ce718a966fbbfc7: Status 404 returned error can't find the container with id 0cac66ba0904d2904e06cc557a764726a1a4844e6c280aa94ce718a966fbbfc7 Jul 09 18:58:52.280635 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.280591 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-km5nl" Jul 09 18:58:52.287695 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:52.287673 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd40ecb0c_9c3c_4fbe_ad50_742b0f848c44.slice/crio-b3797c0cddb78c63826591cbf1832dcd83b8da069fb33980c6b9850bd9ace564 WatchSource:0}: Error finding container b3797c0cddb78c63826591cbf1832dcd83b8da069fb33980c6b9850bd9ace564: Status 404 returned error can't find the container with id b3797c0cddb78c63826591cbf1832dcd83b8da069fb33980c6b9850bd9ace564 Jul 09 18:58:52.297129 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.297111 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:58:52.302862 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.302847 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcfcz" Jul 09 18:58:52.303821 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:52.303801 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb233460_134d_4d9d_8da9_31d89366a4e0.slice/crio-6a5bdc58d4133e4ebf710a62b5d956c696de787c1aa47cd64270b7b772812c5e WatchSource:0}: Error finding container 6a5bdc58d4133e4ebf710a62b5d956c696de787c1aa47cd64270b7b772812c5e: Status 404 returned error can't find the container with id 6a5bdc58d4133e4ebf710a62b5d956c696de787c1aa47cd64270b7b772812c5e Jul 09 18:58:52.310055 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:52.310033 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1f91f8e_9064_4538_a912_6e6354050534.slice/crio-de6dfc9fcd7850b782f08a51275ba9cef3da57223dce40809801a95a17ba3c6f WatchSource:0}: Error finding container de6dfc9fcd7850b782f08a51275ba9cef3da57223dce40809801a95a17ba3c6f: Status 404 returned error can't find the container with id de6dfc9fcd7850b782f08a51275ba9cef3da57223dce40809801a95a17ba3c6f Jul 09 18:58:52.315329 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.315311 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jnt6g" Jul 09 18:58:52.321446 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.321426 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5mwmg" Jul 09 18:58:52.324955 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:52.324937 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c298798_c68a_4450_b9dc_358fa978305a.slice/crio-013cb4c87f84ee7600bafa77044b128d427fbf538633c6637a8e49571cb80b85 WatchSource:0}: Error finding container 013cb4c87f84ee7600bafa77044b128d427fbf538633c6637a8e49571cb80b85: Status 404 returned error can't find the container with id 013cb4c87f84ee7600bafa77044b128d427fbf538633c6637a8e49571cb80b85 Jul 09 18:58:52.329088 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:58:52.329068 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5c65950_27b0_41de_8ee1_e2ced769d6b4.slice/crio-dd50b36958307444f3a321149cff1d8e71f08f0dca14d2f43022b1dc8f908466 WatchSource:0}: Error finding container dd50b36958307444f3a321149cff1d8e71f08f0dca14d2f43022b1dc8f908466: Status 404 returned error can't find the container with id dd50b36958307444f3a321149cff1d8e71f08f0dca14d2f43022b1dc8f908466 Jul 09 18:58:52.599836 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.599757 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7435737c-5954-4ef0-8b41-1889b8231561-original-pull-secret\") pod \"global-pull-secret-syncer-tqtv5\" (UID: \"7435737c-5954-4ef0-8b41-1889b8231561\") " pod="kube-system/global-pull-secret-syncer-tqtv5" Jul 09 18:58:52.599836 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.599805 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30cf5feb-cf9a-40c0-8a6a-de8fc6c60506-metrics-certs\") pod \"network-metrics-daemon-vtc8p\" (UID: \"30cf5feb-cf9a-40c0-8a6a-de8fc6c60506\") " pod="openshift-multus/network-metrics-daemon-vtc8p" Jul 09 18:58:52.600043 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:52.599917 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 18:58:52.600043 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:52.599977 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30cf5feb-cf9a-40c0-8a6a-de8fc6c60506-metrics-certs podName:30cf5feb-cf9a-40c0-8a6a-de8fc6c60506 nodeName:}" failed. No retries permitted until 2026-07-09 18:58:53.599959735 +0000 UTC m=+3.107739486 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30cf5feb-cf9a-40c0-8a6a-de8fc6c60506-metrics-certs") pod "network-metrics-daemon-vtc8p" (UID: "30cf5feb-cf9a-40c0-8a6a-de8fc6c60506") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 18:58:52.600398 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:52.600376 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 18:58:52.600502 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:52.600438 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7435737c-5954-4ef0-8b41-1889b8231561-original-pull-secret podName:7435737c-5954-4ef0-8b41-1889b8231561 nodeName:}" failed. No retries permitted until 2026-07-09 18:58:53.600421667 +0000 UTC m=+3.108201423 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7435737c-5954-4ef0-8b41-1889b8231561-original-pull-secret") pod "global-pull-secret-syncer-tqtv5" (UID: "7435737c-5954-4ef0-8b41-1889b8231561") : object "kube-system"/"original-pull-secret" not registered Jul 09 18:58:52.701588 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.700943 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pntjm\" (UniqueName: \"kubernetes.io/projected/2fe73719-af90-4093-ae73-d4e462e5e43c-kube-api-access-pntjm\") pod \"network-check-target-mtz87\" (UID: \"2fe73719-af90-4093-ae73-d4e462e5e43c\") " pod="openshift-network-diagnostics/network-check-target-mtz87" Jul 09 18:58:52.701588 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:52.701096 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 18:58:52.701588 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:52.701117 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 18:58:52.701588 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:52.701130 2566 projected.go:196] Error preparing data for projected volume kube-api-access-pntjm for pod openshift-network-diagnostics/network-check-target-mtz87: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 18:58:52.701588 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:52.701234 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fe73719-af90-4093-ae73-d4e462e5e43c-kube-api-access-pntjm podName:2fe73719-af90-4093-ae73-d4e462e5e43c nodeName:}" failed. No retries permitted until 2026-07-09 18:58:53.701205539 +0000 UTC m=+3.208985311 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-pntjm" (UniqueName: "kubernetes.io/projected/2fe73719-af90-4093-ae73-d4e462e5e43c-kube-api-access-pntjm") pod "network-check-target-mtz87" (UID: "2fe73719-af90-4093-ae73-d4e462e5e43c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 18:58:52.798251 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:52.798219 2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 18:58:53.015536 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:53.015412 2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 18:53:52 +0000 UTC" deadline="2028-01-03 02:26:07.140464761 +0000 UTC" Jul 09 18:58:53.015536 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:53.015476 2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13015h27m14.124993096s" Jul 09 18:58:53.066101 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:53.066073 2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 18:58:53.073134 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:53.072272 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jnt6g" event={"ID":"5c298798-c68a-4450-b9dc-358fa978305a","Type":"ContainerStarted","Data":"013cb4c87f84ee7600bafa77044b128d427fbf538633c6637a8e49571cb80b85"} Jul 09 18:58:53.086531 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:53.086494 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" event={"ID":"db233460-134d-4d9d-8da9-31d89366a4e0","Type":"ContainerStarted","Data":"6a5bdc58d4133e4ebf710a62b5d956c696de787c1aa47cd64270b7b772812c5e"} Jul 09 18:58:53.115859 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:53.115728 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-km5nl" event={"ID":"d40ecb0c-9c3c-4fbe-ad50-742b0f848c44","Type":"ContainerStarted","Data":"b3797c0cddb78c63826591cbf1832dcd83b8da069fb33980c6b9850bd9ace564"} Jul 09 18:58:53.141098 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:53.140798 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-swdmr" event={"ID":"db52b7ec-a201-4d65-81b3-56e37f0b6d36","Type":"ContainerStarted","Data":"0cac66ba0904d2904e06cc557a764726a1a4844e6c280aa94ce718a966fbbfc7"} Jul 09 18:58:53.153665 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:53.153420 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-x8k65" event={"ID":"0da71879-5de7-47a0-819a-0b1951b1e817","Type":"ContainerStarted","Data":"1728a7d8238d86ea0ec425545c424935d49346ab7a6204f067a7bc3113df1b66"} Jul 09 18:58:53.182847 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:53.182811 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5mwmg" event={"ID":"e5c65950-27b0-41de-8ee1-e2ced769d6b4","Type":"ContainerStarted","Data":"dd50b36958307444f3a321149cff1d8e71f08f0dca14d2f43022b1dc8f908466"} Jul 09 18:58:53.215722 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:53.215685 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcfcz" event={"ID":"b1f91f8e-9064-4538-a912-6e6354050534","Type":"ContainerStarted","Data":"de6dfc9fcd7850b782f08a51275ba9cef3da57223dce40809801a95a17ba3c6f"} Jul 09 18:58:53.220119 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:53.220087 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hrjdx" event={"ID":"47f9bbdb-bfc1-464a-a495-b310124b104b","Type":"ContainerStarted","Data":"aab3882568fadb6f870ec629e9417cd55772888ebbe4f05c1123edf35a9eb8b3"} Jul 09 18:58:53.237225 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:53.237190 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-c2bn7" event={"ID":"51b7fa90-97cc-4b69-80d4-d63ea5f30d6f","Type":"ContainerStarted","Data":"7254f5aca2a39332f9110077e04d8d5d444085fd2e479346bc14b90cc1afbc06"} Jul 09 18:58:53.504391 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:53.504113 2566 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 18:58:53.610089 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:53.610043 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7435737c-5954-4ef0-8b41-1889b8231561-original-pull-secret\") pod \"global-pull-secret-syncer-tqtv5\" (UID: \"7435737c-5954-4ef0-8b41-1889b8231561\") " pod="kube-system/global-pull-secret-syncer-tqtv5" Jul 09 18:58:53.610254 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:53.610104 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30cf5feb-cf9a-40c0-8a6a-de8fc6c60506-metrics-certs\") pod \"network-metrics-daemon-vtc8p\" (UID: \"30cf5feb-cf9a-40c0-8a6a-de8fc6c60506\") " pod="openshift-multus/network-metrics-daemon-vtc8p" Jul 09 18:58:53.610254 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:53.610220 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 18:58:53.610377 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:53.610286 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30cf5feb-cf9a-40c0-8a6a-de8fc6c60506-metrics-certs podName:30cf5feb-cf9a-40c0-8a6a-de8fc6c60506 nodeName:}" failed. No retries permitted until 2026-07-09 18:58:55.610266998 +0000 UTC m=+5.118046753 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30cf5feb-cf9a-40c0-8a6a-de8fc6c60506-metrics-certs") pod "network-metrics-daemon-vtc8p" (UID: "30cf5feb-cf9a-40c0-8a6a-de8fc6c60506") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 18:58:53.610377 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:53.610221 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 18:58:53.610377 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:53.610372 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7435737c-5954-4ef0-8b41-1889b8231561-original-pull-secret podName:7435737c-5954-4ef0-8b41-1889b8231561 nodeName:}" failed. No retries permitted until 2026-07-09 18:58:55.610353659 +0000 UTC m=+5.118133427 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7435737c-5954-4ef0-8b41-1889b8231561-original-pull-secret") pod "global-pull-secret-syncer-tqtv5" (UID: "7435737c-5954-4ef0-8b41-1889b8231561") : object "kube-system"/"original-pull-secret" not registered Jul 09 18:58:53.711571 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:53.710893 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pntjm\" (UniqueName: \"kubernetes.io/projected/2fe73719-af90-4093-ae73-d4e462e5e43c-kube-api-access-pntjm\") pod \"network-check-target-mtz87\" (UID: \"2fe73719-af90-4093-ae73-d4e462e5e43c\") " pod="openshift-network-diagnostics/network-check-target-mtz87" Jul 09 18:58:53.711571 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:53.711109 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 18:58:53.711571 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:53.711133 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 18:58:53.711571 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:53.711147 2566 projected.go:196] Error preparing data for projected volume kube-api-access-pntjm for pod openshift-network-diagnostics/network-check-target-mtz87: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 18:58:53.711571 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:53.711208 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fe73719-af90-4093-ae73-d4e462e5e43c-kube-api-access-pntjm podName:2fe73719-af90-4093-ae73-d4e462e5e43c nodeName:}" failed. No retries permitted until 2026-07-09 18:58:55.711188136 +0000 UTC m=+5.218967913 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-pntjm" (UniqueName: "kubernetes.io/projected/2fe73719-af90-4093-ae73-d4e462e5e43c-kube-api-access-pntjm") pod "network-check-target-mtz87" (UID: "2fe73719-af90-4093-ae73-d4e462e5e43c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 18:58:54.015987 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:54.015935 2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 18:53:52 +0000 UTC" deadline="2028-02-17 17:28:59.405282572 +0000 UTC" Jul 09 18:58:54.015987 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:54.015986 2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14110h30m5.389301319s" Jul 09 18:58:54.051136 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:54.051087 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tqtv5" Jul 09 18:58:54.051307 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:54.051208 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tqtv5" podUID="7435737c-5954-4ef0-8b41-1889b8231561" Jul 09 18:58:54.051486 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:54.051465 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtc8p" Jul 09 18:58:54.051548 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:54.051508 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mtz87" Jul 09 18:58:54.051719 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:54.051686 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtc8p" podUID="30cf5feb-cf9a-40c0-8a6a-de8fc6c60506" Jul 09 18:58:54.051818 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:54.051793 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mtz87" podUID="2fe73719-af90-4093-ae73-d4e462e5e43c" Jul 09 18:58:54.056066 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:54.055783 2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 18:58:55.629173 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:55.628386 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30cf5feb-cf9a-40c0-8a6a-de8fc6c60506-metrics-certs\") pod \"network-metrics-daemon-vtc8p\" (UID: \"30cf5feb-cf9a-40c0-8a6a-de8fc6c60506\") " pod="openshift-multus/network-metrics-daemon-vtc8p" Jul 09 18:58:55.629173 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:55.628486 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7435737c-5954-4ef0-8b41-1889b8231561-original-pull-secret\") pod \"global-pull-secret-syncer-tqtv5\" (UID: \"7435737c-5954-4ef0-8b41-1889b8231561\") " pod="kube-system/global-pull-secret-syncer-tqtv5" Jul 09 18:58:55.629173 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:55.628629 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 18:58:55.629173 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:55.628691 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7435737c-5954-4ef0-8b41-1889b8231561-original-pull-secret podName:7435737c-5954-4ef0-8b41-1889b8231561 nodeName:}" failed. No retries permitted until 2026-07-09 18:58:59.628673391 +0000 UTC m=+9.136453146 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7435737c-5954-4ef0-8b41-1889b8231561-original-pull-secret") pod "global-pull-secret-syncer-tqtv5" (UID: "7435737c-5954-4ef0-8b41-1889b8231561") : object "kube-system"/"original-pull-secret" not registered Jul 09 18:58:55.629173 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:55.629068 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 18:58:55.629173 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:55.629117 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30cf5feb-cf9a-40c0-8a6a-de8fc6c60506-metrics-certs podName:30cf5feb-cf9a-40c0-8a6a-de8fc6c60506 nodeName:}" failed. No retries permitted until 2026-07-09 18:58:59.629101503 +0000 UTC m=+9.136881257 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30cf5feb-cf9a-40c0-8a6a-de8fc6c60506-metrics-certs") pod "network-metrics-daemon-vtc8p" (UID: "30cf5feb-cf9a-40c0-8a6a-de8fc6c60506") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 18:58:55.729182 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:55.729057 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pntjm\" (UniqueName: \"kubernetes.io/projected/2fe73719-af90-4093-ae73-d4e462e5e43c-kube-api-access-pntjm\") pod \"network-check-target-mtz87\" (UID: \"2fe73719-af90-4093-ae73-d4e462e5e43c\") " pod="openshift-network-diagnostics/network-check-target-mtz87" Jul 09 18:58:55.729356 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:55.729259 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 18:58:55.729356 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:55.729289 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 18:58:55.729356 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:55.729304 2566 projected.go:196] Error preparing data for projected volume kube-api-access-pntjm for pod openshift-network-diagnostics/network-check-target-mtz87: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 18:58:55.729516 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:55.729363 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fe73719-af90-4093-ae73-d4e462e5e43c-kube-api-access-pntjm podName:2fe73719-af90-4093-ae73-d4e462e5e43c nodeName:}" failed. No retries permitted until 2026-07-09 18:58:59.729345784 +0000 UTC m=+9.237125555 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-pntjm" (UniqueName: "kubernetes.io/projected/2fe73719-af90-4093-ae73-d4e462e5e43c-kube-api-access-pntjm") pod "network-check-target-mtz87" (UID: "2fe73719-af90-4093-ae73-d4e462e5e43c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 18:58:56.051531 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:56.051489 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtc8p" Jul 09 18:58:56.051713 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:56.051653 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtc8p" podUID="30cf5feb-cf9a-40c0-8a6a-de8fc6c60506" Jul 09 18:58:56.051713 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:56.051489 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tqtv5" Jul 09 18:58:56.051848 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:56.051747 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tqtv5" podUID="7435737c-5954-4ef0-8b41-1889b8231561" Jul 09 18:58:56.051848 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:56.051489 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mtz87" Jul 09 18:58:56.051848 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:56.051823 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mtz87" podUID="2fe73719-af90-4093-ae73-d4e462e5e43c" Jul 09 18:58:58.052009 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:58.051315 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tqtv5" Jul 09 18:58:58.052009 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:58.051454 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tqtv5" podUID="7435737c-5954-4ef0-8b41-1889b8231561" Jul 09 18:58:58.052009 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:58.051324 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mtz87" Jul 09 18:58:58.052009 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:58.051549 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mtz87" podUID="2fe73719-af90-4093-ae73-d4e462e5e43c" Jul 09 18:58:58.052009 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:58.051324 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtc8p" Jul 09 18:58:58.052009 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:58.051647 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtc8p" podUID="30cf5feb-cf9a-40c0-8a6a-de8fc6c60506" Jul 09 18:58:59.663091 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:59.663056 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7435737c-5954-4ef0-8b41-1889b8231561-original-pull-secret\") pod \"global-pull-secret-syncer-tqtv5\" (UID: \"7435737c-5954-4ef0-8b41-1889b8231561\") " pod="kube-system/global-pull-secret-syncer-tqtv5" Jul 09 18:58:59.663091 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:59.663107 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30cf5feb-cf9a-40c0-8a6a-de8fc6c60506-metrics-certs\") pod \"network-metrics-daemon-vtc8p\" (UID: \"30cf5feb-cf9a-40c0-8a6a-de8fc6c60506\") " pod="openshift-multus/network-metrics-daemon-vtc8p" Jul 09 18:58:59.663642 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:59.663219 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 18:58:59.663642 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:59.663272 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30cf5feb-cf9a-40c0-8a6a-de8fc6c60506-metrics-certs podName:30cf5feb-cf9a-40c0-8a6a-de8fc6c60506 nodeName:}" failed. No retries permitted until 2026-07-09 18:59:07.663258438 +0000 UTC m=+17.171038189 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30cf5feb-cf9a-40c0-8a6a-de8fc6c60506-metrics-certs") pod "network-metrics-daemon-vtc8p" (UID: "30cf5feb-cf9a-40c0-8a6a-de8fc6c60506") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 18:58:59.663642 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:59.663219 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 18:58:59.663642 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:59.663339 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7435737c-5954-4ef0-8b41-1889b8231561-original-pull-secret podName:7435737c-5954-4ef0-8b41-1889b8231561 nodeName:}" failed. No retries permitted until 2026-07-09 18:59:07.663327699 +0000 UTC m=+17.171107464 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7435737c-5954-4ef0-8b41-1889b8231561-original-pull-secret") pod "global-pull-secret-syncer-tqtv5" (UID: "7435737c-5954-4ef0-8b41-1889b8231561") : object "kube-system"/"original-pull-secret" not registered Jul 09 18:58:59.764492 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:58:59.764420 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pntjm\" (UniqueName: \"kubernetes.io/projected/2fe73719-af90-4093-ae73-d4e462e5e43c-kube-api-access-pntjm\") pod \"network-check-target-mtz87\" (UID: \"2fe73719-af90-4093-ae73-d4e462e5e43c\") " pod="openshift-network-diagnostics/network-check-target-mtz87" Jul 09 18:58:59.764700 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:59.764628 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 18:58:59.764700 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:59.764657 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 18:58:59.764700 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:59.764674 2566 projected.go:196] Error preparing data for projected volume kube-api-access-pntjm for pod openshift-network-diagnostics/network-check-target-mtz87: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 18:58:59.764868 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:58:59.764740 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fe73719-af90-4093-ae73-d4e462e5e43c-kube-api-access-pntjm podName:2fe73719-af90-4093-ae73-d4e462e5e43c nodeName:}" failed. No retries permitted until 2026-07-09 18:59:07.764720061 +0000 UTC m=+17.272499830 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-pntjm" (UniqueName: "kubernetes.io/projected/2fe73719-af90-4093-ae73-d4e462e5e43c-kube-api-access-pntjm") pod "network-check-target-mtz87" (UID: "2fe73719-af90-4093-ae73-d4e462e5e43c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 18:59:00.051647 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:00.051597 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtc8p" Jul 09 18:59:00.051647 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:00.051651 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tqtv5" Jul 09 18:59:00.051887 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:00.051597 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mtz87" Jul 09 18:59:00.051887 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:00.051750 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtc8p" podUID="30cf5feb-cf9a-40c0-8a6a-de8fc6c60506" Jul 09 18:59:00.051887 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:00.051809 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mtz87" podUID="2fe73719-af90-4093-ae73-d4e462e5e43c" Jul 09 18:59:00.051887 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:00.051875 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tqtv5" podUID="7435737c-5954-4ef0-8b41-1889b8231561" Jul 09 18:59:02.051574 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:02.051198 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tqtv5" Jul 09 18:59:02.051574 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:02.051568 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tqtv5" podUID="7435737c-5954-4ef0-8b41-1889b8231561" Jul 09 18:59:02.052152 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:02.051305 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mtz87" Jul 09 18:59:02.052152 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:02.051683 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mtz87" podUID="2fe73719-af90-4093-ae73-d4e462e5e43c" Jul 09 18:59:02.052152 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:02.051244 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtc8p" Jul 09 18:59:02.052152 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:02.051776 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtc8p" podUID="30cf5feb-cf9a-40c0-8a6a-de8fc6c60506" Jul 09 18:59:04.051204 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:04.051145 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tqtv5" Jul 09 18:59:04.051204 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:04.051184 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtc8p" Jul 09 18:59:04.051742 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:04.051184 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mtz87" Jul 09 18:59:04.051742 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:04.051279 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tqtv5" podUID="7435737c-5954-4ef0-8b41-1889b8231561" Jul 09 18:59:04.051742 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:04.051390 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtc8p" podUID="30cf5feb-cf9a-40c0-8a6a-de8fc6c60506" Jul 09 18:59:04.051742 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:04.051471 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mtz87" podUID="2fe73719-af90-4093-ae73-d4e462e5e43c" Jul 09 18:59:06.051879 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:06.051846 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tqtv5" Jul 09 18:59:06.052360 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:06.051960 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mtz87" Jul 09 18:59:06.052360 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:06.051976 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tqtv5" podUID="7435737c-5954-4ef0-8b41-1889b8231561" Jul 09 18:59:06.052360 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:06.052006 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtc8p" Jul 09 18:59:06.052360 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:06.052090 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mtz87" podUID="2fe73719-af90-4093-ae73-d4e462e5e43c" Jul 09 18:59:06.052360 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:06.052171 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtc8p" podUID="30cf5feb-cf9a-40c0-8a6a-de8fc6c60506" Jul 09 18:59:07.722189 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:07.722157 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7435737c-5954-4ef0-8b41-1889b8231561-original-pull-secret\") pod \"global-pull-secret-syncer-tqtv5\" (UID: \"7435737c-5954-4ef0-8b41-1889b8231561\") " pod="kube-system/global-pull-secret-syncer-tqtv5" Jul 09 18:59:07.722773 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:07.722210 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30cf5feb-cf9a-40c0-8a6a-de8fc6c60506-metrics-certs\") pod \"network-metrics-daemon-vtc8p\" (UID: \"30cf5feb-cf9a-40c0-8a6a-de8fc6c60506\") " pod="openshift-multus/network-metrics-daemon-vtc8p" Jul 09 18:59:07.722773 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:07.722324 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 18:59:07.722773 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:07.722383 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30cf5feb-cf9a-40c0-8a6a-de8fc6c60506-metrics-certs podName:30cf5feb-cf9a-40c0-8a6a-de8fc6c60506 nodeName:}" failed. No retries permitted until 2026-07-09 18:59:23.72237009 +0000 UTC m=+33.230149856 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30cf5feb-cf9a-40c0-8a6a-de8fc6c60506-metrics-certs") pod "network-metrics-daemon-vtc8p" (UID: "30cf5feb-cf9a-40c0-8a6a-de8fc6c60506") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 18:59:07.722773 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:07.722324 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 18:59:07.722773 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:07.722448 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7435737c-5954-4ef0-8b41-1889b8231561-original-pull-secret podName:7435737c-5954-4ef0-8b41-1889b8231561 nodeName:}" failed. No retries permitted until 2026-07-09 18:59:23.722436245 +0000 UTC m=+33.230216010 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7435737c-5954-4ef0-8b41-1889b8231561-original-pull-secret") pod "global-pull-secret-syncer-tqtv5" (UID: "7435737c-5954-4ef0-8b41-1889b8231561") : object "kube-system"/"original-pull-secret" not registered Jul 09 18:59:07.822584 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:07.822548 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pntjm\" (UniqueName: \"kubernetes.io/projected/2fe73719-af90-4093-ae73-d4e462e5e43c-kube-api-access-pntjm\") pod \"network-check-target-mtz87\" (UID: \"2fe73719-af90-4093-ae73-d4e462e5e43c\") " pod="openshift-network-diagnostics/network-check-target-mtz87" Jul 09 18:59:07.822754 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:07.822728 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 18:59:07.822806 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:07.822753 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 18:59:07.822806 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:07.822766 2566 projected.go:196] Error preparing data for projected volume kube-api-access-pntjm for pod openshift-network-diagnostics/network-check-target-mtz87: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 18:59:07.822873 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:07.822824 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fe73719-af90-4093-ae73-d4e462e5e43c-kube-api-access-pntjm podName:2fe73719-af90-4093-ae73-d4e462e5e43c nodeName:}" failed. No retries permitted until 2026-07-09 18:59:23.822810398 +0000 UTC m=+33.330590150 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-pntjm" (UniqueName: "kubernetes.io/projected/2fe73719-af90-4093-ae73-d4e462e5e43c-kube-api-access-pntjm") pod "network-check-target-mtz87" (UID: "2fe73719-af90-4093-ae73-d4e462e5e43c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 18:59:08.051024 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:08.050944 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mtz87" Jul 09 18:59:08.051024 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:08.050969 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtc8p" Jul 09 18:59:08.051238 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:08.050961 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tqtv5" Jul 09 18:59:08.051238 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:08.051073 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mtz87" podUID="2fe73719-af90-4093-ae73-d4e462e5e43c" Jul 09 18:59:08.051238 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:08.051134 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tqtv5" podUID="7435737c-5954-4ef0-8b41-1889b8231561" Jul 09 18:59:08.051238 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:08.051212 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtc8p" podUID="30cf5feb-cf9a-40c0-8a6a-de8fc6c60506" Jul 09 18:59:10.051671 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:10.051636 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtc8p" Jul 09 18:59:10.052097 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:10.051636 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mtz87" Jul 09 18:59:10.052097 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:10.051763 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtc8p" podUID="30cf5feb-cf9a-40c0-8a6a-de8fc6c60506" Jul 09 18:59:10.052097 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:10.051637 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tqtv5" Jul 09 18:59:10.052097 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:10.051817 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mtz87" podUID="2fe73719-af90-4093-ae73-d4e462e5e43c" Jul 09 18:59:10.052097 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:10.051894 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tqtv5" podUID="7435737c-5954-4ef0-8b41-1889b8231561" Jul 09 18:59:12.050827 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:12.050804 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tqtv5" Jul 09 18:59:12.051268 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:12.050844 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtc8p" Jul 09 18:59:12.051268 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:12.050902 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tqtv5" podUID="7435737c-5954-4ef0-8b41-1889b8231561" Jul 09 18:59:12.051268 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:12.050952 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mtz87" Jul 09 18:59:12.051268 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:12.051040 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mtz87" podUID="2fe73719-af90-4093-ae73-d4e462e5e43c" Jul 09 18:59:12.051268 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:12.051130 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtc8p" podUID="30cf5feb-cf9a-40c0-8a6a-de8fc6c60506" Jul 09 18:59:12.284459 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:12.284195 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-x8k65" event={"ID":"0da71879-5de7-47a0-819a-0b1951b1e817","Type":"ContainerStarted","Data":"50dd452e2b52443287c3ab915d97037bb8eb7b9a647e5c899079969aa39590ee"} Jul 09 18:59:12.285857 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:12.285827 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5mwmg" event={"ID":"e5c65950-27b0-41de-8ee1-e2ced769d6b4","Type":"ContainerStarted","Data":"ec60fb695e5b62cc37c2b1ca31a4932fc357ddb646570156867f7de80422c635"} Jul 09 18:59:12.287098 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:12.287078 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcfcz" event={"ID":"b1f91f8e-9064-4538-a912-6e6354050534","Type":"ContainerStarted","Data":"603f5589a0a3cf90b8c7c7c38b8f49153496907111cf32e2fdc225681dd022d4"} Jul 09 18:59:12.288462 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:12.288436 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hrjdx" event={"ID":"47f9bbdb-bfc1-464a-a495-b310124b104b","Type":"ContainerStarted","Data":"74767ac38b3009853e2502471d5aa35dd1f3279a1b5d9c127d775e51e04e4fb0"} Jul 09 18:59:12.289851 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:12.289826 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-c2bn7" event={"ID":"51b7fa90-97cc-4b69-80d4-d63ea5f30d6f","Type":"ContainerStarted","Data":"54eb1cb5fdcdb726a8476a4be6c76a246d29188ff72428e725df654af49b4e3b"} Jul 09 18:59:12.291288 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:12.291266 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-153.ec2.internal" event={"ID":"2bd4f2bfb833729e4a5ed7d4607ff8c1","Type":"ContainerStarted","Data":"8de5442b800e02bc5dd6cd8727b0fedc54d27e0e419b6c040f8e2b029b95a740"} Jul 09 18:59:12.292907 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:12.292886 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-153.ec2.internal" event={"ID":"dbfbad2bcd3a64e3008a9009f32541a5","Type":"ContainerStarted","Data":"c8a8612518b45935165c8a4420bc2cf85d6fc3b38d7888a6b914408c5c33559a"} Jul 09 18:59:12.294373 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:12.294352 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jnt6g" event={"ID":"5c298798-c68a-4450-b9dc-358fa978305a","Type":"ContainerStarted","Data":"0f100326ccb0ea5e652a5b8fde05b5274bac363b63c91daccaa6c4eaa756e33c"} Jul 09 18:59:12.296225 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:12.296209 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6dh7g_db233460-134d-4d9d-8da9-31d89366a4e0/ovn-acl-logging/0.log" Jul 09 18:59:12.296577 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:12.296557 2566 generic.go:356] "Generic (PLEG): container finished" podID="db233460-134d-4d9d-8da9-31d89366a4e0" containerID="7828604f9d2ffd8b4e3c5bf94c68e7a271335d2617fcddd67c0aafddd20c27a0" exitCode=1 Jul 09 18:59:12.296676 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:12.296630 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" event={"ID":"db233460-134d-4d9d-8da9-31d89366a4e0","Type":"ContainerStarted","Data":"b8683fb25e74bf6e0f938fea6b659e48815e0475679deb59db5fcdcd41fd1248"} Jul 09 18:59:12.296717 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:12.296701 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" event={"ID":"db233460-134d-4d9d-8da9-31d89366a4e0","Type":"ContainerDied","Data":"7828604f9d2ffd8b4e3c5bf94c68e7a271335d2617fcddd67c0aafddd20c27a0"} Jul 09 18:59:12.296770 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:12.296756 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" event={"ID":"db233460-134d-4d9d-8da9-31d89366a4e0","Type":"ContainerStarted","Data":"3d63d0156c09aee510e48182d5aece11412ce1e0e6318d6453169cb7f7d33a50"} Jul 09 18:59:12.298096 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:12.298071 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-swdmr" event={"ID":"db52b7ec-a201-4d65-81b3-56e37f0b6d36","Type":"ContainerStarted","Data":"0232efea726695468ec7720b4b49d11800fa4a93909db84604dbfffa06e248a1"} Jul 09 18:59:12.300237 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:12.300193 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-x8k65" podStartSLOduration=1.705126039 podStartE2EDuration="21.300178019s" totalImagesPullingTime="19.59505198s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:58:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:59:12.299278192 +0000 UTC m=+21.807057965" watchObservedRunningTime="2026-07-09 18:59:12.300178019 +0000 UTC m=+21.807957799" Jul 09 18:59:12.370413 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:12.370227 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-hrjdx" podStartSLOduration=1.7545291029999999 podStartE2EDuration="21.370207829s" totalImagesPullingTime="19.615678726s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:58:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:59:12.353050173 +0000 UTC m=+21.860829947" watchObservedRunningTime="2026-07-09 18:59:12.370207829 +0000 UTC m=+21.877987604" Jul 09 18:59:12.371702 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:12.371574 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-5mwmg" podStartSLOduration=1.8362078990000001 podStartE2EDuration="21.371558784s" totalImagesPullingTime="19.535350885s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:58:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:59:12.370134816 +0000 UTC m=+21.877914590" watchObservedRunningTime="2026-07-09 18:59:12.371558784 +0000 UTC m=+21.879338559" Jul 09 18:59:12.386979 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:12.386931 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-153.ec2.internal" podStartSLOduration=21.386914697 podStartE2EDuration="21.386914697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:58:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:59:12.386525098 +0000 UTC m=+21.894304873" watchObservedRunningTime="2026-07-09 18:59:12.386914697 +0000 UTC m=+21.894694471" Jul 09 18:59:12.401540 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:12.401501 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-c2bn7" podStartSLOduration=6.325217413 podStartE2EDuration="21.401486818s" totalImagesPullingTime="15.076269405s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:58:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:59:12.401172699 +0000 UTC m=+21.908952472" watchObservedRunningTime="2026-07-09 18:59:12.401486818 +0000 UTC m=+21.909266592" Jul 09 18:59:12.415267 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:12.415231 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jnt6g" podStartSLOduration=1.908685746 podStartE2EDuration="21.415216524s" totalImagesPullingTime="19.506530778s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:58:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:59:12.41484286 +0000 UTC m=+21.922622643" watchObservedRunningTime="2026-07-09 18:59:12.415216524 +0000 UTC m=+21.922996297" Jul 09 18:59:13.302069 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:13.302031 2566 generic.go:356] "Generic (PLEG): container finished" podID="db52b7ec-a201-4d65-81b3-56e37f0b6d36" containerID="0232efea726695468ec7720b4b49d11800fa4a93909db84604dbfffa06e248a1" exitCode=0 Jul 09 18:59:13.302990 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:13.302096 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-swdmr" event={"ID":"db52b7ec-a201-4d65-81b3-56e37f0b6d36","Type":"ContainerDied","Data":"0232efea726695468ec7720b4b49d11800fa4a93909db84604dbfffa06e248a1"} Jul 09 18:59:13.304287 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:13.304255 2566 generic.go:356] "Generic (PLEG): container finished" podID="dbfbad2bcd3a64e3008a9009f32541a5" containerID="c8a8612518b45935165c8a4420bc2cf85d6fc3b38d7888a6b914408c5c33559a" exitCode=0 Jul 09 18:59:13.304398 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:13.304345 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-153.ec2.internal" event={"ID":"dbfbad2bcd3a64e3008a9009f32541a5","Type":"ContainerDied","Data":"c8a8612518b45935165c8a4420bc2cf85d6fc3b38d7888a6b914408c5c33559a"} Jul 09 18:59:13.304458 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:13.304399 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-153.ec2.internal" event={"ID":"dbfbad2bcd3a64e3008a9009f32541a5","Type":"ContainerStarted","Data":"1b4b0a60c65a2686d5c436c08c6747027108bb8645749093ce97af828117710e"} Jul 09 18:59:13.307661 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:13.307627 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6dh7g_db233460-134d-4d9d-8da9-31d89366a4e0/ovn-acl-logging/0.log" Jul 09 18:59:13.308122 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:13.308086 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" event={"ID":"db233460-134d-4d9d-8da9-31d89366a4e0","Type":"ContainerStarted","Data":"48a4a5177f6670185985214c26e4bc7f1b412775eab98d44480cbaba642c8ba8"} Jul 09 18:59:13.308231 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:13.308182 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" event={"ID":"db233460-134d-4d9d-8da9-31d89366a4e0","Type":"ContainerStarted","Data":"b294e59e4464f19b6c1c5c23dcca3b6985207300a6ae0660083cf29914f92625"} Jul 09 18:59:13.308291 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:13.308232 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" event={"ID":"db233460-134d-4d9d-8da9-31d89366a4e0","Type":"ContainerStarted","Data":"ab1a1697973f8641b7dc431eb2e31dfb5ec7b7774e6074551eb84ea019d90d5b"} Jul 09 18:59:13.339140 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:13.339077 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-153.ec2.internal" podStartSLOduration=22.339057011 podStartE2EDuration="22.339057011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:58:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:59:13.338570331 +0000 UTC m=+22.846350106" watchObservedRunningTime="2026-07-09 18:59:13.339057011 +0000 UTC m=+22.846836784" Jul 09 18:59:13.693170 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:13.693148 2566 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 09 18:59:13.956923 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:13.956837 2566 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-09T18:59:13.693165316Z","UUID":"b4971c46-153a-41eb-b6b4-cc40f0e95055","Handler":null,"Name":"","Endpoint":""} Jul 09 18:59:13.958544 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:13.958517 2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 09 18:59:13.958702 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:13.958552 2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 09 18:59:14.051684 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:14.051659 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tqtv5" Jul 09 18:59:14.051684 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:14.051688 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtc8p" Jul 09 18:59:14.051969 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:14.051688 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mtz87" Jul 09 18:59:14.051969 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:14.051783 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tqtv5" podUID="7435737c-5954-4ef0-8b41-1889b8231561" Jul 09 18:59:14.051969 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:14.051865 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtc8p" podUID="30cf5feb-cf9a-40c0-8a6a-de8fc6c60506" Jul 09 18:59:14.051969 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:14.051945 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mtz87" podUID="2fe73719-af90-4093-ae73-d4e462e5e43c" Jul 09 18:59:14.312762 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:14.312682 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-km5nl" event={"ID":"d40ecb0c-9c3c-4fbe-ad50-742b0f848c44","Type":"ContainerStarted","Data":"655d867e02d94862b84c01d4f8169c447ff447d3e242aa46142ba9d092b7c627"} Jul 09 18:59:14.314967 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:14.314937 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcfcz" event={"ID":"b1f91f8e-9064-4538-a912-6e6354050534","Type":"ContainerStarted","Data":"def212f2f65aba882e51ea608d42d7cc7a22a93a175491d0bf81604fde32ed8a"} Jul 09 18:59:14.328470 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:14.328421 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-km5nl" podStartSLOduration=3.799328352 podStartE2EDuration="23.328404379s" totalImagesPullingTime="19.529076027s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:58:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:59:14.328042867 +0000 UTC m=+23.835822642" watchObservedRunningTime="2026-07-09 18:59:14.328404379 +0000 UTC m=+23.836184153" Jul 09 18:59:15.253630 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:15.253386 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-c2bn7" Jul 09 18:59:15.253973 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:15.253946 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-c2bn7" Jul 09 18:59:15.321018 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:15.320953 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6dh7g_db233460-134d-4d9d-8da9-31d89366a4e0/ovn-acl-logging/0.log" Jul 09 18:59:15.321489 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:15.321459 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" event={"ID":"db233460-134d-4d9d-8da9-31d89366a4e0","Type":"ContainerStarted","Data":"4627f4dfd19b0be55eff8865e3326c2a18765b84645f7596762ac364f5b11f6d"} Jul 09 18:59:15.323678 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:15.323644 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcfcz" event={"ID":"b1f91f8e-9064-4538-a912-6e6354050534","Type":"ContainerStarted","Data":"2cf79709f4ba44eae96688023150f75ad81668f7896269d8d407e7305a287f38"} Jul 09 18:59:15.341953 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:15.341908 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gcfcz" podStartSLOduration=2.145011127 podStartE2EDuration="24.341892064s" totalImagesPullingTime="22.196880937s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:58:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:59:15.341534645 +0000 UTC m=+24.849314419" watchObservedRunningTime="2026-07-09 18:59:15.341892064 +0000 UTC m=+24.849671838" Jul 09 18:59:16.051246 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:16.051208 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tqtv5" Jul 09 18:59:16.051246 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:16.051230 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mtz87" Jul 09 18:59:16.051528 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:16.051230 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtc8p" Jul 09 18:59:16.051528 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:16.051348 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tqtv5" podUID="7435737c-5954-4ef0-8b41-1889b8231561" Jul 09 18:59:16.051528 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:16.051409 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mtz87" podUID="2fe73719-af90-4093-ae73-d4e462e5e43c" Jul 09 18:59:16.051528 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:16.051472 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtc8p" podUID="30cf5feb-cf9a-40c0-8a6a-de8fc6c60506" Jul 09 18:59:18.051217 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:18.051194 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tqtv5" Jul 09 18:59:18.051732 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:18.051231 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtc8p" Jul 09 18:59:18.051732 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:18.051307 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mtz87" Jul 09 18:59:18.051732 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:18.051315 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tqtv5" podUID="7435737c-5954-4ef0-8b41-1889b8231561" Jul 09 18:59:18.051732 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:18.051382 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mtz87" podUID="2fe73719-af90-4093-ae73-d4e462e5e43c" Jul 09 18:59:18.051732 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:18.051491 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtc8p" podUID="30cf5feb-cf9a-40c0-8a6a-de8fc6c60506" Jul 09 18:59:18.304703 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:18.302409 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-c2bn7" Jul 09 18:59:18.304703 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:18.302837 2566 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 09 18:59:18.305692 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:18.305668 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-c2bn7" Jul 09 18:59:18.333095 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:18.333077 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6dh7g_db233460-134d-4d9d-8da9-31d89366a4e0/ovn-acl-logging/0.log" Jul 09 18:59:18.333510 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:18.333486 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" event={"ID":"db233460-134d-4d9d-8da9-31d89366a4e0","Type":"ContainerStarted","Data":"abd0538cca50a33f02cead124491679c01dae0b43dc4cccd1df8eb24d82453ca"} Jul 09 18:59:18.333744 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:18.333728 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:59:18.333875 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:18.333857 2566 scope.go:117] "RemoveContainer" containerID="7828604f9d2ffd8b4e3c5bf94c68e7a271335d2617fcddd67c0aafddd20c27a0" Jul 09 18:59:18.349472 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:18.349452 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:59:19.342766 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:19.342697 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6dh7g_db233460-134d-4d9d-8da9-31d89366a4e0/ovn-acl-logging/0.log" Jul 09 18:59:19.344002 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:19.343970 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" event={"ID":"db233460-134d-4d9d-8da9-31d89366a4e0","Type":"ContainerStarted","Data":"151086f73f79083db4abb95541f4f72ff30e806de168a20bf2301248b4b70068"} Jul 09 18:59:19.344389 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:19.344305 2566 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 09 18:59:19.345806 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:19.345316 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:59:19.346882 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:19.346859 2566 generic.go:356] "Generic (PLEG): container finished" podID="db52b7ec-a201-4d65-81b3-56e37f0b6d36" containerID="8f875f0d216be57f64cc836c1274342193dc95a4be9e509e9c3e94eb9f7ec857" exitCode=0 Jul 09 18:59:19.347066 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:19.347036 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-swdmr" event={"ID":"db52b7ec-a201-4d65-81b3-56e37f0b6d36","Type":"ContainerDied","Data":"8f875f0d216be57f64cc836c1274342193dc95a4be9e509e9c3e94eb9f7ec857"} Jul 09 18:59:19.372656 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:19.372600 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:59:19.409040 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:19.408987 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" podStartSLOduration=8.734824939 podStartE2EDuration="28.408971503s" totalImagesPullingTime="19.674146564s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:58:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:59:19.383753079 +0000 UTC m=+28.891532853" watchObservedRunningTime="2026-07-09 18:59:19.408971503 +0000 UTC m=+28.916751275" Jul 09 18:59:20.051118 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:20.051087 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tqtv5" Jul 09 18:59:20.051277 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:20.051179 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtc8p" Jul 09 18:59:20.051277 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:20.051186 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mtz87" Jul 09 18:59:20.051277 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:20.051185 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tqtv5" podUID="7435737c-5954-4ef0-8b41-1889b8231561" Jul 09 18:59:20.051277 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:20.051269 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtc8p" podUID="30cf5feb-cf9a-40c0-8a6a-de8fc6c60506" Jul 09 18:59:20.051414 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:20.051319 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mtz87" podUID="2fe73719-af90-4093-ae73-d4e462e5e43c" Jul 09 18:59:20.120024 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:20.119995 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vtc8p"] Jul 09 18:59:20.120588 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:20.120562 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tqtv5"] Jul 09 18:59:20.132306 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:20.132280 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mtz87"] Jul 09 18:59:20.348869 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:20.348844 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mtz87" Jul 09 18:59:20.348869 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:20.348870 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtc8p" Jul 09 18:59:20.349298 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:20.348927 2566 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 09 18:59:20.349298 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:20.348853 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tqtv5" Jul 09 18:59:20.349298 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:20.348937 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mtz87" podUID="2fe73719-af90-4093-ae73-d4e462e5e43c" Jul 09 18:59:20.349298 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:20.348998 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtc8p" podUID="30cf5feb-cf9a-40c0-8a6a-de8fc6c60506" Jul 09 18:59:20.349430 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:20.349332 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tqtv5" podUID="7435737c-5954-4ef0-8b41-1889b8231561" Jul 09 18:59:21.353567 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:21.353383 2566 generic.go:356] "Generic (PLEG): container finished" podID="db52b7ec-a201-4d65-81b3-56e37f0b6d36" containerID="9010f42910d66e359f34187b87e308a7d7c13ebf222eccd4e10bb1b496464b72" exitCode=0 Jul 09 18:59:21.354080 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:21.353462 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-swdmr" event={"ID":"db52b7ec-a201-4d65-81b3-56e37f0b6d36","Type":"ContainerDied","Data":"9010f42910d66e359f34187b87e308a7d7c13ebf222eccd4e10bb1b496464b72"} Jul 09 18:59:21.354080 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:21.353708 2566 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 09 18:59:21.948133 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:21.948091 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:59:22.051894 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:22.051859 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mtz87" Jul 09 18:59:22.052065 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:22.051909 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tqtv5" Jul 09 18:59:22.052065 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:22.051866 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtc8p" Jul 09 18:59:22.052065 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:22.051978 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mtz87" podUID="2fe73719-af90-4093-ae73-d4e462e5e43c" Jul 09 18:59:22.052065 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:22.052060 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtc8p" podUID="30cf5feb-cf9a-40c0-8a6a-de8fc6c60506" Jul 09 18:59:22.052308 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:22.052140 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tqtv5" podUID="7435737c-5954-4ef0-8b41-1889b8231561" Jul 09 18:59:23.361963 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:23.361931 2566 generic.go:356] "Generic (PLEG): container finished" podID="db52b7ec-a201-4d65-81b3-56e37f0b6d36" containerID="bd668f8b3408c5847566f4903bbdf16b55e23f4791548069551100398419bff2" exitCode=0 Jul 09 18:59:23.362459 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:23.362007 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-swdmr" event={"ID":"db52b7ec-a201-4d65-81b3-56e37f0b6d36","Type":"ContainerDied","Data":"bd668f8b3408c5847566f4903bbdf16b55e23f4791548069551100398419bff2"} Jul 09 18:59:23.745333 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:23.745303 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30cf5feb-cf9a-40c0-8a6a-de8fc6c60506-metrics-certs\") pod \"network-metrics-daemon-vtc8p\" (UID: \"30cf5feb-cf9a-40c0-8a6a-de8fc6c60506\") " pod="openshift-multus/network-metrics-daemon-vtc8p" Jul 09 18:59:23.745494 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:23.745378 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7435737c-5954-4ef0-8b41-1889b8231561-original-pull-secret\") pod \"global-pull-secret-syncer-tqtv5\" (UID: \"7435737c-5954-4ef0-8b41-1889b8231561\") " pod="kube-system/global-pull-secret-syncer-tqtv5" Jul 09 18:59:23.745494 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:23.745443 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 18:59:23.745494 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:23.745452 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 18:59:23.745580 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:23.745498 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7435737c-5954-4ef0-8b41-1889b8231561-original-pull-secret podName:7435737c-5954-4ef0-8b41-1889b8231561 nodeName:}" failed. No retries permitted until 2026-07-09 18:59:55.745485256 +0000 UTC m=+65.253265008 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7435737c-5954-4ef0-8b41-1889b8231561-original-pull-secret") pod "global-pull-secret-syncer-tqtv5" (UID: "7435737c-5954-4ef0-8b41-1889b8231561") : object "kube-system"/"original-pull-secret" not registered Jul 09 18:59:23.745580 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:23.745514 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30cf5feb-cf9a-40c0-8a6a-de8fc6c60506-metrics-certs podName:30cf5feb-cf9a-40c0-8a6a-de8fc6c60506 nodeName:}" failed. No retries permitted until 2026-07-09 18:59:55.745506378 +0000 UTC m=+65.253286129 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30cf5feb-cf9a-40c0-8a6a-de8fc6c60506-metrics-certs") pod "network-metrics-daemon-vtc8p" (UID: "30cf5feb-cf9a-40c0-8a6a-de8fc6c60506") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 18:59:23.845937 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:23.845891 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pntjm\" (UniqueName: \"kubernetes.io/projected/2fe73719-af90-4093-ae73-d4e462e5e43c-kube-api-access-pntjm\") pod \"network-check-target-mtz87\" (UID: \"2fe73719-af90-4093-ae73-d4e462e5e43c\") " pod="openshift-network-diagnostics/network-check-target-mtz87" Jul 09 18:59:23.846102 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:23.846027 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 18:59:23.846102 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:23.846041 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 18:59:23.846102 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:23.846050 2566 projected.go:196] Error preparing data for projected volume kube-api-access-pntjm for pod openshift-network-diagnostics/network-check-target-mtz87: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 18:59:23.846102 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:23.846093 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fe73719-af90-4093-ae73-d4e462e5e43c-kube-api-access-pntjm podName:2fe73719-af90-4093-ae73-d4e462e5e43c nodeName:}" failed. No retries permitted until 2026-07-09 18:59:55.846080083 +0000 UTC m=+65.353859835 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-pntjm" (UniqueName: "kubernetes.io/projected/2fe73719-af90-4093-ae73-d4e462e5e43c-kube-api-access-pntjm") pod "network-check-target-mtz87" (UID: "2fe73719-af90-4093-ae73-d4e462e5e43c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 18:59:24.051351 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:24.051273 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mtz87" Jul 09 18:59:24.051351 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:24.051295 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tqtv5" Jul 09 18:59:24.051557 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:24.051273 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtc8p" Jul 09 18:59:24.051557 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:24.051384 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mtz87" podUID="2fe73719-af90-4093-ae73-d4e462e5e43c" Jul 09 18:59:24.051557 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:24.051441 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tqtv5" podUID="7435737c-5954-4ef0-8b41-1889b8231561" Jul 09 18:59:24.051557 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:24.051512 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vtc8p" podUID="30cf5feb-cf9a-40c0-8a6a-de8fc6c60506" Jul 09 18:59:25.822920 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:25.822892 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-153.ec2.internal" event="NodeReady" Jul 09 18:59:25.823345 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:25.823043 2566 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 09 18:59:25.870785 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:25.870751 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9xn8w"] Jul 09 18:59:25.874472 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:25.874441 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9xn8w" Jul 09 18:59:25.874623 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:25.874507 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7dgq9"] Jul 09 18:59:25.877276 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:25.877245 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 09 18:59:25.877555 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:25.877538 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bw6lk\"" Jul 09 18:59:25.877555 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:25.877547 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 09 18:59:25.878335 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:25.878315 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7dgq9" Jul 09 18:59:25.881062 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:25.881036 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-65b7z\"" Jul 09 18:59:25.881166 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:25.881036 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 09 18:59:25.881166 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:25.881115 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 09 18:59:25.881166 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:25.881046 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 09 18:59:25.884690 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:25.884658 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9xn8w"] Jul 09 18:59:25.888409 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:25.888355 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7dgq9"] Jul 09 18:59:25.960399 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:25.960368 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a209a84f-cc92-4f17-a299-4f4161322946-cert\") pod \"ingress-canary-7dgq9\" (UID: \"a209a84f-cc92-4f17-a299-4f4161322946\") " pod="openshift-ingress-canary/ingress-canary-7dgq9" Jul 09 18:59:25.960399 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:25.960400 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa9f1324-a4e6-4f69-93f5-101c3aed7a0c-tmp-dir\") pod \"dns-default-9xn8w\" (UID: \"aa9f1324-a4e6-4f69-93f5-101c3aed7a0c\") " pod="openshift-dns/dns-default-9xn8w" Jul 09 18:59:25.960570 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:25.960420 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/aa9f1324-a4e6-4f69-93f5-101c3aed7a0c-config-volume\") pod \"dns-default-9xn8w\" (UID: \"aa9f1324-a4e6-4f69-93f5-101c3aed7a0c\") " pod="openshift-dns/dns-default-9xn8w" Jul 09 18:59:25.960570 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:25.960438 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aa9f1324-a4e6-4f69-93f5-101c3aed7a0c-metrics-tls\") pod \"dns-default-9xn8w\" (UID: \"aa9f1324-a4e6-4f69-93f5-101c3aed7a0c\") " pod="openshift-dns/dns-default-9xn8w" Jul 09 18:59:25.960570 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:25.960516 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hg8d\" (UniqueName: \"kubernetes.io/projected/aa9f1324-a4e6-4f69-93f5-101c3aed7a0c-kube-api-access-2hg8d\") pod \"dns-default-9xn8w\" (UID: \"aa9f1324-a4e6-4f69-93f5-101c3aed7a0c\") " pod="openshift-dns/dns-default-9xn8w" Jul 09 18:59:25.960570 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:25.960551 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpnn7\" (UniqueName: \"kubernetes.io/projected/a209a84f-cc92-4f17-a299-4f4161322946-kube-api-access-bpnn7\") pod \"ingress-canary-7dgq9\" (UID: \"a209a84f-cc92-4f17-a299-4f4161322946\") " pod="openshift-ingress-canary/ingress-canary-7dgq9" Jul 09 18:59:26.051091 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:26.051051 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mtz87" Jul 09 18:59:26.051091 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:26.051093 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tqtv5" Jul 09 18:59:26.051322 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:26.051094 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtc8p" Jul 09 18:59:26.054261 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:26.054240 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 09 18:59:26.054624 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:26.054586 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 09 18:59:26.054741 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:26.054638 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s4jjw\"" Jul 09 18:59:26.054741 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:26.054680 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 09 18:59:26.054741 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:26.054709 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rzlbw\"" Jul 09 18:59:26.054741 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:26.054713 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 09 18:59:26.060822 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:26.060802 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a209a84f-cc92-4f17-a299-4f4161322946-cert\") pod \"ingress-canary-7dgq9\" (UID: \"a209a84f-cc92-4f17-a299-4f4161322946\") " pod="openshift-ingress-canary/ingress-canary-7dgq9" Jul 09 18:59:26.060913 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:26.060833 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa9f1324-a4e6-4f69-93f5-101c3aed7a0c-tmp-dir\") pod \"dns-default-9xn8w\" (UID: \"aa9f1324-a4e6-4f69-93f5-101c3aed7a0c\") " pod="openshift-dns/dns-default-9xn8w" Jul 09 18:59:26.060913 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:26.060862 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/aa9f1324-a4e6-4f69-93f5-101c3aed7a0c-config-volume\") pod \"dns-default-9xn8w\" (UID: \"aa9f1324-a4e6-4f69-93f5-101c3aed7a0c\") " pod="openshift-dns/dns-default-9xn8w" Jul 09 18:59:26.060913 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:26.060888 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aa9f1324-a4e6-4f69-93f5-101c3aed7a0c-metrics-tls\") pod \"dns-default-9xn8w\" (UID: \"aa9f1324-a4e6-4f69-93f5-101c3aed7a0c\") " pod="openshift-dns/dns-default-9xn8w" Jul 09 18:59:26.061048 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:26.060918 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hg8d\" (UniqueName: \"kubernetes.io/projected/aa9f1324-a4e6-4f69-93f5-101c3aed7a0c-kube-api-access-2hg8d\") pod \"dns-default-9xn8w\" (UID: \"aa9f1324-a4e6-4f69-93f5-101c3aed7a0c\") " pod="openshift-dns/dns-default-9xn8w" Jul 09 18:59:26.061048 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:26.060962 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpnn7\" (UniqueName: \"kubernetes.io/projected/a209a84f-cc92-4f17-a299-4f4161322946-kube-api-access-bpnn7\") pod \"ingress-canary-7dgq9\" (UID: \"a209a84f-cc92-4f17-a299-4f4161322946\") " pod="openshift-ingress-canary/ingress-canary-7dgq9" Jul 09 18:59:26.061157 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:26.061121 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 18:59:26.061204 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:26.061157 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa9f1324-a4e6-4f69-93f5-101c3aed7a0c-tmp-dir\") pod \"dns-default-9xn8w\" (UID: \"aa9f1324-a4e6-4f69-93f5-101c3aed7a0c\") " pod="openshift-dns/dns-default-9xn8w" Jul 09 18:59:26.061256 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:26.061204 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa9f1324-a4e6-4f69-93f5-101c3aed7a0c-metrics-tls podName:aa9f1324-a4e6-4f69-93f5-101c3aed7a0c nodeName:}" failed. No retries permitted until 2026-07-09 18:59:26.561173759 +0000 UTC m=+36.068953528 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aa9f1324-a4e6-4f69-93f5-101c3aed7a0c-metrics-tls") pod "dns-default-9xn8w" (UID: "aa9f1324-a4e6-4f69-93f5-101c3aed7a0c") : secret "dns-default-metrics-tls" not found Jul 09 18:59:26.061399 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:26.061383 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/aa9f1324-a4e6-4f69-93f5-101c3aed7a0c-config-volume\") pod \"dns-default-9xn8w\" (UID: \"aa9f1324-a4e6-4f69-93f5-101c3aed7a0c\") " pod="openshift-dns/dns-default-9xn8w" Jul 09 18:59:26.065299 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:26.065282 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a209a84f-cc92-4f17-a299-4f4161322946-cert\") pod \"ingress-canary-7dgq9\" (UID: \"a209a84f-cc92-4f17-a299-4f4161322946\") " pod="openshift-ingress-canary/ingress-canary-7dgq9" Jul 09 18:59:26.069454 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:26.069429 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hg8d\" (UniqueName: \"kubernetes.io/projected/aa9f1324-a4e6-4f69-93f5-101c3aed7a0c-kube-api-access-2hg8d\") pod \"dns-default-9xn8w\" (UID: \"aa9f1324-a4e6-4f69-93f5-101c3aed7a0c\") " pod="openshift-dns/dns-default-9xn8w" Jul 09 18:59:26.069964 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:26.069947 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpnn7\" (UniqueName: \"kubernetes.io/projected/a209a84f-cc92-4f17-a299-4f4161322946-kube-api-access-bpnn7\") pod \"ingress-canary-7dgq9\" (UID: \"a209a84f-cc92-4f17-a299-4f4161322946\") " pod="openshift-ingress-canary/ingress-canary-7dgq9" Jul 09 18:59:26.193900 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:26.193869 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7dgq9" Jul 09 18:59:26.346246 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:26.345938 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7dgq9"] Jul 09 18:59:26.349691 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:59:26.349663 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda209a84f_cc92_4f17_a299_4f4161322946.slice/crio-5a6bdb0e60b0d268db2309e3905c1d8d7f33ee3541f5d2efe1c5c7c1eff3caa0 WatchSource:0}: Error finding container 5a6bdb0e60b0d268db2309e3905c1d8d7f33ee3541f5d2efe1c5c7c1eff3caa0: Status 404 returned error can't find the container with id 5a6bdb0e60b0d268db2309e3905c1d8d7f33ee3541f5d2efe1c5c7c1eff3caa0 Jul 09 18:59:26.369639 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:26.369598 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7dgq9" event={"ID":"a209a84f-cc92-4f17-a299-4f4161322946","Type":"ContainerStarted","Data":"5a6bdb0e60b0d268db2309e3905c1d8d7f33ee3541f5d2efe1c5c7c1eff3caa0"} Jul 09 18:59:26.567841 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:26.565362 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aa9f1324-a4e6-4f69-93f5-101c3aed7a0c-metrics-tls\") pod \"dns-default-9xn8w\" (UID: \"aa9f1324-a4e6-4f69-93f5-101c3aed7a0c\") " pod="openshift-dns/dns-default-9xn8w" Jul 09 18:59:26.567841 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:26.565536 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 18:59:26.567841 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:26.565635 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa9f1324-a4e6-4f69-93f5-101c3aed7a0c-metrics-tls podName:aa9f1324-a4e6-4f69-93f5-101c3aed7a0c nodeName:}" failed. No retries permitted until 2026-07-09 18:59:27.565591229 +0000 UTC m=+37.073370987 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aa9f1324-a4e6-4f69-93f5-101c3aed7a0c-metrics-tls") pod "dns-default-9xn8w" (UID: "aa9f1324-a4e6-4f69-93f5-101c3aed7a0c") : secret "dns-default-metrics-tls" not found Jul 09 18:59:27.573906 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:27.573863 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aa9f1324-a4e6-4f69-93f5-101c3aed7a0c-metrics-tls\") pod \"dns-default-9xn8w\" (UID: \"aa9f1324-a4e6-4f69-93f5-101c3aed7a0c\") " pod="openshift-dns/dns-default-9xn8w" Jul 09 18:59:27.574396 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:27.574038 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 18:59:27.574396 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:27.574137 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa9f1324-a4e6-4f69-93f5-101c3aed7a0c-metrics-tls podName:aa9f1324-a4e6-4f69-93f5-101c3aed7a0c nodeName:}" failed. No retries permitted until 2026-07-09 18:59:29.574097433 +0000 UTC m=+39.081877206 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aa9f1324-a4e6-4f69-93f5-101c3aed7a0c-metrics-tls") pod "dns-default-9xn8w" (UID: "aa9f1324-a4e6-4f69-93f5-101c3aed7a0c") : secret "dns-default-metrics-tls" not found Jul 09 18:59:28.379982 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:28.379945 2566 generic.go:356] "Generic (PLEG): container finished" podID="db52b7ec-a201-4d65-81b3-56e37f0b6d36" containerID="63f98e817df8fe3c546699165be89ef4d0b64de4b328df42658b35f731fa6304" exitCode=0 Jul 09 18:59:28.380143 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:28.380001 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-swdmr" event={"ID":"db52b7ec-a201-4d65-81b3-56e37f0b6d36","Type":"ContainerDied","Data":"63f98e817df8fe3c546699165be89ef4d0b64de4b328df42658b35f731fa6304"} Jul 09 18:59:29.386200 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:29.385952 2566 generic.go:356] "Generic (PLEG): container finished" podID="db52b7ec-a201-4d65-81b3-56e37f0b6d36" containerID="34aec1a13d5b6cfa6fc3d1d13c52d9be6933ef4c409df2027947ba1bb26de9e9" exitCode=0 Jul 09 18:59:29.386573 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:29.386035 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-swdmr" event={"ID":"db52b7ec-a201-4d65-81b3-56e37f0b6d36","Type":"ContainerDied","Data":"34aec1a13d5b6cfa6fc3d1d13c52d9be6933ef4c409df2027947ba1bb26de9e9"} Jul 09 18:59:29.588650 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:29.588597 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aa9f1324-a4e6-4f69-93f5-101c3aed7a0c-metrics-tls\") pod \"dns-default-9xn8w\" (UID: \"aa9f1324-a4e6-4f69-93f5-101c3aed7a0c\") " pod="openshift-dns/dns-default-9xn8w" Jul 09 18:59:29.588817 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:29.588720 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 18:59:29.588817 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:29.588781 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa9f1324-a4e6-4f69-93f5-101c3aed7a0c-metrics-tls podName:aa9f1324-a4e6-4f69-93f5-101c3aed7a0c nodeName:}" failed. No retries permitted until 2026-07-09 18:59:33.588766356 +0000 UTC m=+43.096546108 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aa9f1324-a4e6-4f69-93f5-101c3aed7a0c-metrics-tls") pod "dns-default-9xn8w" (UID: "aa9f1324-a4e6-4f69-93f5-101c3aed7a0c") : secret "dns-default-metrics-tls" not found Jul 09 18:59:30.394861 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:30.394822 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-swdmr" event={"ID":"db52b7ec-a201-4d65-81b3-56e37f0b6d36","Type":"ContainerStarted","Data":"068a9d94256f55bb6ef36839fd2ab49f4b098c359e2be0b4e57bce6039c8984c"} Jul 09 18:59:30.418070 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:30.418021 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-swdmr" podStartSLOduration=7.719845832 podStartE2EDuration="39.41800154s" totalImagesPullingTime="31.698155708s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:58:51 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:59:30.416297118 +0000 UTC m=+39.924076903" watchObservedRunningTime="2026-07-09 18:59:30.41800154 +0000 UTC m=+39.925781295" Jul 09 18:59:31.398145 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:31.398102 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7dgq9" event={"ID":"a209a84f-cc92-4f17-a299-4f4161322946","Type":"ContainerStarted","Data":"53e9c7da61b82accd0acd9c4562d8f7adde2f6a93c011fd265b32f935a882b16"} Jul 09 18:59:31.415939 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:31.415893 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7dgq9" podStartSLOduration=1.978798329 podStartE2EDuration="6.415877931s" totalImagesPullingTime="4.437079602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:59:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:59:31.415298888 +0000 UTC m=+40.923078675" watchObservedRunningTime="2026-07-09 18:59:31.415877931 +0000 UTC m=+40.923657704" Jul 09 18:59:33.616314 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:33.616275 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aa9f1324-a4e6-4f69-93f5-101c3aed7a0c-metrics-tls\") pod \"dns-default-9xn8w\" (UID: \"aa9f1324-a4e6-4f69-93f5-101c3aed7a0c\") " pod="openshift-dns/dns-default-9xn8w" Jul 09 18:59:33.616689 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:33.616432 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 18:59:33.616689 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:33.616494 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa9f1324-a4e6-4f69-93f5-101c3aed7a0c-metrics-tls podName:aa9f1324-a4e6-4f69-93f5-101c3aed7a0c nodeName:}" failed. No retries permitted until 2026-07-09 18:59:41.616479917 +0000 UTC m=+51.124259669 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aa9f1324-a4e6-4f69-93f5-101c3aed7a0c-metrics-tls") pod "dns-default-9xn8w" (UID: "aa9f1324-a4e6-4f69-93f5-101c3aed7a0c") : secret "dns-default-metrics-tls" not found Jul 09 18:59:41.674401 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:41.674366 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aa9f1324-a4e6-4f69-93f5-101c3aed7a0c-metrics-tls\") pod \"dns-default-9xn8w\" (UID: \"aa9f1324-a4e6-4f69-93f5-101c3aed7a0c\") " pod="openshift-dns/dns-default-9xn8w" Jul 09 18:59:41.674865 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:41.674478 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 18:59:41.674865 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:41.674537 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa9f1324-a4e6-4f69-93f5-101c3aed7a0c-metrics-tls podName:aa9f1324-a4e6-4f69-93f5-101c3aed7a0c nodeName:}" failed. No retries permitted until 2026-07-09 18:59:57.67452319 +0000 UTC m=+67.182302942 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aa9f1324-a4e6-4f69-93f5-101c3aed7a0c-metrics-tls") pod "dns-default-9xn8w" (UID: "aa9f1324-a4e6-4f69-93f5-101c3aed7a0c") : secret "dns-default-metrics-tls" not found Jul 09 18:59:52.366481 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:52.366452 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6dh7g" Jul 09 18:59:55.769570 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:55.769532 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7435737c-5954-4ef0-8b41-1889b8231561-original-pull-secret\") pod \"global-pull-secret-syncer-tqtv5\" (UID: \"7435737c-5954-4ef0-8b41-1889b8231561\") " pod="kube-system/global-pull-secret-syncer-tqtv5" Jul 09 18:59:55.769570 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:55.769575 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30cf5feb-cf9a-40c0-8a6a-de8fc6c60506-metrics-certs\") pod \"network-metrics-daemon-vtc8p\" (UID: \"30cf5feb-cf9a-40c0-8a6a-de8fc6c60506\") " pod="openshift-multus/network-metrics-daemon-vtc8p" Jul 09 18:59:55.772643 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:55.772626 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 09 18:59:55.772721 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:55.772629 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 09 18:59:55.780145 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:55.780125 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 09 18:59:55.780244 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:55.780198 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30cf5feb-cf9a-40c0-8a6a-de8fc6c60506-metrics-certs podName:30cf5feb-cf9a-40c0-8a6a-de8fc6c60506 nodeName:}" failed. No retries permitted until 2026-07-09 19:00:59.780176713 +0000 UTC m=+129.287956465 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30cf5feb-cf9a-40c0-8a6a-de8fc6c60506-metrics-certs") pod "network-metrics-daemon-vtc8p" (UID: "30cf5feb-cf9a-40c0-8a6a-de8fc6c60506") : secret "metrics-daemon-secret" not found Jul 09 18:59:55.783763 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:55.783738 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7435737c-5954-4ef0-8b41-1889b8231561-original-pull-secret\") pod \"global-pull-secret-syncer-tqtv5\" (UID: \"7435737c-5954-4ef0-8b41-1889b8231561\") " pod="kube-system/global-pull-secret-syncer-tqtv5" Jul 09 18:59:55.870348 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:55.870308 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pntjm\" (UniqueName: \"kubernetes.io/projected/2fe73719-af90-4093-ae73-d4e462e5e43c-kube-api-access-pntjm\") pod \"network-check-target-mtz87\" (UID: \"2fe73719-af90-4093-ae73-d4e462e5e43c\") " pod="openshift-network-diagnostics/network-check-target-mtz87" Jul 09 18:59:55.873396 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:55.873375 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 09 18:59:55.883701 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:55.883678 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 09 18:59:55.893374 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:55.893356 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pntjm\" (UniqueName: \"kubernetes.io/projected/2fe73719-af90-4093-ae73-d4e462e5e43c-kube-api-access-pntjm\") pod \"network-check-target-mtz87\" (UID: \"2fe73719-af90-4093-ae73-d4e462e5e43c\") " pod="openshift-network-diagnostics/network-check-target-mtz87" Jul 09 18:59:56.063653 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:56.063554 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rzlbw\"" Jul 09 18:59:56.071569 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:56.071549 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mtz87" Jul 09 18:59:56.080208 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:56.080189 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tqtv5" Jul 09 18:59:56.203657 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:56.203627 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mtz87"] Jul 09 18:59:56.206622 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:59:56.206568 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fe73719_af90_4093_ae73_d4e462e5e43c.slice/crio-ebf5b76faf60925220c2bf8950a685897a822a0f3e42f4c6e82f7428cf4234db WatchSource:0}: Error finding container ebf5b76faf60925220c2bf8950a685897a822a0f3e42f4c6e82f7428cf4234db: Status 404 returned error can't find the container with id ebf5b76faf60925220c2bf8950a685897a822a0f3e42f4c6e82f7428cf4234db Jul 09 18:59:56.216126 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:56.216103 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tqtv5"] Jul 09 18:59:56.218781 ip-10-0-140-153 kubenswrapper[2566]: W0709 18:59:56.218759 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7435737c_5954_4ef0_8b41_1889b8231561.slice/crio-3437b9d4a82500630c6b952427dcf70d9a9c9d7d878f2195a3be7b16dcc37905 WatchSource:0}: Error finding container 3437b9d4a82500630c6b952427dcf70d9a9c9d7d878f2195a3be7b16dcc37905: Status 404 returned error can't find the container with id 3437b9d4a82500630c6b952427dcf70d9a9c9d7d878f2195a3be7b16dcc37905 Jul 09 18:59:56.455051 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:56.455016 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tqtv5" event={"ID":"7435737c-5954-4ef0-8b41-1889b8231561","Type":"ContainerStarted","Data":"3437b9d4a82500630c6b952427dcf70d9a9c9d7d878f2195a3be7b16dcc37905"} Jul 09 18:59:56.456131 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:56.456105 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mtz87" event={"ID":"2fe73719-af90-4093-ae73-d4e462e5e43c","Type":"ContainerStarted","Data":"ebf5b76faf60925220c2bf8950a685897a822a0f3e42f4c6e82f7428cf4234db"} Jul 09 18:59:57.681542 ip-10-0-140-153 kubenswrapper[2566]: I0709 18:59:57.681500 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aa9f1324-a4e6-4f69-93f5-101c3aed7a0c-metrics-tls\") pod \"dns-default-9xn8w\" (UID: \"aa9f1324-a4e6-4f69-93f5-101c3aed7a0c\") " pod="openshift-dns/dns-default-9xn8w" Jul 09 18:59:57.682023 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:57.681694 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 18:59:57.682023 ip-10-0-140-153 kubenswrapper[2566]: E0709 18:59:57.681774 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa9f1324-a4e6-4f69-93f5-101c3aed7a0c-metrics-tls podName:aa9f1324-a4e6-4f69-93f5-101c3aed7a0c nodeName:}" failed. No retries permitted until 2026-07-09 19:00:29.681751981 +0000 UTC m=+99.189531751 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aa9f1324-a4e6-4f69-93f5-101c3aed7a0c-metrics-tls") pod "dns-default-9xn8w" (UID: "aa9f1324-a4e6-4f69-93f5-101c3aed7a0c") : secret "dns-default-metrics-tls" not found Jul 09 19:00:01.475072 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:00:01.475025 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tqtv5" event={"ID":"7435737c-5954-4ef0-8b41-1889b8231561","Type":"ContainerStarted","Data":"8665e9fcced1639319e4a3fcb2df340dc2f63ba632be7e398ee7e675f7981a76"} Jul 09 19:00:01.476527 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:00:01.476504 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mtz87" event={"ID":"2fe73719-af90-4093-ae73-d4e462e5e43c","Type":"ContainerStarted","Data":"dec22aa2a49a47a2bd45be0c15c58f9cc3dd6e94ba055de57ce231f5928e5d71"} Jul 09 19:00:01.476677 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:00:01.476660 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mtz87" Jul 09 19:00:01.505328 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:00:01.505287 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-tqtv5" podStartSLOduration=65.652429021 podStartE2EDuration="1m10.505272392s" totalImagesPullingTime="4.852843371s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:58:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:00:01.490800464 +0000 UTC m=+70.998580237" watchObservedRunningTime="2026-07-09 19:00:01.505272392 +0000 UTC m=+71.013052171" Jul 09 19:00:10.791528 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:00:10.791470 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mtz87" podStartSLOduration=74.92744409 podStartE2EDuration="1m19.791451732s" totalImagesPullingTime="4.864007642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:58:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:00:01.505025105 +0000 UTC m=+71.012804878" watchObservedRunningTime="2026-07-09 19:00:10.791451732 +0000 UTC m=+80.299231505" Jul 09 19:00:10.792042 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:00:10.791973 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7ddf5bcd8-n2jkw"] Jul 09 19:00:10.793787 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:00:10.793772 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7ddf5bcd8-n2jkw" Jul 09 19:00:10.798622 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:00:10.798580 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 09 19:00:10.798779 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:00:10.798653 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 09 19:00:10.798779 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:00:10.798580 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 09 19:00:10.798779 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:00:10.798703 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 09 19:00:10.799311 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:00:10.799286 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-k64nb\"" Jul 09 19:00:10.809029 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:00:10.809006 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7ddf5bcd8-n2jkw"] Jul 09 19:00:10.872166 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:00:10.872137 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt749\" (UniqueName: \"kubernetes.io/projected/75d1a79e-e18e-47e6-9fc6-c5fb01e96002-kube-api-access-gt749\") pod \"managed-serviceaccount-addon-agent-7ddf5bcd8-n2jkw\" (UID: \"75d1a79e-e18e-47e6-9fc6-c5fb01e96002\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7ddf5bcd8-n2jkw" Jul 09 19:00:10.872331 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:00:10.872174 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/75d1a79e-e18e-47e6-9fc6-c5fb01e96002-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7ddf5bcd8-n2jkw\" (UID: \"75d1a79e-e18e-47e6-9fc6-c5fb01e96002\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7ddf5bcd8-n2jkw" Jul 09 19:00:10.972589 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:00:10.972547 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt749\" (UniqueName: \"kubernetes.io/projected/75d1a79e-e18e-47e6-9fc6-c5fb01e96002-kube-api-access-gt749\") pod \"managed-serviceaccount-addon-agent-7ddf5bcd8-n2jkw\" (UID: \"75d1a79e-e18e-47e6-9fc6-c5fb01e96002\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7ddf5bcd8-n2jkw" Jul 09 19:00:10.972589 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:00:10.972593 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/75d1a79e-e18e-47e6-9fc6-c5fb01e96002-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7ddf5bcd8-n2jkw\" (UID: \"75d1a79e-e18e-47e6-9fc6-c5fb01e96002\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7ddf5bcd8-n2jkw" Jul 09 19:00:10.975054 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:00:10.975026 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/75d1a79e-e18e-47e6-9fc6-c5fb01e96002-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7ddf5bcd8-n2jkw\" (UID: \"75d1a79e-e18e-47e6-9fc6-c5fb01e96002\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7ddf5bcd8-n2jkw" Jul 09 19:00:10.982496 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:00:10.982469 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt749\" (UniqueName: \"kubernetes.io/projected/75d1a79e-e18e-47e6-9fc6-c5fb01e96002-kube-api-access-gt749\") pod \"managed-serviceaccount-addon-agent-7ddf5bcd8-n2jkw\" (UID: \"75d1a79e-e18e-47e6-9fc6-c5fb01e96002\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7ddf5bcd8-n2jkw" Jul 09 19:00:11.117927 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:00:11.117847 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7ddf5bcd8-n2jkw" Jul 09 19:00:11.236214 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:00:11.236184 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7ddf5bcd8-n2jkw"] Jul 09 19:00:11.239458 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:00:11.239416 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75d1a79e_e18e_47e6_9fc6_c5fb01e96002.slice/crio-36e4e4e7b72ea9ce7ee55d358b118324ca29f48a26e5bd789edd1b0ca449d337 WatchSource:0}: Error finding container 36e4e4e7b72ea9ce7ee55d358b118324ca29f48a26e5bd789edd1b0ca449d337: Status 404 returned error can't find the container with id 36e4e4e7b72ea9ce7ee55d358b118324ca29f48a26e5bd789edd1b0ca449d337 Jul 09 19:00:11.501457 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:00:11.501426 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7ddf5bcd8-n2jkw" event={"ID":"75d1a79e-e18e-47e6-9fc6-c5fb01e96002","Type":"ContainerStarted","Data":"36e4e4e7b72ea9ce7ee55d358b118324ca29f48a26e5bd789edd1b0ca449d337"} Jul 09 19:00:16.518327 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:00:16.518293 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7ddf5bcd8-n2jkw" event={"ID":"75d1a79e-e18e-47e6-9fc6-c5fb01e96002","Type":"ContainerStarted","Data":"5844d5f1ac592597f2b908e5491b4b6d1b8f14bca7db1a339f96278fbbc50843"} Jul 09 19:00:16.534800 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:00:16.534749 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7ddf5bcd8-n2jkw" podStartSLOduration=1.586328161 podStartE2EDuration="6.534729577s" totalImagesPullingTime="4.948401416s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:00:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:00:16.53426074 +0000 UTC m=+86.042040513" watchObservedRunningTime="2026-07-09 19:00:16.534729577 +0000 UTC m=+86.042509351" Jul 09 19:00:29.697331 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:00:29.697299 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aa9f1324-a4e6-4f69-93f5-101c3aed7a0c-metrics-tls\") pod \"dns-default-9xn8w\" (UID: \"aa9f1324-a4e6-4f69-93f5-101c3aed7a0c\") " pod="openshift-dns/dns-default-9xn8w" Jul 09 19:00:29.697686 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:00:29.697421 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 19:00:29.697686 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:00:29.697482 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa9f1324-a4e6-4f69-93f5-101c3aed7a0c-metrics-tls podName:aa9f1324-a4e6-4f69-93f5-101c3aed7a0c nodeName:}" failed. No retries permitted until 2026-07-09 19:01:33.697469914 +0000 UTC m=+163.205249665 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aa9f1324-a4e6-4f69-93f5-101c3aed7a0c-metrics-tls") pod "dns-default-9xn8w" (UID: "aa9f1324-a4e6-4f69-93f5-101c3aed7a0c") : secret "dns-default-metrics-tls" not found Jul 09 19:00:32.481376 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:00:32.481342 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mtz87" Jul 09 19:00:59.800502 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:00:59.800463 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30cf5feb-cf9a-40c0-8a6a-de8fc6c60506-metrics-certs\") pod \"network-metrics-daemon-vtc8p\" (UID: \"30cf5feb-cf9a-40c0-8a6a-de8fc6c60506\") " pod="openshift-multus/network-metrics-daemon-vtc8p" Jul 09 19:00:59.800996 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:00:59.800580 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 09 19:00:59.800996 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:00:59.800661 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30cf5feb-cf9a-40c0-8a6a-de8fc6c60506-metrics-certs podName:30cf5feb-cf9a-40c0-8a6a-de8fc6c60506 nodeName:}" failed. No retries permitted until 2026-07-09 19:03:01.800645654 +0000 UTC m=+251.308425406 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30cf5feb-cf9a-40c0-8a6a-de8fc6c60506-metrics-certs") pod "network-metrics-daemon-vtc8p" (UID: "30cf5feb-cf9a-40c0-8a6a-de8fc6c60506") : secret "metrics-daemon-secret" not found Jul 09 19:01:28.887367 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:01:28.887310 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-9xn8w" podUID="aa9f1324-a4e6-4f69-93f5-101c3aed7a0c" Jul 09 19:01:29.067047 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:01:29.067010 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-vtc8p" podUID="30cf5feb-cf9a-40c0-8a6a-de8fc6c60506" Jul 09 19:01:29.691941 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:01:29.691910 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9xn8w" Jul 09 19:01:33.728065 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:01:33.728023 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aa9f1324-a4e6-4f69-93f5-101c3aed7a0c-metrics-tls\") pod \"dns-default-9xn8w\" (UID: \"aa9f1324-a4e6-4f69-93f5-101c3aed7a0c\") " pod="openshift-dns/dns-default-9xn8w" Jul 09 19:01:33.728514 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:01:33.728157 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 19:01:33.728514 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:01:33.728218 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa9f1324-a4e6-4f69-93f5-101c3aed7a0c-metrics-tls podName:aa9f1324-a4e6-4f69-93f5-101c3aed7a0c nodeName:}" failed. No retries permitted until 2026-07-09 19:03:35.728204334 +0000 UTC m=+285.235984087 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aa9f1324-a4e6-4f69-93f5-101c3aed7a0c-metrics-tls") pod "dns-default-9xn8w" (UID: "aa9f1324-a4e6-4f69-93f5-101c3aed7a0c") : secret "dns-default-metrics-tls" not found Jul 09 19:01:44.050965 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:01:44.050879 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtc8p" Jul 09 19:02:16.804549 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:16.804515 2566 generic.go:356] "Generic (PLEG): container finished" podID="75d1a79e-e18e-47e6-9fc6-c5fb01e96002" containerID="5844d5f1ac592597f2b908e5491b4b6d1b8f14bca7db1a339f96278fbbc50843" exitCode=255 Jul 09 19:02:16.804966 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:16.804572 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7ddf5bcd8-n2jkw" event={"ID":"75d1a79e-e18e-47e6-9fc6-c5fb01e96002","Type":"ContainerDied","Data":"5844d5f1ac592597f2b908e5491b4b6d1b8f14bca7db1a339f96278fbbc50843"} Jul 09 19:02:16.810290 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:16.810274 2566 scope.go:117] "RemoveContainer" containerID="5844d5f1ac592597f2b908e5491b4b6d1b8f14bca7db1a339f96278fbbc50843" Jul 09 19:02:17.809196 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:17.809164 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7ddf5bcd8-n2jkw" event={"ID":"75d1a79e-e18e-47e6-9fc6-c5fb01e96002","Type":"ContainerStarted","Data":"f7cc9769bcfbdc400a9fbc2f0d1e78b35f987d5f5519ed7aeb34506ed50a6c9b"} Jul 09 19:02:39.914196 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:39.914168 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7866497494-vws4c"] Jul 09 19:02:39.917128 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:39.917111 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-vws4c" Jul 09 19:02:39.920088 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:39.920058 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 09 19:02:39.920088 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:39.920058 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 09 19:02:39.920276 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:39.920099 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 09 19:02:39.921246 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:39.921228 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 09 19:02:39.921325 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:39.921291 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-79nh5\"" Jul 09 19:02:39.926850 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:39.926821 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 09 19:02:39.931208 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:39.931180 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-vws4c"] Jul 09 19:02:39.964140 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:39.964113 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7cf4af17-5218-4b92-bceb-addbefcc03cb-config\") pod \"console-operator-7866497494-vws4c\" (UID: \"7cf4af17-5218-4b92-bceb-addbefcc03cb\") " pod="openshift-console-operator/console-operator-7866497494-vws4c" Jul 09 19:02:39.964140 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:39.964140 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqkqn\" (UniqueName: \"kubernetes.io/projected/7cf4af17-5218-4b92-bceb-addbefcc03cb-kube-api-access-tqkqn\") pod \"console-operator-7866497494-vws4c\" (UID: \"7cf4af17-5218-4b92-bceb-addbefcc03cb\") " pod="openshift-console-operator/console-operator-7866497494-vws4c" Jul 09 19:02:39.964289 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:39.964188 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7cf4af17-5218-4b92-bceb-addbefcc03cb-serving-cert\") pod \"console-operator-7866497494-vws4c\" (UID: \"7cf4af17-5218-4b92-bceb-addbefcc03cb\") " pod="openshift-console-operator/console-operator-7866497494-vws4c" Jul 09 19:02:39.964289 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:39.964259 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7cf4af17-5218-4b92-bceb-addbefcc03cb-trusted-ca\") pod \"console-operator-7866497494-vws4c\" (UID: \"7cf4af17-5218-4b92-bceb-addbefcc03cb\") " pod="openshift-console-operator/console-operator-7866497494-vws4c" Jul 09 19:02:40.004746 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.004715 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-8blf2"] Jul 09 19:02:40.007330 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.007312 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c7cmn"] Jul 09 19:02:40.007462 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.007446 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-8blf2" Jul 09 19:02:40.010198 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.010176 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c7cmn" Jul 09 19:02:40.010905 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.010889 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-lnlct\"" Jul 09 19:02:40.011232 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.011213 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-l7m79"] Jul 09 19:02:40.013538 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.013520 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 09 19:02:40.013861 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.013843 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 09 19:02:40.014111 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.013999 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-v2vdt\"" Jul 09 19:02:40.014111 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.014048 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-bwggg"] Jul 09 19:02:40.014214 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.014192 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-l7m79" Jul 09 19:02:40.014671 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.014493 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 09 19:02:40.016732 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.016718 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-66b97f775b-6x2dl"] Jul 09 19:02:40.016825 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.016812 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-bwggg" Jul 09 19:02:40.019538 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.019521 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66b97f775b-6x2dl" Jul 09 19:02:40.022776 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.022754 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 09 19:02:40.023276 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.022962 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 09 19:02:40.023276 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.023026 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 09 19:02:40.023276 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.023135 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 09 19:02:40.023276 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.023142 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 09 19:02:40.023276 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.023143 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 09 19:02:40.024191 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.023786 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 09 19:02:40.024191 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.023865 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 09 19:02:40.024191 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.024015 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 09 19:02:40.024191 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.024039 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-zlvzt\"" Jul 09 19:02:40.024443 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.024366 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-l7m79"] Jul 09 19:02:40.025077 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.024868 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-4tmhw\"" Jul 09 19:02:40.025211 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.025089 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xh4zm\"" Jul 09 19:02:40.025537 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.025519 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-8blf2"] Jul 09 19:02:40.028887 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.028866 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c7cmn"] Jul 09 19:02:40.032626 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.032593 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 09 19:02:40.033357 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.033331 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 09 19:02:40.033506 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.033469 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-bwggg"] Jul 09 19:02:40.036164 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.036145 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-66b97f775b-6x2dl"] Jul 09 19:02:40.064742 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.064716 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7dc9a1c9-93e9-4e98-b49d-90f25331cea6-snapshots\") pod \"insights-operator-68d9dd94cf-bwggg\" (UID: \"7dc9a1c9-93e9-4e98-b49d-90f25331cea6\") " pod="openshift-insights/insights-operator-68d9dd94cf-bwggg" Jul 09 19:02:40.064850 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.064742 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/31207519-aadd-4eb5-8cef-af759079efb8-registry-tls\") pod \"image-registry-66b97f775b-6x2dl\" (UID: \"31207519-aadd-4eb5-8cef-af759079efb8\") " pod="openshift-image-registry/image-registry-66b97f775b-6x2dl" Jul 09 19:02:40.064850 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.064771 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7dc9a1c9-93e9-4e98-b49d-90f25331cea6-serving-cert\") pod \"insights-operator-68d9dd94cf-bwggg\" (UID: \"7dc9a1c9-93e9-4e98-b49d-90f25331cea6\") " pod="openshift-insights/insights-operator-68d9dd94cf-bwggg" Jul 09 19:02:40.064850 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.064795 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/31207519-aadd-4eb5-8cef-af759079efb8-image-registry-private-configuration\") pod \"image-registry-66b97f775b-6x2dl\" (UID: \"31207519-aadd-4eb5-8cef-af759079efb8\") " pod="openshift-image-registry/image-registry-66b97f775b-6x2dl" Jul 09 19:02:40.065001 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.064848 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7cf4af17-5218-4b92-bceb-addbefcc03cb-serving-cert\") pod \"console-operator-7866497494-vws4c\" (UID: \"7cf4af17-5218-4b92-bceb-addbefcc03cb\") " pod="openshift-console-operator/console-operator-7866497494-vws4c" Jul 09 19:02:40.065001 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.064871 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dc9a1c9-93e9-4e98-b49d-90f25331cea6-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-bwggg\" (UID: \"7dc9a1c9-93e9-4e98-b49d-90f25331cea6\") " pod="openshift-insights/insights-operator-68d9dd94cf-bwggg" Jul 09 19:02:40.065001 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.064944 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7cf4af17-5218-4b92-bceb-addbefcc03cb-trusted-ca\") pod \"console-operator-7866497494-vws4c\" (UID: \"7cf4af17-5218-4b92-bceb-addbefcc03cb\") " pod="openshift-console-operator/console-operator-7866497494-vws4c" Jul 09 19:02:40.065001 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.064978 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/31207519-aadd-4eb5-8cef-af759079efb8-trusted-ca\") pod \"image-registry-66b97f775b-6x2dl\" (UID: \"31207519-aadd-4eb5-8cef-af759079efb8\") " pod="openshift-image-registry/image-registry-66b97f775b-6x2dl" Jul 09 19:02:40.065151 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.065011 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dc9a1c9-93e9-4e98-b49d-90f25331cea6-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-bwggg\" (UID: \"7dc9a1c9-93e9-4e98-b49d-90f25331cea6\") " pod="openshift-insights/insights-operator-68d9dd94cf-bwggg" Jul 09 19:02:40.065151 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.065070 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5a082e9f-d712-4727-b6a4-41e3dec95e75-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-c7cmn\" (UID: \"5a082e9f-d712-4727-b6a4-41e3dec95e75\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c7cmn" Jul 09 19:02:40.065151 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.065117 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbf8j\" (UniqueName: \"kubernetes.io/projected/7dc9a1c9-93e9-4e98-b49d-90f25331cea6-kube-api-access-qbf8j\") pod \"insights-operator-68d9dd94cf-bwggg\" (UID: \"7dc9a1c9-93e9-4e98-b49d-90f25331cea6\") " pod="openshift-insights/insights-operator-68d9dd94cf-bwggg" Jul 09 19:02:40.065267 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.065154 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qb2c\" (UniqueName: \"kubernetes.io/projected/31207519-aadd-4eb5-8cef-af759079efb8-kube-api-access-2qb2c\") pod \"image-registry-66b97f775b-6x2dl\" (UID: \"31207519-aadd-4eb5-8cef-af759079efb8\") " pod="openshift-image-registry/image-registry-66b97f775b-6x2dl" Jul 09 19:02:40.065267 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.065211 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt7pm\" (UniqueName: \"kubernetes.io/projected/01d01444-63e0-4637-9edb-9fbe2273de23-kube-api-access-pt7pm\") pod \"volume-data-source-validator-78495597f6-l7m79\" (UID: \"01d01444-63e0-4637-9edb-9fbe2273de23\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-l7m79" Jul 09 19:02:40.065267 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.065260 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/31207519-aadd-4eb5-8cef-af759079efb8-installation-pull-secrets\") pod \"image-registry-66b97f775b-6x2dl\" (UID: \"31207519-aadd-4eb5-8cef-af759079efb8\") " pod="openshift-image-registry/image-registry-66b97f775b-6x2dl" Jul 09 19:02:40.065366 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.065297 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/31207519-aadd-4eb5-8cef-af759079efb8-registry-certificates\") pod \"image-registry-66b97f775b-6x2dl\" (UID: \"31207519-aadd-4eb5-8cef-af759079efb8\") " pod="openshift-image-registry/image-registry-66b97f775b-6x2dl" Jul 09 19:02:40.065366 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.065312 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/31207519-aadd-4eb5-8cef-af759079efb8-bound-sa-token\") pod \"image-registry-66b97f775b-6x2dl\" (UID: \"31207519-aadd-4eb5-8cef-af759079efb8\") " pod="openshift-image-registry/image-registry-66b97f775b-6x2dl" Jul 09 19:02:40.065366 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.065330 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fzd6\" (UniqueName: \"kubernetes.io/projected/a3c51ab0-12a0-41fa-bb08-f1dca367becd-kube-api-access-4fzd6\") pod \"network-check-source-869dcfcdcd-8blf2\" (UID: \"a3c51ab0-12a0-41fa-bb08-f1dca367becd\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-8blf2" Jul 09 19:02:40.065366 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.065360 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp7dw\" (UniqueName: \"kubernetes.io/projected/5a082e9f-d712-4727-b6a4-41e3dec95e75-kube-api-access-hp7dw\") pod \"cluster-samples-operator-65db858894-c7cmn\" (UID: \"5a082e9f-d712-4727-b6a4-41e3dec95e75\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c7cmn" Jul 09 19:02:40.065663 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.065382 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7dc9a1c9-93e9-4e98-b49d-90f25331cea6-tmp\") pod \"insights-operator-68d9dd94cf-bwggg\" (UID: \"7dc9a1c9-93e9-4e98-b49d-90f25331cea6\") " pod="openshift-insights/insights-operator-68d9dd94cf-bwggg" Jul 09 19:02:40.065663 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.065399 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/31207519-aadd-4eb5-8cef-af759079efb8-ca-trust-extracted\") pod \"image-registry-66b97f775b-6x2dl\" (UID: \"31207519-aadd-4eb5-8cef-af759079efb8\") " pod="openshift-image-registry/image-registry-66b97f775b-6x2dl" Jul 09 19:02:40.065663 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.065461 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqkqn\" (UniqueName: \"kubernetes.io/projected/7cf4af17-5218-4b92-bceb-addbefcc03cb-kube-api-access-tqkqn\") pod \"console-operator-7866497494-vws4c\" (UID: \"7cf4af17-5218-4b92-bceb-addbefcc03cb\") " pod="openshift-console-operator/console-operator-7866497494-vws4c" Jul 09 19:02:40.065663 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.065597 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7cf4af17-5218-4b92-bceb-addbefcc03cb-config\") pod \"console-operator-7866497494-vws4c\" (UID: \"7cf4af17-5218-4b92-bceb-addbefcc03cb\") " pod="openshift-console-operator/console-operator-7866497494-vws4c" Jul 09 19:02:40.065891 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.065875 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7cf4af17-5218-4b92-bceb-addbefcc03cb-trusted-ca\") pod \"console-operator-7866497494-vws4c\" (UID: \"7cf4af17-5218-4b92-bceb-addbefcc03cb\") " pod="openshift-console-operator/console-operator-7866497494-vws4c" Jul 09 19:02:40.066130 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.066114 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7cf4af17-5218-4b92-bceb-addbefcc03cb-config\") pod \"console-operator-7866497494-vws4c\" (UID: \"7cf4af17-5218-4b92-bceb-addbefcc03cb\") " pod="openshift-console-operator/console-operator-7866497494-vws4c" Jul 09 19:02:40.067076 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.067060 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7cf4af17-5218-4b92-bceb-addbefcc03cb-serving-cert\") pod \"console-operator-7866497494-vws4c\" (UID: \"7cf4af17-5218-4b92-bceb-addbefcc03cb\") " pod="openshift-console-operator/console-operator-7866497494-vws4c" Jul 09 19:02:40.076339 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.076305 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqkqn\" (UniqueName: \"kubernetes.io/projected/7cf4af17-5218-4b92-bceb-addbefcc03cb-kube-api-access-tqkqn\") pod \"console-operator-7866497494-vws4c\" (UID: \"7cf4af17-5218-4b92-bceb-addbefcc03cb\") " pod="openshift-console-operator/console-operator-7866497494-vws4c" Jul 09 19:02:40.166406 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.166329 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/31207519-aadd-4eb5-8cef-af759079efb8-registry-certificates\") pod \"image-registry-66b97f775b-6x2dl\" (UID: \"31207519-aadd-4eb5-8cef-af759079efb8\") " pod="openshift-image-registry/image-registry-66b97f775b-6x2dl" Jul 09 19:02:40.166406 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.166361 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/31207519-aadd-4eb5-8cef-af759079efb8-bound-sa-token\") pod \"image-registry-66b97f775b-6x2dl\" (UID: \"31207519-aadd-4eb5-8cef-af759079efb8\") " pod="openshift-image-registry/image-registry-66b97f775b-6x2dl" Jul 09 19:02:40.166406 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.166381 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fzd6\" (UniqueName: \"kubernetes.io/projected/a3c51ab0-12a0-41fa-bb08-f1dca367becd-kube-api-access-4fzd6\") pod \"network-check-source-869dcfcdcd-8blf2\" (UID: \"a3c51ab0-12a0-41fa-bb08-f1dca367becd\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-8blf2" Jul 09 19:02:40.166406 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.166400 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp7dw\" (UniqueName: \"kubernetes.io/projected/5a082e9f-d712-4727-b6a4-41e3dec95e75-kube-api-access-hp7dw\") pod \"cluster-samples-operator-65db858894-c7cmn\" (UID: \"5a082e9f-d712-4727-b6a4-41e3dec95e75\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c7cmn" Jul 09 19:02:40.166754 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.166521 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7dc9a1c9-93e9-4e98-b49d-90f25331cea6-tmp\") pod \"insights-operator-68d9dd94cf-bwggg\" (UID: \"7dc9a1c9-93e9-4e98-b49d-90f25331cea6\") " pod="openshift-insights/insights-operator-68d9dd94cf-bwggg" Jul 09 19:02:40.166754 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.166561 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/31207519-aadd-4eb5-8cef-af759079efb8-ca-trust-extracted\") pod \"image-registry-66b97f775b-6x2dl\" (UID: \"31207519-aadd-4eb5-8cef-af759079efb8\") " pod="openshift-image-registry/image-registry-66b97f775b-6x2dl" Jul 09 19:02:40.170346 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.166926 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7dc9a1c9-93e9-4e98-b49d-90f25331cea6-snapshots\") pod \"insights-operator-68d9dd94cf-bwggg\" (UID: \"7dc9a1c9-93e9-4e98-b49d-90f25331cea6\") " pod="openshift-insights/insights-operator-68d9dd94cf-bwggg" Jul 09 19:02:40.170346 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.167099 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/31207519-aadd-4eb5-8cef-af759079efb8-registry-tls\") pod \"image-registry-66b97f775b-6x2dl\" (UID: \"31207519-aadd-4eb5-8cef-af759079efb8\") " pod="openshift-image-registry/image-registry-66b97f775b-6x2dl" Jul 09 19:02:40.170346 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:02:40.167250 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 19:02:40.170346 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:02:40.167267 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66b97f775b-6x2dl: secret "image-registry-tls" not found Jul 09 19:02:40.170346 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:02:40.167331 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/31207519-aadd-4eb5-8cef-af759079efb8-registry-tls podName:31207519-aadd-4eb5-8cef-af759079efb8 nodeName:}" failed. No retries permitted until 2026-07-09 19:02:40.667311893 +0000 UTC m=+230.175091662 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/31207519-aadd-4eb5-8cef-af759079efb8-registry-tls") pod "image-registry-66b97f775b-6x2dl" (UID: "31207519-aadd-4eb5-8cef-af759079efb8") : secret "image-registry-tls" not found Jul 09 19:02:40.170346 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.167167 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7dc9a1c9-93e9-4e98-b49d-90f25331cea6-serving-cert\") pod \"insights-operator-68d9dd94cf-bwggg\" (UID: \"7dc9a1c9-93e9-4e98-b49d-90f25331cea6\") " pod="openshift-insights/insights-operator-68d9dd94cf-bwggg" Jul 09 19:02:40.170346 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.167386 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/31207519-aadd-4eb5-8cef-af759079efb8-image-registry-private-configuration\") pod \"image-registry-66b97f775b-6x2dl\" (UID: \"31207519-aadd-4eb5-8cef-af759079efb8\") " pod="openshift-image-registry/image-registry-66b97f775b-6x2dl" Jul 09 19:02:40.170346 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.167407 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7dc9a1c9-93e9-4e98-b49d-90f25331cea6-tmp\") pod \"insights-operator-68d9dd94cf-bwggg\" (UID: \"7dc9a1c9-93e9-4e98-b49d-90f25331cea6\") " pod="openshift-insights/insights-operator-68d9dd94cf-bwggg" Jul 09 19:02:40.170346 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.167430 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dc9a1c9-93e9-4e98-b49d-90f25331cea6-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-bwggg\" (UID: \"7dc9a1c9-93e9-4e98-b49d-90f25331cea6\") " pod="openshift-insights/insights-operator-68d9dd94cf-bwggg" Jul 09 19:02:40.170346 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.167486 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/31207519-aadd-4eb5-8cef-af759079efb8-registry-certificates\") pod \"image-registry-66b97f775b-6x2dl\" (UID: \"31207519-aadd-4eb5-8cef-af759079efb8\") " pod="openshift-image-registry/image-registry-66b97f775b-6x2dl" Jul 09 19:02:40.170346 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.167576 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/31207519-aadd-4eb5-8cef-af759079efb8-ca-trust-extracted\") pod \"image-registry-66b97f775b-6x2dl\" (UID: \"31207519-aadd-4eb5-8cef-af759079efb8\") " pod="openshift-image-registry/image-registry-66b97f775b-6x2dl" Jul 09 19:02:40.170346 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.167631 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/31207519-aadd-4eb5-8cef-af759079efb8-trusted-ca\") pod \"image-registry-66b97f775b-6x2dl\" (UID: \"31207519-aadd-4eb5-8cef-af759079efb8\") " pod="openshift-image-registry/image-registry-66b97f775b-6x2dl" Jul 09 19:02:40.170346 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.167721 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dc9a1c9-93e9-4e98-b49d-90f25331cea6-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-bwggg\" (UID: \"7dc9a1c9-93e9-4e98-b49d-90f25331cea6\") " pod="openshift-insights/insights-operator-68d9dd94cf-bwggg" Jul 09 19:02:40.170346 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.167917 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5a082e9f-d712-4727-b6a4-41e3dec95e75-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-c7cmn\" (UID: \"5a082e9f-d712-4727-b6a4-41e3dec95e75\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c7cmn" Jul 09 19:02:40.170346 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.168058 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbf8j\" (UniqueName: \"kubernetes.io/projected/7dc9a1c9-93e9-4e98-b49d-90f25331cea6-kube-api-access-qbf8j\") pod \"insights-operator-68d9dd94cf-bwggg\" (UID: \"7dc9a1c9-93e9-4e98-b49d-90f25331cea6\") " pod="openshift-insights/insights-operator-68d9dd94cf-bwggg" Jul 09 19:02:40.170346 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.168194 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qb2c\" (UniqueName: \"kubernetes.io/projected/31207519-aadd-4eb5-8cef-af759079efb8-kube-api-access-2qb2c\") pod \"image-registry-66b97f775b-6x2dl\" (UID: \"31207519-aadd-4eb5-8cef-af759079efb8\") " pod="openshift-image-registry/image-registry-66b97f775b-6x2dl" Jul 09 19:02:40.172204 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.168215 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7dc9a1c9-93e9-4e98-b49d-90f25331cea6-snapshots\") pod \"insights-operator-68d9dd94cf-bwggg\" (UID: \"7dc9a1c9-93e9-4e98-b49d-90f25331cea6\") " pod="openshift-insights/insights-operator-68d9dd94cf-bwggg" Jul 09 19:02:40.172204 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.168317 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt7pm\" (UniqueName: \"kubernetes.io/projected/01d01444-63e0-4637-9edb-9fbe2273de23-kube-api-access-pt7pm\") pod \"volume-data-source-validator-78495597f6-l7m79\" (UID: \"01d01444-63e0-4637-9edb-9fbe2273de23\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-l7m79" Jul 09 19:02:40.172204 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.168410 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/31207519-aadd-4eb5-8cef-af759079efb8-installation-pull-secrets\") pod \"image-registry-66b97f775b-6x2dl\" (UID: \"31207519-aadd-4eb5-8cef-af759079efb8\") " pod="openshift-image-registry/image-registry-66b97f775b-6x2dl" Jul 09 19:02:40.172204 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.169288 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dc9a1c9-93e9-4e98-b49d-90f25331cea6-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-bwggg\" (UID: \"7dc9a1c9-93e9-4e98-b49d-90f25331cea6\") " pod="openshift-insights/insights-operator-68d9dd94cf-bwggg" Jul 09 19:02:40.172204 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:02:40.169435 2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 19:02:40.172204 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:02:40.170087 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a082e9f-d712-4727-b6a4-41e3dec95e75-samples-operator-tls podName:5a082e9f-d712-4727-b6a4-41e3dec95e75 nodeName:}" failed. No retries permitted until 2026-07-09 19:02:40.670058696 +0000 UTC m=+230.177838458 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5a082e9f-d712-4727-b6a4-41e3dec95e75-samples-operator-tls") pod "cluster-samples-operator-65db858894-c7cmn" (UID: "5a082e9f-d712-4727-b6a4-41e3dec95e75") : secret "samples-operator-tls" not found Jul 09 19:02:40.172204 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.171511 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/31207519-aadd-4eb5-8cef-af759079efb8-trusted-ca\") pod \"image-registry-66b97f775b-6x2dl\" (UID: \"31207519-aadd-4eb5-8cef-af759079efb8\") " pod="openshift-image-registry/image-registry-66b97f775b-6x2dl" Jul 09 19:02:40.172204 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.171647 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7dc9a1c9-93e9-4e98-b49d-90f25331cea6-serving-cert\") pod \"insights-operator-68d9dd94cf-bwggg\" (UID: \"7dc9a1c9-93e9-4e98-b49d-90f25331cea6\") " pod="openshift-insights/insights-operator-68d9dd94cf-bwggg" Jul 09 19:02:40.172653 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.172227 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/31207519-aadd-4eb5-8cef-af759079efb8-installation-pull-secrets\") pod \"image-registry-66b97f775b-6x2dl\" (UID: \"31207519-aadd-4eb5-8cef-af759079efb8\") " pod="openshift-image-registry/image-registry-66b97f775b-6x2dl" Jul 09 19:02:40.172653 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.172264 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dc9a1c9-93e9-4e98-b49d-90f25331cea6-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-bwggg\" (UID: \"7dc9a1c9-93e9-4e98-b49d-90f25331cea6\") " pod="openshift-insights/insights-operator-68d9dd94cf-bwggg" Jul 09 19:02:40.173322 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.173288 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/31207519-aadd-4eb5-8cef-af759079efb8-image-registry-private-configuration\") pod \"image-registry-66b97f775b-6x2dl\" (UID: \"31207519-aadd-4eb5-8cef-af759079efb8\") " pod="openshift-image-registry/image-registry-66b97f775b-6x2dl" Jul 09 19:02:40.175624 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.175580 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fzd6\" (UniqueName: \"kubernetes.io/projected/a3c51ab0-12a0-41fa-bb08-f1dca367becd-kube-api-access-4fzd6\") pod \"network-check-source-869dcfcdcd-8blf2\" (UID: \"a3c51ab0-12a0-41fa-bb08-f1dca367becd\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-8blf2" Jul 09 19:02:40.175802 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.175779 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp7dw\" (UniqueName: \"kubernetes.io/projected/5a082e9f-d712-4727-b6a4-41e3dec95e75-kube-api-access-hp7dw\") pod \"cluster-samples-operator-65db858894-c7cmn\" (UID: \"5a082e9f-d712-4727-b6a4-41e3dec95e75\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c7cmn" Jul 09 19:02:40.175892 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.175876 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/31207519-aadd-4eb5-8cef-af759079efb8-bound-sa-token\") pod \"image-registry-66b97f775b-6x2dl\" (UID: \"31207519-aadd-4eb5-8cef-af759079efb8\") " pod="openshift-image-registry/image-registry-66b97f775b-6x2dl" Jul 09 19:02:40.176804 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.176786 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbf8j\" (UniqueName: \"kubernetes.io/projected/7dc9a1c9-93e9-4e98-b49d-90f25331cea6-kube-api-access-qbf8j\") pod \"insights-operator-68d9dd94cf-bwggg\" (UID: \"7dc9a1c9-93e9-4e98-b49d-90f25331cea6\") " pod="openshift-insights/insights-operator-68d9dd94cf-bwggg" Jul 09 19:02:40.176871 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.176837 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt7pm\" (UniqueName: \"kubernetes.io/projected/01d01444-63e0-4637-9edb-9fbe2273de23-kube-api-access-pt7pm\") pod \"volume-data-source-validator-78495597f6-l7m79\" (UID: \"01d01444-63e0-4637-9edb-9fbe2273de23\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-l7m79" Jul 09 19:02:40.180461 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.180441 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qb2c\" (UniqueName: \"kubernetes.io/projected/31207519-aadd-4eb5-8cef-af759079efb8-kube-api-access-2qb2c\") pod \"image-registry-66b97f775b-6x2dl\" (UID: \"31207519-aadd-4eb5-8cef-af759079efb8\") " pod="openshift-image-registry/image-registry-66b97f775b-6x2dl" Jul 09 19:02:40.226412 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.226374 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-vws4c" Jul 09 19:02:40.317930 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.317884 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-8blf2" Jul 09 19:02:40.332147 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.332119 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-l7m79" Jul 09 19:02:40.340088 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.340063 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-bwggg" Jul 09 19:02:40.349879 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.349824 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-vws4c"] Jul 09 19:02:40.355336 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:02:40.355298 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cf4af17_5218_4b92_bceb_addbefcc03cb.slice/crio-a798cd97ae2ce7489472733dd856235f744cd9d95723bb47fbd0778f32c6ab1b WatchSource:0}: Error finding container a798cd97ae2ce7489472733dd856235f744cd9d95723bb47fbd0778f32c6ab1b: Status 404 returned error can't find the container with id a798cd97ae2ce7489472733dd856235f744cd9d95723bb47fbd0778f32c6ab1b Jul 09 19:02:40.460357 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.460326 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-8blf2"] Jul 09 19:02:40.463757 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:02:40.463719 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3c51ab0_12a0_41fa_bb08_f1dca367becd.slice/crio-960211d579db49ad485b16b16b61e181f02f4967f8e0f8677074d3e2fb8df56b WatchSource:0}: Error finding container 960211d579db49ad485b16b16b61e181f02f4967f8e0f8677074d3e2fb8df56b: Status 404 returned error can't find the container with id 960211d579db49ad485b16b16b61e181f02f4967f8e0f8677074d3e2fb8df56b Jul 09 19:02:40.478160 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.478134 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-l7m79"] Jul 09 19:02:40.481309 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:02:40.481284 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01d01444_63e0_4637_9edb_9fbe2273de23.slice/crio-52d9eea72e79b30f4d193b525705430da5e5dbcb89afc63dc6b56d7838902700 WatchSource:0}: Error finding container 52d9eea72e79b30f4d193b525705430da5e5dbcb89afc63dc6b56d7838902700: Status 404 returned error can't find the container with id 52d9eea72e79b30f4d193b525705430da5e5dbcb89afc63dc6b56d7838902700 Jul 09 19:02:40.495285 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.495265 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-bwggg"] Jul 09 19:02:40.499183 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:02:40.499159 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dc9a1c9_93e9_4e98_b49d_90f25331cea6.slice/crio-ba871a59b435e4dd7f0be6a071ab9d9c45c5ac4a56fd6cdf5c811518003c8694 WatchSource:0}: Error finding container ba871a59b435e4dd7f0be6a071ab9d9c45c5ac4a56fd6cdf5c811518003c8694: Status 404 returned error can't find the container with id ba871a59b435e4dd7f0be6a071ab9d9c45c5ac4a56fd6cdf5c811518003c8694 Jul 09 19:02:40.673811 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.673773 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5a082e9f-d712-4727-b6a4-41e3dec95e75-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-c7cmn\" (UID: \"5a082e9f-d712-4727-b6a4-41e3dec95e75\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c7cmn" Jul 09 19:02:40.674011 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.673835 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/31207519-aadd-4eb5-8cef-af759079efb8-registry-tls\") pod \"image-registry-66b97f775b-6x2dl\" (UID: \"31207519-aadd-4eb5-8cef-af759079efb8\") " pod="openshift-image-registry/image-registry-66b97f775b-6x2dl" Jul 09 19:02:40.674011 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:02:40.673931 2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 19:02:40.674011 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:02:40.673992 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a082e9f-d712-4727-b6a4-41e3dec95e75-samples-operator-tls podName:5a082e9f-d712-4727-b6a4-41e3dec95e75 nodeName:}" failed. No retries permitted until 2026-07-09 19:02:41.673977319 +0000 UTC m=+231.181757072 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5a082e9f-d712-4727-b6a4-41e3dec95e75-samples-operator-tls") pod "cluster-samples-operator-65db858894-c7cmn" (UID: "5a082e9f-d712-4727-b6a4-41e3dec95e75") : secret "samples-operator-tls" not found Jul 09 19:02:40.674172 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:02:40.673936 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 19:02:40.674172 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:02:40.674031 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66b97f775b-6x2dl: secret "image-registry-tls" not found Jul 09 19:02:40.674172 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:02:40.674080 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/31207519-aadd-4eb5-8cef-af759079efb8-registry-tls podName:31207519-aadd-4eb5-8cef-af759079efb8 nodeName:}" failed. No retries permitted until 2026-07-09 19:02:41.674065951 +0000 UTC m=+231.181845722 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/31207519-aadd-4eb5-8cef-af759079efb8-registry-tls") pod "image-registry-66b97f775b-6x2dl" (UID: "31207519-aadd-4eb5-8cef-af759079efb8") : secret "image-registry-tls" not found Jul 09 19:02:40.876152 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.876114 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-l7m79" event={"ID":"01d01444-63e0-4637-9edb-9fbe2273de23","Type":"ContainerStarted","Data":"52d9eea72e79b30f4d193b525705430da5e5dbcb89afc63dc6b56d7838902700"} Jul 09 19:02:40.877340 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.877299 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-bwggg" event={"ID":"7dc9a1c9-93e9-4e98-b49d-90f25331cea6","Type":"ContainerStarted","Data":"ba871a59b435e4dd7f0be6a071ab9d9c45c5ac4a56fd6cdf5c811518003c8694"} Jul 09 19:02:40.878526 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.878499 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-vws4c" event={"ID":"7cf4af17-5218-4b92-bceb-addbefcc03cb","Type":"ContainerStarted","Data":"a798cd97ae2ce7489472733dd856235f744cd9d95723bb47fbd0778f32c6ab1b"} Jul 09 19:02:40.880131 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.880107 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-8blf2" event={"ID":"a3c51ab0-12a0-41fa-bb08-f1dca367becd","Type":"ContainerStarted","Data":"824aa7ae885fb48dd92ee9985867ea73ae5cde7647dd7e228b59382b030f0ea5"} Jul 09 19:02:40.880317 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.880148 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-8blf2" event={"ID":"a3c51ab0-12a0-41fa-bb08-f1dca367becd","Type":"ContainerStarted","Data":"960211d579db49ad485b16b16b61e181f02f4967f8e0f8677074d3e2fb8df56b"} Jul 09 19:02:40.898687 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:40.898634 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-8blf2" podStartSLOduration=1.898598209 podStartE2EDuration="1.898598209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:02:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:02:40.897994566 +0000 UTC m=+230.405774343" watchObservedRunningTime="2026-07-09 19:02:40.898598209 +0000 UTC m=+230.406377964" Jul 09 19:02:41.681057 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:41.681017 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5a082e9f-d712-4727-b6a4-41e3dec95e75-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-c7cmn\" (UID: \"5a082e9f-d712-4727-b6a4-41e3dec95e75\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c7cmn" Jul 09 19:02:41.681534 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:41.681116 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/31207519-aadd-4eb5-8cef-af759079efb8-registry-tls\") pod \"image-registry-66b97f775b-6x2dl\" (UID: \"31207519-aadd-4eb5-8cef-af759079efb8\") " pod="openshift-image-registry/image-registry-66b97f775b-6x2dl" Jul 09 19:02:41.681534 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:02:41.681184 2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 19:02:41.681534 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:02:41.681249 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a082e9f-d712-4727-b6a4-41e3dec95e75-samples-operator-tls podName:5a082e9f-d712-4727-b6a4-41e3dec95e75 nodeName:}" failed. No retries permitted until 2026-07-09 19:02:43.681234207 +0000 UTC m=+233.189013959 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5a082e9f-d712-4727-b6a4-41e3dec95e75-samples-operator-tls") pod "cluster-samples-operator-65db858894-c7cmn" (UID: "5a082e9f-d712-4727-b6a4-41e3dec95e75") : secret "samples-operator-tls" not found Jul 09 19:02:41.681534 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:02:41.681261 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 19:02:41.681534 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:02:41.681272 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66b97f775b-6x2dl: secret "image-registry-tls" not found Jul 09 19:02:41.681534 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:02:41.681316 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/31207519-aadd-4eb5-8cef-af759079efb8-registry-tls podName:31207519-aadd-4eb5-8cef-af759079efb8 nodeName:}" failed. No retries permitted until 2026-07-09 19:02:43.681302214 +0000 UTC m=+233.189081967 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/31207519-aadd-4eb5-8cef-af759079efb8-registry-tls") pod "image-registry-66b97f775b-6x2dl" (UID: "31207519-aadd-4eb5-8cef-af759079efb8") : secret "image-registry-tls" not found Jul 09 19:02:43.700008 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:43.699971 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5a082e9f-d712-4727-b6a4-41e3dec95e75-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-c7cmn\" (UID: \"5a082e9f-d712-4727-b6a4-41e3dec95e75\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c7cmn" Jul 09 19:02:43.700390 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:43.700047 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/31207519-aadd-4eb5-8cef-af759079efb8-registry-tls\") pod \"image-registry-66b97f775b-6x2dl\" (UID: \"31207519-aadd-4eb5-8cef-af759079efb8\") " pod="openshift-image-registry/image-registry-66b97f775b-6x2dl" Jul 09 19:02:43.700390 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:02:43.700136 2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 19:02:43.700390 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:02:43.700148 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 19:02:43.700390 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:02:43.700159 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66b97f775b-6x2dl: secret "image-registry-tls" not found Jul 09 19:02:43.700390 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:02:43.700205 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a082e9f-d712-4727-b6a4-41e3dec95e75-samples-operator-tls podName:5a082e9f-d712-4727-b6a4-41e3dec95e75 nodeName:}" failed. No retries permitted until 2026-07-09 19:02:47.700189982 +0000 UTC m=+237.207969745 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5a082e9f-d712-4727-b6a4-41e3dec95e75-samples-operator-tls") pod "cluster-samples-operator-65db858894-c7cmn" (UID: "5a082e9f-d712-4727-b6a4-41e3dec95e75") : secret "samples-operator-tls" not found Jul 09 19:02:43.700390 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:02:43.700221 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/31207519-aadd-4eb5-8cef-af759079efb8-registry-tls podName:31207519-aadd-4eb5-8cef-af759079efb8 nodeName:}" failed. No retries permitted until 2026-07-09 19:02:47.700215198 +0000 UTC m=+237.207994950 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/31207519-aadd-4eb5-8cef-af759079efb8-registry-tls") pod "image-registry-66b97f775b-6x2dl" (UID: "31207519-aadd-4eb5-8cef-af759079efb8") : secret "image-registry-tls" not found Jul 09 19:02:43.892690 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:43.892655 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-bwggg" event={"ID":"7dc9a1c9-93e9-4e98-b49d-90f25331cea6","Type":"ContainerStarted","Data":"a2a41e35da487c862947668f329f8b20b6f9c96dd56970ec82f4e2b9dc9beb39"} Jul 09 19:02:43.894383 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:43.894362 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-vws4c_7cf4af17-5218-4b92-bceb-addbefcc03cb/console-operator/0.log" Jul 09 19:02:43.894506 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:43.894394 2566 generic.go:356] "Generic (PLEG): container finished" podID="7cf4af17-5218-4b92-bceb-addbefcc03cb" containerID="3f50bca91a114e08403c4e4d0a30c5edb84fa8e50222ba0fabc459976c23a198" exitCode=255 Jul 09 19:02:43.894569 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:43.894490 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-vws4c" event={"ID":"7cf4af17-5218-4b92-bceb-addbefcc03cb","Type":"ContainerDied","Data":"3f50bca91a114e08403c4e4d0a30c5edb84fa8e50222ba0fabc459976c23a198"} Jul 09 19:02:43.894700 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:43.894684 2566 scope.go:117] "RemoveContainer" containerID="3f50bca91a114e08403c4e4d0a30c5edb84fa8e50222ba0fabc459976c23a198" Jul 09 19:02:43.896006 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:43.895982 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-l7m79" event={"ID":"01d01444-63e0-4637-9edb-9fbe2273de23","Type":"ContainerStarted","Data":"e9f7eb4523dfce223bb1ee0236310f24a6905c4ab8ef46cf92826e35a477f0bb"} Jul 09 19:02:43.914961 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:43.914913 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-68d9dd94cf-bwggg" podStartSLOduration=2.154380819 podStartE2EDuration="4.91489569s" totalImagesPullingTime="2.760514871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:02:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:02:43.913761492 +0000 UTC m=+233.421541266" watchObservedRunningTime="2026-07-09 19:02:43.91489569 +0000 UTC m=+233.422675466" Jul 09 19:02:43.935191 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:43.935150 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-l7m79" podStartSLOduration=2.162545559 podStartE2EDuration="4.935135577s" totalImagesPullingTime="2.772590018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:02:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:02:43.934324657 +0000 UTC m=+233.442104429" watchObservedRunningTime="2026-07-09 19:02:43.935135577 +0000 UTC m=+233.442915352" Jul 09 19:02:44.313050 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:44.313017 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8zh4c"] Jul 09 19:02:44.316281 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:44.316254 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8zh4c" Jul 09 19:02:44.318935 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:44.318906 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 09 19:02:44.319116 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:44.318906 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 09 19:02:44.319181 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:44.319165 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rw89p\"" Jul 09 19:02:44.328302 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:44.328279 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8zh4c"] Jul 09 19:02:44.406491 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:44.406464 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtwmd\" (UniqueName: \"kubernetes.io/projected/e057c3f1-8832-4c08-b9c2-7e1353a8a82c-kube-api-access-qtwmd\") pod \"insights-runtime-extractor-8zh4c\" (UID: \"e057c3f1-8832-4c08-b9c2-7e1353a8a82c\") " pod="openshift-insights/insights-runtime-extractor-8zh4c" Jul 09 19:02:44.406666 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:44.406517 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e057c3f1-8832-4c08-b9c2-7e1353a8a82c-crio-socket\") pod \"insights-runtime-extractor-8zh4c\" (UID: \"e057c3f1-8832-4c08-b9c2-7e1353a8a82c\") " pod="openshift-insights/insights-runtime-extractor-8zh4c" Jul 09 19:02:44.406666 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:44.406577 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e057c3f1-8832-4c08-b9c2-7e1353a8a82c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8zh4c\" (UID: \"e057c3f1-8832-4c08-b9c2-7e1353a8a82c\") " pod="openshift-insights/insights-runtime-extractor-8zh4c" Jul 09 19:02:44.406666 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:44.406627 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e057c3f1-8832-4c08-b9c2-7e1353a8a82c-data-volume\") pod \"insights-runtime-extractor-8zh4c\" (UID: \"e057c3f1-8832-4c08-b9c2-7e1353a8a82c\") " pod="openshift-insights/insights-runtime-extractor-8zh4c" Jul 09 19:02:44.406666 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:44.406661 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e057c3f1-8832-4c08-b9c2-7e1353a8a82c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8zh4c\" (UID: \"e057c3f1-8832-4c08-b9c2-7e1353a8a82c\") " pod="openshift-insights/insights-runtime-extractor-8zh4c" Jul 09 19:02:44.507429 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:44.507396 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e057c3f1-8832-4c08-b9c2-7e1353a8a82c-crio-socket\") pod \"insights-runtime-extractor-8zh4c\" (UID: \"e057c3f1-8832-4c08-b9c2-7e1353a8a82c\") " pod="openshift-insights/insights-runtime-extractor-8zh4c" Jul 09 19:02:44.507591 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:44.507438 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e057c3f1-8832-4c08-b9c2-7e1353a8a82c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8zh4c\" (UID: \"e057c3f1-8832-4c08-b9c2-7e1353a8a82c\") " pod="openshift-insights/insights-runtime-extractor-8zh4c" Jul 09 19:02:44.507591 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:44.507457 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e057c3f1-8832-4c08-b9c2-7e1353a8a82c-data-volume\") pod \"insights-runtime-extractor-8zh4c\" (UID: \"e057c3f1-8832-4c08-b9c2-7e1353a8a82c\") " pod="openshift-insights/insights-runtime-extractor-8zh4c" Jul 09 19:02:44.507591 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:44.507474 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e057c3f1-8832-4c08-b9c2-7e1353a8a82c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8zh4c\" (UID: \"e057c3f1-8832-4c08-b9c2-7e1353a8a82c\") " pod="openshift-insights/insights-runtime-extractor-8zh4c" Jul 09 19:02:44.507591 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:44.507534 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e057c3f1-8832-4c08-b9c2-7e1353a8a82c-crio-socket\") pod \"insights-runtime-extractor-8zh4c\" (UID: \"e057c3f1-8832-4c08-b9c2-7e1353a8a82c\") " pod="openshift-insights/insights-runtime-extractor-8zh4c" Jul 09 19:02:44.507591 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:02:44.507554 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 19:02:44.507591 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:44.507540 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtwmd\" (UniqueName: \"kubernetes.io/projected/e057c3f1-8832-4c08-b9c2-7e1353a8a82c-kube-api-access-qtwmd\") pod \"insights-runtime-extractor-8zh4c\" (UID: \"e057c3f1-8832-4c08-b9c2-7e1353a8a82c\") " pod="openshift-insights/insights-runtime-extractor-8zh4c" Jul 09 19:02:44.507834 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:02:44.507657 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 19:02:44.507834 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:02:44.507707 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e057c3f1-8832-4c08-b9c2-7e1353a8a82c-insights-runtime-extractor-tls podName:e057c3f1-8832-4c08-b9c2-7e1353a8a82c nodeName:}" failed. No retries permitted until 2026-07-09 19:02:45.007687733 +0000 UTC m=+234.515467494 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e057c3f1-8832-4c08-b9c2-7e1353a8a82c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8zh4c" (UID: "e057c3f1-8832-4c08-b9c2-7e1353a8a82c") : secret "insights-runtime-extractor-tls" not found Jul 09 19:02:44.507834 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:02:44.507726 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e057c3f1-8832-4c08-b9c2-7e1353a8a82c-kube-rbac-proxy-cm podName:e057c3f1-8832-4c08-b9c2-7e1353a8a82c nodeName:}" failed. No retries permitted until 2026-07-09 19:02:45.007717268 +0000 UTC m=+234.515497021 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e057c3f1-8832-4c08-b9c2-7e1353a8a82c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8zh4c" (UID: "e057c3f1-8832-4c08-b9c2-7e1353a8a82c") : configmap "kube-rbac-proxy" not found Jul 09 19:02:44.507834 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:44.507825 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e057c3f1-8832-4c08-b9c2-7e1353a8a82c-data-volume\") pod \"insights-runtime-extractor-8zh4c\" (UID: \"e057c3f1-8832-4c08-b9c2-7e1353a8a82c\") " pod="openshift-insights/insights-runtime-extractor-8zh4c" Jul 09 19:02:44.518266 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:44.518227 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtwmd\" (UniqueName: \"kubernetes.io/projected/e057c3f1-8832-4c08-b9c2-7e1353a8a82c-kube-api-access-qtwmd\") pod \"insights-runtime-extractor-8zh4c\" (UID: \"e057c3f1-8832-4c08-b9c2-7e1353a8a82c\") " pod="openshift-insights/insights-runtime-extractor-8zh4c" Jul 09 19:02:44.899858 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:44.899830 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-vws4c_7cf4af17-5218-4b92-bceb-addbefcc03cb/console-operator/1.log" Jul 09 19:02:44.900248 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:44.900198 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-vws4c_7cf4af17-5218-4b92-bceb-addbefcc03cb/console-operator/0.log" Jul 09 19:02:44.900248 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:44.900230 2566 generic.go:356] "Generic (PLEG): container finished" podID="7cf4af17-5218-4b92-bceb-addbefcc03cb" containerID="e02ec39b234acd125f1ee181ca22a17ebac6c0e5eccdcebdad14cd4f2075fca0" exitCode=255 Jul 09 19:02:44.900336 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:44.900301 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-vws4c" event={"ID":"7cf4af17-5218-4b92-bceb-addbefcc03cb","Type":"ContainerDied","Data":"e02ec39b234acd125f1ee181ca22a17ebac6c0e5eccdcebdad14cd4f2075fca0"} Jul 09 19:02:44.900379 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:44.900352 2566 scope.go:117] "RemoveContainer" containerID="3f50bca91a114e08403c4e4d0a30c5edb84fa8e50222ba0fabc459976c23a198" Jul 09 19:02:44.900634 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:44.900594 2566 scope.go:117] "RemoveContainer" containerID="e02ec39b234acd125f1ee181ca22a17ebac6c0e5eccdcebdad14cd4f2075fca0" Jul 09 19:02:44.900829 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:02:44.900808 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-vws4c_openshift-console-operator(7cf4af17-5218-4b92-bceb-addbefcc03cb)\"" pod="openshift-console-operator/console-operator-7866497494-vws4c" podUID="7cf4af17-5218-4b92-bceb-addbefcc03cb" Jul 09 19:02:45.011262 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:45.011231 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e057c3f1-8832-4c08-b9c2-7e1353a8a82c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8zh4c\" (UID: \"e057c3f1-8832-4c08-b9c2-7e1353a8a82c\") " pod="openshift-insights/insights-runtime-extractor-8zh4c" Jul 09 19:02:45.011458 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:45.011294 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e057c3f1-8832-4c08-b9c2-7e1353a8a82c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8zh4c\" (UID: \"e057c3f1-8832-4c08-b9c2-7e1353a8a82c\") " pod="openshift-insights/insights-runtime-extractor-8zh4c" Jul 09 19:02:45.011458 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:02:45.011391 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 19:02:45.011458 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:02:45.011413 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 19:02:45.011650 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:02:45.011462 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e057c3f1-8832-4c08-b9c2-7e1353a8a82c-kube-rbac-proxy-cm podName:e057c3f1-8832-4c08-b9c2-7e1353a8a82c nodeName:}" failed. No retries permitted until 2026-07-09 19:02:46.011443663 +0000 UTC m=+235.519223433 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e057c3f1-8832-4c08-b9c2-7e1353a8a82c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8zh4c" (UID: "e057c3f1-8832-4c08-b9c2-7e1353a8a82c") : configmap "kube-rbac-proxy" not found Jul 09 19:02:45.011650 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:02:45.011483 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e057c3f1-8832-4c08-b9c2-7e1353a8a82c-insights-runtime-extractor-tls podName:e057c3f1-8832-4c08-b9c2-7e1353a8a82c nodeName:}" failed. No retries permitted until 2026-07-09 19:02:46.011472978 +0000 UTC m=+235.519252731 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e057c3f1-8832-4c08-b9c2-7e1353a8a82c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8zh4c" (UID: "e057c3f1-8832-4c08-b9c2-7e1353a8a82c") : secret "insights-runtime-extractor-tls" not found Jul 09 19:02:45.196175 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:45.196149 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-z4t8w"] Jul 09 19:02:45.200113 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:45.200097 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-z4t8w" Jul 09 19:02:45.202938 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:45.202915 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-qwd2l\"" Jul 09 19:02:45.203056 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:45.202966 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 09 19:02:45.203056 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:45.202970 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 09 19:02:45.208164 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:45.208142 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-z4t8w"] Jul 09 19:02:45.313956 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:45.313913 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbkcr\" (UniqueName: \"kubernetes.io/projected/fbfea8e9-4071-48a1-894c-6c6e0549e5cd-kube-api-access-dbkcr\") pod \"migrator-7f59b54f6c-z4t8w\" (UID: \"fbfea8e9-4071-48a1-894c-6c6e0549e5cd\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-z4t8w" Jul 09 19:02:45.414579 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:45.414535 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbkcr\" (UniqueName: \"kubernetes.io/projected/fbfea8e9-4071-48a1-894c-6c6e0549e5cd-kube-api-access-dbkcr\") pod \"migrator-7f59b54f6c-z4t8w\" (UID: \"fbfea8e9-4071-48a1-894c-6c6e0549e5cd\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-z4t8w" Jul 09 19:02:45.423172 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:45.423154 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbkcr\" (UniqueName: \"kubernetes.io/projected/fbfea8e9-4071-48a1-894c-6c6e0549e5cd-kube-api-access-dbkcr\") pod \"migrator-7f59b54f6c-z4t8w\" (UID: \"fbfea8e9-4071-48a1-894c-6c6e0549e5cd\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-z4t8w" Jul 09 19:02:45.509473 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:45.509393 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-z4t8w" Jul 09 19:02:45.634236 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:45.630929 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-z4t8w"] Jul 09 19:02:45.636890 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:02:45.636866 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbfea8e9_4071_48a1_894c_6c6e0549e5cd.slice/crio-5c7b1ca3e67d75eefbceadd75fe93bc8b60ec80f74e461bb7c1fb74bd19bac8f WatchSource:0}: Error finding container 5c7b1ca3e67d75eefbceadd75fe93bc8b60ec80f74e461bb7c1fb74bd19bac8f: Status 404 returned error can't find the container with id 5c7b1ca3e67d75eefbceadd75fe93bc8b60ec80f74e461bb7c1fb74bd19bac8f Jul 09 19:02:45.904666 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:45.904583 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-vws4c_7cf4af17-5218-4b92-bceb-addbefcc03cb/console-operator/1.log" Jul 09 19:02:45.905031 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:45.904944 2566 scope.go:117] "RemoveContainer" containerID="e02ec39b234acd125f1ee181ca22a17ebac6c0e5eccdcebdad14cd4f2075fca0" Jul 09 19:02:45.905155 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:02:45.905132 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-vws4c_openshift-console-operator(7cf4af17-5218-4b92-bceb-addbefcc03cb)\"" pod="openshift-console-operator/console-operator-7866497494-vws4c" podUID="7cf4af17-5218-4b92-bceb-addbefcc03cb" Jul 09 19:02:45.905764 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:45.905734 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-z4t8w" event={"ID":"fbfea8e9-4071-48a1-894c-6c6e0549e5cd","Type":"ContainerStarted","Data":"5c7b1ca3e67d75eefbceadd75fe93bc8b60ec80f74e461bb7c1fb74bd19bac8f"} Jul 09 19:02:46.019720 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:46.019686 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e057c3f1-8832-4c08-b9c2-7e1353a8a82c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8zh4c\" (UID: \"e057c3f1-8832-4c08-b9c2-7e1353a8a82c\") " pod="openshift-insights/insights-runtime-extractor-8zh4c" Jul 09 19:02:46.019906 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:46.019732 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e057c3f1-8832-4c08-b9c2-7e1353a8a82c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8zh4c\" (UID: \"e057c3f1-8832-4c08-b9c2-7e1353a8a82c\") " pod="openshift-insights/insights-runtime-extractor-8zh4c" Jul 09 19:02:46.019906 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:02:46.019795 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 19:02:46.019906 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:02:46.019848 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 19:02:46.019906 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:02:46.019865 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e057c3f1-8832-4c08-b9c2-7e1353a8a82c-insights-runtime-extractor-tls podName:e057c3f1-8832-4c08-b9c2-7e1353a8a82c nodeName:}" failed. No retries permitted until 2026-07-09 19:02:48.019845111 +0000 UTC m=+237.527624874 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e057c3f1-8832-4c08-b9c2-7e1353a8a82c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8zh4c" (UID: "e057c3f1-8832-4c08-b9c2-7e1353a8a82c") : secret "insights-runtime-extractor-tls" not found Jul 09 19:02:46.019906 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:02:46.019906 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e057c3f1-8832-4c08-b9c2-7e1353a8a82c-kube-rbac-proxy-cm podName:e057c3f1-8832-4c08-b9c2-7e1353a8a82c nodeName:}" failed. No retries permitted until 2026-07-09 19:02:48.019889153 +0000 UTC m=+237.527668920 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e057c3f1-8832-4c08-b9c2-7e1353a8a82c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8zh4c" (UID: "e057c3f1-8832-4c08-b9c2-7e1353a8a82c") : configmap "kube-rbac-proxy" not found Jul 09 19:02:46.576389 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:46.576362 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-x8k65_0da71879-5de7-47a0-819a-0b1951b1e817/dns-node-resolver/0.log" Jul 09 19:02:47.378499 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:47.378424 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jnt6g_5c298798-c68a-4450-b9dc-358fa978305a/node-ca/0.log" Jul 09 19:02:47.732619 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:47.732570 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5a082e9f-d712-4727-b6a4-41e3dec95e75-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-c7cmn\" (UID: \"5a082e9f-d712-4727-b6a4-41e3dec95e75\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c7cmn" Jul 09 19:02:47.732780 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:47.732672 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/31207519-aadd-4eb5-8cef-af759079efb8-registry-tls\") pod \"image-registry-66b97f775b-6x2dl\" (UID: \"31207519-aadd-4eb5-8cef-af759079efb8\") " pod="openshift-image-registry/image-registry-66b97f775b-6x2dl" Jul 09 19:02:47.732780 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:02:47.732724 2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 19:02:47.732780 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:02:47.732760 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 19:02:47.732780 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:02:47.732770 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66b97f775b-6x2dl: secret "image-registry-tls" not found Jul 09 19:02:47.732906 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:02:47.732787 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a082e9f-d712-4727-b6a4-41e3dec95e75-samples-operator-tls podName:5a082e9f-d712-4727-b6a4-41e3dec95e75 nodeName:}" failed. No retries permitted until 2026-07-09 19:02:55.732772218 +0000 UTC m=+245.240551970 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5a082e9f-d712-4727-b6a4-41e3dec95e75-samples-operator-tls") pod "cluster-samples-operator-65db858894-c7cmn" (UID: "5a082e9f-d712-4727-b6a4-41e3dec95e75") : secret "samples-operator-tls" not found Jul 09 19:02:47.732906 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:02:47.732805 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/31207519-aadd-4eb5-8cef-af759079efb8-registry-tls podName:31207519-aadd-4eb5-8cef-af759079efb8 nodeName:}" failed. No retries permitted until 2026-07-09 19:02:55.732794907 +0000 UTC m=+245.240574658 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/31207519-aadd-4eb5-8cef-af759079efb8-registry-tls") pod "image-registry-66b97f775b-6x2dl" (UID: "31207519-aadd-4eb5-8cef-af759079efb8") : secret "image-registry-tls" not found Jul 09 19:02:47.913456 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:47.913418 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-z4t8w" event={"ID":"fbfea8e9-4071-48a1-894c-6c6e0549e5cd","Type":"ContainerStarted","Data":"fc30d52034b10bcb8a14454288b50b13e6662454abc1dea439913d2659d47d83"} Jul 09 19:02:47.913650 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:47.913484 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-z4t8w" event={"ID":"fbfea8e9-4071-48a1-894c-6c6e0549e5cd","Type":"ContainerStarted","Data":"83a5aae5ac4d0228b87c0f32aff3e1225bfc2fd883617ec4b4de4e6d2a96b271"} Jul 09 19:02:47.934487 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:47.934442 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-z4t8w" podStartSLOduration=1.514173395 podStartE2EDuration="2.934428292s" totalImagesPullingTime="1.420254897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:02:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:02:47.933404906 +0000 UTC m=+237.441184680" watchObservedRunningTime="2026-07-09 19:02:47.934428292 +0000 UTC m=+237.442208066" Jul 09 19:02:47.981353 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:47.981331 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7dgq9_a209a84f-cc92-4f17-a299-4f4161322946/serve-healthcheck-canary/0.log" Jul 09 19:02:48.035966 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:48.035889 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e057c3f1-8832-4c08-b9c2-7e1353a8a82c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8zh4c\" (UID: \"e057c3f1-8832-4c08-b9c2-7e1353a8a82c\") " pod="openshift-insights/insights-runtime-extractor-8zh4c" Jul 09 19:02:48.035966 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:48.035952 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e057c3f1-8832-4c08-b9c2-7e1353a8a82c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8zh4c\" (UID: \"e057c3f1-8832-4c08-b9c2-7e1353a8a82c\") " pod="openshift-insights/insights-runtime-extractor-8zh4c" Jul 09 19:02:48.036151 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:02:48.036040 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 19:02:48.036151 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:02:48.036084 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e057c3f1-8832-4c08-b9c2-7e1353a8a82c-insights-runtime-extractor-tls podName:e057c3f1-8832-4c08-b9c2-7e1353a8a82c nodeName:}" failed. No retries permitted until 2026-07-09 19:02:52.036071937 +0000 UTC m=+241.543851689 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e057c3f1-8832-4c08-b9c2-7e1353a8a82c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8zh4c" (UID: "e057c3f1-8832-4c08-b9c2-7e1353a8a82c") : secret "insights-runtime-extractor-tls" not found Jul 09 19:02:48.036151 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:02:48.036113 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 19:02:48.036291 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:02:48.036167 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e057c3f1-8832-4c08-b9c2-7e1353a8a82c-kube-rbac-proxy-cm podName:e057c3f1-8832-4c08-b9c2-7e1353a8a82c nodeName:}" failed. No retries permitted until 2026-07-09 19:02:52.036154496 +0000 UTC m=+241.543934249 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e057c3f1-8832-4c08-b9c2-7e1353a8a82c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8zh4c" (UID: "e057c3f1-8832-4c08-b9c2-7e1353a8a82c") : configmap "kube-rbac-proxy" not found Jul 09 19:02:50.226544 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:50.226512 2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-vws4c" Jul 09 19:02:50.226913 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:50.226564 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-vws4c" Jul 09 19:02:50.226913 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:50.226892 2566 scope.go:117] "RemoveContainer" containerID="e02ec39b234acd125f1ee181ca22a17ebac6c0e5eccdcebdad14cd4f2075fca0" Jul 09 19:02:50.227065 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:02:50.227044 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-vws4c_openshift-console-operator(7cf4af17-5218-4b92-bceb-addbefcc03cb)\"" pod="openshift-console-operator/console-operator-7866497494-vws4c" podUID="7cf4af17-5218-4b92-bceb-addbefcc03cb" Jul 09 19:02:52.066268 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:52.066229 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e057c3f1-8832-4c08-b9c2-7e1353a8a82c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8zh4c\" (UID: \"e057c3f1-8832-4c08-b9c2-7e1353a8a82c\") " pod="openshift-insights/insights-runtime-extractor-8zh4c" Jul 09 19:02:52.066268 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:52.066268 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e057c3f1-8832-4c08-b9c2-7e1353a8a82c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8zh4c\" (UID: \"e057c3f1-8832-4c08-b9c2-7e1353a8a82c\") " pod="openshift-insights/insights-runtime-extractor-8zh4c" Jul 09 19:02:52.066692 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:02:52.066384 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 19:02:52.066692 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:02:52.066414 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 19:02:52.066692 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:02:52.066446 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e057c3f1-8832-4c08-b9c2-7e1353a8a82c-insights-runtime-extractor-tls podName:e057c3f1-8832-4c08-b9c2-7e1353a8a82c nodeName:}" failed. No retries permitted until 2026-07-09 19:03:00.066428884 +0000 UTC m=+249.574208649 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e057c3f1-8832-4c08-b9c2-7e1353a8a82c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8zh4c" (UID: "e057c3f1-8832-4c08-b9c2-7e1353a8a82c") : secret "insights-runtime-extractor-tls" not found Jul 09 19:02:52.066692 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:02:52.066480 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e057c3f1-8832-4c08-b9c2-7e1353a8a82c-kube-rbac-proxy-cm podName:e057c3f1-8832-4c08-b9c2-7e1353a8a82c nodeName:}" failed. No retries permitted until 2026-07-09 19:03:00.066467269 +0000 UTC m=+249.574247026 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e057c3f1-8832-4c08-b9c2-7e1353a8a82c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8zh4c" (UID: "e057c3f1-8832-4c08-b9c2-7e1353a8a82c") : configmap "kube-rbac-proxy" not found Jul 09 19:02:55.798583 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:55.798545 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/31207519-aadd-4eb5-8cef-af759079efb8-registry-tls\") pod \"image-registry-66b97f775b-6x2dl\" (UID: \"31207519-aadd-4eb5-8cef-af759079efb8\") " pod="openshift-image-registry/image-registry-66b97f775b-6x2dl" Jul 09 19:02:55.798976 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:55.798620 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5a082e9f-d712-4727-b6a4-41e3dec95e75-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-c7cmn\" (UID: \"5a082e9f-d712-4727-b6a4-41e3dec95e75\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c7cmn" Jul 09 19:02:55.800921 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:55.800901 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5a082e9f-d712-4727-b6a4-41e3dec95e75-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-c7cmn\" (UID: \"5a082e9f-d712-4727-b6a4-41e3dec95e75\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c7cmn" Jul 09 19:02:55.801000 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:55.800933 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/31207519-aadd-4eb5-8cef-af759079efb8-registry-tls\") pod \"image-registry-66b97f775b-6x2dl\" (UID: \"31207519-aadd-4eb5-8cef-af759079efb8\") " pod="openshift-image-registry/image-registry-66b97f775b-6x2dl" Jul 09 19:02:55.926469 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:55.926437 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-v2vdt\"" Jul 09 19:02:55.933747 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:55.933730 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c7cmn" Jul 09 19:02:55.949015 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:55.948992 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xh4zm\"" Jul 09 19:02:55.955751 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:55.955730 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66b97f775b-6x2dl" Jul 09 19:02:56.075840 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:56.075770 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c7cmn"] Jul 09 19:02:56.096968 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:56.096942 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-66b97f775b-6x2dl"] Jul 09 19:02:56.099204 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:02:56.099181 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31207519_aadd_4eb5_8cef_af759079efb8.slice/crio-6337aecc27af6392c7780a641cb4df54024d5bade2da76fb3398df3e7aeca07b WatchSource:0}: Error finding container 6337aecc27af6392c7780a641cb4df54024d5bade2da76fb3398df3e7aeca07b: Status 404 returned error can't find the container with id 6337aecc27af6392c7780a641cb4df54024d5bade2da76fb3398df3e7aeca07b Jul 09 19:02:56.942335 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:56.942299 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66b97f775b-6x2dl" event={"ID":"31207519-aadd-4eb5-8cef-af759079efb8","Type":"ContainerStarted","Data":"ec77586e1ed361ed56896b4f70470c8a3774ee78e4bbf50b225bc9f6f1998d2a"} Jul 09 19:02:56.942778 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:56.942350 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66b97f775b-6x2dl" event={"ID":"31207519-aadd-4eb5-8cef-af759079efb8","Type":"ContainerStarted","Data":"6337aecc27af6392c7780a641cb4df54024d5bade2da76fb3398df3e7aeca07b"} Jul 09 19:02:56.942778 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:56.942397 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-66b97f775b-6x2dl" Jul 09 19:02:56.943412 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:56.943389 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c7cmn" event={"ID":"5a082e9f-d712-4727-b6a4-41e3dec95e75","Type":"ContainerStarted","Data":"a8306056aa6227cd13aad89220f2193d6a58c15e8aa55c2b3d9ee7ecfb511fe9"} Jul 09 19:02:56.962262 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:56.962218 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-66b97f775b-6x2dl" podStartSLOduration=17.962203785 podStartE2EDuration="17.962203785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:02:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:02:56.961818446 +0000 UTC m=+246.469598221" watchObservedRunningTime="2026-07-09 19:02:56.962203785 +0000 UTC m=+246.469983558" Jul 09 19:02:58.952774 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:58.952742 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c7cmn" event={"ID":"5a082e9f-d712-4727-b6a4-41e3dec95e75","Type":"ContainerStarted","Data":"7ab2ba5dc63a6ed8a11e3514d05077edb0a087da54afa88f2b9f5a580dfb5fbb"} Jul 09 19:02:58.953126 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:58.952800 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c7cmn" event={"ID":"5a082e9f-d712-4727-b6a4-41e3dec95e75","Type":"ContainerStarted","Data":"3ebf16bbd242cab020cfaf06752d9f77a049217a8beb7a919fcc502485dd1d0b"} Jul 09 19:02:58.968494 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:02:58.968453 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c7cmn" podStartSLOduration=17.921949587 podStartE2EDuration="19.968439226s" totalImagesPullingTime="2.046489639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:02:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:02:58.967847527 +0000 UTC m=+248.475627301" watchObservedRunningTime="2026-07-09 19:02:58.968439226 +0000 UTC m=+248.476219000" Jul 09 19:03:00.133802 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:00.133758 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e057c3f1-8832-4c08-b9c2-7e1353a8a82c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8zh4c\" (UID: \"e057c3f1-8832-4c08-b9c2-7e1353a8a82c\") " pod="openshift-insights/insights-runtime-extractor-8zh4c" Jul 09 19:03:00.133802 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:00.133806 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e057c3f1-8832-4c08-b9c2-7e1353a8a82c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8zh4c\" (UID: \"e057c3f1-8832-4c08-b9c2-7e1353a8a82c\") " pod="openshift-insights/insights-runtime-extractor-8zh4c" Jul 09 19:03:00.134266 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:00.134249 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e057c3f1-8832-4c08-b9c2-7e1353a8a82c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8zh4c\" (UID: \"e057c3f1-8832-4c08-b9c2-7e1353a8a82c\") " pod="openshift-insights/insights-runtime-extractor-8zh4c" Jul 09 19:03:00.136008 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:00.135987 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e057c3f1-8832-4c08-b9c2-7e1353a8a82c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8zh4c\" (UID: \"e057c3f1-8832-4c08-b9c2-7e1353a8a82c\") " pod="openshift-insights/insights-runtime-extractor-8zh4c" Jul 09 19:03:00.228040 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:00.228015 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rw89p\"" Jul 09 19:03:00.235361 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:00.235340 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8zh4c" Jul 09 19:03:00.359948 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:00.359918 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8zh4c"] Jul 09 19:03:00.363553 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:03:00.363526 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode057c3f1_8832_4c08_b9c2_7e1353a8a82c.slice/crio-bebd8371d9fb9c35509717aa374a9cf62ffabc240ba6c5f941e0056cda975789 WatchSource:0}: Error finding container bebd8371d9fb9c35509717aa374a9cf62ffabc240ba6c5f941e0056cda975789: Status 404 returned error can't find the container with id bebd8371d9fb9c35509717aa374a9cf62ffabc240ba6c5f941e0056cda975789 Jul 09 19:03:00.968340 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:00.968299 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8zh4c" event={"ID":"e057c3f1-8832-4c08-b9c2-7e1353a8a82c","Type":"ContainerStarted","Data":"bebd8371d9fb9c35509717aa374a9cf62ffabc240ba6c5f941e0056cda975789"} Jul 09 19:03:01.053917 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:01.053885 2566 scope.go:117] "RemoveContainer" containerID="e02ec39b234acd125f1ee181ca22a17ebac6c0e5eccdcebdad14cd4f2075fca0" Jul 09 19:03:01.849370 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:01.849332 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30cf5feb-cf9a-40c0-8a6a-de8fc6c60506-metrics-certs\") pod \"network-metrics-daemon-vtc8p\" (UID: \"30cf5feb-cf9a-40c0-8a6a-de8fc6c60506\") " pod="openshift-multus/network-metrics-daemon-vtc8p" Jul 09 19:03:01.851817 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:01.851787 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30cf5feb-cf9a-40c0-8a6a-de8fc6c60506-metrics-certs\") pod \"network-metrics-daemon-vtc8p\" (UID: \"30cf5feb-cf9a-40c0-8a6a-de8fc6c60506\") " pod="openshift-multus/network-metrics-daemon-vtc8p" Jul 09 19:03:01.974267 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:01.974240 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-vws4c_7cf4af17-5218-4b92-bceb-addbefcc03cb/console-operator/1.log" Jul 09 19:03:01.974414 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:01.974313 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-vws4c" event={"ID":"7cf4af17-5218-4b92-bceb-addbefcc03cb","Type":"ContainerStarted","Data":"3dfb7a6560a682f57554a13c8a8ea3f7cc02eb6dc9b1e563b6b22f2c79a9806d"} Jul 09 19:03:01.974626 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:01.974590 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-vws4c" Jul 09 19:03:01.979505 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:01.979480 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7866497494-vws4c" Jul 09 19:03:01.993977 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:01.993930 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7866497494-vws4c" podStartSLOduration=20.097232109 podStartE2EDuration="22.993913834s" totalImagesPullingTime="2.896681725s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:02:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:03:01.993051477 +0000 UTC m=+251.500831263" watchObservedRunningTime="2026-07-09 19:03:01.993913834 +0000 UTC m=+251.501693607" Jul 09 19:03:02.054843 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:02.054816 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s4jjw\"" Jul 09 19:03:02.062326 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:02.062303 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vtc8p" Jul 09 19:03:04.293912 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:04.293874 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vtc8p"] Jul 09 19:03:04.296465 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:03:04.296443 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30cf5feb_cf9a_40c0_8a6a_de8fc6c60506.slice/crio-2e23cc7e254e75e202755a3cf6a022b4971c82bc9be5130c4dcd2fb37e191eed WatchSource:0}: Error finding container 2e23cc7e254e75e202755a3cf6a022b4971c82bc9be5130c4dcd2fb37e191eed: Status 404 returned error can't find the container with id 2e23cc7e254e75e202755a3cf6a022b4971c82bc9be5130c4dcd2fb37e191eed Jul 09 19:03:04.989337 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:04.989302 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8zh4c" event={"ID":"e057c3f1-8832-4c08-b9c2-7e1353a8a82c","Type":"ContainerStarted","Data":"ff81df23127664cdd69bc741f97d83fbd7c4a5578e4e39dce1a220e063ba2c18"} Jul 09 19:03:04.990645 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:04.990600 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vtc8p" event={"ID":"30cf5feb-cf9a-40c0-8a6a-de8fc6c60506","Type":"ContainerStarted","Data":"2e23cc7e254e75e202755a3cf6a022b4971c82bc9be5130c4dcd2fb37e191eed"} Jul 09 19:03:05.996951 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:05.996915 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8zh4c" event={"ID":"e057c3f1-8832-4c08-b9c2-7e1353a8a82c","Type":"ContainerStarted","Data":"8f5684b2a4d4f7008c4990ddc50812b5af526c4fe0150cb2a98eb86cb3c3f319"} Jul 09 19:03:07.002543 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:07.002502 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vtc8p" event={"ID":"30cf5feb-cf9a-40c0-8a6a-de8fc6c60506","Type":"ContainerStarted","Data":"7a9cb38737e98c4e12356bedb34ccdad091130c1b7545ee0349c203a2e6bea4d"} Jul 09 19:03:07.003020 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:07.002568 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vtc8p" event={"ID":"30cf5feb-cf9a-40c0-8a6a-de8fc6c60506","Type":"ContainerStarted","Data":"c332cca183c2b47a019cffdaac698f4dd46dfeeccc93f9c5fb691b488763de3c"} Jul 09 19:03:07.020598 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:07.020541 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vtc8p" podStartSLOduration=254.046384264 podStartE2EDuration="4m16.020521962s" totalImagesPullingTime="1.974137698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:58:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:03:07.018120354 +0000 UTC m=+256.525900128" watchObservedRunningTime="2026-07-09 19:03:07.020521962 +0000 UTC m=+256.528301736" Jul 09 19:03:08.008138 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:08.008103 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8zh4c" event={"ID":"e057c3f1-8832-4c08-b9c2-7e1353a8a82c","Type":"ContainerStarted","Data":"18d4209fe8e4ba4daaf1f7a3894e3850b831e643a5eec9a584b26524a378bf2d"} Jul 09 19:03:08.027433 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:08.027391 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8zh4c" podStartSLOduration=16.767965302 podStartE2EDuration="24.027376436s" totalImagesPullingTime="7.259411134s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 19:02:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:03:08.026028089 +0000 UTC m=+257.533807877" watchObservedRunningTime="2026-07-09 19:03:08.027376436 +0000 UTC m=+257.535156209" Jul 09 19:03:08.710721 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:08.710690 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-74dbl"] Jul 09 19:03:08.713833 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:08.713817 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-74dbl" Jul 09 19:03:08.716523 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:08.716500 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 09 19:03:08.716644 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:08.716503 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-g9r9m\"" Jul 09 19:03:08.716844 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:08.716829 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 09 19:03:08.724132 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:08.724109 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-74dbl"] Jul 09 19:03:08.730467 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:08.730448 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-66b97f775b-6x2dl"] Jul 09 19:03:08.771348 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:08.771319 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-c5759b49-2l8kr"] Jul 09 19:03:08.774301 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:08.774284 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c5759b49-2l8kr" Jul 09 19:03:08.790269 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:08.790244 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c5759b49-2l8kr"] Jul 09 19:03:08.805820 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:08.805770 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6jrf\" (UniqueName: \"kubernetes.io/projected/acc4f579-a12a-488f-b17a-8503ac4be7b4-kube-api-access-h6jrf\") pod \"downloads-68c9cbb497-74dbl\" (UID: \"acc4f579-a12a-488f-b17a-8503ac4be7b4\") " pod="openshift-console/downloads-68c9cbb497-74dbl" Jul 09 19:03:08.906350 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:08.906318 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4cb1ea83-2644-457c-b6dd-e730c35effe1-trusted-ca\") pod \"image-registry-c5759b49-2l8kr\" (UID: \"4cb1ea83-2644-457c-b6dd-e730c35effe1\") " pod="openshift-image-registry/image-registry-c5759b49-2l8kr" Jul 09 19:03:08.906541 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:08.906367 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4cb1ea83-2644-457c-b6dd-e730c35effe1-ca-trust-extracted\") pod \"image-registry-c5759b49-2l8kr\" (UID: \"4cb1ea83-2644-457c-b6dd-e730c35effe1\") " pod="openshift-image-registry/image-registry-c5759b49-2l8kr" Jul 09 19:03:08.906541 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:08.906433 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4cb1ea83-2644-457c-b6dd-e730c35effe1-image-registry-private-configuration\") pod \"image-registry-c5759b49-2l8kr\" (UID: \"4cb1ea83-2644-457c-b6dd-e730c35effe1\") " pod="openshift-image-registry/image-registry-c5759b49-2l8kr" Jul 09 19:03:08.906541 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:08.906483 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4cb1ea83-2644-457c-b6dd-e730c35effe1-registry-tls\") pod \"image-registry-c5759b49-2l8kr\" (UID: \"4cb1ea83-2644-457c-b6dd-e730c35effe1\") " pod="openshift-image-registry/image-registry-c5759b49-2l8kr" Jul 09 19:03:08.906541 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:08.906511 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4cb1ea83-2644-457c-b6dd-e730c35effe1-bound-sa-token\") pod \"image-registry-c5759b49-2l8kr\" (UID: \"4cb1ea83-2644-457c-b6dd-e730c35effe1\") " pod="openshift-image-registry/image-registry-c5759b49-2l8kr" Jul 09 19:03:08.906759 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:08.906561 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4cb1ea83-2644-457c-b6dd-e730c35effe1-installation-pull-secrets\") pod \"image-registry-c5759b49-2l8kr\" (UID: \"4cb1ea83-2644-457c-b6dd-e730c35effe1\") " pod="openshift-image-registry/image-registry-c5759b49-2l8kr" Jul 09 19:03:08.906759 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:08.906583 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4cb1ea83-2644-457c-b6dd-e730c35effe1-registry-certificates\") pod \"image-registry-c5759b49-2l8kr\" (UID: \"4cb1ea83-2644-457c-b6dd-e730c35effe1\") " pod="openshift-image-registry/image-registry-c5759b49-2l8kr" Jul 09 19:03:08.906759 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:08.906625 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8gcw\" (UniqueName: \"kubernetes.io/projected/4cb1ea83-2644-457c-b6dd-e730c35effe1-kube-api-access-d8gcw\") pod \"image-registry-c5759b49-2l8kr\" (UID: \"4cb1ea83-2644-457c-b6dd-e730c35effe1\") " pod="openshift-image-registry/image-registry-c5759b49-2l8kr" Jul 09 19:03:08.906759 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:08.906684 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6jrf\" (UniqueName: \"kubernetes.io/projected/acc4f579-a12a-488f-b17a-8503ac4be7b4-kube-api-access-h6jrf\") pod \"downloads-68c9cbb497-74dbl\" (UID: \"acc4f579-a12a-488f-b17a-8503ac4be7b4\") " pod="openshift-console/downloads-68c9cbb497-74dbl" Jul 09 19:03:08.915176 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:08.915157 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6jrf\" (UniqueName: \"kubernetes.io/projected/acc4f579-a12a-488f-b17a-8503ac4be7b4-kube-api-access-h6jrf\") pod \"downloads-68c9cbb497-74dbl\" (UID: \"acc4f579-a12a-488f-b17a-8503ac4be7b4\") " pod="openshift-console/downloads-68c9cbb497-74dbl" Jul 09 19:03:09.008058 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:09.007981 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4cb1ea83-2644-457c-b6dd-e730c35effe1-installation-pull-secrets\") pod \"image-registry-c5759b49-2l8kr\" (UID: \"4cb1ea83-2644-457c-b6dd-e730c35effe1\") " pod="openshift-image-registry/image-registry-c5759b49-2l8kr" Jul 09 19:03:09.008058 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:09.008013 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4cb1ea83-2644-457c-b6dd-e730c35effe1-registry-certificates\") pod \"image-registry-c5759b49-2l8kr\" (UID: \"4cb1ea83-2644-457c-b6dd-e730c35effe1\") " pod="openshift-image-registry/image-registry-c5759b49-2l8kr" Jul 09 19:03:09.008058 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:09.008032 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8gcw\" (UniqueName: \"kubernetes.io/projected/4cb1ea83-2644-457c-b6dd-e730c35effe1-kube-api-access-d8gcw\") pod \"image-registry-c5759b49-2l8kr\" (UID: \"4cb1ea83-2644-457c-b6dd-e730c35effe1\") " pod="openshift-image-registry/image-registry-c5759b49-2l8kr" Jul 09 19:03:09.008058 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:09.008058 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4cb1ea83-2644-457c-b6dd-e730c35effe1-trusted-ca\") pod \"image-registry-c5759b49-2l8kr\" (UID: \"4cb1ea83-2644-457c-b6dd-e730c35effe1\") " pod="openshift-image-registry/image-registry-c5759b49-2l8kr" Jul 09 19:03:09.008731 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:09.008092 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4cb1ea83-2644-457c-b6dd-e730c35effe1-ca-trust-extracted\") pod \"image-registry-c5759b49-2l8kr\" (UID: \"4cb1ea83-2644-457c-b6dd-e730c35effe1\") " pod="openshift-image-registry/image-registry-c5759b49-2l8kr" Jul 09 19:03:09.008731 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:09.008238 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4cb1ea83-2644-457c-b6dd-e730c35effe1-image-registry-private-configuration\") pod \"image-registry-c5759b49-2l8kr\" (UID: \"4cb1ea83-2644-457c-b6dd-e730c35effe1\") " pod="openshift-image-registry/image-registry-c5759b49-2l8kr" Jul 09 19:03:09.008731 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:09.008293 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4cb1ea83-2644-457c-b6dd-e730c35effe1-registry-tls\") pod \"image-registry-c5759b49-2l8kr\" (UID: \"4cb1ea83-2644-457c-b6dd-e730c35effe1\") " pod="openshift-image-registry/image-registry-c5759b49-2l8kr" Jul 09 19:03:09.008731 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:09.008333 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4cb1ea83-2644-457c-b6dd-e730c35effe1-bound-sa-token\") pod \"image-registry-c5759b49-2l8kr\" (UID: \"4cb1ea83-2644-457c-b6dd-e730c35effe1\") " pod="openshift-image-registry/image-registry-c5759b49-2l8kr" Jul 09 19:03:09.008731 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:09.008639 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4cb1ea83-2644-457c-b6dd-e730c35effe1-ca-trust-extracted\") pod \"image-registry-c5759b49-2l8kr\" (UID: \"4cb1ea83-2644-457c-b6dd-e730c35effe1\") " pod="openshift-image-registry/image-registry-c5759b49-2l8kr" Jul 09 19:03:09.009117 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:09.009088 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4cb1ea83-2644-457c-b6dd-e730c35effe1-registry-certificates\") pod \"image-registry-c5759b49-2l8kr\" (UID: \"4cb1ea83-2644-457c-b6dd-e730c35effe1\") " pod="openshift-image-registry/image-registry-c5759b49-2l8kr" Jul 09 19:03:09.009255 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:09.009216 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4cb1ea83-2644-457c-b6dd-e730c35effe1-trusted-ca\") pod \"image-registry-c5759b49-2l8kr\" (UID: \"4cb1ea83-2644-457c-b6dd-e730c35effe1\") " pod="openshift-image-registry/image-registry-c5759b49-2l8kr" Jul 09 19:03:09.011044 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:09.011015 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4cb1ea83-2644-457c-b6dd-e730c35effe1-image-registry-private-configuration\") pod \"image-registry-c5759b49-2l8kr\" (UID: \"4cb1ea83-2644-457c-b6dd-e730c35effe1\") " pod="openshift-image-registry/image-registry-c5759b49-2l8kr" Jul 09 19:03:09.011143 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:09.011069 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4cb1ea83-2644-457c-b6dd-e730c35effe1-registry-tls\") pod \"image-registry-c5759b49-2l8kr\" (UID: \"4cb1ea83-2644-457c-b6dd-e730c35effe1\") " pod="openshift-image-registry/image-registry-c5759b49-2l8kr" Jul 09 19:03:09.011204 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:09.011186 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4cb1ea83-2644-457c-b6dd-e730c35effe1-installation-pull-secrets\") pod \"image-registry-c5759b49-2l8kr\" (UID: \"4cb1ea83-2644-457c-b6dd-e730c35effe1\") " pod="openshift-image-registry/image-registry-c5759b49-2l8kr" Jul 09 19:03:09.017044 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:09.016984 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4cb1ea83-2644-457c-b6dd-e730c35effe1-bound-sa-token\") pod \"image-registry-c5759b49-2l8kr\" (UID: \"4cb1ea83-2644-457c-b6dd-e730c35effe1\") " pod="openshift-image-registry/image-registry-c5759b49-2l8kr" Jul 09 19:03:09.017377 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:09.017360 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8gcw\" (UniqueName: \"kubernetes.io/projected/4cb1ea83-2644-457c-b6dd-e730c35effe1-kube-api-access-d8gcw\") pod \"image-registry-c5759b49-2l8kr\" (UID: \"4cb1ea83-2644-457c-b6dd-e730c35effe1\") " pod="openshift-image-registry/image-registry-c5759b49-2l8kr" Jul 09 19:03:09.022107 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:09.022073 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-74dbl" Jul 09 19:03:09.083027 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:09.082996 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c5759b49-2l8kr" Jul 09 19:03:09.154586 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:09.154564 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-74dbl"] Jul 09 19:03:09.157259 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:03:09.157231 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacc4f579_a12a_488f_b17a_8503ac4be7b4.slice/crio-cf51ecded73f5dfa5744d06f85d07f7c818e253695700609a6299723adfaef02 WatchSource:0}: Error finding container cf51ecded73f5dfa5744d06f85d07f7c818e253695700609a6299723adfaef02: Status 404 returned error can't find the container with id cf51ecded73f5dfa5744d06f85d07f7c818e253695700609a6299723adfaef02 Jul 09 19:03:09.218930 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:09.216894 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c5759b49-2l8kr"] Jul 09 19:03:09.222681 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:03:09.222641 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cb1ea83_2644_457c_b6dd_e730c35effe1.slice/crio-135890a867320c394aedad3c7e5aa1c93f47a87b362d0f33bb1d5f05b70acbea WatchSource:0}: Error finding container 135890a867320c394aedad3c7e5aa1c93f47a87b362d0f33bb1d5f05b70acbea: Status 404 returned error can't find the container with id 135890a867320c394aedad3c7e5aa1c93f47a87b362d0f33bb1d5f05b70acbea Jul 09 19:03:10.017431 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:10.017395 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-74dbl" event={"ID":"acc4f579-a12a-488f-b17a-8503ac4be7b4","Type":"ContainerStarted","Data":"cf51ecded73f5dfa5744d06f85d07f7c818e253695700609a6299723adfaef02"} Jul 09 19:03:10.019074 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:10.019043 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c5759b49-2l8kr" event={"ID":"4cb1ea83-2644-457c-b6dd-e730c35effe1","Type":"ContainerStarted","Data":"39cdaf2c3ba0412fdd4d031b2fa0eb5d0483765eae2faa84500e854881c106e4"} Jul 09 19:03:10.019201 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:10.019090 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c5759b49-2l8kr" event={"ID":"4cb1ea83-2644-457c-b6dd-e730c35effe1","Type":"ContainerStarted","Data":"135890a867320c394aedad3c7e5aa1c93f47a87b362d0f33bb1d5f05b70acbea"} Jul 09 19:03:10.019201 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:10.019160 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-c5759b49-2l8kr" Jul 09 19:03:10.040492 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:10.040450 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-c5759b49-2l8kr" podStartSLOduration=2.040426795 podStartE2EDuration="2.040426795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:03:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:03:10.038837263 +0000 UTC m=+259.546617052" watchObservedRunningTime="2026-07-09 19:03:10.040426795 +0000 UTC m=+259.548206568" Jul 09 19:03:18.736550 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:18.736468 2566 patch_prober.go:28] interesting pod/image-registry-66b97f775b-6x2dl container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 09 19:03:18.736550 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:18.736526 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-66b97f775b-6x2dl" podUID="31207519-aadd-4eb5-8cef-af759079efb8" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 19:03:22.915477 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:22.915439 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-wnllf"] Jul 09 19:03:22.947787 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:22.947746 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-wnllf"] Jul 09 19:03:22.947958 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:22.947802 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-cz67s"] Jul 09 19:03:22.947958 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:22.947889 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wnllf" Jul 09 19:03:22.950741 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:22.950709 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-bb2mk\"" Jul 09 19:03:22.952014 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:22.951871 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 09 19:03:22.952014 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:22.951881 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 09 19:03:22.952014 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:22.951970 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 09 19:03:22.952014 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:22.951870 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 09 19:03:22.952304 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:22.951969 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 09 19:03:22.961807 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:22.961776 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-kvwlg"] Jul 09 19:03:22.961947 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:22.961909 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-cz67s" Jul 09 19:03:22.965036 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:22.964865 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 09 19:03:22.965036 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:22.964886 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 09 19:03:22.965036 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:22.964903 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 09 19:03:22.965036 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:22.964931 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-b86pt\"" Jul 09 19:03:22.980016 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:22.979907 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-cz67s"] Jul 09 19:03:22.980152 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:22.980040 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kvwlg" Jul 09 19:03:22.983018 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:22.982996 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 09 19:03:22.983123 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:22.983097 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 09 19:03:22.983197 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:22.983154 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 09 19:03:22.983197 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:22.983159 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fv76w\"" Jul 09 19:03:23.017275 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.017244 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/40fbb88a-0919-45cb-88e9-53a6f4177875-node-exporter-wtmp\") pod \"node-exporter-kvwlg\" (UID: \"40fbb88a-0919-45cb-88e9-53a6f4177875\") " pod="openshift-monitoring/node-exporter-kvwlg" Jul 09 19:03:23.017275 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.017280 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/40fbb88a-0919-45cb-88e9-53a6f4177875-root\") pod \"node-exporter-kvwlg\" (UID: \"40fbb88a-0919-45cb-88e9-53a6f4177875\") " pod="openshift-monitoring/node-exporter-kvwlg" Jul 09 19:03:23.017506 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.017303 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/40fbb88a-0919-45cb-88e9-53a6f4177875-node-exporter-textfile\") pod \"node-exporter-kvwlg\" (UID: \"40fbb88a-0919-45cb-88e9-53a6f4177875\") " pod="openshift-monitoring/node-exporter-kvwlg" Jul 09 19:03:23.017506 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.017355 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6402e20a-0bd8-4e91-bf65-e5de9a1a6f50-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-wnllf\" (UID: \"6402e20a-0bd8-4e91-bf65-e5de9a1a6f50\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wnllf" Jul 09 19:03:23.017506 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.017378 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9cb3de9f-2c0d-4e83-a0a1-fb96c9fb5281-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-cz67s\" (UID: \"9cb3de9f-2c0d-4e83-a0a1-fb96c9fb5281\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-cz67s" Jul 09 19:03:23.017506 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.017402 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6402e20a-0bd8-4e91-bf65-e5de9a1a6f50-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-wnllf\" (UID: \"6402e20a-0bd8-4e91-bf65-e5de9a1a6f50\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wnllf" Jul 09 19:03:23.017506 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.017426 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/40fbb88a-0919-45cb-88e9-53a6f4177875-node-exporter-accelerators-collector-config\") pod \"node-exporter-kvwlg\" (UID: \"40fbb88a-0919-45cb-88e9-53a6f4177875\") " pod="openshift-monitoring/node-exporter-kvwlg" Jul 09 19:03:23.017506 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.017450 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n6pp\" (UniqueName: \"kubernetes.io/projected/6402e20a-0bd8-4e91-bf65-e5de9a1a6f50-kube-api-access-6n6pp\") pod \"openshift-state-metrics-84bd68b896-wnllf\" (UID: \"6402e20a-0bd8-4e91-bf65-e5de9a1a6f50\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wnllf" Jul 09 19:03:23.017506 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.017466 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9cb3de9f-2c0d-4e83-a0a1-fb96c9fb5281-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-cz67s\" (UID: \"9cb3de9f-2c0d-4e83-a0a1-fb96c9fb5281\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-cz67s" Jul 09 19:03:23.017506 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.017483 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9cb3de9f-2c0d-4e83-a0a1-fb96c9fb5281-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-cz67s\" (UID: \"9cb3de9f-2c0d-4e83-a0a1-fb96c9fb5281\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-cz67s" Jul 09 19:03:23.017936 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.017562 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6402e20a-0bd8-4e91-bf65-e5de9a1a6f50-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-wnllf\" (UID: \"6402e20a-0bd8-4e91-bf65-e5de9a1a6f50\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wnllf" Jul 09 19:03:23.017936 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.017599 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/40fbb88a-0919-45cb-88e9-53a6f4177875-node-exporter-tls\") pod \"node-exporter-kvwlg\" (UID: \"40fbb88a-0919-45cb-88e9-53a6f4177875\") " pod="openshift-monitoring/node-exporter-kvwlg" Jul 09 19:03:23.017936 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.017665 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9cb3de9f-2c0d-4e83-a0a1-fb96c9fb5281-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-cz67s\" (UID: \"9cb3de9f-2c0d-4e83-a0a1-fb96c9fb5281\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-cz67s" Jul 09 19:03:23.017936 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.017705 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9cb3de9f-2c0d-4e83-a0a1-fb96c9fb5281-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-cz67s\" (UID: \"9cb3de9f-2c0d-4e83-a0a1-fb96c9fb5281\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-cz67s" Jul 09 19:03:23.017936 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.017737 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd6fc\" (UniqueName: \"kubernetes.io/projected/9cb3de9f-2c0d-4e83-a0a1-fb96c9fb5281-kube-api-access-vd6fc\") pod \"kube-state-metrics-785778ff74-cz67s\" (UID: \"9cb3de9f-2c0d-4e83-a0a1-fb96c9fb5281\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-cz67s" Jul 09 19:03:23.017936 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.017774 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/40fbb88a-0919-45cb-88e9-53a6f4177875-sys\") pod \"node-exporter-kvwlg\" (UID: \"40fbb88a-0919-45cb-88e9-53a6f4177875\") " pod="openshift-monitoring/node-exporter-kvwlg" Jul 09 19:03:23.017936 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.017810 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40fbb88a-0919-45cb-88e9-53a6f4177875-metrics-client-ca\") pod \"node-exporter-kvwlg\" (UID: \"40fbb88a-0919-45cb-88e9-53a6f4177875\") " pod="openshift-monitoring/node-exporter-kvwlg" Jul 09 19:03:23.017936 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.017836 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/40fbb88a-0919-45cb-88e9-53a6f4177875-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kvwlg\" (UID: \"40fbb88a-0919-45cb-88e9-53a6f4177875\") " pod="openshift-monitoring/node-exporter-kvwlg" Jul 09 19:03:23.017936 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.017863 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj5w4\" (UniqueName: \"kubernetes.io/projected/40fbb88a-0919-45cb-88e9-53a6f4177875-kube-api-access-hj5w4\") pod \"node-exporter-kvwlg\" (UID: \"40fbb88a-0919-45cb-88e9-53a6f4177875\") " pod="openshift-monitoring/node-exporter-kvwlg" Jul 09 19:03:23.118932 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.118894 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n6pp\" (UniqueName: \"kubernetes.io/projected/6402e20a-0bd8-4e91-bf65-e5de9a1a6f50-kube-api-access-6n6pp\") pod \"openshift-state-metrics-84bd68b896-wnllf\" (UID: \"6402e20a-0bd8-4e91-bf65-e5de9a1a6f50\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wnllf" Jul 09 19:03:23.118932 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.118934 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9cb3de9f-2c0d-4e83-a0a1-fb96c9fb5281-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-cz67s\" (UID: \"9cb3de9f-2c0d-4e83-a0a1-fb96c9fb5281\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-cz67s" Jul 09 19:03:23.119182 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.118959 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9cb3de9f-2c0d-4e83-a0a1-fb96c9fb5281-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-cz67s\" (UID: \"9cb3de9f-2c0d-4e83-a0a1-fb96c9fb5281\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-cz67s" Jul 09 19:03:23.119182 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.118993 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6402e20a-0bd8-4e91-bf65-e5de9a1a6f50-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-wnllf\" (UID: \"6402e20a-0bd8-4e91-bf65-e5de9a1a6f50\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wnllf" Jul 09 19:03:23.119182 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.119015 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/40fbb88a-0919-45cb-88e9-53a6f4177875-node-exporter-tls\") pod \"node-exporter-kvwlg\" (UID: \"40fbb88a-0919-45cb-88e9-53a6f4177875\") " pod="openshift-monitoring/node-exporter-kvwlg" Jul 09 19:03:23.119182 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.119058 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9cb3de9f-2c0d-4e83-a0a1-fb96c9fb5281-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-cz67s\" (UID: \"9cb3de9f-2c0d-4e83-a0a1-fb96c9fb5281\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-cz67s" Jul 09 19:03:23.119182 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.119091 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9cb3de9f-2c0d-4e83-a0a1-fb96c9fb5281-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-cz67s\" (UID: \"9cb3de9f-2c0d-4e83-a0a1-fb96c9fb5281\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-cz67s" Jul 09 19:03:23.119182 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.119121 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd6fc\" (UniqueName: \"kubernetes.io/projected/9cb3de9f-2c0d-4e83-a0a1-fb96c9fb5281-kube-api-access-vd6fc\") pod \"kube-state-metrics-785778ff74-cz67s\" (UID: \"9cb3de9f-2c0d-4e83-a0a1-fb96c9fb5281\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-cz67s" Jul 09 19:03:23.119182 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.119148 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/40fbb88a-0919-45cb-88e9-53a6f4177875-sys\") pod \"node-exporter-kvwlg\" (UID: \"40fbb88a-0919-45cb-88e9-53a6f4177875\") " pod="openshift-monitoring/node-exporter-kvwlg" Jul 09 19:03:23.119182 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.119177 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40fbb88a-0919-45cb-88e9-53a6f4177875-metrics-client-ca\") pod \"node-exporter-kvwlg\" (UID: \"40fbb88a-0919-45cb-88e9-53a6f4177875\") " pod="openshift-monitoring/node-exporter-kvwlg" Jul 09 19:03:23.119554 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.119201 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/40fbb88a-0919-45cb-88e9-53a6f4177875-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kvwlg\" (UID: \"40fbb88a-0919-45cb-88e9-53a6f4177875\") " pod="openshift-monitoring/node-exporter-kvwlg" Jul 09 19:03:23.119554 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.119223 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj5w4\" (UniqueName: \"kubernetes.io/projected/40fbb88a-0919-45cb-88e9-53a6f4177875-kube-api-access-hj5w4\") pod \"node-exporter-kvwlg\" (UID: \"40fbb88a-0919-45cb-88e9-53a6f4177875\") " pod="openshift-monitoring/node-exporter-kvwlg" Jul 09 19:03:23.119554 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.119255 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/40fbb88a-0919-45cb-88e9-53a6f4177875-node-exporter-wtmp\") pod \"node-exporter-kvwlg\" (UID: \"40fbb88a-0919-45cb-88e9-53a6f4177875\") " pod="openshift-monitoring/node-exporter-kvwlg" Jul 09 19:03:23.119554 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.119281 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/40fbb88a-0919-45cb-88e9-53a6f4177875-root\") pod \"node-exporter-kvwlg\" (UID: \"40fbb88a-0919-45cb-88e9-53a6f4177875\") " pod="openshift-monitoring/node-exporter-kvwlg" Jul 09 19:03:23.119554 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.119306 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/40fbb88a-0919-45cb-88e9-53a6f4177875-node-exporter-textfile\") pod \"node-exporter-kvwlg\" (UID: \"40fbb88a-0919-45cb-88e9-53a6f4177875\") " pod="openshift-monitoring/node-exporter-kvwlg" Jul 09 19:03:23.119554 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.119340 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6402e20a-0bd8-4e91-bf65-e5de9a1a6f50-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-wnllf\" (UID: \"6402e20a-0bd8-4e91-bf65-e5de9a1a6f50\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wnllf" Jul 09 19:03:23.119554 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.119361 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9cb3de9f-2c0d-4e83-a0a1-fb96c9fb5281-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-cz67s\" (UID: \"9cb3de9f-2c0d-4e83-a0a1-fb96c9fb5281\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-cz67s" Jul 09 19:03:23.119554 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.119371 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9cb3de9f-2c0d-4e83-a0a1-fb96c9fb5281-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-cz67s\" (UID: \"9cb3de9f-2c0d-4e83-a0a1-fb96c9fb5281\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-cz67s" Jul 09 19:03:23.119554 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.119398 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6402e20a-0bd8-4e91-bf65-e5de9a1a6f50-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-wnllf\" (UID: \"6402e20a-0bd8-4e91-bf65-e5de9a1a6f50\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wnllf" Jul 09 19:03:23.119554 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.119426 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/40fbb88a-0919-45cb-88e9-53a6f4177875-node-exporter-accelerators-collector-config\") pod \"node-exporter-kvwlg\" (UID: \"40fbb88a-0919-45cb-88e9-53a6f4177875\") " pod="openshift-monitoring/node-exporter-kvwlg" Jul 09 19:03:23.120061 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.119909 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/40fbb88a-0919-45cb-88e9-53a6f4177875-sys\") pod \"node-exporter-kvwlg\" (UID: \"40fbb88a-0919-45cb-88e9-53a6f4177875\") " pod="openshift-monitoring/node-exporter-kvwlg" Jul 09 19:03:23.120061 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.119954 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9cb3de9f-2c0d-4e83-a0a1-fb96c9fb5281-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-cz67s\" (UID: \"9cb3de9f-2c0d-4e83-a0a1-fb96c9fb5281\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-cz67s" Jul 09 19:03:23.120061 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.120018 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/40fbb88a-0919-45cb-88e9-53a6f4177875-node-exporter-accelerators-collector-config\") pod \"node-exporter-kvwlg\" (UID: \"40fbb88a-0919-45cb-88e9-53a6f4177875\") " pod="openshift-monitoring/node-exporter-kvwlg" Jul 09 19:03:23.120216 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.120148 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/40fbb88a-0919-45cb-88e9-53a6f4177875-node-exporter-wtmp\") pod \"node-exporter-kvwlg\" (UID: \"40fbb88a-0919-45cb-88e9-53a6f4177875\") " pod="openshift-monitoring/node-exporter-kvwlg" Jul 09 19:03:23.120216 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.120193 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/40fbb88a-0919-45cb-88e9-53a6f4177875-root\") pod \"node-exporter-kvwlg\" (UID: \"40fbb88a-0919-45cb-88e9-53a6f4177875\") " pod="openshift-monitoring/node-exporter-kvwlg" Jul 09 19:03:23.120983 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.120428 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/40fbb88a-0919-45cb-88e9-53a6f4177875-node-exporter-textfile\") pod \"node-exporter-kvwlg\" (UID: \"40fbb88a-0919-45cb-88e9-53a6f4177875\") " pod="openshift-monitoring/node-exporter-kvwlg" Jul 09 19:03:23.120983 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.120477 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40fbb88a-0919-45cb-88e9-53a6f4177875-metrics-client-ca\") pod \"node-exporter-kvwlg\" (UID: \"40fbb88a-0919-45cb-88e9-53a6f4177875\") " pod="openshift-monitoring/node-exporter-kvwlg" Jul 09 19:03:23.120983 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:03:23.120641 2566 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 09 19:03:23.120983 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:03:23.120707 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6402e20a-0bd8-4e91-bf65-e5de9a1a6f50-openshift-state-metrics-tls podName:6402e20a-0bd8-4e91-bf65-e5de9a1a6f50 nodeName:}" failed. No retries permitted until 2026-07-09 19:03:23.620687924 +0000 UTC m=+273.128467678 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/6402e20a-0bd8-4e91-bf65-e5de9a1a6f50-openshift-state-metrics-tls") pod "openshift-state-metrics-84bd68b896-wnllf" (UID: "6402e20a-0bd8-4e91-bf65-e5de9a1a6f50") : secret "openshift-state-metrics-tls" not found Jul 09 19:03:23.121331 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.121111 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9cb3de9f-2c0d-4e83-a0a1-fb96c9fb5281-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-cz67s\" (UID: \"9cb3de9f-2c0d-4e83-a0a1-fb96c9fb5281\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-cz67s" Jul 09 19:03:23.121331 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.121153 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6402e20a-0bd8-4e91-bf65-e5de9a1a6f50-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-wnllf\" (UID: \"6402e20a-0bd8-4e91-bf65-e5de9a1a6f50\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wnllf" Jul 09 19:03:23.123064 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.123026 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/40fbb88a-0919-45cb-88e9-53a6f4177875-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kvwlg\" (UID: \"40fbb88a-0919-45cb-88e9-53a6f4177875\") " pod="openshift-monitoring/node-exporter-kvwlg" Jul 09 19:03:23.123668 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.123643 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9cb3de9f-2c0d-4e83-a0a1-fb96c9fb5281-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-cz67s\" (UID: \"9cb3de9f-2c0d-4e83-a0a1-fb96c9fb5281\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-cz67s" Jul 09 19:03:23.123873 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.123848 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6402e20a-0bd8-4e91-bf65-e5de9a1a6f50-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-wnllf\" (UID: \"6402e20a-0bd8-4e91-bf65-e5de9a1a6f50\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wnllf" Jul 09 19:03:23.123952 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.123866 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9cb3de9f-2c0d-4e83-a0a1-fb96c9fb5281-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-cz67s\" (UID: \"9cb3de9f-2c0d-4e83-a0a1-fb96c9fb5281\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-cz67s" Jul 09 19:03:23.124289 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.124271 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/40fbb88a-0919-45cb-88e9-53a6f4177875-node-exporter-tls\") pod \"node-exporter-kvwlg\" (UID: \"40fbb88a-0919-45cb-88e9-53a6f4177875\") " pod="openshift-monitoring/node-exporter-kvwlg" Jul 09 19:03:23.130318 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.130294 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd6fc\" (UniqueName: \"kubernetes.io/projected/9cb3de9f-2c0d-4e83-a0a1-fb96c9fb5281-kube-api-access-vd6fc\") pod \"kube-state-metrics-785778ff74-cz67s\" (UID: \"9cb3de9f-2c0d-4e83-a0a1-fb96c9fb5281\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-cz67s" Jul 09 19:03:23.130888 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.130862 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n6pp\" (UniqueName: \"kubernetes.io/projected/6402e20a-0bd8-4e91-bf65-e5de9a1a6f50-kube-api-access-6n6pp\") pod \"openshift-state-metrics-84bd68b896-wnllf\" (UID: \"6402e20a-0bd8-4e91-bf65-e5de9a1a6f50\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wnllf" Jul 09 19:03:23.131023 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.131000 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj5w4\" (UniqueName: \"kubernetes.io/projected/40fbb88a-0919-45cb-88e9-53a6f4177875-kube-api-access-hj5w4\") pod \"node-exporter-kvwlg\" (UID: \"40fbb88a-0919-45cb-88e9-53a6f4177875\") " pod="openshift-monitoring/node-exporter-kvwlg" Jul 09 19:03:23.271922 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.271889 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-cz67s" Jul 09 19:03:23.289702 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.289674 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kvwlg" Jul 09 19:03:23.623726 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.623646 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6402e20a-0bd8-4e91-bf65-e5de9a1a6f50-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-wnllf\" (UID: \"6402e20a-0bd8-4e91-bf65-e5de9a1a6f50\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wnllf" Jul 09 19:03:23.626296 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.626272 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6402e20a-0bd8-4e91-bf65-e5de9a1a6f50-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-wnllf\" (UID: \"6402e20a-0bd8-4e91-bf65-e5de9a1a6f50\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wnllf" Jul 09 19:03:23.860614 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:23.860557 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wnllf" Jul 09 19:03:27.704700 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:27.704659 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-xtr7d"] Jul 09 19:03:27.707696 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:27.707667 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-xtr7d" Jul 09 19:03:27.710454 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:27.710430 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 09 19:03:27.710649 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:27.710629 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-6fs5n\"" Jul 09 19:03:27.716878 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:27.716818 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-xtr7d"] Jul 09 19:03:27.752622 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:03:27.752572 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40fbb88a_0919_45cb_88e9_53a6f4177875.slice/crio-9ec8fe2e52b76b8285581601c62a1f73a46d6a75743b2a74dd4a3c6149dcaa4a WatchSource:0}: Error finding container 9ec8fe2e52b76b8285581601c62a1f73a46d6a75743b2a74dd4a3c6149dcaa4a: Status 404 returned error can't find the container with id 9ec8fe2e52b76b8285581601c62a1f73a46d6a75743b2a74dd4a3c6149dcaa4a Jul 09 19:03:27.761169 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:27.760974 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/13d71f7d-dc42-44c6-98be-d8ed4d2173c9-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-xtr7d\" (UID: \"13d71f7d-dc42-44c6-98be-d8ed4d2173c9\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-xtr7d" Jul 09 19:03:27.761169 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:27.761031 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdckv\" (UniqueName: \"kubernetes.io/projected/13d71f7d-dc42-44c6-98be-d8ed4d2173c9-kube-api-access-kdckv\") pod \"monitoring-plugin-5f68df7f58-xtr7d\" (UID: \"13d71f7d-dc42-44c6-98be-d8ed4d2173c9\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-xtr7d" Jul 09 19:03:27.862279 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:27.861864 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/13d71f7d-dc42-44c6-98be-d8ed4d2173c9-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-xtr7d\" (UID: \"13d71f7d-dc42-44c6-98be-d8ed4d2173c9\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-xtr7d" Jul 09 19:03:27.862279 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:27.861913 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdckv\" (UniqueName: \"kubernetes.io/projected/13d71f7d-dc42-44c6-98be-d8ed4d2173c9-kube-api-access-kdckv\") pod \"monitoring-plugin-5f68df7f58-xtr7d\" (UID: \"13d71f7d-dc42-44c6-98be-d8ed4d2173c9\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-xtr7d" Jul 09 19:03:27.862279 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:03:27.862109 2566 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 09 19:03:27.862279 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:03:27.862178 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13d71f7d-dc42-44c6-98be-d8ed4d2173c9-monitoring-plugin-cert podName:13d71f7d-dc42-44c6-98be-d8ed4d2173c9 nodeName:}" failed. No retries permitted until 2026-07-09 19:03:28.362157764 +0000 UTC m=+277.869937522 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/13d71f7d-dc42-44c6-98be-d8ed4d2173c9-monitoring-plugin-cert") pod "monitoring-plugin-5f68df7f58-xtr7d" (UID: "13d71f7d-dc42-44c6-98be-d8ed4d2173c9") : secret "monitoring-plugin-cert" not found Jul 09 19:03:27.873560 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:27.873532 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdckv\" (UniqueName: \"kubernetes.io/projected/13d71f7d-dc42-44c6-98be-d8ed4d2173c9-kube-api-access-kdckv\") pod \"monitoring-plugin-5f68df7f58-xtr7d\" (UID: \"13d71f7d-dc42-44c6-98be-d8ed4d2173c9\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-xtr7d" Jul 09 19:03:27.893104 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:27.893081 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-cz67s"] Jul 09 19:03:27.895979 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:03:27.895948 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cb3de9f_2c0d_4e83_a0a1_fb96c9fb5281.slice/crio-27b79561cc2964c7f860e6ba03c41ac4472b7d2629c53b5e80e076ffbed648b2 WatchSource:0}: Error finding container 27b79561cc2964c7f860e6ba03c41ac4472b7d2629c53b5e80e076ffbed648b2: Status 404 returned error can't find the container with id 27b79561cc2964c7f860e6ba03c41ac4472b7d2629c53b5e80e076ffbed648b2 Jul 09 19:03:27.908382 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:27.908359 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-wnllf"] Jul 09 19:03:27.911441 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:03:27.911421 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6402e20a_0bd8_4e91_bf65_e5de9a1a6f50.slice/crio-962d5834147a0d9e1aa1821ad084162568890b42e0023d127252809b633d2176 WatchSource:0}: Error finding container 962d5834147a0d9e1aa1821ad084162568890b42e0023d127252809b633d2176: Status 404 returned error can't find the container with id 962d5834147a0d9e1aa1821ad084162568890b42e0023d127252809b633d2176 Jul 09 19:03:28.084654 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:28.084590 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-cz67s" event={"ID":"9cb3de9f-2c0d-4e83-a0a1-fb96c9fb5281","Type":"ContainerStarted","Data":"27b79561cc2964c7f860e6ba03c41ac4472b7d2629c53b5e80e076ffbed648b2"} Jul 09 19:03:28.085973 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:28.085940 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kvwlg" event={"ID":"40fbb88a-0919-45cb-88e9-53a6f4177875","Type":"ContainerStarted","Data":"9ec8fe2e52b76b8285581601c62a1f73a46d6a75743b2a74dd4a3c6149dcaa4a"} Jul 09 19:03:28.087765 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:28.087743 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wnllf" event={"ID":"6402e20a-0bd8-4e91-bf65-e5de9a1a6f50","Type":"ContainerStarted","Data":"bc62cf3ef0a17cfcbd0b2a86a11b0517b64b1c598ccd1fdc894a767183227403"} Jul 09 19:03:28.087875 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:28.087794 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wnllf" event={"ID":"6402e20a-0bd8-4e91-bf65-e5de9a1a6f50","Type":"ContainerStarted","Data":"86a69285cf9c67948094e550245c13309c0b6e35d11e0e2ebe56e2f470e14469"} Jul 09 19:03:28.087875 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:28.087823 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wnllf" event={"ID":"6402e20a-0bd8-4e91-bf65-e5de9a1a6f50","Type":"ContainerStarted","Data":"962d5834147a0d9e1aa1821ad084162568890b42e0023d127252809b633d2176"} Jul 09 19:03:28.089363 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:28.089340 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-74dbl" event={"ID":"acc4f579-a12a-488f-b17a-8503ac4be7b4","Type":"ContainerStarted","Data":"3101074de51c2f1bb2342215fd55ac763b343e9b2a086e1814c863146e6657d8"} Jul 09 19:03:28.089644 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:28.089620 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-74dbl" Jul 09 19:03:28.109270 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:28.109224 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-74dbl" podStartSLOduration=1.454763754 podStartE2EDuration="20.109209166s" totalImagesPullingTime="18.654445412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:03:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:03:28.107546944 +0000 UTC m=+277.615326720" watchObservedRunningTime="2026-07-09 19:03:28.109209166 +0000 UTC m=+277.616988939" Jul 09 19:03:28.111491 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:28.111405 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-74dbl" Jul 09 19:03:28.368400 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:28.368291 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/13d71f7d-dc42-44c6-98be-d8ed4d2173c9-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-xtr7d\" (UID: \"13d71f7d-dc42-44c6-98be-d8ed4d2173c9\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-xtr7d" Jul 09 19:03:28.371542 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:28.371449 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/13d71f7d-dc42-44c6-98be-d8ed4d2173c9-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-xtr7d\" (UID: \"13d71f7d-dc42-44c6-98be-d8ed4d2173c9\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-xtr7d" Jul 09 19:03:28.619768 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:28.619674 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-xtr7d" Jul 09 19:03:28.737427 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:28.737173 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-66b97f775b-6x2dl" Jul 09 19:03:29.089854 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:29.089823 2566 patch_prober.go:28] interesting pod/image-registry-c5759b49-2l8kr container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 09 19:03:29.090138 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:29.089879 2566 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-c5759b49-2l8kr" podUID="4cb1ea83-2644-457c-b6dd-e730c35effe1" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 19:03:29.104520 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:29.104491 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-xtr7d"] Jul 09 19:03:29.306817 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:03:29.306780 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13d71f7d_dc42_44c6_98be_d8ed4d2173c9.slice/crio-58a1db53984d6e3fae792feb7181b187a6fb8a531b61d24f046ca8a3c76e2b07 WatchSource:0}: Error finding container 58a1db53984d6e3fae792feb7181b187a6fb8a531b61d24f046ca8a3c76e2b07: Status 404 returned error can't find the container with id 58a1db53984d6e3fae792feb7181b187a6fb8a531b61d24f046ca8a3c76e2b07 Jul 09 19:03:30.108899 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:30.108533 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kvwlg" event={"ID":"40fbb88a-0919-45cb-88e9-53a6f4177875","Type":"ContainerStarted","Data":"098029e459291186684e6914c24145b61315afe3152639e046bf9627128d0c9a"} Jul 09 19:03:30.114159 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:30.114127 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-xtr7d" event={"ID":"13d71f7d-dc42-44c6-98be-d8ed4d2173c9","Type":"ContainerStarted","Data":"58a1db53984d6e3fae792feb7181b187a6fb8a531b61d24f046ca8a3c76e2b07"} Jul 09 19:03:31.027385 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:31.027353 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-c5759b49-2l8kr" Jul 09 19:03:31.124199 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:31.124150 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-cz67s" event={"ID":"9cb3de9f-2c0d-4e83-a0a1-fb96c9fb5281","Type":"ContainerStarted","Data":"90c8c538aa2f32621ca1a98e387d124c5af4417dd21027acff39bcf758c2d8e3"} Jul 09 19:03:31.124691 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:31.124222 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-cz67s" event={"ID":"9cb3de9f-2c0d-4e83-a0a1-fb96c9fb5281","Type":"ContainerStarted","Data":"69d2ffcc7d2d2f9fa3c4ee193d3fb047dd37521c66101c46c8f3520f3cb438df"} Jul 09 19:03:31.124691 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:31.124255 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-cz67s" event={"ID":"9cb3de9f-2c0d-4e83-a0a1-fb96c9fb5281","Type":"ContainerStarted","Data":"4752267c86c77243a5eabe6663942779133a2b39d9dd23ac6462a6b0f020df99"} Jul 09 19:03:31.126138 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:31.126104 2566 generic.go:356] "Generic (PLEG): container finished" podID="40fbb88a-0919-45cb-88e9-53a6f4177875" containerID="098029e459291186684e6914c24145b61315afe3152639e046bf9627128d0c9a" exitCode=0 Jul 09 19:03:31.126268 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:31.126182 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kvwlg" event={"ID":"40fbb88a-0919-45cb-88e9-53a6f4177875","Type":"ContainerDied","Data":"098029e459291186684e6914c24145b61315afe3152639e046bf9627128d0c9a"} Jul 09 19:03:31.130036 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:31.130011 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wnllf" event={"ID":"6402e20a-0bd8-4e91-bf65-e5de9a1a6f50","Type":"ContainerStarted","Data":"5d52e22def001b576d3f0efdee198f87a3d25e4beb6dc34e6b6031a7289aee6d"} Jul 09 19:03:31.165639 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:31.165566 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-cz67s" podStartSLOduration=7.090267587 podStartE2EDuration="9.165544566s" totalImagesPullingTime="2.075276979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:03:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:03:31.164387626 +0000 UTC m=+280.672167401" watchObservedRunningTime="2026-07-09 19:03:31.165544566 +0000 UTC m=+280.673324341" Jul 09 19:03:31.207504 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:31.207448 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-wnllf" podStartSLOduration=7.276260122 podStartE2EDuration="9.207431483s" totalImagesPullingTime="1.931171361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:03:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:03:31.205582212 +0000 UTC m=+280.713361988" watchObservedRunningTime="2026-07-09 19:03:31.207431483 +0000 UTC m=+280.715211257" Jul 09 19:03:32.137670 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:32.137638 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kvwlg" event={"ID":"40fbb88a-0919-45cb-88e9-53a6f4177875","Type":"ContainerStarted","Data":"6da12355ef0b0fda8959ffc536525f122459ce32bee1adc02ade70a83684b262"} Jul 09 19:03:32.138076 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:32.137702 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kvwlg" event={"ID":"40fbb88a-0919-45cb-88e9-53a6f4177875","Type":"ContainerStarted","Data":"469fedbab5f8014ba6b51176eccae3ff00fe238d244610667f46dc4bbdfc2b45"} Jul 09 19:03:32.159781 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:32.159734 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-kvwlg" podStartSLOduration=8.860377954 podStartE2EDuration="10.159716336s" totalImagesPullingTime="1.18004131s" totalInitContainerRuntime="119.297072ms" isStatefulPod=true podCreationTimestamp="2026-07-09 19:03:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:03:32.158071841 +0000 UTC m=+281.665851616" watchObservedRunningTime="2026-07-09 19:03:32.159716336 +0000 UTC m=+281.667496109" Jul 09 19:03:32.692789 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:03:32.692737 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-9xn8w" podUID="aa9f1324-a4e6-4f69-93f5-101c3aed7a0c" Jul 09 19:03:33.146584 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:33.146498 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-xtr7d" event={"ID":"13d71f7d-dc42-44c6-98be-d8ed4d2173c9","Type":"ContainerStarted","Data":"b3a5c87d262d2b129a5cf97217ceb6830f3c32d84e14f4b607313ad897de1f9f"} Jul 09 19:03:33.146584 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:33.146517 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9xn8w" Jul 09 19:03:33.165089 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:33.165018 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-xtr7d" podStartSLOduration=3.455382931 podStartE2EDuration="6.164997257s" totalImagesPullingTime="2.709614326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:03:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:03:33.164486894 +0000 UTC m=+282.672266668" watchObservedRunningTime="2026-07-09 19:03:33.164997257 +0000 UTC m=+282.672777033" Jul 09 19:03:33.751958 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:33.751897 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-66b97f775b-6x2dl" podUID="31207519-aadd-4eb5-8cef-af759079efb8" containerName="registry" containerID="cri-o://ec77586e1ed361ed56896b4f70470c8a3774ee78e4bbf50b225bc9f6f1998d2a" gracePeriod=30 Jul 09 19:03:34.058181 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:34.058142 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66b97f775b-6x2dl" Jul 09 19:03:34.123041 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:34.123004 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qb2c\" (UniqueName: \"kubernetes.io/projected/31207519-aadd-4eb5-8cef-af759079efb8-kube-api-access-2qb2c\") pod \"31207519-aadd-4eb5-8cef-af759079efb8\" (UID: \"31207519-aadd-4eb5-8cef-af759079efb8\") " Jul 09 19:03:34.123236 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:34.123061 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/31207519-aadd-4eb5-8cef-af759079efb8-registry-tls\") pod \"31207519-aadd-4eb5-8cef-af759079efb8\" (UID: \"31207519-aadd-4eb5-8cef-af759079efb8\") " Jul 09 19:03:34.123236 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:34.123092 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/31207519-aadd-4eb5-8cef-af759079efb8-image-registry-private-configuration\") pod \"31207519-aadd-4eb5-8cef-af759079efb8\" (UID: \"31207519-aadd-4eb5-8cef-af759079efb8\") " Jul 09 19:03:34.123236 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:34.123124 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/31207519-aadd-4eb5-8cef-af759079efb8-registry-certificates\") pod \"31207519-aadd-4eb5-8cef-af759079efb8\" (UID: \"31207519-aadd-4eb5-8cef-af759079efb8\") " Jul 09 19:03:34.123236 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:34.123152 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/31207519-aadd-4eb5-8cef-af759079efb8-ca-trust-extracted\") pod \"31207519-aadd-4eb5-8cef-af759079efb8\" (UID: \"31207519-aadd-4eb5-8cef-af759079efb8\") " Jul 09 19:03:34.123236 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:34.123196 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/31207519-aadd-4eb5-8cef-af759079efb8-trusted-ca\") pod \"31207519-aadd-4eb5-8cef-af759079efb8\" (UID: \"31207519-aadd-4eb5-8cef-af759079efb8\") " Jul 09 19:03:34.123509 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:34.123239 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/31207519-aadd-4eb5-8cef-af759079efb8-bound-sa-token\") pod \"31207519-aadd-4eb5-8cef-af759079efb8\" (UID: \"31207519-aadd-4eb5-8cef-af759079efb8\") " Jul 09 19:03:34.123509 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:34.123274 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/31207519-aadd-4eb5-8cef-af759079efb8-installation-pull-secrets\") pod \"31207519-aadd-4eb5-8cef-af759079efb8\" (UID: \"31207519-aadd-4eb5-8cef-af759079efb8\") " Jul 09 19:03:34.124026 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:34.123743 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31207519-aadd-4eb5-8cef-af759079efb8-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "31207519-aadd-4eb5-8cef-af759079efb8" (UID: "31207519-aadd-4eb5-8cef-af759079efb8"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 19:03:34.124026 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:34.123986 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31207519-aadd-4eb5-8cef-af759079efb8-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "31207519-aadd-4eb5-8cef-af759079efb8" (UID: "31207519-aadd-4eb5-8cef-af759079efb8"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 19:03:34.125991 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:34.125962 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31207519-aadd-4eb5-8cef-af759079efb8-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "31207519-aadd-4eb5-8cef-af759079efb8" (UID: "31207519-aadd-4eb5-8cef-af759079efb8"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 19:03:34.126191 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:34.126149 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31207519-aadd-4eb5-8cef-af759079efb8-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "31207519-aadd-4eb5-8cef-af759079efb8" (UID: "31207519-aadd-4eb5-8cef-af759079efb8"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 19:03:34.126297 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:34.126197 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31207519-aadd-4eb5-8cef-af759079efb8-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "31207519-aadd-4eb5-8cef-af759079efb8" (UID: "31207519-aadd-4eb5-8cef-af759079efb8"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:03:34.127164 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:34.127102 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31207519-aadd-4eb5-8cef-af759079efb8-kube-api-access-2qb2c" (OuterVolumeSpecName: "kube-api-access-2qb2c") pod "31207519-aadd-4eb5-8cef-af759079efb8" (UID: "31207519-aadd-4eb5-8cef-af759079efb8"). InnerVolumeSpecName "kube-api-access-2qb2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:03:34.127164 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:34.127126 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31207519-aadd-4eb5-8cef-af759079efb8-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "31207519-aadd-4eb5-8cef-af759079efb8" (UID: "31207519-aadd-4eb5-8cef-af759079efb8"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:03:34.133589 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:34.133561 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31207519-aadd-4eb5-8cef-af759079efb8-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "31207519-aadd-4eb5-8cef-af759079efb8" (UID: "31207519-aadd-4eb5-8cef-af759079efb8"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:03:34.153058 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:34.153021 2566 generic.go:356] "Generic (PLEG): container finished" podID="31207519-aadd-4eb5-8cef-af759079efb8" containerID="ec77586e1ed361ed56896b4f70470c8a3774ee78e4bbf50b225bc9f6f1998d2a" exitCode=0 Jul 09 19:03:34.153487 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:34.153092 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66b97f775b-6x2dl" Jul 09 19:03:34.153487 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:34.153108 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66b97f775b-6x2dl" event={"ID":"31207519-aadd-4eb5-8cef-af759079efb8","Type":"ContainerDied","Data":"ec77586e1ed361ed56896b4f70470c8a3774ee78e4bbf50b225bc9f6f1998d2a"} Jul 09 19:03:34.153487 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:34.153221 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66b97f775b-6x2dl" event={"ID":"31207519-aadd-4eb5-8cef-af759079efb8","Type":"ContainerDied","Data":"6337aecc27af6392c7780a641cb4df54024d5bade2da76fb3398df3e7aeca07b"} Jul 09 19:03:34.153487 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:34.153245 2566 scope.go:117] "RemoveContainer" containerID="ec77586e1ed361ed56896b4f70470c8a3774ee78e4bbf50b225bc9f6f1998d2a" Jul 09 19:03:34.153954 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:34.153936 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-xtr7d" Jul 09 19:03:34.160497 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:34.160475 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-xtr7d" Jul 09 19:03:34.164366 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:34.164334 2566 scope.go:117] "RemoveContainer" containerID="ec77586e1ed361ed56896b4f70470c8a3774ee78e4bbf50b225bc9f6f1998d2a" Jul 09 19:03:34.164677 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:03:34.164650 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec77586e1ed361ed56896b4f70470c8a3774ee78e4bbf50b225bc9f6f1998d2a\": container with ID starting with ec77586e1ed361ed56896b4f70470c8a3774ee78e4bbf50b225bc9f6f1998d2a not found: ID does not exist" containerID="ec77586e1ed361ed56896b4f70470c8a3774ee78e4bbf50b225bc9f6f1998d2a" Jul 09 19:03:34.164760 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:34.164687 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec77586e1ed361ed56896b4f70470c8a3774ee78e4bbf50b225bc9f6f1998d2a"} err="failed to get container status \"ec77586e1ed361ed56896b4f70470c8a3774ee78e4bbf50b225bc9f6f1998d2a\": rpc error: code = NotFound desc = could not find container \"ec77586e1ed361ed56896b4f70470c8a3774ee78e4bbf50b225bc9f6f1998d2a\": container with ID starting with ec77586e1ed361ed56896b4f70470c8a3774ee78e4bbf50b225bc9f6f1998d2a not found: ID does not exist" Jul 09 19:03:34.193774 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:34.193742 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-66b97f775b-6x2dl"] Jul 09 19:03:34.199839 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:34.199812 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-66b97f775b-6x2dl"] Jul 09 19:03:34.224629 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:34.224535 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qb2c\" (UniqueName: \"kubernetes.io/projected/31207519-aadd-4eb5-8cef-af759079efb8-kube-api-access-2qb2c\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:03:34.224629 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:34.224569 2566 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/31207519-aadd-4eb5-8cef-af759079efb8-registry-tls\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:03:34.224629 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:34.224584 2566 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/31207519-aadd-4eb5-8cef-af759079efb8-image-registry-private-configuration\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:03:34.224629 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:34.224598 2566 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/31207519-aadd-4eb5-8cef-af759079efb8-registry-certificates\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:03:34.224629 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:34.224629 2566 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/31207519-aadd-4eb5-8cef-af759079efb8-ca-trust-extracted\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:03:34.224993 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:34.224642 2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/31207519-aadd-4eb5-8cef-af759079efb8-trusted-ca\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:03:34.224993 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:34.224659 2566 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/31207519-aadd-4eb5-8cef-af759079efb8-bound-sa-token\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:03:34.224993 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:34.224672 2566 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/31207519-aadd-4eb5-8cef-af759079efb8-installation-pull-secrets\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:03:35.057044 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:35.057010 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31207519-aadd-4eb5-8cef-af759079efb8" path="/var/lib/kubelet/pods/31207519-aadd-4eb5-8cef-af759079efb8/volumes" Jul 09 19:03:35.737813 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:35.737775 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aa9f1324-a4e6-4f69-93f5-101c3aed7a0c-metrics-tls\") pod \"dns-default-9xn8w\" (UID: \"aa9f1324-a4e6-4f69-93f5-101c3aed7a0c\") " pod="openshift-dns/dns-default-9xn8w" Jul 09 19:03:35.740542 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:35.740517 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aa9f1324-a4e6-4f69-93f5-101c3aed7a0c-metrics-tls\") pod \"dns-default-9xn8w\" (UID: \"aa9f1324-a4e6-4f69-93f5-101c3aed7a0c\") " pod="openshift-dns/dns-default-9xn8w" Jul 09 19:03:35.850821 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:35.850786 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bw6lk\"" Jul 09 19:03:35.858346 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:35.858313 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9xn8w" Jul 09 19:03:36.046148 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:36.046113 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9xn8w"] Jul 09 19:03:36.049683 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:03:36.049638 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa9f1324_a4e6_4f69_93f5_101c3aed7a0c.slice/crio-a649609410ff16658ca99c3d65543ee0532c4c04903b7cdd374174d7885deecd WatchSource:0}: Error finding container a649609410ff16658ca99c3d65543ee0532c4c04903b7cdd374174d7885deecd: Status 404 returned error can't find the container with id a649609410ff16658ca99c3d65543ee0532c4c04903b7cdd374174d7885deecd Jul 09 19:03:36.165152 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:36.165119 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9xn8w" event={"ID":"aa9f1324-a4e6-4f69-93f5-101c3aed7a0c","Type":"ContainerStarted","Data":"a649609410ff16658ca99c3d65543ee0532c4c04903b7cdd374174d7885deecd"} Jul 09 19:03:39.178251 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:39.178218 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9xn8w" event={"ID":"aa9f1324-a4e6-4f69-93f5-101c3aed7a0c","Type":"ContainerStarted","Data":"ae73ae1a6d630b9f3462e94d3fa761380d45f4565e75fe6f8106ec82839f038a"} Jul 09 19:03:39.178713 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:39.178277 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9xn8w" event={"ID":"aa9f1324-a4e6-4f69-93f5-101c3aed7a0c","Type":"ContainerStarted","Data":"b70f4d9bca60d58f39b0e48f8d59cb1eeed9c3ebb748a3b013dbddfb11799898"} Jul 09 19:03:39.178713 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:39.178307 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9xn8w" Jul 09 19:03:39.196532 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:39.196485 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-9xn8w" podStartSLOduration=251.765797134 podStartE2EDuration="4m14.196471698s" totalImagesPullingTime="2.430674564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:59:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:03:39.195364138 +0000 UTC m=+288.703143913" watchObservedRunningTime="2026-07-09 19:03:39.196471698 +0000 UTC m=+288.704251472" Jul 09 19:03:49.184161 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:49.184128 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9xn8w" Jul 09 19:03:50.898341 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:50.898273 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-vws4c_7cf4af17-5218-4b92-bceb-addbefcc03cb/console-operator/1.log" Jul 09 19:03:50.906341 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:50.906316 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-vws4c_7cf4af17-5218-4b92-bceb-addbefcc03cb/console-operator/1.log" Jul 09 19:03:50.910489 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:50.910467 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6dh7g_db233460-134d-4d9d-8da9-31d89366a4e0/ovn-acl-logging/0.log" Jul 09 19:03:50.912158 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:50.912135 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6dh7g_db233460-134d-4d9d-8da9-31d89366a4e0/ovn-acl-logging/0.log" Jul 09 19:03:50.917049 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:50.917031 2566 kubelet.go:1648] "Image garbage collection succeeded" Jul 09 19:03:59.252001 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:59.251967 2566 generic.go:356] "Generic (PLEG): container finished" podID="7dc9a1c9-93e9-4e98-b49d-90f25331cea6" containerID="a2a41e35da487c862947668f329f8b20b6f9c96dd56970ec82f4e2b9dc9beb39" exitCode=0 Jul 09 19:03:59.252408 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:59.252041 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-bwggg" event={"ID":"7dc9a1c9-93e9-4e98-b49d-90f25331cea6","Type":"ContainerDied","Data":"a2a41e35da487c862947668f329f8b20b6f9c96dd56970ec82f4e2b9dc9beb39"} Jul 09 19:03:59.252408 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:03:59.252355 2566 scope.go:117] "RemoveContainer" containerID="a2a41e35da487c862947668f329f8b20b6f9c96dd56970ec82f4e2b9dc9beb39" Jul 09 19:04:00.258731 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:04:00.258699 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-bwggg" event={"ID":"7dc9a1c9-93e9-4e98-b49d-90f25331cea6","Type":"ContainerStarted","Data":"9575b9623937698cc3f300f230493f1bf1c8c4cfe39a5f6a8e2bbc21b07dbedc"} Jul 09 19:05:47.159322 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:05:47.159287 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jg57t"] Jul 09 19:05:47.159821 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:05:47.159583 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31207519-aadd-4eb5-8cef-af759079efb8" containerName="registry" Jul 09 19:05:47.159821 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:05:47.159595 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="31207519-aadd-4eb5-8cef-af759079efb8" containerName="registry" Jul 09 19:05:47.159821 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:05:47.159676 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="31207519-aadd-4eb5-8cef-af759079efb8" containerName="registry" Jul 09 19:05:47.161661 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:05:47.161641 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jg57t" Jul 09 19:05:47.164548 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:05:47.164527 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lwdhr\"" Jul 09 19:05:47.164867 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:05:47.164850 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 09 19:05:47.165844 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:05:47.165829 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 09 19:05:47.173215 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:05:47.173186 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jg57t"] Jul 09 19:05:47.263236 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:05:47.263197 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxkhp\" (UniqueName: \"kubernetes.io/projected/3bcfa0d2-ec44-458b-a6d1-43c300d829e3-kube-api-access-wxkhp\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jg57t\" (UID: \"3bcfa0d2-ec44-458b-a6d1-43c300d829e3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jg57t" Jul 09 19:05:47.263236 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:05:47.263239 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3bcfa0d2-ec44-458b-a6d1-43c300d829e3-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jg57t\" (UID: \"3bcfa0d2-ec44-458b-a6d1-43c300d829e3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jg57t" Jul 09 19:05:47.263443 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:05:47.263282 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3bcfa0d2-ec44-458b-a6d1-43c300d829e3-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jg57t\" (UID: \"3bcfa0d2-ec44-458b-a6d1-43c300d829e3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jg57t" Jul 09 19:05:47.363616 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:05:47.363579 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxkhp\" (UniqueName: \"kubernetes.io/projected/3bcfa0d2-ec44-458b-a6d1-43c300d829e3-kube-api-access-wxkhp\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jg57t\" (UID: \"3bcfa0d2-ec44-458b-a6d1-43c300d829e3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jg57t" Jul 09 19:05:47.363773 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:05:47.363638 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3bcfa0d2-ec44-458b-a6d1-43c300d829e3-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jg57t\" (UID: \"3bcfa0d2-ec44-458b-a6d1-43c300d829e3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jg57t" Jul 09 19:05:47.363773 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:05:47.363666 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3bcfa0d2-ec44-458b-a6d1-43c300d829e3-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jg57t\" (UID: \"3bcfa0d2-ec44-458b-a6d1-43c300d829e3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jg57t" Jul 09 19:05:47.363991 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:05:47.363977 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3bcfa0d2-ec44-458b-a6d1-43c300d829e3-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jg57t\" (UID: \"3bcfa0d2-ec44-458b-a6d1-43c300d829e3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jg57t" Jul 09 19:05:47.364026 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:05:47.364012 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3bcfa0d2-ec44-458b-a6d1-43c300d829e3-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jg57t\" (UID: \"3bcfa0d2-ec44-458b-a6d1-43c300d829e3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jg57t" Jul 09 19:05:47.373304 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:05:47.373267 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxkhp\" (UniqueName: \"kubernetes.io/projected/3bcfa0d2-ec44-458b-a6d1-43c300d829e3-kube-api-access-wxkhp\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jg57t\" (UID: \"3bcfa0d2-ec44-458b-a6d1-43c300d829e3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jg57t" Jul 09 19:05:47.471356 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:05:47.471332 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jg57t" Jul 09 19:05:47.601736 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:05:47.601649 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jg57t"] Jul 09 19:05:47.607089 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:05:47.607065 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bcfa0d2_ec44_458b_a6d1_43c300d829e3.slice/crio-6f6c802bcbc3193df6b05763745656c6eff2ad2a07a017a967fc35b34b5d66e2 WatchSource:0}: Error finding container 6f6c802bcbc3193df6b05763745656c6eff2ad2a07a017a967fc35b34b5d66e2: Status 404 returned error can't find the container with id 6f6c802bcbc3193df6b05763745656c6eff2ad2a07a017a967fc35b34b5d66e2 Jul 09 19:05:47.608901 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:05:47.608884 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 19:05:47.655728 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:05:47.655694 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jg57t" event={"ID":"3bcfa0d2-ec44-458b-a6d1-43c300d829e3","Type":"ContainerStarted","Data":"6f6c802bcbc3193df6b05763745656c6eff2ad2a07a017a967fc35b34b5d66e2"} Jul 09 19:05:55.689152 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:05:55.689116 2566 generic.go:356] "Generic (PLEG): container finished" podID="3bcfa0d2-ec44-458b-a6d1-43c300d829e3" containerID="a06053d52c18186ff0ca746ca2573d46f402bf88653092364e839e44b266f1b6" exitCode=0 Jul 09 19:05:55.689633 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:05:55.689188 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jg57t" event={"ID":"3bcfa0d2-ec44-458b-a6d1-43c300d829e3","Type":"ContainerDied","Data":"a06053d52c18186ff0ca746ca2573d46f402bf88653092364e839e44b266f1b6"} Jul 09 19:05:58.703204 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:05:58.703170 2566 generic.go:356] "Generic (PLEG): container finished" podID="3bcfa0d2-ec44-458b-a6d1-43c300d829e3" containerID="2f4055522e5b1ae7b77419d990163e3fc9410df07e25e0256bfd8bcadfc6f4b1" exitCode=0 Jul 09 19:05:58.703698 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:05:58.703227 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jg57t" event={"ID":"3bcfa0d2-ec44-458b-a6d1-43c300d829e3","Type":"ContainerDied","Data":"2f4055522e5b1ae7b77419d990163e3fc9410df07e25e0256bfd8bcadfc6f4b1"} Jul 09 19:06:06.739666 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:06.739631 2566 generic.go:356] "Generic (PLEG): container finished" podID="3bcfa0d2-ec44-458b-a6d1-43c300d829e3" containerID="cd4e47a3c26021a2dc3acb68bfa0a1e6695a077ac05497bde4ed9d0d098a8ebe" exitCode=0 Jul 09 19:06:06.740038 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:06.739719 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jg57t" event={"ID":"3bcfa0d2-ec44-458b-a6d1-43c300d829e3","Type":"ContainerDied","Data":"cd4e47a3c26021a2dc3acb68bfa0a1e6695a077ac05497bde4ed9d0d098a8ebe"} Jul 09 19:06:07.866420 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:07.866396 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jg57t" Jul 09 19:06:07.934497 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:07.934467 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3bcfa0d2-ec44-458b-a6d1-43c300d829e3-util\") pod \"3bcfa0d2-ec44-458b-a6d1-43c300d829e3\" (UID: \"3bcfa0d2-ec44-458b-a6d1-43c300d829e3\") " Jul 09 19:06:07.934669 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:07.934518 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxkhp\" (UniqueName: \"kubernetes.io/projected/3bcfa0d2-ec44-458b-a6d1-43c300d829e3-kube-api-access-wxkhp\") pod \"3bcfa0d2-ec44-458b-a6d1-43c300d829e3\" (UID: \"3bcfa0d2-ec44-458b-a6d1-43c300d829e3\") " Jul 09 19:06:07.934669 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:07.934639 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3bcfa0d2-ec44-458b-a6d1-43c300d829e3-bundle\") pod \"3bcfa0d2-ec44-458b-a6d1-43c300d829e3\" (UID: \"3bcfa0d2-ec44-458b-a6d1-43c300d829e3\") " Jul 09 19:06:07.935272 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:07.935246 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bcfa0d2-ec44-458b-a6d1-43c300d829e3-bundle" (OuterVolumeSpecName: "bundle") pod "3bcfa0d2-ec44-458b-a6d1-43c300d829e3" (UID: "3bcfa0d2-ec44-458b-a6d1-43c300d829e3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:06:07.936799 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:07.936773 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bcfa0d2-ec44-458b-a6d1-43c300d829e3-kube-api-access-wxkhp" (OuterVolumeSpecName: "kube-api-access-wxkhp") pod "3bcfa0d2-ec44-458b-a6d1-43c300d829e3" (UID: "3bcfa0d2-ec44-458b-a6d1-43c300d829e3"). InnerVolumeSpecName "kube-api-access-wxkhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:06:07.940103 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:07.940079 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bcfa0d2-ec44-458b-a6d1-43c300d829e3-util" (OuterVolumeSpecName: "util") pod "3bcfa0d2-ec44-458b-a6d1-43c300d829e3" (UID: "3bcfa0d2-ec44-458b-a6d1-43c300d829e3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:06:08.035541 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:08.035455 2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3bcfa0d2-ec44-458b-a6d1-43c300d829e3-bundle\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:06:08.035541 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:08.035481 2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3bcfa0d2-ec44-458b-a6d1-43c300d829e3-util\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:06:08.035541 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:08.035491 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxkhp\" (UniqueName: \"kubernetes.io/projected/3bcfa0d2-ec44-458b-a6d1-43c300d829e3-kube-api-access-wxkhp\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:06:08.749514 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:08.749487 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jg57t" Jul 09 19:06:08.749514 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:08.749508 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jg57t" event={"ID":"3bcfa0d2-ec44-458b-a6d1-43c300d829e3","Type":"ContainerDied","Data":"6f6c802bcbc3193df6b05763745656c6eff2ad2a07a017a967fc35b34b5d66e2"} Jul 09 19:06:08.749745 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:08.749577 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f6c802bcbc3193df6b05763745656c6eff2ad2a07a017a967fc35b34b5d66e2" Jul 09 19:06:18.575444 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:18.575355 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzv4qb"] Jul 09 19:06:18.576034 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:18.575791 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bcfa0d2-ec44-458b-a6d1-43c300d829e3" containerName="pull" Jul 09 19:06:18.576034 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:18.575809 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bcfa0d2-ec44-458b-a6d1-43c300d829e3" containerName="pull" Jul 09 19:06:18.576034 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:18.575823 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bcfa0d2-ec44-458b-a6d1-43c300d829e3" containerName="util" Jul 09 19:06:18.576034 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:18.575831 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bcfa0d2-ec44-458b-a6d1-43c300d829e3" containerName="util" Jul 09 19:06:18.576034 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:18.575843 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bcfa0d2-ec44-458b-a6d1-43c300d829e3" containerName="extract" Jul 09 19:06:18.576034 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:18.575851 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bcfa0d2-ec44-458b-a6d1-43c300d829e3" containerName="extract" Jul 09 19:06:18.576034 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:18.575971 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bcfa0d2-ec44-458b-a6d1-43c300d829e3" containerName="extract" Jul 09 19:06:18.578750 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:18.578728 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzv4qb" Jul 09 19:06:18.582149 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:18.582128 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 09 19:06:18.582240 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:18.582182 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 09 19:06:18.583442 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:18.583410 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lwdhr\"" Jul 09 19:06:18.593960 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:18.593931 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzv4qb"] Jul 09 19:06:18.722785 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:18.722750 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8df7bf3e-341c-44b2-a94c-39b91ff07791-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzv4qb\" (UID: \"8df7bf3e-341c-44b2-a94c-39b91ff07791\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzv4qb" Jul 09 19:06:18.722785 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:18.722787 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8df7bf3e-341c-44b2-a94c-39b91ff07791-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzv4qb\" (UID: \"8df7bf3e-341c-44b2-a94c-39b91ff07791\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzv4qb" Jul 09 19:06:18.723009 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:18.722836 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4khkd\" (UniqueName: \"kubernetes.io/projected/8df7bf3e-341c-44b2-a94c-39b91ff07791-kube-api-access-4khkd\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzv4qb\" (UID: \"8df7bf3e-341c-44b2-a94c-39b91ff07791\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzv4qb" Jul 09 19:06:18.808738 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:18.808708 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-kxql5"] Jul 09 19:06:18.811810 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:18.811795 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-kxql5" Jul 09 19:06:18.815006 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:18.814979 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 09 19:06:18.815006 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:18.814979 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 09 19:06:18.815186 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:18.815147 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-fkfdq\"" Jul 09 19:06:18.823406 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:18.823385 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8df7bf3e-341c-44b2-a94c-39b91ff07791-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzv4qb\" (UID: \"8df7bf3e-341c-44b2-a94c-39b91ff07791\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzv4qb" Jul 09 19:06:18.823504 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:18.823415 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8df7bf3e-341c-44b2-a94c-39b91ff07791-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzv4qb\" (UID: \"8df7bf3e-341c-44b2-a94c-39b91ff07791\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzv4qb" Jul 09 19:06:18.823504 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:18.823469 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4khkd\" (UniqueName: \"kubernetes.io/projected/8df7bf3e-341c-44b2-a94c-39b91ff07791-kube-api-access-4khkd\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzv4qb\" (UID: \"8df7bf3e-341c-44b2-a94c-39b91ff07791\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzv4qb" Jul 09 19:06:18.823805 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:18.823788 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8df7bf3e-341c-44b2-a94c-39b91ff07791-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzv4qb\" (UID: \"8df7bf3e-341c-44b2-a94c-39b91ff07791\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzv4qb" Jul 09 19:06:18.823895 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:18.823833 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8df7bf3e-341c-44b2-a94c-39b91ff07791-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzv4qb\" (UID: \"8df7bf3e-341c-44b2-a94c-39b91ff07791\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzv4qb" Jul 09 19:06:18.828673 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:18.828625 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-kxql5"] Jul 09 19:06:18.841122 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:18.841098 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4khkd\" (UniqueName: \"kubernetes.io/projected/8df7bf3e-341c-44b2-a94c-39b91ff07791-kube-api-access-4khkd\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzv4qb\" (UID: \"8df7bf3e-341c-44b2-a94c-39b91ff07791\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzv4qb" Jul 09 19:06:18.887668 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:18.887644 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzv4qb" Jul 09 19:06:18.925013 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:18.924929 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9j52\" (UniqueName: \"kubernetes.io/projected/7b0cf7c3-90ca-4a86-9b4c-7eab1caf16b2-kube-api-access-r9j52\") pod \"cert-manager-webhook-6486ffd997-kxql5\" (UID: \"7b0cf7c3-90ca-4a86-9b4c-7eab1caf16b2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kxql5" Jul 09 19:06:18.925168 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:18.925025 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7b0cf7c3-90ca-4a86-9b4c-7eab1caf16b2-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-kxql5\" (UID: \"7b0cf7c3-90ca-4a86-9b4c-7eab1caf16b2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kxql5" Jul 09 19:06:19.025480 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:19.025445 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9j52\" (UniqueName: \"kubernetes.io/projected/7b0cf7c3-90ca-4a86-9b4c-7eab1caf16b2-kube-api-access-r9j52\") pod \"cert-manager-webhook-6486ffd997-kxql5\" (UID: \"7b0cf7c3-90ca-4a86-9b4c-7eab1caf16b2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kxql5" Jul 09 19:06:19.025668 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:19.025535 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7b0cf7c3-90ca-4a86-9b4c-7eab1caf16b2-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-kxql5\" (UID: \"7b0cf7c3-90ca-4a86-9b4c-7eab1caf16b2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kxql5" Jul 09 19:06:19.045048 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:19.045026 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzv4qb"] Jul 09 19:06:19.047185 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:06:19.047155 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8df7bf3e_341c_44b2_a94c_39b91ff07791.slice/crio-62a586ddb348c76049e0aa75a22a9f6b19f7a738c6dcc071593025780f8cc73b WatchSource:0}: Error finding container 62a586ddb348c76049e0aa75a22a9f6b19f7a738c6dcc071593025780f8cc73b: Status 404 returned error can't find the container with id 62a586ddb348c76049e0aa75a22a9f6b19f7a738c6dcc071593025780f8cc73b Jul 09 19:06:19.049035 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:19.049020 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9j52\" (UniqueName: \"kubernetes.io/projected/7b0cf7c3-90ca-4a86-9b4c-7eab1caf16b2-kube-api-access-r9j52\") pod \"cert-manager-webhook-6486ffd997-kxql5\" (UID: \"7b0cf7c3-90ca-4a86-9b4c-7eab1caf16b2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kxql5" Jul 09 19:06:19.049280 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:19.049256 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7b0cf7c3-90ca-4a86-9b4c-7eab1caf16b2-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-kxql5\" (UID: \"7b0cf7c3-90ca-4a86-9b4c-7eab1caf16b2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kxql5" Jul 09 19:06:19.120487 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:19.120456 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-kxql5" Jul 09 19:06:19.258301 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:19.258270 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-kxql5"] Jul 09 19:06:19.261094 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:06:19.261061 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b0cf7c3_90ca_4a86_9b4c_7eab1caf16b2.slice/crio-06d30568d37d3fff1c0f3395393501fc3068461e351c7a714416b5275f6daf59 WatchSource:0}: Error finding container 06d30568d37d3fff1c0f3395393501fc3068461e351c7a714416b5275f6daf59: Status 404 returned error can't find the container with id 06d30568d37d3fff1c0f3395393501fc3068461e351c7a714416b5275f6daf59 Jul 09 19:06:19.790681 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:19.790644 2566 generic.go:356] "Generic (PLEG): container finished" podID="8df7bf3e-341c-44b2-a94c-39b91ff07791" containerID="1f877e9da93af97e5b005da058f396a6bae2b395659f7a4f1fce83a52bd034ac" exitCode=0 Jul 09 19:06:19.791109 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:19.790683 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzv4qb" event={"ID":"8df7bf3e-341c-44b2-a94c-39b91ff07791","Type":"ContainerDied","Data":"1f877e9da93af97e5b005da058f396a6bae2b395659f7a4f1fce83a52bd034ac"} Jul 09 19:06:19.791109 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:19.790757 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzv4qb" event={"ID":"8df7bf3e-341c-44b2-a94c-39b91ff07791","Type":"ContainerStarted","Data":"62a586ddb348c76049e0aa75a22a9f6b19f7a738c6dcc071593025780f8cc73b"} Jul 09 19:06:19.792087 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:19.792052 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-kxql5" event={"ID":"7b0cf7c3-90ca-4a86-9b4c-7eab1caf16b2","Type":"ContainerStarted","Data":"06d30568d37d3fff1c0f3395393501fc3068461e351c7a714416b5275f6daf59"} Jul 09 19:06:22.805991 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:22.805951 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-kxql5" event={"ID":"7b0cf7c3-90ca-4a86-9b4c-7eab1caf16b2","Type":"ContainerStarted","Data":"19784ff36a86a511cd4b5d3064b4a70f6b052da60afd98a1466c967f6243fa4b"} Jul 09 19:06:22.806403 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:22.806103 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-kxql5" Jul 09 19:06:22.835838 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:22.835780 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-kxql5" podStartSLOduration=2.259126177 podStartE2EDuration="4.835759211s" totalImagesPullingTime="2.576633034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:06:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:06:22.833895033 +0000 UTC m=+452.341674807" watchObservedRunningTime="2026-07-09 19:06:22.835759211 +0000 UTC m=+452.343538988" Jul 09 19:06:23.811974 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:23.811936 2566 generic.go:356] "Generic (PLEG): container finished" podID="8df7bf3e-341c-44b2-a94c-39b91ff07791" containerID="e26d25e9060ce3c72b8fc53b60e4c97328264d6d11c0aa40ac0c5f549805b637" exitCode=0 Jul 09 19:06:23.812379 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:23.812016 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzv4qb" event={"ID":"8df7bf3e-341c-44b2-a94c-39b91ff07791","Type":"ContainerDied","Data":"e26d25e9060ce3c72b8fc53b60e4c97328264d6d11c0aa40ac0c5f549805b637"} Jul 09 19:06:24.817369 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:24.817337 2566 generic.go:356] "Generic (PLEG): container finished" podID="8df7bf3e-341c-44b2-a94c-39b91ff07791" containerID="1b5d4c91fbbab847282474e8d216ac5159da18f188cf42a32980d4bcd673c566" exitCode=0 Jul 09 19:06:24.817729 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:24.817430 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzv4qb" event={"ID":"8df7bf3e-341c-44b2-a94c-39b91ff07791","Type":"ContainerDied","Data":"1b5d4c91fbbab847282474e8d216ac5159da18f188cf42a32980d4bcd673c566"} Jul 09 19:06:25.960432 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:25.960408 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzv4qb" Jul 09 19:06:26.082562 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:26.082482 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8df7bf3e-341c-44b2-a94c-39b91ff07791-bundle\") pod \"8df7bf3e-341c-44b2-a94c-39b91ff07791\" (UID: \"8df7bf3e-341c-44b2-a94c-39b91ff07791\") " Jul 09 19:06:26.082562 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:26.082539 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4khkd\" (UniqueName: \"kubernetes.io/projected/8df7bf3e-341c-44b2-a94c-39b91ff07791-kube-api-access-4khkd\") pod \"8df7bf3e-341c-44b2-a94c-39b91ff07791\" (UID: \"8df7bf3e-341c-44b2-a94c-39b91ff07791\") " Jul 09 19:06:26.082810 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:26.082568 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8df7bf3e-341c-44b2-a94c-39b91ff07791-util\") pod \"8df7bf3e-341c-44b2-a94c-39b91ff07791\" (UID: \"8df7bf3e-341c-44b2-a94c-39b91ff07791\") " Jul 09 19:06:26.082920 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:26.082896 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8df7bf3e-341c-44b2-a94c-39b91ff07791-bundle" (OuterVolumeSpecName: "bundle") pod "8df7bf3e-341c-44b2-a94c-39b91ff07791" (UID: "8df7bf3e-341c-44b2-a94c-39b91ff07791"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:06:26.084634 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:26.084596 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8df7bf3e-341c-44b2-a94c-39b91ff07791-kube-api-access-4khkd" (OuterVolumeSpecName: "kube-api-access-4khkd") pod "8df7bf3e-341c-44b2-a94c-39b91ff07791" (UID: "8df7bf3e-341c-44b2-a94c-39b91ff07791"). InnerVolumeSpecName "kube-api-access-4khkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:06:26.089467 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:26.089425 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8df7bf3e-341c-44b2-a94c-39b91ff07791-util" (OuterVolumeSpecName: "util") pod "8df7bf3e-341c-44b2-a94c-39b91ff07791" (UID: "8df7bf3e-341c-44b2-a94c-39b91ff07791"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:06:26.183978 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:26.183945 2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8df7bf3e-341c-44b2-a94c-39b91ff07791-bundle\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:06:26.183978 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:26.183973 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4khkd\" (UniqueName: \"kubernetes.io/projected/8df7bf3e-341c-44b2-a94c-39b91ff07791-kube-api-access-4khkd\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:06:26.183978 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:26.183982 2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8df7bf3e-341c-44b2-a94c-39b91ff07791-util\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:06:26.826283 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:26.826253 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzv4qb" Jul 09 19:06:26.826283 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:26.826277 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzv4qb" event={"ID":"8df7bf3e-341c-44b2-a94c-39b91ff07791","Type":"ContainerDied","Data":"62a586ddb348c76049e0aa75a22a9f6b19f7a738c6dcc071593025780f8cc73b"} Jul 09 19:06:26.826475 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:26.826321 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62a586ddb348c76049e0aa75a22a9f6b19f7a738c6dcc071593025780f8cc73b" Jul 09 19:06:28.814730 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:28.814702 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-kxql5" Jul 09 19:06:29.265983 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:29.265952 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-jhktz"] Jul 09 19:06:29.266272 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:29.266260 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8df7bf3e-341c-44b2-a94c-39b91ff07791" containerName="extract" Jul 09 19:06:29.266314 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:29.266274 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8df7bf3e-341c-44b2-a94c-39b91ff07791" containerName="extract" Jul 09 19:06:29.266314 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:29.266287 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8df7bf3e-341c-44b2-a94c-39b91ff07791" containerName="util" Jul 09 19:06:29.266314 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:29.266293 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8df7bf3e-341c-44b2-a94c-39b91ff07791" containerName="util" Jul 09 19:06:29.266314 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:29.266301 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8df7bf3e-341c-44b2-a94c-39b91ff07791" containerName="pull" Jul 09 19:06:29.266314 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:29.266306 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8df7bf3e-341c-44b2-a94c-39b91ff07791" containerName="pull" Jul 09 19:06:29.266461 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:29.266348 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8df7bf3e-341c-44b2-a94c-39b91ff07791" containerName="extract" Jul 09 19:06:29.268124 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:29.268107 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-jhktz" Jul 09 19:06:29.270755 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:29.270737 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-z89nm\"" Jul 09 19:06:29.278879 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:29.278858 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-jhktz"] Jul 09 19:06:29.413183 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:29.413156 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lglpx\" (UniqueName: \"kubernetes.io/projected/a63d8fe6-2e68-461c-8b9f-c675b2f6c6c5-kube-api-access-lglpx\") pod \"cert-manager-55b4d954d5-jhktz\" (UID: \"a63d8fe6-2e68-461c-8b9f-c675b2f6c6c5\") " pod="cert-manager/cert-manager-55b4d954d5-jhktz" Jul 09 19:06:29.413346 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:29.413208 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a63d8fe6-2e68-461c-8b9f-c675b2f6c6c5-bound-sa-token\") pod \"cert-manager-55b4d954d5-jhktz\" (UID: \"a63d8fe6-2e68-461c-8b9f-c675b2f6c6c5\") " pod="cert-manager/cert-manager-55b4d954d5-jhktz" Jul 09 19:06:29.514345 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:29.514307 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a63d8fe6-2e68-461c-8b9f-c675b2f6c6c5-bound-sa-token\") pod \"cert-manager-55b4d954d5-jhktz\" (UID: \"a63d8fe6-2e68-461c-8b9f-c675b2f6c6c5\") " pod="cert-manager/cert-manager-55b4d954d5-jhktz" Jul 09 19:06:29.514527 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:29.514388 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lglpx\" (UniqueName: \"kubernetes.io/projected/a63d8fe6-2e68-461c-8b9f-c675b2f6c6c5-kube-api-access-lglpx\") pod \"cert-manager-55b4d954d5-jhktz\" (UID: \"a63d8fe6-2e68-461c-8b9f-c675b2f6c6c5\") " pod="cert-manager/cert-manager-55b4d954d5-jhktz" Jul 09 19:06:29.523517 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:29.523453 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a63d8fe6-2e68-461c-8b9f-c675b2f6c6c5-bound-sa-token\") pod \"cert-manager-55b4d954d5-jhktz\" (UID: \"a63d8fe6-2e68-461c-8b9f-c675b2f6c6c5\") " pod="cert-manager/cert-manager-55b4d954d5-jhktz" Jul 09 19:06:29.523680 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:29.523658 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lglpx\" (UniqueName: \"kubernetes.io/projected/a63d8fe6-2e68-461c-8b9f-c675b2f6c6c5-kube-api-access-lglpx\") pod \"cert-manager-55b4d954d5-jhktz\" (UID: \"a63d8fe6-2e68-461c-8b9f-c675b2f6c6c5\") " pod="cert-manager/cert-manager-55b4d954d5-jhktz" Jul 09 19:06:29.577075 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:29.577047 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-jhktz" Jul 09 19:06:29.702668 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:29.702641 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-jhktz"] Jul 09 19:06:29.705397 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:06:29.705363 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda63d8fe6_2e68_461c_8b9f_c675b2f6c6c5.slice/crio-84cc504edef454623c7cc633509a892aee4e4946c8d82161e4fb799e8bb5f6ca WatchSource:0}: Error finding container 84cc504edef454623c7cc633509a892aee4e4946c8d82161e4fb799e8bb5f6ca: Status 404 returned error can't find the container with id 84cc504edef454623c7cc633509a892aee4e4946c8d82161e4fb799e8bb5f6ca Jul 09 19:06:29.843747 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:29.843663 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-jhktz" event={"ID":"a63d8fe6-2e68-461c-8b9f-c675b2f6c6c5","Type":"ContainerStarted","Data":"ba922bac57956d44c94e6e1c122db32c88e2a30a3e82f67e9d9fcadfba280b28"} Jul 09 19:06:29.843747 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:29.843703 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-jhktz" event={"ID":"a63d8fe6-2e68-461c-8b9f-c675b2f6c6c5","Type":"ContainerStarted","Data":"84cc504edef454623c7cc633509a892aee4e4946c8d82161e4fb799e8bb5f6ca"} Jul 09 19:06:29.879466 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:29.879413 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-jhktz" podStartSLOduration=0.879398589 podStartE2EDuration="879.398589ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:06:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:06:29.878641114 +0000 UTC m=+459.386420889" watchObservedRunningTime="2026-07-09 19:06:29.879398589 +0000 UTC m=+459.387178363" Jul 09 19:06:37.947000 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:37.946966 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcb7w6"] Jul 09 19:06:37.988943 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:37.988910 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcb7w6"] Jul 09 19:06:37.989090 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:37.989032 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcb7w6" Jul 09 19:06:37.992317 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:37.992288 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 09 19:06:37.993727 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:37.993710 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lwdhr\"" Jul 09 19:06:37.993862 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:37.993750 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 09 19:06:38.078646 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:38.078593 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/42ea2d6f-9ea7-4320-8be0-2b137d2ecc13-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcb7w6\" (UID: \"42ea2d6f-9ea7-4320-8be0-2b137d2ecc13\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcb7w6" Jul 09 19:06:38.078789 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:38.078666 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/42ea2d6f-9ea7-4320-8be0-2b137d2ecc13-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcb7w6\" (UID: \"42ea2d6f-9ea7-4320-8be0-2b137d2ecc13\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcb7w6" Jul 09 19:06:38.078789 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:38.078704 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5whzd\" (UniqueName: \"kubernetes.io/projected/42ea2d6f-9ea7-4320-8be0-2b137d2ecc13-kube-api-access-5whzd\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcb7w6\" (UID: \"42ea2d6f-9ea7-4320-8be0-2b137d2ecc13\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcb7w6" Jul 09 19:06:38.179507 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:38.179476 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/42ea2d6f-9ea7-4320-8be0-2b137d2ecc13-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcb7w6\" (UID: \"42ea2d6f-9ea7-4320-8be0-2b137d2ecc13\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcb7w6" Jul 09 19:06:38.179688 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:38.179550 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5whzd\" (UniqueName: \"kubernetes.io/projected/42ea2d6f-9ea7-4320-8be0-2b137d2ecc13-kube-api-access-5whzd\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcb7w6\" (UID: \"42ea2d6f-9ea7-4320-8be0-2b137d2ecc13\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcb7w6" Jul 09 19:06:38.179688 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:38.179593 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/42ea2d6f-9ea7-4320-8be0-2b137d2ecc13-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcb7w6\" (UID: \"42ea2d6f-9ea7-4320-8be0-2b137d2ecc13\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcb7w6" Jul 09 19:06:38.179879 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:38.179851 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/42ea2d6f-9ea7-4320-8be0-2b137d2ecc13-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcb7w6\" (UID: \"42ea2d6f-9ea7-4320-8be0-2b137d2ecc13\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcb7w6" Jul 09 19:06:38.179918 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:38.179879 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/42ea2d6f-9ea7-4320-8be0-2b137d2ecc13-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcb7w6\" (UID: \"42ea2d6f-9ea7-4320-8be0-2b137d2ecc13\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcb7w6" Jul 09 19:06:38.187693 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:38.187669 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5whzd\" (UniqueName: \"kubernetes.io/projected/42ea2d6f-9ea7-4320-8be0-2b137d2ecc13-kube-api-access-5whzd\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcb7w6\" (UID: \"42ea2d6f-9ea7-4320-8be0-2b137d2ecc13\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcb7w6" Jul 09 19:06:38.298663 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:38.298562 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcb7w6" Jul 09 19:06:38.445063 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:38.445032 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcb7w6"] Jul 09 19:06:38.447821 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:06:38.447793 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42ea2d6f_9ea7_4320_8be0_2b137d2ecc13.slice/crio-c67fcff9f44ee57f3ee44a4b6633402499918750d1d49e38b82f566ea6fb88e8 WatchSource:0}: Error finding container c67fcff9f44ee57f3ee44a4b6633402499918750d1d49e38b82f566ea6fb88e8: Status 404 returned error can't find the container with id c67fcff9f44ee57f3ee44a4b6633402499918750d1d49e38b82f566ea6fb88e8 Jul 09 19:06:38.881183 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:38.881087 2566 generic.go:356] "Generic (PLEG): container finished" podID="42ea2d6f-9ea7-4320-8be0-2b137d2ecc13" containerID="930bc4c27dbd8700fd32dfd08491d18a8c035407c033b8f29eca344f4fa768ad" exitCode=0 Jul 09 19:06:38.881183 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:38.881159 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcb7w6" event={"ID":"42ea2d6f-9ea7-4320-8be0-2b137d2ecc13","Type":"ContainerDied","Data":"930bc4c27dbd8700fd32dfd08491d18a8c035407c033b8f29eca344f4fa768ad"} Jul 09 19:06:38.881371 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:38.881208 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcb7w6" event={"ID":"42ea2d6f-9ea7-4320-8be0-2b137d2ecc13","Type":"ContainerStarted","Data":"c67fcff9f44ee57f3ee44a4b6633402499918750d1d49e38b82f566ea6fb88e8"} Jul 09 19:06:40.893038 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:40.893005 2566 generic.go:356] "Generic (PLEG): container finished" podID="42ea2d6f-9ea7-4320-8be0-2b137d2ecc13" containerID="32316c06f990fbcd16a53266f21945d84c17de0ae9f6b75f1774e22d24a558c0" exitCode=0 Jul 09 19:06:40.893424 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:40.893058 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcb7w6" event={"ID":"42ea2d6f-9ea7-4320-8be0-2b137d2ecc13","Type":"ContainerDied","Data":"32316c06f990fbcd16a53266f21945d84c17de0ae9f6b75f1774e22d24a558c0"} Jul 09 19:06:41.898947 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:41.898914 2566 generic.go:356] "Generic (PLEG): container finished" podID="42ea2d6f-9ea7-4320-8be0-2b137d2ecc13" containerID="964eaf0b76b9826982780f7aef037389df54b78ecbfced6ed3edccb55471498e" exitCode=0 Jul 09 19:06:41.899335 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:41.898953 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcb7w6" event={"ID":"42ea2d6f-9ea7-4320-8be0-2b137d2ecc13","Type":"ContainerDied","Data":"964eaf0b76b9826982780f7aef037389df54b78ecbfced6ed3edccb55471498e"} Jul 09 19:06:43.026968 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:43.026944 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcb7w6" Jul 09 19:06:43.116290 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:43.116260 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5whzd\" (UniqueName: \"kubernetes.io/projected/42ea2d6f-9ea7-4320-8be0-2b137d2ecc13-kube-api-access-5whzd\") pod \"42ea2d6f-9ea7-4320-8be0-2b137d2ecc13\" (UID: \"42ea2d6f-9ea7-4320-8be0-2b137d2ecc13\") " Jul 09 19:06:43.116457 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:43.116333 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/42ea2d6f-9ea7-4320-8be0-2b137d2ecc13-util\") pod \"42ea2d6f-9ea7-4320-8be0-2b137d2ecc13\" (UID: \"42ea2d6f-9ea7-4320-8be0-2b137d2ecc13\") " Jul 09 19:06:43.116457 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:43.116363 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/42ea2d6f-9ea7-4320-8be0-2b137d2ecc13-bundle\") pod \"42ea2d6f-9ea7-4320-8be0-2b137d2ecc13\" (UID: \"42ea2d6f-9ea7-4320-8be0-2b137d2ecc13\") " Jul 09 19:06:43.117177 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:43.117148 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42ea2d6f-9ea7-4320-8be0-2b137d2ecc13-bundle" (OuterVolumeSpecName: "bundle") pod "42ea2d6f-9ea7-4320-8be0-2b137d2ecc13" (UID: "42ea2d6f-9ea7-4320-8be0-2b137d2ecc13"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:06:43.118400 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:43.118369 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42ea2d6f-9ea7-4320-8be0-2b137d2ecc13-kube-api-access-5whzd" (OuterVolumeSpecName: "kube-api-access-5whzd") pod "42ea2d6f-9ea7-4320-8be0-2b137d2ecc13" (UID: "42ea2d6f-9ea7-4320-8be0-2b137d2ecc13"). InnerVolumeSpecName "kube-api-access-5whzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:06:43.121971 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:43.121948 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42ea2d6f-9ea7-4320-8be0-2b137d2ecc13-util" (OuterVolumeSpecName: "util") pod "42ea2d6f-9ea7-4320-8be0-2b137d2ecc13" (UID: "42ea2d6f-9ea7-4320-8be0-2b137d2ecc13"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:06:43.217293 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:43.217266 2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/42ea2d6f-9ea7-4320-8be0-2b137d2ecc13-util\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:06:43.217293 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:43.217290 2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/42ea2d6f-9ea7-4320-8be0-2b137d2ecc13-bundle\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:06:43.217463 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:43.217300 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5whzd\" (UniqueName: \"kubernetes.io/projected/42ea2d6f-9ea7-4320-8be0-2b137d2ecc13-kube-api-access-5whzd\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:06:43.908244 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:43.908214 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcb7w6" Jul 09 19:06:43.908408 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:43.908220 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcb7w6" event={"ID":"42ea2d6f-9ea7-4320-8be0-2b137d2ecc13","Type":"ContainerDied","Data":"c67fcff9f44ee57f3ee44a4b6633402499918750d1d49e38b82f566ea6fb88e8"} Jul 09 19:06:43.908408 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:43.908336 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c67fcff9f44ee57f3ee44a4b6633402499918750d1d49e38b82f566ea6fb88e8" Jul 09 19:06:53.917841 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:53.917807 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-799d55cdcd-82dxc"] Jul 09 19:06:53.918368 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:53.918256 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42ea2d6f-9ea7-4320-8be0-2b137d2ecc13" containerName="util" Jul 09 19:06:53.918368 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:53.918275 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="42ea2d6f-9ea7-4320-8be0-2b137d2ecc13" containerName="util" Jul 09 19:06:53.918368 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:53.918290 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42ea2d6f-9ea7-4320-8be0-2b137d2ecc13" containerName="extract" Jul 09 19:06:53.918368 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:53.918298 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="42ea2d6f-9ea7-4320-8be0-2b137d2ecc13" containerName="extract" Jul 09 19:06:53.918368 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:53.918313 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42ea2d6f-9ea7-4320-8be0-2b137d2ecc13" containerName="pull" Jul 09 19:06:53.918368 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:53.918322 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="42ea2d6f-9ea7-4320-8be0-2b137d2ecc13" containerName="pull" Jul 09 19:06:53.918761 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:53.918409 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="42ea2d6f-9ea7-4320-8be0-2b137d2ecc13" containerName="extract" Jul 09 19:06:53.921095 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:53.921075 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-799d55cdcd-82dxc" Jul 09 19:06:53.923980 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:53.923959 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 09 19:06:53.924327 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:53.924306 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 09 19:06:53.924427 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:53.924336 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-ffx4d\"" Jul 09 19:06:53.924427 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:53.924313 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 09 19:06:53.924555 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:53.924500 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 09 19:06:53.938058 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:53.938027 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-799d55cdcd-82dxc"] Jul 09 19:06:53.998236 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:53.998204 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8483e7be-9ac3-4f1a-834d-cc410f402b93-tmp\") pod \"opendatahub-operator-controller-manager-799d55cdcd-82dxc\" (UID: \"8483e7be-9ac3-4f1a-834d-cc410f402b93\") " pod="opendatahub/opendatahub-operator-controller-manager-799d55cdcd-82dxc" Jul 09 19:06:53.998425 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:53.998248 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqrm5\" (UniqueName: \"kubernetes.io/projected/8483e7be-9ac3-4f1a-834d-cc410f402b93-kube-api-access-nqrm5\") pod \"opendatahub-operator-controller-manager-799d55cdcd-82dxc\" (UID: \"8483e7be-9ac3-4f1a-834d-cc410f402b93\") " pod="opendatahub/opendatahub-operator-controller-manager-799d55cdcd-82dxc" Jul 09 19:06:53.998425 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:53.998278 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8483e7be-9ac3-4f1a-834d-cc410f402b93-apiservice-cert\") pod \"opendatahub-operator-controller-manager-799d55cdcd-82dxc\" (UID: \"8483e7be-9ac3-4f1a-834d-cc410f402b93\") " pod="opendatahub/opendatahub-operator-controller-manager-799d55cdcd-82dxc" Jul 09 19:06:53.998425 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:53.998343 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8483e7be-9ac3-4f1a-834d-cc410f402b93-manifests\") pod \"opendatahub-operator-controller-manager-799d55cdcd-82dxc\" (UID: \"8483e7be-9ac3-4f1a-834d-cc410f402b93\") " pod="opendatahub/opendatahub-operator-controller-manager-799d55cdcd-82dxc" Jul 09 19:06:53.998560 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:53.998438 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8483e7be-9ac3-4f1a-834d-cc410f402b93-webhook-cert\") pod \"opendatahub-operator-controller-manager-799d55cdcd-82dxc\" (UID: \"8483e7be-9ac3-4f1a-834d-cc410f402b93\") " pod="opendatahub/opendatahub-operator-controller-manager-799d55cdcd-82dxc" Jul 09 19:06:54.099759 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:54.099721 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8483e7be-9ac3-4f1a-834d-cc410f402b93-tmp\") pod \"opendatahub-operator-controller-manager-799d55cdcd-82dxc\" (UID: \"8483e7be-9ac3-4f1a-834d-cc410f402b93\") " pod="opendatahub/opendatahub-operator-controller-manager-799d55cdcd-82dxc" Jul 09 19:06:54.099938 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:54.099781 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqrm5\" (UniqueName: \"kubernetes.io/projected/8483e7be-9ac3-4f1a-834d-cc410f402b93-kube-api-access-nqrm5\") pod \"opendatahub-operator-controller-manager-799d55cdcd-82dxc\" (UID: \"8483e7be-9ac3-4f1a-834d-cc410f402b93\") " pod="opendatahub/opendatahub-operator-controller-manager-799d55cdcd-82dxc" Jul 09 19:06:54.099938 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:54.099820 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8483e7be-9ac3-4f1a-834d-cc410f402b93-apiservice-cert\") pod \"opendatahub-operator-controller-manager-799d55cdcd-82dxc\" (UID: \"8483e7be-9ac3-4f1a-834d-cc410f402b93\") " pod="opendatahub/opendatahub-operator-controller-manager-799d55cdcd-82dxc" Jul 09 19:06:54.099938 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:54.099849 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8483e7be-9ac3-4f1a-834d-cc410f402b93-manifests\") pod \"opendatahub-operator-controller-manager-799d55cdcd-82dxc\" (UID: \"8483e7be-9ac3-4f1a-834d-cc410f402b93\") " pod="opendatahub/opendatahub-operator-controller-manager-799d55cdcd-82dxc" Jul 09 19:06:54.099938 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:54.099908 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8483e7be-9ac3-4f1a-834d-cc410f402b93-webhook-cert\") pod \"opendatahub-operator-controller-manager-799d55cdcd-82dxc\" (UID: \"8483e7be-9ac3-4f1a-834d-cc410f402b93\") " pod="opendatahub/opendatahub-operator-controller-manager-799d55cdcd-82dxc" Jul 09 19:06:54.100242 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:54.100220 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8483e7be-9ac3-4f1a-834d-cc410f402b93-manifests\") pod \"opendatahub-operator-controller-manager-799d55cdcd-82dxc\" (UID: \"8483e7be-9ac3-4f1a-834d-cc410f402b93\") " pod="opendatahub/opendatahub-operator-controller-manager-799d55cdcd-82dxc" Jul 09 19:06:54.102175 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:54.102142 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8483e7be-9ac3-4f1a-834d-cc410f402b93-tmp\") pod \"opendatahub-operator-controller-manager-799d55cdcd-82dxc\" (UID: \"8483e7be-9ac3-4f1a-834d-cc410f402b93\") " pod="opendatahub/opendatahub-operator-controller-manager-799d55cdcd-82dxc" Jul 09 19:06:54.102278 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:54.102234 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8483e7be-9ac3-4f1a-834d-cc410f402b93-apiservice-cert\") pod \"opendatahub-operator-controller-manager-799d55cdcd-82dxc\" (UID: \"8483e7be-9ac3-4f1a-834d-cc410f402b93\") " pod="opendatahub/opendatahub-operator-controller-manager-799d55cdcd-82dxc" Jul 09 19:06:54.102350 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:54.102332 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8483e7be-9ac3-4f1a-834d-cc410f402b93-webhook-cert\") pod \"opendatahub-operator-controller-manager-799d55cdcd-82dxc\" (UID: \"8483e7be-9ac3-4f1a-834d-cc410f402b93\") " pod="opendatahub/opendatahub-operator-controller-manager-799d55cdcd-82dxc" Jul 09 19:06:54.111156 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:54.111136 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqrm5\" (UniqueName: \"kubernetes.io/projected/8483e7be-9ac3-4f1a-834d-cc410f402b93-kube-api-access-nqrm5\") pod \"opendatahub-operator-controller-manager-799d55cdcd-82dxc\" (UID: \"8483e7be-9ac3-4f1a-834d-cc410f402b93\") " pod="opendatahub/opendatahub-operator-controller-manager-799d55cdcd-82dxc" Jul 09 19:06:54.231154 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:54.231112 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-799d55cdcd-82dxc" Jul 09 19:06:54.403671 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:54.403645 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-799d55cdcd-82dxc"] Jul 09 19:06:54.407098 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:06:54.407063 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8483e7be_9ac3_4f1a_834d_cc410f402b93.slice/crio-e5d61b42ac51d085c053038422e09dbeb648dc914fc90aabdd1bd9626f6851cc WatchSource:0}: Error finding container e5d61b42ac51d085c053038422e09dbeb648dc914fc90aabdd1bd9626f6851cc: Status 404 returned error can't find the container with id e5d61b42ac51d085c053038422e09dbeb648dc914fc90aabdd1bd9626f6851cc Jul 09 19:06:54.955778 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:54.955740 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-799d55cdcd-82dxc" event={"ID":"8483e7be-9ac3-4f1a-834d-cc410f402b93","Type":"ContainerStarted","Data":"e5d61b42ac51d085c053038422e09dbeb648dc914fc90aabdd1bd9626f6851cc"} Jul 09 19:06:57.972017 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:57.971978 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-799d55cdcd-82dxc" event={"ID":"8483e7be-9ac3-4f1a-834d-cc410f402b93","Type":"ContainerStarted","Data":"0b242d15e0ca5276d1de4821d42232f732c82d220398276b92ced742038ac7a3"} Jul 09 19:06:58.977525 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:58.977494 2566 generic.go:356] "Generic (PLEG): container finished" podID="8483e7be-9ac3-4f1a-834d-cc410f402b93" containerID="0b242d15e0ca5276d1de4821d42232f732c82d220398276b92ced742038ac7a3" exitCode=0 Jul 09 19:06:58.977926 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:58.977589 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-799d55cdcd-82dxc" event={"ID":"8483e7be-9ac3-4f1a-834d-cc410f402b93","Type":"ContainerDied","Data":"0b242d15e0ca5276d1de4821d42232f732c82d220398276b92ced742038ac7a3"} Jul 09 19:06:59.983688 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:59.983648 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-799d55cdcd-82dxc" event={"ID":"8483e7be-9ac3-4f1a-834d-cc410f402b93","Type":"ContainerStarted","Data":"c49b7ad751a8362b3d80a2c67eec61906a537ffd40fc63c5a942d7ecfdab1dc8"} Jul 09 19:06:59.984057 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:06:59.983733 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-799d55cdcd-82dxc" Jul 09 19:07:00.008793 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:00.008748 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-799d55cdcd-82dxc" podStartSLOduration=2.588825337 podStartE2EDuration="7.008733369s" totalImagesPullingTime="3.416103807s" totalInitContainerRuntime="1.003804225s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:06:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:07:00.00627764 +0000 UTC m=+489.514057405" watchObservedRunningTime="2026-07-09 19:07:00.008733369 +0000 UTC m=+489.516513144" Jul 09 19:07:10.991102 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:10.991071 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-799d55cdcd-82dxc" Jul 09 19:07:13.492533 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:13.492498 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-db49944d8-vqbn4"] Jul 09 19:07:13.495479 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:13.495462 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-db49944d8-vqbn4" Jul 09 19:07:13.498810 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:13.498789 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 09 19:07:13.498926 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:13.498839 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 09 19:07:13.500272 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:13.500249 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-44rlt\"" Jul 09 19:07:13.500329 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:13.500299 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 09 19:07:13.500329 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:13.500313 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 09 19:07:13.506405 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:13.506387 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-db49944d8-vqbn4"] Jul 09 19:07:13.560858 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:13.560825 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a42cd05-1cb0-4b3b-9140-76554fdc5202-tmp\") pod \"kube-auth-proxy-db49944d8-vqbn4\" (UID: \"9a42cd05-1cb0-4b3b-9140-76554fdc5202\") " pod="openshift-ingress/kube-auth-proxy-db49944d8-vqbn4" Jul 09 19:07:13.560858 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:13.560862 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gqc2\" (UniqueName: \"kubernetes.io/projected/9a42cd05-1cb0-4b3b-9140-76554fdc5202-kube-api-access-7gqc2\") pod \"kube-auth-proxy-db49944d8-vqbn4\" (UID: \"9a42cd05-1cb0-4b3b-9140-76554fdc5202\") " pod="openshift-ingress/kube-auth-proxy-db49944d8-vqbn4" Jul 09 19:07:13.561072 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:13.560963 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a42cd05-1cb0-4b3b-9140-76554fdc5202-tls-certs\") pod \"kube-auth-proxy-db49944d8-vqbn4\" (UID: \"9a42cd05-1cb0-4b3b-9140-76554fdc5202\") " pod="openshift-ingress/kube-auth-proxy-db49944d8-vqbn4" Jul 09 19:07:13.662231 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:13.662126 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a42cd05-1cb0-4b3b-9140-76554fdc5202-tls-certs\") pod \"kube-auth-proxy-db49944d8-vqbn4\" (UID: \"9a42cd05-1cb0-4b3b-9140-76554fdc5202\") " pod="openshift-ingress/kube-auth-proxy-db49944d8-vqbn4" Jul 09 19:07:13.662231 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:07:13.662210 2566 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 09 19:07:13.662461 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:13.662281 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a42cd05-1cb0-4b3b-9140-76554fdc5202-tmp\") pod \"kube-auth-proxy-db49944d8-vqbn4\" (UID: \"9a42cd05-1cb0-4b3b-9140-76554fdc5202\") " pod="openshift-ingress/kube-auth-proxy-db49944d8-vqbn4" Jul 09 19:07:13.662461 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:07:13.662297 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a42cd05-1cb0-4b3b-9140-76554fdc5202-tls-certs podName:9a42cd05-1cb0-4b3b-9140-76554fdc5202 nodeName:}" failed. No retries permitted until 2026-07-09 19:07:14.162275884 +0000 UTC m=+503.670055649 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/9a42cd05-1cb0-4b3b-9140-76554fdc5202-tls-certs") pod "kube-auth-proxy-db49944d8-vqbn4" (UID: "9a42cd05-1cb0-4b3b-9140-76554fdc5202") : secret "kube-auth-proxy-tls" not found Jul 09 19:07:13.662461 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:13.662337 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gqc2\" (UniqueName: \"kubernetes.io/projected/9a42cd05-1cb0-4b3b-9140-76554fdc5202-kube-api-access-7gqc2\") pod \"kube-auth-proxy-db49944d8-vqbn4\" (UID: \"9a42cd05-1cb0-4b3b-9140-76554fdc5202\") " pod="openshift-ingress/kube-auth-proxy-db49944d8-vqbn4" Jul 09 19:07:13.664478 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:13.664458 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a42cd05-1cb0-4b3b-9140-76554fdc5202-tmp\") pod \"kube-auth-proxy-db49944d8-vqbn4\" (UID: \"9a42cd05-1cb0-4b3b-9140-76554fdc5202\") " pod="openshift-ingress/kube-auth-proxy-db49944d8-vqbn4" Jul 09 19:07:13.673183 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:13.673154 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gqc2\" (UniqueName: \"kubernetes.io/projected/9a42cd05-1cb0-4b3b-9140-76554fdc5202-kube-api-access-7gqc2\") pod \"kube-auth-proxy-db49944d8-vqbn4\" (UID: \"9a42cd05-1cb0-4b3b-9140-76554fdc5202\") " pod="openshift-ingress/kube-auth-proxy-db49944d8-vqbn4" Jul 09 19:07:13.738483 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:13.738451 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-wpdz8"] Jul 09 19:07:13.741675 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:13.741655 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-wpdz8" Jul 09 19:07:13.744832 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:13.744772 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 09 19:07:13.744832 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:13.744793 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-gldf6\"" Jul 09 19:07:13.752999 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:13.752981 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-wpdz8"] Jul 09 19:07:13.864285 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:13.864256 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr2ck\" (UniqueName: \"kubernetes.io/projected/f9b48c49-05c5-4319-80d3-5b827d22c1a0-kube-api-access-zr2ck\") pod \"odh-model-controller-7998c67dc4-wpdz8\" (UID: \"f9b48c49-05c5-4319-80d3-5b827d22c1a0\") " pod="opendatahub/odh-model-controller-7998c67dc4-wpdz8" Jul 09 19:07:13.864445 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:13.864351 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9b48c49-05c5-4319-80d3-5b827d22c1a0-cert\") pod \"odh-model-controller-7998c67dc4-wpdz8\" (UID: \"f9b48c49-05c5-4319-80d3-5b827d22c1a0\") " pod="opendatahub/odh-model-controller-7998c67dc4-wpdz8" Jul 09 19:07:13.964898 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:13.964866 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9b48c49-05c5-4319-80d3-5b827d22c1a0-cert\") pod \"odh-model-controller-7998c67dc4-wpdz8\" (UID: \"f9b48c49-05c5-4319-80d3-5b827d22c1a0\") " pod="opendatahub/odh-model-controller-7998c67dc4-wpdz8" Jul 09 19:07:13.965058 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:13.964933 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr2ck\" (UniqueName: \"kubernetes.io/projected/f9b48c49-05c5-4319-80d3-5b827d22c1a0-kube-api-access-zr2ck\") pod \"odh-model-controller-7998c67dc4-wpdz8\" (UID: \"f9b48c49-05c5-4319-80d3-5b827d22c1a0\") " pod="opendatahub/odh-model-controller-7998c67dc4-wpdz8" Jul 09 19:07:13.965058 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:07:13.965018 2566 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 09 19:07:13.965148 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:07:13.965085 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9b48c49-05c5-4319-80d3-5b827d22c1a0-cert podName:f9b48c49-05c5-4319-80d3-5b827d22c1a0 nodeName:}" failed. No retries permitted until 2026-07-09 19:07:14.465067405 +0000 UTC m=+503.972847164 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f9b48c49-05c5-4319-80d3-5b827d22c1a0-cert") pod "odh-model-controller-7998c67dc4-wpdz8" (UID: "f9b48c49-05c5-4319-80d3-5b827d22c1a0") : secret "odh-model-controller-webhook-cert" not found Jul 09 19:07:13.984949 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:13.984914 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr2ck\" (UniqueName: \"kubernetes.io/projected/f9b48c49-05c5-4319-80d3-5b827d22c1a0-kube-api-access-zr2ck\") pod \"odh-model-controller-7998c67dc4-wpdz8\" (UID: \"f9b48c49-05c5-4319-80d3-5b827d22c1a0\") " pod="opendatahub/odh-model-controller-7998c67dc4-wpdz8" Jul 09 19:07:14.167002 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:14.166916 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a42cd05-1cb0-4b3b-9140-76554fdc5202-tls-certs\") pod \"kube-auth-proxy-db49944d8-vqbn4\" (UID: \"9a42cd05-1cb0-4b3b-9140-76554fdc5202\") " pod="openshift-ingress/kube-auth-proxy-db49944d8-vqbn4" Jul 09 19:07:14.169343 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:14.169314 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a42cd05-1cb0-4b3b-9140-76554fdc5202-tls-certs\") pod \"kube-auth-proxy-db49944d8-vqbn4\" (UID: \"9a42cd05-1cb0-4b3b-9140-76554fdc5202\") " pod="openshift-ingress/kube-auth-proxy-db49944d8-vqbn4" Jul 09 19:07:14.406934 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:14.406904 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-db49944d8-vqbn4" Jul 09 19:07:14.469514 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:14.469412 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9b48c49-05c5-4319-80d3-5b827d22c1a0-cert\") pod \"odh-model-controller-7998c67dc4-wpdz8\" (UID: \"f9b48c49-05c5-4319-80d3-5b827d22c1a0\") " pod="opendatahub/odh-model-controller-7998c67dc4-wpdz8" Jul 09 19:07:14.472362 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:14.472308 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9b48c49-05c5-4319-80d3-5b827d22c1a0-cert\") pod \"odh-model-controller-7998c67dc4-wpdz8\" (UID: \"f9b48c49-05c5-4319-80d3-5b827d22c1a0\") " pod="opendatahub/odh-model-controller-7998c67dc4-wpdz8" Jul 09 19:07:14.540378 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:14.540354 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-db49944d8-vqbn4"] Jul 09 19:07:14.542910 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:07:14.542874 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a42cd05_1cb0_4b3b_9140_76554fdc5202.slice/crio-5908a06d4c81b4540a1392e3d82a1a10595f19dd06c6ac0b0718e93e2193e17d WatchSource:0}: Error finding container 5908a06d4c81b4540a1392e3d82a1a10595f19dd06c6ac0b0718e93e2193e17d: Status 404 returned error can't find the container with id 5908a06d4c81b4540a1392e3d82a1a10595f19dd06c6ac0b0718e93e2193e17d Jul 09 19:07:14.556997 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:14.556966 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-28d7z"] Jul 09 19:07:14.561253 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:14.561235 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-28d7z" Jul 09 19:07:14.567297 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:14.567260 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 09 19:07:14.567528 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:14.567445 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 09 19:07:14.567528 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:14.567445 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 09 19:07:14.567528 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:14.567509 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-76vfb\"" Jul 09 19:07:14.567975 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:14.567757 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 09 19:07:14.577688 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:14.577668 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-28d7z"] Jul 09 19:07:14.652182 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:14.652149 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-wpdz8" Jul 09 19:07:14.671233 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:14.671164 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9hrj\" (UniqueName: \"kubernetes.io/projected/a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc-kube-api-access-h9hrj\") pod \"istiod-openshift-gateway-7574459656-28d7z\" (UID: \"a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-28d7z" Jul 09 19:07:14.671233 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:14.671211 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc-local-certs\") pod \"istiod-openshift-gateway-7574459656-28d7z\" (UID: \"a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-28d7z" Jul 09 19:07:14.671399 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:14.671255 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc-istio-token\") pod \"istiod-openshift-gateway-7574459656-28d7z\" (UID: \"a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-28d7z" Jul 09 19:07:14.671399 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:14.671283 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-28d7z\" (UID: \"a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-28d7z" Jul 09 19:07:14.671399 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:14.671309 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-28d7z\" (UID: \"a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-28d7z" Jul 09 19:07:14.671544 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:14.671404 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc-cacerts\") pod \"istiod-openshift-gateway-7574459656-28d7z\" (UID: \"a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-28d7z" Jul 09 19:07:14.671544 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:14.671447 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-28d7z\" (UID: \"a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-28d7z" Jul 09 19:07:14.772436 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:14.772397 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9hrj\" (UniqueName: \"kubernetes.io/projected/a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc-kube-api-access-h9hrj\") pod \"istiod-openshift-gateway-7574459656-28d7z\" (UID: \"a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-28d7z" Jul 09 19:07:14.772436 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:14.772434 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc-local-certs\") pod \"istiod-openshift-gateway-7574459656-28d7z\" (UID: \"a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-28d7z" Jul 09 19:07:14.772666 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:14.772560 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc-istio-token\") pod \"istiod-openshift-gateway-7574459656-28d7z\" (UID: \"a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-28d7z" Jul 09 19:07:14.772666 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:14.772598 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-28d7z\" (UID: \"a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-28d7z" Jul 09 19:07:14.772666 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:14.772648 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-28d7z\" (UID: \"a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-28d7z" Jul 09 19:07:14.772841 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:14.772809 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc-cacerts\") pod \"istiod-openshift-gateway-7574459656-28d7z\" (UID: \"a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-28d7z" Jul 09 19:07:14.772893 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:14.772870 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-28d7z\" (UID: \"a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-28d7z" Jul 09 19:07:14.773341 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:14.773311 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-28d7z\" (UID: \"a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-28d7z" Jul 09 19:07:14.774731 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:14.774705 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc-local-certs\") pod \"istiod-openshift-gateway-7574459656-28d7z\" (UID: \"a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-28d7z" Jul 09 19:07:14.774921 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:14.774901 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc-cacerts\") pod \"istiod-openshift-gateway-7574459656-28d7z\" (UID: \"a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-28d7z" Jul 09 19:07:14.775000 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:14.774986 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-28d7z\" (UID: \"a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-28d7z" Jul 09 19:07:14.775794 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:14.775778 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-28d7z\" (UID: \"a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-28d7z" Jul 09 19:07:14.796025 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:14.795996 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc-istio-token\") pod \"istiod-openshift-gateway-7574459656-28d7z\" (UID: \"a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-28d7z" Jul 09 19:07:14.801097 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:14.801064 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9hrj\" (UniqueName: \"kubernetes.io/projected/a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc-kube-api-access-h9hrj\") pod \"istiod-openshift-gateway-7574459656-28d7z\" (UID: \"a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-28d7z" Jul 09 19:07:14.809967 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:14.809944 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-wpdz8"] Jul 09 19:07:14.812817 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:07:14.812795 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9b48c49_05c5_4319_80d3_5b827d22c1a0.slice/crio-8e8381e71caeb4cde0dd3f6f23d6c67b83e98e64a083aa47995f4e2012345cc2 WatchSource:0}: Error finding container 8e8381e71caeb4cde0dd3f6f23d6c67b83e98e64a083aa47995f4e2012345cc2: Status 404 returned error can't find the container with id 8e8381e71caeb4cde0dd3f6f23d6c67b83e98e64a083aa47995f4e2012345cc2 Jul 09 19:07:14.870861 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:14.870834 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-28d7z" Jul 09 19:07:15.006586 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:15.006560 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-28d7z"] Jul 09 19:07:15.009122 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:07:15.009098 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1a5fd99_cb3b_489c_82ff_0649ccfbc0cc.slice/crio-6a81493d91ed6936098e307b08ec58bd42d94e6ef746816d6eae842d103b5cac WatchSource:0}: Error finding container 6a81493d91ed6936098e307b08ec58bd42d94e6ef746816d6eae842d103b5cac: Status 404 returned error can't find the container with id 6a81493d91ed6936098e307b08ec58bd42d94e6ef746816d6eae842d103b5cac Jul 09 19:07:15.060052 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:15.060018 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-28d7z" event={"ID":"a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc","Type":"ContainerStarted","Data":"6a81493d91ed6936098e307b08ec58bd42d94e6ef746816d6eae842d103b5cac"} Jul 09 19:07:15.060052 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:15.060065 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-db49944d8-vqbn4" event={"ID":"9a42cd05-1cb0-4b3b-9140-76554fdc5202","Type":"ContainerStarted","Data":"5908a06d4c81b4540a1392e3d82a1a10595f19dd06c6ac0b0718e93e2193e17d"} Jul 09 19:07:15.060685 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:15.060661 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-wpdz8" event={"ID":"f9b48c49-05c5-4319-80d3-5b827d22c1a0","Type":"ContainerStarted","Data":"8e8381e71caeb4cde0dd3f6f23d6c67b83e98e64a083aa47995f4e2012345cc2"} Jul 09 19:07:19.084731 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:19.084693 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-db49944d8-vqbn4" event={"ID":"9a42cd05-1cb0-4b3b-9140-76554fdc5202","Type":"ContainerStarted","Data":"7294b107b27466564703c8d2c21acaad021cb1024635fa5bf9f505eeb00e1293"} Jul 09 19:07:19.086380 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:19.086353 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-wpdz8" event={"ID":"f9b48c49-05c5-4319-80d3-5b827d22c1a0","Type":"ContainerStarted","Data":"6b78b6ef1b6644ecb2925fadfffd2dce10bd021345e49e6d478107e53bd1ff7d"} Jul 09 19:07:19.086527 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:19.086420 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-wpdz8" Jul 09 19:07:19.104137 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:19.104094 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-db49944d8-vqbn4" podStartSLOduration=1.968560685 podStartE2EDuration="6.104082697s" totalImagesPullingTime="4.135522012s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:07:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:07:19.102087449 +0000 UTC m=+508.609867223" watchObservedRunningTime="2026-07-09 19:07:19.104082697 +0000 UTC m=+508.611862473" Jul 09 19:07:19.121491 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:19.121429 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-wpdz8" podStartSLOduration=2.255581916 podStartE2EDuration="6.121407217s" totalImagesPullingTime="3.865825301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:07:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:07:19.119926911 +0000 UTC m=+508.627706685" watchObservedRunningTime="2026-07-09 19:07:19.121407217 +0000 UTC m=+508.629186992" Jul 09 19:07:21.017682 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:21.017646 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:07:21.017998 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:21.017721 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:07:21.103921 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:21.103887 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-28d7z" event={"ID":"a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc","Type":"ContainerStarted","Data":"4f65897f2e9ae0db8b27707e00f2756564733060db5da1336cc58a70dc9f169f"} Jul 09 19:07:21.104069 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:21.104036 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-28d7z" Jul 09 19:07:21.145059 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:21.145001 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-28d7z" podStartSLOduration=1.13880217 podStartE2EDuration="7.144981102s" totalImagesPullingTime="6.006178932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:07:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:07:21.142308681 +0000 UTC m=+510.650088455" watchObservedRunningTime="2026-07-09 19:07:21.144981102 +0000 UTC m=+510.652760877" Jul 09 19:07:22.110717 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:22.110683 2566 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-28d7z container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 09 19:07:22.111099 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:22.110738 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-28d7z" podUID="a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 19:07:22.612012 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:22.611979 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4z4jll"] Jul 09 19:07:22.615256 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:22.615240 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4z4jll" Jul 09 19:07:22.623992 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:22.623974 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-vsj7z\"" Jul 09 19:07:22.637328 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:22.637299 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4z4jll"] Jul 09 19:07:22.749019 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:22.748981 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a05d1fbf-cb86-4936-99c7-fc3811c2192a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4z4jll\" (UID: \"a05d1fbf-cb86-4936-99c7-fc3811c2192a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4z4jll" Jul 09 19:07:22.749196 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:22.749026 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a05d1fbf-cb86-4936-99c7-fc3811c2192a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4z4jll\" (UID: \"a05d1fbf-cb86-4936-99c7-fc3811c2192a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4z4jll" Jul 09 19:07:22.749196 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:22.749091 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a05d1fbf-cb86-4936-99c7-fc3811c2192a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4z4jll\" (UID: \"a05d1fbf-cb86-4936-99c7-fc3811c2192a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4z4jll" Jul 09 19:07:22.749196 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:22.749136 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a05d1fbf-cb86-4936-99c7-fc3811c2192a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4z4jll\" (UID: \"a05d1fbf-cb86-4936-99c7-fc3811c2192a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4z4jll" Jul 09 19:07:22.749196 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:22.749160 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a05d1fbf-cb86-4936-99c7-fc3811c2192a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4z4jll\" (UID: \"a05d1fbf-cb86-4936-99c7-fc3811c2192a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4z4jll" Jul 09 19:07:22.749353 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:22.749233 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbdh7\" (UniqueName: \"kubernetes.io/projected/a05d1fbf-cb86-4936-99c7-fc3811c2192a-kube-api-access-pbdh7\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4z4jll\" (UID: \"a05d1fbf-cb86-4936-99c7-fc3811c2192a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4z4jll" Jul 09 19:07:22.749353 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:22.749267 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a05d1fbf-cb86-4936-99c7-fc3811c2192a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4z4jll\" (UID: \"a05d1fbf-cb86-4936-99c7-fc3811c2192a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4z4jll" Jul 09 19:07:22.749353 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:22.749299 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a05d1fbf-cb86-4936-99c7-fc3811c2192a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4z4jll\" (UID: \"a05d1fbf-cb86-4936-99c7-fc3811c2192a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4z4jll" Jul 09 19:07:22.749353 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:22.749343 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a05d1fbf-cb86-4936-99c7-fc3811c2192a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4z4jll\" (UID: \"a05d1fbf-cb86-4936-99c7-fc3811c2192a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4z4jll" Jul 09 19:07:22.850352 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:22.850322 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a05d1fbf-cb86-4936-99c7-fc3811c2192a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4z4jll\" (UID: \"a05d1fbf-cb86-4936-99c7-fc3811c2192a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4z4jll" Jul 09 19:07:22.850547 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:22.850365 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a05d1fbf-cb86-4936-99c7-fc3811c2192a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4z4jll\" (UID: \"a05d1fbf-cb86-4936-99c7-fc3811c2192a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4z4jll" Jul 09 19:07:22.850547 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:22.850386 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a05d1fbf-cb86-4936-99c7-fc3811c2192a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4z4jll\" (UID: \"a05d1fbf-cb86-4936-99c7-fc3811c2192a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4z4jll" Jul 09 19:07:22.850547 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:22.850414 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a05d1fbf-cb86-4936-99c7-fc3811c2192a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4z4jll\" (UID: \"a05d1fbf-cb86-4936-99c7-fc3811c2192a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4z4jll" Jul 09 19:07:22.850547 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:22.850446 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a05d1fbf-cb86-4936-99c7-fc3811c2192a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4z4jll\" (UID: \"a05d1fbf-cb86-4936-99c7-fc3811c2192a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4z4jll" Jul 09 19:07:22.850547 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:22.850473 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a05d1fbf-cb86-4936-99c7-fc3811c2192a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4z4jll\" (UID: \"a05d1fbf-cb86-4936-99c7-fc3811c2192a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4z4jll" Jul 09 19:07:22.850547 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:22.850512 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbdh7\" (UniqueName: \"kubernetes.io/projected/a05d1fbf-cb86-4936-99c7-fc3811c2192a-kube-api-access-pbdh7\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4z4jll\" (UID: \"a05d1fbf-cb86-4936-99c7-fc3811c2192a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4z4jll" Jul 09 19:07:22.850547 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:22.850545 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a05d1fbf-cb86-4936-99c7-fc3811c2192a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4z4jll\" (UID: \"a05d1fbf-cb86-4936-99c7-fc3811c2192a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4z4jll" Jul 09 19:07:22.850942 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:22.850574 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a05d1fbf-cb86-4936-99c7-fc3811c2192a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4z4jll\" (UID: \"a05d1fbf-cb86-4936-99c7-fc3811c2192a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4z4jll" Jul 09 19:07:22.850942 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:22.850841 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a05d1fbf-cb86-4936-99c7-fc3811c2192a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4z4jll\" (UID: \"a05d1fbf-cb86-4936-99c7-fc3811c2192a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4z4jll" Jul 09 19:07:22.850942 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:22.850848 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a05d1fbf-cb86-4936-99c7-fc3811c2192a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4z4jll\" (UID: \"a05d1fbf-cb86-4936-99c7-fc3811c2192a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4z4jll" Jul 09 19:07:22.850942 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:22.850919 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a05d1fbf-cb86-4936-99c7-fc3811c2192a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4z4jll\" (UID: \"a05d1fbf-cb86-4936-99c7-fc3811c2192a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4z4jll" Jul 09 19:07:22.851155 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:22.851069 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a05d1fbf-cb86-4936-99c7-fc3811c2192a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4z4jll\" (UID: \"a05d1fbf-cb86-4936-99c7-fc3811c2192a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4z4jll" Jul 09 19:07:22.851323 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:22.851299 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a05d1fbf-cb86-4936-99c7-fc3811c2192a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4z4jll\" (UID: \"a05d1fbf-cb86-4936-99c7-fc3811c2192a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4z4jll" Jul 09 19:07:22.852691 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:22.852672 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a05d1fbf-cb86-4936-99c7-fc3811c2192a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4z4jll\" (UID: \"a05d1fbf-cb86-4936-99c7-fc3811c2192a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4z4jll" Jul 09 19:07:22.852973 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:22.852953 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a05d1fbf-cb86-4936-99c7-fc3811c2192a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4z4jll\" (UID: \"a05d1fbf-cb86-4936-99c7-fc3811c2192a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4z4jll" Jul 09 19:07:22.858953 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:22.858927 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbdh7\" (UniqueName: \"kubernetes.io/projected/a05d1fbf-cb86-4936-99c7-fc3811c2192a-kube-api-access-pbdh7\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4z4jll\" (UID: \"a05d1fbf-cb86-4936-99c7-fc3811c2192a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4z4jll" Jul 09 19:07:22.859059 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:22.859015 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a05d1fbf-cb86-4936-99c7-fc3811c2192a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4z4jll\" (UID: \"a05d1fbf-cb86-4936-99c7-fc3811c2192a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4z4jll" Jul 09 19:07:22.925485 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:22.925412 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4z4jll" Jul 09 19:07:23.090081 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:23.090055 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4z4jll"] Jul 09 19:07:23.091628 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:07:23.091588 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda05d1fbf_cb86_4936_99c7_fc3811c2192a.slice/crio-e7c9464061ab5bd81ce1a9402c81b1b6bb9f6bf2855bd71c0a0f6c9029e2d2ac WatchSource:0}: Error finding container e7c9464061ab5bd81ce1a9402c81b1b6bb9f6bf2855bd71c0a0f6c9029e2d2ac: Status 404 returned error can't find the container with id e7c9464061ab5bd81ce1a9402c81b1b6bb9f6bf2855bd71c0a0f6c9029e2d2ac Jul 09 19:07:23.115161 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:23.115128 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4z4jll" event={"ID":"a05d1fbf-cb86-4936-99c7-fc3811c2192a","Type":"ContainerStarted","Data":"e7c9464061ab5bd81ce1a9402c81b1b6bb9f6bf2855bd71c0a0f6c9029e2d2ac"} Jul 09 19:07:23.775312 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:23.775280 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-6sn6c"] Jul 09 19:07:23.779793 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:23.779772 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-6sn6c" Jul 09 19:07:23.782546 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:23.782528 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 09 19:07:23.783015 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:23.782918 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-fkkjf\"" Jul 09 19:07:23.788267 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:23.788248 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-6sn6c"] Jul 09 19:07:23.794959 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:23.794940 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-cjd52"] Jul 09 19:07:23.798175 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:23.798160 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cjd52" Jul 09 19:07:23.801155 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:23.801136 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 09 19:07:23.801535 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:23.801515 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-5kplm\"" Jul 09 19:07:23.808570 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:23.808546 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-cjd52"] Jul 09 19:07:23.961400 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:23.961370 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/adc02e16-794f-4f41-ad98-1ca745d50b62-cert\") pod \"kserve-controller-manager-d6f4b65b8-6sn6c\" (UID: \"adc02e16-794f-4f41-ad98-1ca745d50b62\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-6sn6c" Jul 09 19:07:23.961574 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:23.961422 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr66h\" (UniqueName: \"kubernetes.io/projected/adc02e16-794f-4f41-ad98-1ca745d50b62-kube-api-access-hr66h\") pod \"kserve-controller-manager-d6f4b65b8-6sn6c\" (UID: \"adc02e16-794f-4f41-ad98-1ca745d50b62\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-6sn6c" Jul 09 19:07:23.961574 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:23.961444 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfg8x\" (UniqueName: \"kubernetes.io/projected/1c649895-8c6a-4f67-ae72-872f80af8f79-kube-api-access-dfg8x\") pod \"llmisvc-controller-manager-777c6bd7d5-cjd52\" (UID: \"1c649895-8c6a-4f67-ae72-872f80af8f79\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cjd52" Jul 09 19:07:23.961574 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:23.961530 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1c649895-8c6a-4f67-ae72-872f80af8f79-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-cjd52\" (UID: \"1c649895-8c6a-4f67-ae72-872f80af8f79\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cjd52" Jul 09 19:07:24.062883 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:24.062801 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1c649895-8c6a-4f67-ae72-872f80af8f79-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-cjd52\" (UID: \"1c649895-8c6a-4f67-ae72-872f80af8f79\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cjd52" Jul 09 19:07:24.062883 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:24.062847 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/adc02e16-794f-4f41-ad98-1ca745d50b62-cert\") pod \"kserve-controller-manager-d6f4b65b8-6sn6c\" (UID: \"adc02e16-794f-4f41-ad98-1ca745d50b62\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-6sn6c" Jul 09 19:07:24.063105 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:24.062906 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr66h\" (UniqueName: \"kubernetes.io/projected/adc02e16-794f-4f41-ad98-1ca745d50b62-kube-api-access-hr66h\") pod \"kserve-controller-manager-d6f4b65b8-6sn6c\" (UID: \"adc02e16-794f-4f41-ad98-1ca745d50b62\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-6sn6c" Jul 09 19:07:24.063105 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:24.062939 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfg8x\" (UniqueName: \"kubernetes.io/projected/1c649895-8c6a-4f67-ae72-872f80af8f79-kube-api-access-dfg8x\") pod \"llmisvc-controller-manager-777c6bd7d5-cjd52\" (UID: \"1c649895-8c6a-4f67-ae72-872f80af8f79\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cjd52" Jul 09 19:07:24.065222 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:24.065201 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/adc02e16-794f-4f41-ad98-1ca745d50b62-cert\") pod \"kserve-controller-manager-d6f4b65b8-6sn6c\" (UID: \"adc02e16-794f-4f41-ad98-1ca745d50b62\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-6sn6c" Jul 09 19:07:24.065326 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:24.065309 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1c649895-8c6a-4f67-ae72-872f80af8f79-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-cjd52\" (UID: \"1c649895-8c6a-4f67-ae72-872f80af8f79\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cjd52" Jul 09 19:07:24.083598 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:24.083575 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr66h\" (UniqueName: \"kubernetes.io/projected/adc02e16-794f-4f41-ad98-1ca745d50b62-kube-api-access-hr66h\") pod \"kserve-controller-manager-d6f4b65b8-6sn6c\" (UID: \"adc02e16-794f-4f41-ad98-1ca745d50b62\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-6sn6c" Jul 09 19:07:24.085085 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:24.085062 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfg8x\" (UniqueName: \"kubernetes.io/projected/1c649895-8c6a-4f67-ae72-872f80af8f79-kube-api-access-dfg8x\") pod \"llmisvc-controller-manager-777c6bd7d5-cjd52\" (UID: \"1c649895-8c6a-4f67-ae72-872f80af8f79\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cjd52" Jul 09 19:07:24.091279 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:24.091259 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-6sn6c" Jul 09 19:07:24.109700 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:24.109680 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cjd52" Jul 09 19:07:24.289683 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:24.289649 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-6sn6c"] Jul 09 19:07:24.292545 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:07:24.292505 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadc02e16_794f_4f41_ad98_1ca745d50b62.slice/crio-936d7eae55785b4ba63511a55fa0dcdc6906795571ec4711041efe303a0a4dbb WatchSource:0}: Error finding container 936d7eae55785b4ba63511a55fa0dcdc6906795571ec4711041efe303a0a4dbb: Status 404 returned error can't find the container with id 936d7eae55785b4ba63511a55fa0dcdc6906795571ec4711041efe303a0a4dbb Jul 09 19:07:24.303709 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:24.303685 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-cjd52"] Jul 09 19:07:24.305885 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:07:24.305831 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1c649895_8c6a_4f67_ae72_872f80af8f79.slice/crio-dd5426c8713ccd47cb3b699cd545d1795cb9d8a72d8b995a01df05e1c3d13843 WatchSource:0}: Error finding container dd5426c8713ccd47cb3b699cd545d1795cb9d8a72d8b995a01df05e1c3d13843: Status 404 returned error can't find the container with id dd5426c8713ccd47cb3b699cd545d1795cb9d8a72d8b995a01df05e1c3d13843 Jul 09 19:07:25.110153 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:25.110124 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-28d7z" Jul 09 19:07:25.129879 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:25.129786 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cjd52" event={"ID":"1c649895-8c6a-4f67-ae72-872f80af8f79","Type":"ContainerStarted","Data":"dd5426c8713ccd47cb3b699cd545d1795cb9d8a72d8b995a01df05e1c3d13843"} Jul 09 19:07:25.131469 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:25.131440 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-6sn6c" event={"ID":"adc02e16-794f-4f41-ad98-1ca745d50b62","Type":"ContainerStarted","Data":"936d7eae55785b4ba63511a55fa0dcdc6906795571ec4711041efe303a0a4dbb"} Jul 09 19:07:25.751202 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:25.751158 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-28d7z"] Jul 09 19:07:25.751664 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:25.751415 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-28d7z" podUID="a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc" containerName="discovery" containerID="cri-o://4f65897f2e9ae0db8b27707e00f2756564733060db5da1336cc58a70dc9f169f" gracePeriod=30 Jul 09 19:07:26.102548 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:26.102521 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-28d7z" Jul 09 19:07:26.142667 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:26.142631 2566 generic.go:356] "Generic (PLEG): container finished" podID="a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc" containerID="4f65897f2e9ae0db8b27707e00f2756564733060db5da1336cc58a70dc9f169f" exitCode=0 Jul 09 19:07:26.142853 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:26.142719 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-28d7z" event={"ID":"a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc","Type":"ContainerDied","Data":"4f65897f2e9ae0db8b27707e00f2756564733060db5da1336cc58a70dc9f169f"} Jul 09 19:07:26.142853 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:26.142732 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-28d7z" Jul 09 19:07:26.142853 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:26.142776 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-28d7z" event={"ID":"a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc","Type":"ContainerDied","Data":"6a81493d91ed6936098e307b08ec58bd42d94e6ef746816d6eae842d103b5cac"} Jul 09 19:07:26.142853 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:26.142808 2566 scope.go:117] "RemoveContainer" containerID="4f65897f2e9ae0db8b27707e00f2756564733060db5da1336cc58a70dc9f169f" Jul 09 19:07:26.154582 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:26.154556 2566 scope.go:117] "RemoveContainer" containerID="4f65897f2e9ae0db8b27707e00f2756564733060db5da1336cc58a70dc9f169f" Jul 09 19:07:26.155112 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:07:26.155082 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f65897f2e9ae0db8b27707e00f2756564733060db5da1336cc58a70dc9f169f\": container with ID starting with 4f65897f2e9ae0db8b27707e00f2756564733060db5da1336cc58a70dc9f169f not found: ID does not exist" containerID="4f65897f2e9ae0db8b27707e00f2756564733060db5da1336cc58a70dc9f169f" Jul 09 19:07:26.155207 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:26.155121 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f65897f2e9ae0db8b27707e00f2756564733060db5da1336cc58a70dc9f169f"} err="failed to get container status \"4f65897f2e9ae0db8b27707e00f2756564733060db5da1336cc58a70dc9f169f\": rpc error: code = NotFound desc = could not find container \"4f65897f2e9ae0db8b27707e00f2756564733060db5da1336cc58a70dc9f169f\": container with ID starting with 4f65897f2e9ae0db8b27707e00f2756564733060db5da1336cc58a70dc9f169f not found: ID does not exist" Jul 09 19:07:26.285263 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:26.285101 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc-local-certs\") pod \"a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc\" (UID: \"a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc\") " Jul 09 19:07:26.285263 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:26.285151 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc-istio-csr-dns-cert\") pod \"a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc\" (UID: \"a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc\") " Jul 09 19:07:26.285263 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:26.285205 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc-istio-kubeconfig\") pod \"a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc\" (UID: \"a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc\") " Jul 09 19:07:26.285263 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:26.285252 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc-istio-token\") pod \"a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc\" (UID: \"a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc\") " Jul 09 19:07:26.285555 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:26.285289 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc-istio-csr-ca-configmap\") pod \"a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc\" (UID: \"a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc\") " Jul 09 19:07:26.285555 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:26.285314 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc-cacerts\") pod \"a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc\" (UID: \"a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc\") " Jul 09 19:07:26.285555 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:26.285366 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9hrj\" (UniqueName: \"kubernetes.io/projected/a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc-kube-api-access-h9hrj\") pod \"a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc\" (UID: \"a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc\") " Jul 09 19:07:26.286135 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:26.286101 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc" (UID: "a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 19:07:26.287980 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:26.287954 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc" (UID: "a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 19:07:26.288117 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:26.288089 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc-local-certs" (OuterVolumeSpecName: "local-certs") pod "a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc" (UID: "a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:07:26.288224 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:26.288112 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc-kube-api-access-h9hrj" (OuterVolumeSpecName: "kube-api-access-h9hrj") pod "a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc" (UID: "a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc"). InnerVolumeSpecName "kube-api-access-h9hrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:07:26.288572 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:26.288547 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc-cacerts" (OuterVolumeSpecName: "cacerts") pod "a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc" (UID: "a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 19:07:26.288731 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:26.288713 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc" (UID: "a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 19:07:26.288797 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:26.288749 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc-istio-token" (OuterVolumeSpecName: "istio-token") pod "a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc" (UID: "a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:07:26.386808 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:26.386770 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9hrj\" (UniqueName: \"kubernetes.io/projected/a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc-kube-api-access-h9hrj\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:07:26.386808 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:26.386809 2566 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc-local-certs\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:07:26.387058 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:26.386823 2566 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc-istio-csr-dns-cert\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:07:26.387058 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:26.386837 2566 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc-istio-kubeconfig\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:07:26.387058 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:26.386851 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc-istio-token\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:07:26.387058 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:26.386865 2566 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc-istio-csr-ca-configmap\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:07:26.387058 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:26.386878 2566 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc-cacerts\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:07:26.472034 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:26.472002 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-28d7z"] Jul 09 19:07:26.474880 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:26.474830 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-28d7z"] Jul 09 19:07:27.058787 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:27.058748 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc" path="/var/lib/kubelet/pods/a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc/volumes" Jul 09 19:07:28.870802 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:28.870766 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:07:28.871210 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:28.870853 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:07:28.871210 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:28.870901 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:07:29.163498 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:29.163411 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cjd52" event={"ID":"1c649895-8c6a-4f67-ae72-872f80af8f79","Type":"ContainerStarted","Data":"75d9f140745e5a582f9d62e27fbc78881d82ccbc0670b19401aed0d9fc2defa6"} Jul 09 19:07:29.163659 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:29.163510 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cjd52" Jul 09 19:07:29.165100 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:29.165072 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-6sn6c" event={"ID":"adc02e16-794f-4f41-ad98-1ca745d50b62","Type":"ContainerStarted","Data":"2dd0562b7e638d4f2b7d753d29540e4f29fda5121bf482925e7cc161ed0accbf"} Jul 09 19:07:29.165322 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:29.165138 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-6sn6c" Jul 09 19:07:29.166644 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:29.166623 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4z4jll" event={"ID":"a05d1fbf-cb86-4936-99c7-fc3811c2192a","Type":"ContainerStarted","Data":"c2fc0744736a7e0b833093891f3a60f5be32b135690bce4449b267fbb3c65a38"} Jul 09 19:07:29.183628 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:29.183546 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cjd52" podStartSLOduration=1.6151899410000001 podStartE2EDuration="6.183527028s" totalImagesPullingTime="4.568337087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:07:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:07:29.183043754 +0000 UTC m=+518.690823551" watchObservedRunningTime="2026-07-09 19:07:29.183527028 +0000 UTC m=+518.691306802" Jul 09 19:07:29.205425 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:29.205365 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4z4jll" podStartSLOduration=1.4286552860000001 podStartE2EDuration="7.205345351s" totalImagesPullingTime="5.776690065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:07:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:07:29.204194544 +0000 UTC m=+518.711974646" watchObservedRunningTime="2026-07-09 19:07:29.205345351 +0000 UTC m=+518.713125127" Jul 09 19:07:29.221122 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:29.221064 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-6sn6c" podStartSLOduration=1.645424899 podStartE2EDuration="6.22104989s" totalImagesPullingTime="4.575624991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:07:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:07:29.219917768 +0000 UTC m=+518.727697541" watchObservedRunningTime="2026-07-09 19:07:29.22104989 +0000 UTC m=+518.728829663" Jul 09 19:07:29.925936 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:29.925902 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4z4jll" Jul 09 19:07:29.930733 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:29.930710 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4z4jll" Jul 09 19:07:30.094229 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:30.094201 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-wpdz8" Jul 09 19:07:30.172954 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:30.172925 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4z4jll" Jul 09 19:07:30.173939 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:07:30.173916 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4z4jll" Jul 09 19:08:00.174944 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:00.174908 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-cjd52" Jul 09 19:08:00.178116 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:00.178095 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-6sn6c" Jul 09 19:08:14.199850 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:14.199818 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2gswg"] Jul 09 19:08:14.200294 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:14.200156 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc" containerName="discovery" Jul 09 19:08:14.200294 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:14.200166 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc" containerName="discovery" Jul 09 19:08:14.200294 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:14.200233 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1a5fd99-cb3b-489c-82ff-0649ccfbc0cc" containerName="discovery" Jul 09 19:08:14.202187 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:14.202166 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2gswg" Jul 09 19:08:14.206678 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:14.206651 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 09 19:08:14.207055 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:14.207037 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-29f2n\"" Jul 09 19:08:14.208028 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:14.208008 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 09 19:08:14.208528 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:14.208495 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6r7r\" (UniqueName: \"kubernetes.io/projected/85b0845c-e274-4a8e-bdb5-cfac693da112-kube-api-access-c6r7r\") pod \"kuadrant-operator-catalog-2gswg\" (UID: \"85b0845c-e274-4a8e-bdb5-cfac693da112\") " pod="kuadrant-system/kuadrant-operator-catalog-2gswg" Jul 09 19:08:14.225286 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:14.225259 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2gswg"] Jul 09 19:08:14.309365 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:14.309329 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6r7r\" (UniqueName: \"kubernetes.io/projected/85b0845c-e274-4a8e-bdb5-cfac693da112-kube-api-access-c6r7r\") pod \"kuadrant-operator-catalog-2gswg\" (UID: \"85b0845c-e274-4a8e-bdb5-cfac693da112\") " pod="kuadrant-system/kuadrant-operator-catalog-2gswg" Jul 09 19:08:14.324960 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:14.324925 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6r7r\" (UniqueName: \"kubernetes.io/projected/85b0845c-e274-4a8e-bdb5-cfac693da112-kube-api-access-c6r7r\") pod \"kuadrant-operator-catalog-2gswg\" (UID: \"85b0845c-e274-4a8e-bdb5-cfac693da112\") " pod="kuadrant-system/kuadrant-operator-catalog-2gswg" Jul 09 19:08:14.512333 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:14.512240 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2gswg" Jul 09 19:08:14.549129 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:14.549102 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2gswg"] Jul 09 19:08:14.643065 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:14.643043 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2gswg"] Jul 09 19:08:14.645754 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:08:14.645727 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85b0845c_e274_4a8e_bdb5_cfac693da112.slice/crio-1abb155a1ca097e30ddf3d8ca553a24eeab08c2064c6802d7081d3d540c160ae WatchSource:0}: Error finding container 1abb155a1ca097e30ddf3d8ca553a24eeab08c2064c6802d7081d3d540c160ae: Status 404 returned error can't find the container with id 1abb155a1ca097e30ddf3d8ca553a24eeab08c2064c6802d7081d3d540c160ae Jul 09 19:08:14.756526 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:14.756496 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xxpcc"] Jul 09 19:08:14.759367 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:14.759351 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-xxpcc" Jul 09 19:08:14.767372 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:14.767348 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xxpcc"] Jul 09 19:08:14.814243 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:14.814214 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8swv5\" (UniqueName: \"kubernetes.io/projected/25daeb84-f792-4cb4-b20b-ae07589df438-kube-api-access-8swv5\") pod \"kuadrant-operator-catalog-xxpcc\" (UID: \"25daeb84-f792-4cb4-b20b-ae07589df438\") " pod="kuadrant-system/kuadrant-operator-catalog-xxpcc" Jul 09 19:08:14.915493 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:14.915460 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8swv5\" (UniqueName: \"kubernetes.io/projected/25daeb84-f792-4cb4-b20b-ae07589df438-kube-api-access-8swv5\") pod \"kuadrant-operator-catalog-xxpcc\" (UID: \"25daeb84-f792-4cb4-b20b-ae07589df438\") " pod="kuadrant-system/kuadrant-operator-catalog-xxpcc" Jul 09 19:08:14.924818 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:14.924789 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8swv5\" (UniqueName: \"kubernetes.io/projected/25daeb84-f792-4cb4-b20b-ae07589df438-kube-api-access-8swv5\") pod \"kuadrant-operator-catalog-xxpcc\" (UID: \"25daeb84-f792-4cb4-b20b-ae07589df438\") " pod="kuadrant-system/kuadrant-operator-catalog-xxpcc" Jul 09 19:08:15.069668 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:15.069575 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-xxpcc" Jul 09 19:08:15.200224 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:15.200195 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xxpcc"] Jul 09 19:08:15.204186 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:08:15.204157 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25daeb84_f792_4cb4_b20b_ae07589df438.slice/crio-92887093cdd76b637ba59442d6bc0992b0f0af81299033dfd20b907e96a2ee04 WatchSource:0}: Error finding container 92887093cdd76b637ba59442d6bc0992b0f0af81299033dfd20b907e96a2ee04: Status 404 returned error can't find the container with id 92887093cdd76b637ba59442d6bc0992b0f0af81299033dfd20b907e96a2ee04 Jul 09 19:08:15.378723 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:15.378643 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2gswg" event={"ID":"85b0845c-e274-4a8e-bdb5-cfac693da112","Type":"ContainerStarted","Data":"1abb155a1ca097e30ddf3d8ca553a24eeab08c2064c6802d7081d3d540c160ae"} Jul 09 19:08:15.379785 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:15.379765 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-xxpcc" event={"ID":"25daeb84-f792-4cb4-b20b-ae07589df438","Type":"ContainerStarted","Data":"92887093cdd76b637ba59442d6bc0992b0f0af81299033dfd20b907e96a2ee04"} Jul 09 19:08:17.391248 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:17.391135 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2gswg" event={"ID":"85b0845c-e274-4a8e-bdb5-cfac693da112","Type":"ContainerStarted","Data":"33c7134ba728e97f3fc75971aa9d3fbb550d52d5f1014b7dbf0e25870a0f12b2"} Jul 09 19:08:17.391248 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:17.391163 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-2gswg" podUID="85b0845c-e274-4a8e-bdb5-cfac693da112" containerName="registry-server" containerID="cri-o://33c7134ba728e97f3fc75971aa9d3fbb550d52d5f1014b7dbf0e25870a0f12b2" gracePeriod=2 Jul 09 19:08:17.392778 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:17.392756 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-xxpcc" event={"ID":"25daeb84-f792-4cb4-b20b-ae07589df438","Type":"ContainerStarted","Data":"29f863771bc0d79a2d376253654db1d43806ae0fcb4896c68ad14cda533d9448"} Jul 09 19:08:17.406898 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:17.406859 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-2gswg" podStartSLOduration=0.985324213 podStartE2EDuration="3.406844016s" totalImagesPullingTime="2.421519803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:08:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:08:17.406141096 +0000 UTC m=+566.913920871" watchObservedRunningTime="2026-07-09 19:08:17.406844016 +0000 UTC m=+566.914623789" Jul 09 19:08:17.434099 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:17.434054 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-xxpcc" podStartSLOduration=1.568264058 podStartE2EDuration="3.434030756s" totalImagesPullingTime="1.865766698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:08:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:08:17.433433472 +0000 UTC m=+566.941213246" watchObservedRunningTime="2026-07-09 19:08:17.434030756 +0000 UTC m=+566.941810530" Jul 09 19:08:17.636828 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:17.636802 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2gswg" Jul 09 19:08:17.739457 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:17.739425 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6r7r\" (UniqueName: \"kubernetes.io/projected/85b0845c-e274-4a8e-bdb5-cfac693da112-kube-api-access-c6r7r\") pod \"85b0845c-e274-4a8e-bdb5-cfac693da112\" (UID: \"85b0845c-e274-4a8e-bdb5-cfac693da112\") " Jul 09 19:08:17.741542 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:17.741515 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85b0845c-e274-4a8e-bdb5-cfac693da112-kube-api-access-c6r7r" (OuterVolumeSpecName: "kube-api-access-c6r7r") pod "85b0845c-e274-4a8e-bdb5-cfac693da112" (UID: "85b0845c-e274-4a8e-bdb5-cfac693da112"). InnerVolumeSpecName "kube-api-access-c6r7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:08:17.840072 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:17.840041 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6r7r\" (UniqueName: \"kubernetes.io/projected/85b0845c-e274-4a8e-bdb5-cfac693da112-kube-api-access-c6r7r\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:08:18.398158 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:18.398076 2566 generic.go:356] "Generic (PLEG): container finished" podID="85b0845c-e274-4a8e-bdb5-cfac693da112" containerID="33c7134ba728e97f3fc75971aa9d3fbb550d52d5f1014b7dbf0e25870a0f12b2" exitCode=0 Jul 09 19:08:18.398158 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:18.398130 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2gswg" Jul 09 19:08:18.398641 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:18.398163 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2gswg" event={"ID":"85b0845c-e274-4a8e-bdb5-cfac693da112","Type":"ContainerDied","Data":"33c7134ba728e97f3fc75971aa9d3fbb550d52d5f1014b7dbf0e25870a0f12b2"} Jul 09 19:08:18.398641 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:18.398232 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2gswg" event={"ID":"85b0845c-e274-4a8e-bdb5-cfac693da112","Type":"ContainerDied","Data":"1abb155a1ca097e30ddf3d8ca553a24eeab08c2064c6802d7081d3d540c160ae"} Jul 09 19:08:18.398641 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:18.398252 2566 scope.go:117] "RemoveContainer" containerID="33c7134ba728e97f3fc75971aa9d3fbb550d52d5f1014b7dbf0e25870a0f12b2" Jul 09 19:08:18.407365 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:18.407349 2566 scope.go:117] "RemoveContainer" containerID="33c7134ba728e97f3fc75971aa9d3fbb550d52d5f1014b7dbf0e25870a0f12b2" Jul 09 19:08:18.407717 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:08:18.407695 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33c7134ba728e97f3fc75971aa9d3fbb550d52d5f1014b7dbf0e25870a0f12b2\": container with ID starting with 33c7134ba728e97f3fc75971aa9d3fbb550d52d5f1014b7dbf0e25870a0f12b2 not found: ID does not exist" containerID="33c7134ba728e97f3fc75971aa9d3fbb550d52d5f1014b7dbf0e25870a0f12b2" Jul 09 19:08:18.407811 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:18.407725 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33c7134ba728e97f3fc75971aa9d3fbb550d52d5f1014b7dbf0e25870a0f12b2"} err="failed to get container status \"33c7134ba728e97f3fc75971aa9d3fbb550d52d5f1014b7dbf0e25870a0f12b2\": rpc error: code = NotFound desc = could not find container \"33c7134ba728e97f3fc75971aa9d3fbb550d52d5f1014b7dbf0e25870a0f12b2\": container with ID starting with 33c7134ba728e97f3fc75971aa9d3fbb550d52d5f1014b7dbf0e25870a0f12b2 not found: ID does not exist" Jul 09 19:08:18.418507 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:18.418481 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2gswg"] Jul 09 19:08:18.422975 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:18.422955 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2gswg"] Jul 09 19:08:19.057212 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:19.057171 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85b0845c-e274-4a8e-bdb5-cfac693da112" path="/var/lib/kubelet/pods/85b0845c-e274-4a8e-bdb5-cfac693da112/volumes" Jul 09 19:08:25.070657 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:25.070601 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-xxpcc" Jul 09 19:08:25.071247 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:25.070686 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-xxpcc" Jul 09 19:08:25.092660 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:25.092637 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-xxpcc" Jul 09 19:08:25.454400 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:25.454375 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-xxpcc" Jul 09 19:08:29.550555 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:29.550519 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xvp5x"] Jul 09 19:08:29.550937 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:29.550887 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85b0845c-e274-4a8e-bdb5-cfac693da112" containerName="registry-server" Jul 09 19:08:29.550937 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:29.550900 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="85b0845c-e274-4a8e-bdb5-cfac693da112" containerName="registry-server" Jul 09 19:08:29.551009 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:29.550973 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="85b0845c-e274-4a8e-bdb5-cfac693da112" containerName="registry-server" Jul 09 19:08:29.555730 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:29.555713 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xvp5x" Jul 09 19:08:29.558538 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:29.558522 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-4wjtj\"" Jul 09 19:08:29.562748 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:29.562726 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xvp5x"] Jul 09 19:08:29.626235 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:29.626195 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbzv8\" (UniqueName: \"kubernetes.io/projected/bbfbf153-e930-4709-bb2a-f5d20eb37cc5-kube-api-access-xbzv8\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xvp5x\" (UID: \"bbfbf153-e930-4709-bb2a-f5d20eb37cc5\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xvp5x" Jul 09 19:08:29.626235 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:29.626238 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bbfbf153-e930-4709-bb2a-f5d20eb37cc5-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xvp5x\" (UID: \"bbfbf153-e930-4709-bb2a-f5d20eb37cc5\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xvp5x" Jul 09 19:08:29.626482 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:29.626320 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bbfbf153-e930-4709-bb2a-f5d20eb37cc5-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xvp5x\" (UID: \"bbfbf153-e930-4709-bb2a-f5d20eb37cc5\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xvp5x" Jul 09 19:08:29.727544 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:29.727507 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bbfbf153-e930-4709-bb2a-f5d20eb37cc5-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xvp5x\" (UID: \"bbfbf153-e930-4709-bb2a-f5d20eb37cc5\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xvp5x" Jul 09 19:08:29.727747 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:29.727573 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbzv8\" (UniqueName: \"kubernetes.io/projected/bbfbf153-e930-4709-bb2a-f5d20eb37cc5-kube-api-access-xbzv8\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xvp5x\" (UID: \"bbfbf153-e930-4709-bb2a-f5d20eb37cc5\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xvp5x" Jul 09 19:08:29.727747 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:29.727602 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bbfbf153-e930-4709-bb2a-f5d20eb37cc5-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xvp5x\" (UID: \"bbfbf153-e930-4709-bb2a-f5d20eb37cc5\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xvp5x" Jul 09 19:08:29.727987 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:29.727964 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bbfbf153-e930-4709-bb2a-f5d20eb37cc5-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xvp5x\" (UID: \"bbfbf153-e930-4709-bb2a-f5d20eb37cc5\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xvp5x" Jul 09 19:08:29.728053 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:29.727984 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bbfbf153-e930-4709-bb2a-f5d20eb37cc5-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xvp5x\" (UID: \"bbfbf153-e930-4709-bb2a-f5d20eb37cc5\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xvp5x" Jul 09 19:08:29.738129 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:29.738099 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbzv8\" (UniqueName: \"kubernetes.io/projected/bbfbf153-e930-4709-bb2a-f5d20eb37cc5-kube-api-access-xbzv8\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xvp5x\" (UID: \"bbfbf153-e930-4709-bb2a-f5d20eb37cc5\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xvp5x" Jul 09 19:08:29.866533 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:29.866455 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xvp5x" Jul 09 19:08:29.999319 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:29.999157 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xvp5x"] Jul 09 19:08:30.001975 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:08:30.001946 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbfbf153_e930_4709_bb2a_f5d20eb37cc5.slice/crio-29c13411a76b0a7ad62c13ba9261703130c4a8dc45b7bebc9c8f54dd3a500a89 WatchSource:0}: Error finding container 29c13411a76b0a7ad62c13ba9261703130c4a8dc45b7bebc9c8f54dd3a500a89: Status 404 returned error can't find the container with id 29c13411a76b0a7ad62c13ba9261703130c4a8dc45b7bebc9c8f54dd3a500a89 Jul 09 19:08:30.353089 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:30.353053 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dt6dn"] Jul 09 19:08:30.356557 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:30.356534 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dt6dn" Jul 09 19:08:30.366473 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:30.366448 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dt6dn"] Jul 09 19:08:30.433364 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:30.433332 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab362fb3-ec00-4d48-b5ae-494b00c79e84-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dt6dn\" (UID: \"ab362fb3-ec00-4d48-b5ae-494b00c79e84\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dt6dn" Jul 09 19:08:30.433364 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:30.433368 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpxd2\" (UniqueName: \"kubernetes.io/projected/ab362fb3-ec00-4d48-b5ae-494b00c79e84-kube-api-access-vpxd2\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dt6dn\" (UID: \"ab362fb3-ec00-4d48-b5ae-494b00c79e84\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dt6dn" Jul 09 19:08:30.433543 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:30.433408 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab362fb3-ec00-4d48-b5ae-494b00c79e84-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dt6dn\" (UID: \"ab362fb3-ec00-4d48-b5ae-494b00c79e84\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dt6dn" Jul 09 19:08:30.457208 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:30.457178 2566 generic.go:356] "Generic (PLEG): container finished" podID="bbfbf153-e930-4709-bb2a-f5d20eb37cc5" containerID="de5d3af2d253405224358c527e0815a245ae286b0a8d9b47077ecba922122115" exitCode=0 Jul 09 19:08:30.457325 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:30.457216 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xvp5x" event={"ID":"bbfbf153-e930-4709-bb2a-f5d20eb37cc5","Type":"ContainerDied","Data":"de5d3af2d253405224358c527e0815a245ae286b0a8d9b47077ecba922122115"} Jul 09 19:08:30.457325 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:30.457258 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xvp5x" event={"ID":"bbfbf153-e930-4709-bb2a-f5d20eb37cc5","Type":"ContainerStarted","Data":"29c13411a76b0a7ad62c13ba9261703130c4a8dc45b7bebc9c8f54dd3a500a89"} Jul 09 19:08:30.534102 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:30.534062 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab362fb3-ec00-4d48-b5ae-494b00c79e84-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dt6dn\" (UID: \"ab362fb3-ec00-4d48-b5ae-494b00c79e84\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dt6dn" Jul 09 19:08:30.534274 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:30.534171 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab362fb3-ec00-4d48-b5ae-494b00c79e84-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dt6dn\" (UID: \"ab362fb3-ec00-4d48-b5ae-494b00c79e84\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dt6dn" Jul 09 19:08:30.534274 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:30.534200 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpxd2\" (UniqueName: \"kubernetes.io/projected/ab362fb3-ec00-4d48-b5ae-494b00c79e84-kube-api-access-vpxd2\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dt6dn\" (UID: \"ab362fb3-ec00-4d48-b5ae-494b00c79e84\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dt6dn" Jul 09 19:08:30.534452 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:30.534428 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab362fb3-ec00-4d48-b5ae-494b00c79e84-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dt6dn\" (UID: \"ab362fb3-ec00-4d48-b5ae-494b00c79e84\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dt6dn" Jul 09 19:08:30.534521 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:30.534464 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab362fb3-ec00-4d48-b5ae-494b00c79e84-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dt6dn\" (UID: \"ab362fb3-ec00-4d48-b5ae-494b00c79e84\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dt6dn" Jul 09 19:08:30.543016 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:30.542986 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpxd2\" (UniqueName: \"kubernetes.io/projected/ab362fb3-ec00-4d48-b5ae-494b00c79e84-kube-api-access-vpxd2\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dt6dn\" (UID: \"ab362fb3-ec00-4d48-b5ae-494b00c79e84\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dt6dn" Jul 09 19:08:30.665964 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:30.665881 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dt6dn" Jul 09 19:08:30.800959 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:30.800936 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dt6dn"] Jul 09 19:08:30.803063 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:08:30.803036 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab362fb3_ec00_4d48_b5ae_494b00c79e84.slice/crio-ebb0aa04466be309a5fa4e630533944b1abe576eb1c28bb813e1bb7e9743e24e WatchSource:0}: Error finding container ebb0aa04466be309a5fa4e630533944b1abe576eb1c28bb813e1bb7e9743e24e: Status 404 returned error can't find the container with id ebb0aa04466be309a5fa4e630533944b1abe576eb1c28bb813e1bb7e9743e24e Jul 09 19:08:31.153193 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:31.153165 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04qjbb"] Jul 09 19:08:31.158632 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:31.158527 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04qjbb" Jul 09 19:08:31.166537 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:31.166497 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04qjbb"] Jul 09 19:08:31.240007 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:31.239976 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53a4270c-599b-4314-95f4-e0ce293d6568-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04qjbb\" (UID: \"53a4270c-599b-4314-95f4-e0ce293d6568\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04qjbb" Jul 09 19:08:31.240169 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:31.240032 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53a4270c-599b-4314-95f4-e0ce293d6568-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04qjbb\" (UID: \"53a4270c-599b-4314-95f4-e0ce293d6568\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04qjbb" Jul 09 19:08:31.240169 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:31.240092 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vvpg\" (UniqueName: \"kubernetes.io/projected/53a4270c-599b-4314-95f4-e0ce293d6568-kube-api-access-9vvpg\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04qjbb\" (UID: \"53a4270c-599b-4314-95f4-e0ce293d6568\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04qjbb" Jul 09 19:08:31.340479 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:31.340454 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53a4270c-599b-4314-95f4-e0ce293d6568-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04qjbb\" (UID: \"53a4270c-599b-4314-95f4-e0ce293d6568\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04qjbb" Jul 09 19:08:31.340587 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:31.340499 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vvpg\" (UniqueName: \"kubernetes.io/projected/53a4270c-599b-4314-95f4-e0ce293d6568-kube-api-access-9vvpg\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04qjbb\" (UID: \"53a4270c-599b-4314-95f4-e0ce293d6568\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04qjbb" Jul 09 19:08:31.340587 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:31.340538 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53a4270c-599b-4314-95f4-e0ce293d6568-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04qjbb\" (UID: \"53a4270c-599b-4314-95f4-e0ce293d6568\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04qjbb" Jul 09 19:08:31.340891 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:31.340873 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53a4270c-599b-4314-95f4-e0ce293d6568-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04qjbb\" (UID: \"53a4270c-599b-4314-95f4-e0ce293d6568\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04qjbb" Jul 09 19:08:31.340942 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:31.340880 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53a4270c-599b-4314-95f4-e0ce293d6568-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04qjbb\" (UID: \"53a4270c-599b-4314-95f4-e0ce293d6568\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04qjbb" Jul 09 19:08:31.348147 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:31.348102 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vvpg\" (UniqueName: \"kubernetes.io/projected/53a4270c-599b-4314-95f4-e0ce293d6568-kube-api-access-9vvpg\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04qjbb\" (UID: \"53a4270c-599b-4314-95f4-e0ce293d6568\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04qjbb" Jul 09 19:08:31.466491 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:31.466448 2566 generic.go:356] "Generic (PLEG): container finished" podID="ab362fb3-ec00-4d48-b5ae-494b00c79e84" containerID="c5d49620b147e02cc9b10b3d4a35acd9201c9e5fa00f9271eaca9e629046f8f6" exitCode=0 Jul 09 19:08:31.466674 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:31.466530 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dt6dn" event={"ID":"ab362fb3-ec00-4d48-b5ae-494b00c79e84","Type":"ContainerDied","Data":"c5d49620b147e02cc9b10b3d4a35acd9201c9e5fa00f9271eaca9e629046f8f6"} Jul 09 19:08:31.466674 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:31.466581 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dt6dn" event={"ID":"ab362fb3-ec00-4d48-b5ae-494b00c79e84","Type":"ContainerStarted","Data":"ebb0aa04466be309a5fa4e630533944b1abe576eb1c28bb813e1bb7e9743e24e"} Jul 09 19:08:31.468407 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:31.468383 2566 generic.go:356] "Generic (PLEG): container finished" podID="bbfbf153-e930-4709-bb2a-f5d20eb37cc5" containerID="8602bdcf47aec1a31782dc6f80cb11f1640e5b83b2785a06ec399c490db9289b" exitCode=0 Jul 09 19:08:31.468514 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:31.468449 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xvp5x" event={"ID":"bbfbf153-e930-4709-bb2a-f5d20eb37cc5","Type":"ContainerDied","Data":"8602bdcf47aec1a31782dc6f80cb11f1640e5b83b2785a06ec399c490db9289b"} Jul 09 19:08:31.469670 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:31.469656 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04qjbb" Jul 09 19:08:31.613153 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:31.613125 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04qjbb"] Jul 09 19:08:31.615569 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:08:31.615541 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53a4270c_599b_4314_95f4_e0ce293d6568.slice/crio-133a228c6c2f8f9fe6a897b3b2c7a5c27056bfc4a640ce1b1481498c0113eba5 WatchSource:0}: Error finding container 133a228c6c2f8f9fe6a897b3b2c7a5c27056bfc4a640ce1b1481498c0113eba5: Status 404 returned error can't find the container with id 133a228c6c2f8f9fe6a897b3b2c7a5c27056bfc4a640ce1b1481498c0113eba5 Jul 09 19:08:31.746793 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:31.746765 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfmnp"] Jul 09 19:08:31.751997 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:31.751977 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfmnp" Jul 09 19:08:31.764483 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:31.764452 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfmnp"] Jul 09 19:08:31.843927 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:31.843895 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ccc3e7f-f900-431b-8310-8945c6fb00ca-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfmnp\" (UID: \"1ccc3e7f-f900-431b-8310-8945c6fb00ca\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfmnp" Jul 09 19:08:31.844070 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:31.843950 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jvpl\" (UniqueName: \"kubernetes.io/projected/1ccc3e7f-f900-431b-8310-8945c6fb00ca-kube-api-access-9jvpl\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfmnp\" (UID: \"1ccc3e7f-f900-431b-8310-8945c6fb00ca\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfmnp" Jul 09 19:08:31.844070 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:31.843974 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ccc3e7f-f900-431b-8310-8945c6fb00ca-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfmnp\" (UID: \"1ccc3e7f-f900-431b-8310-8945c6fb00ca\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfmnp" Jul 09 19:08:31.945075 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:31.945043 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ccc3e7f-f900-431b-8310-8945c6fb00ca-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfmnp\" (UID: \"1ccc3e7f-f900-431b-8310-8945c6fb00ca\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfmnp" Jul 09 19:08:31.945284 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:31.945108 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jvpl\" (UniqueName: \"kubernetes.io/projected/1ccc3e7f-f900-431b-8310-8945c6fb00ca-kube-api-access-9jvpl\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfmnp\" (UID: \"1ccc3e7f-f900-431b-8310-8945c6fb00ca\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfmnp" Jul 09 19:08:31.945284 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:31.945144 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ccc3e7f-f900-431b-8310-8945c6fb00ca-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfmnp\" (UID: \"1ccc3e7f-f900-431b-8310-8945c6fb00ca\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfmnp" Jul 09 19:08:31.945498 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:31.945473 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ccc3e7f-f900-431b-8310-8945c6fb00ca-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfmnp\" (UID: \"1ccc3e7f-f900-431b-8310-8945c6fb00ca\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfmnp" Jul 09 19:08:31.945557 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:31.945491 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ccc3e7f-f900-431b-8310-8945c6fb00ca-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfmnp\" (UID: \"1ccc3e7f-f900-431b-8310-8945c6fb00ca\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfmnp" Jul 09 19:08:31.953778 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:31.953753 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jvpl\" (UniqueName: \"kubernetes.io/projected/1ccc3e7f-f900-431b-8310-8945c6fb00ca-kube-api-access-9jvpl\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfmnp\" (UID: \"1ccc3e7f-f900-431b-8310-8945c6fb00ca\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfmnp" Jul 09 19:08:32.061870 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:32.061801 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfmnp" Jul 09 19:08:32.201594 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:32.201571 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfmnp"] Jul 09 19:08:32.247828 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:08:32.247790 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ccc3e7f_f900_431b_8310_8945c6fb00ca.slice/crio-03782534e0a66c757c28ad31af55eb5bc31ee3bf8535250674564877c260feff WatchSource:0}: Error finding container 03782534e0a66c757c28ad31af55eb5bc31ee3bf8535250674564877c260feff: Status 404 returned error can't find the container with id 03782534e0a66c757c28ad31af55eb5bc31ee3bf8535250674564877c260feff Jul 09 19:08:32.408586 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:08:32.408564 2566 helpers.go:245] readString: Failed to read "/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab362fb3_ec00_4d48_b5ae_494b00c79e84.slice/crio-conmon-c203a1a2bc4a01284ae1b0672b76f98902b85f1568931d77562566ed02c63c24.scope/memory.swap.max": read /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab362fb3_ec00_4d48_b5ae_494b00c79e84.slice/crio-conmon-c203a1a2bc4a01284ae1b0672b76f98902b85f1568931d77562566ed02c63c24.scope/memory.swap.max: no such device Jul 09 19:08:32.474620 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:32.474573 2566 generic.go:356] "Generic (PLEG): container finished" podID="53a4270c-599b-4314-95f4-e0ce293d6568" containerID="bfd19e0a0e96488043a1b2da449c6badfd21d5df1019659a826b218333c99d88" exitCode=0 Jul 09 19:08:32.474749 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:32.474664 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04qjbb" event={"ID":"53a4270c-599b-4314-95f4-e0ce293d6568","Type":"ContainerDied","Data":"bfd19e0a0e96488043a1b2da449c6badfd21d5df1019659a826b218333c99d88"} Jul 09 19:08:32.474749 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:32.474714 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04qjbb" event={"ID":"53a4270c-599b-4314-95f4-e0ce293d6568","Type":"ContainerStarted","Data":"133a228c6c2f8f9fe6a897b3b2c7a5c27056bfc4a640ce1b1481498c0113eba5"} Jul 09 19:08:32.476600 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:32.476568 2566 generic.go:356] "Generic (PLEG): container finished" podID="ab362fb3-ec00-4d48-b5ae-494b00c79e84" containerID="c203a1a2bc4a01284ae1b0672b76f98902b85f1568931d77562566ed02c63c24" exitCode=0 Jul 09 19:08:32.476719 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:32.476616 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dt6dn" event={"ID":"ab362fb3-ec00-4d48-b5ae-494b00c79e84","Type":"ContainerDied","Data":"c203a1a2bc4a01284ae1b0672b76f98902b85f1568931d77562566ed02c63c24"} Jul 09 19:08:32.478870 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:32.478848 2566 generic.go:356] "Generic (PLEG): container finished" podID="bbfbf153-e930-4709-bb2a-f5d20eb37cc5" containerID="1d5841a8e05e62c1e43f60eaaa9c9b61bbd1fe51ff3c522ec20c4c0012cb44ee" exitCode=0 Jul 09 19:08:32.479196 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:32.479169 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xvp5x" event={"ID":"bbfbf153-e930-4709-bb2a-f5d20eb37cc5","Type":"ContainerDied","Data":"1d5841a8e05e62c1e43f60eaaa9c9b61bbd1fe51ff3c522ec20c4c0012cb44ee"} Jul 09 19:08:32.480480 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:32.480459 2566 generic.go:356] "Generic (PLEG): container finished" podID="1ccc3e7f-f900-431b-8310-8945c6fb00ca" containerID="2d81d7377b6422ec10592502ae19c66f1842022e7dcd7aa945de046304d8e1e1" exitCode=0 Jul 09 19:08:32.480581 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:32.480517 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfmnp" event={"ID":"1ccc3e7f-f900-431b-8310-8945c6fb00ca","Type":"ContainerDied","Data":"2d81d7377b6422ec10592502ae19c66f1842022e7dcd7aa945de046304d8e1e1"} Jul 09 19:08:32.480581 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:32.480550 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfmnp" event={"ID":"1ccc3e7f-f900-431b-8310-8945c6fb00ca","Type":"ContainerStarted","Data":"03782534e0a66c757c28ad31af55eb5bc31ee3bf8535250674564877c260feff"} Jul 09 19:08:33.488065 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:33.488028 2566 generic.go:356] "Generic (PLEG): container finished" podID="ab362fb3-ec00-4d48-b5ae-494b00c79e84" containerID="dd88ae7dadb6abf9f6ca572fe951c341d319f03a7f214dc0160d439d459632ac" exitCode=0 Jul 09 19:08:33.488450 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:33.488108 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dt6dn" event={"ID":"ab362fb3-ec00-4d48-b5ae-494b00c79e84","Type":"ContainerDied","Data":"dd88ae7dadb6abf9f6ca572fe951c341d319f03a7f214dc0160d439d459632ac"} Jul 09 19:08:33.489969 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:33.489944 2566 generic.go:356] "Generic (PLEG): container finished" podID="1ccc3e7f-f900-431b-8310-8945c6fb00ca" containerID="73c4594627524a4e046a2f0283f22fd869b35ff601bfd44e422d0be7e5d559bf" exitCode=0 Jul 09 19:08:33.490095 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:33.490023 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfmnp" event={"ID":"1ccc3e7f-f900-431b-8310-8945c6fb00ca","Type":"ContainerDied","Data":"73c4594627524a4e046a2f0283f22fd869b35ff601bfd44e422d0be7e5d559bf"} Jul 09 19:08:33.631492 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:33.631466 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xvp5x" Jul 09 19:08:33.660524 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:33.660494 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bbfbf153-e930-4709-bb2a-f5d20eb37cc5-bundle\") pod \"bbfbf153-e930-4709-bb2a-f5d20eb37cc5\" (UID: \"bbfbf153-e930-4709-bb2a-f5d20eb37cc5\") " Jul 09 19:08:33.660700 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:33.660590 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bbfbf153-e930-4709-bb2a-f5d20eb37cc5-util\") pod \"bbfbf153-e930-4709-bb2a-f5d20eb37cc5\" (UID: \"bbfbf153-e930-4709-bb2a-f5d20eb37cc5\") " Jul 09 19:08:33.660700 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:33.660654 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xbzv8\" (UniqueName: \"kubernetes.io/projected/bbfbf153-e930-4709-bb2a-f5d20eb37cc5-kube-api-access-xbzv8\") pod \"bbfbf153-e930-4709-bb2a-f5d20eb37cc5\" (UID: \"bbfbf153-e930-4709-bb2a-f5d20eb37cc5\") " Jul 09 19:08:33.661252 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:33.661150 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbfbf153-e930-4709-bb2a-f5d20eb37cc5-bundle" (OuterVolumeSpecName: "bundle") pod "bbfbf153-e930-4709-bb2a-f5d20eb37cc5" (UID: "bbfbf153-e930-4709-bb2a-f5d20eb37cc5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:08:33.663410 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:33.663385 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbfbf153-e930-4709-bb2a-f5d20eb37cc5-kube-api-access-xbzv8" (OuterVolumeSpecName: "kube-api-access-xbzv8") pod "bbfbf153-e930-4709-bb2a-f5d20eb37cc5" (UID: "bbfbf153-e930-4709-bb2a-f5d20eb37cc5"). InnerVolumeSpecName "kube-api-access-xbzv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:08:33.670328 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:33.670290 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbfbf153-e930-4709-bb2a-f5d20eb37cc5-util" (OuterVolumeSpecName: "util") pod "bbfbf153-e930-4709-bb2a-f5d20eb37cc5" (UID: "bbfbf153-e930-4709-bb2a-f5d20eb37cc5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:08:33.761830 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:33.761800 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbzv8\" (UniqueName: \"kubernetes.io/projected/bbfbf153-e930-4709-bb2a-f5d20eb37cc5-kube-api-access-xbzv8\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:08:33.761830 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:33.761827 2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bbfbf153-e930-4709-bb2a-f5d20eb37cc5-bundle\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:08:33.762032 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:33.761839 2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bbfbf153-e930-4709-bb2a-f5d20eb37cc5-util\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:08:34.497052 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:34.497015 2566 generic.go:356] "Generic (PLEG): container finished" podID="53a4270c-599b-4314-95f4-e0ce293d6568" containerID="7b51036b03965e0de6b411d802a920f8bba6b6275530ca21a45eaca86fec5842" exitCode=0 Jul 09 19:08:34.497519 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:34.497104 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04qjbb" event={"ID":"53a4270c-599b-4314-95f4-e0ce293d6568","Type":"ContainerDied","Data":"7b51036b03965e0de6b411d802a920f8bba6b6275530ca21a45eaca86fec5842"} Jul 09 19:08:34.499272 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:34.499257 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xvp5x" Jul 09 19:08:34.499272 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:34.499257 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xvp5x" event={"ID":"bbfbf153-e930-4709-bb2a-f5d20eb37cc5","Type":"ContainerDied","Data":"29c13411a76b0a7ad62c13ba9261703130c4a8dc45b7bebc9c8f54dd3a500a89"} Jul 09 19:08:34.499404 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:34.499299 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29c13411a76b0a7ad62c13ba9261703130c4a8dc45b7bebc9c8f54dd3a500a89" Jul 09 19:08:34.501662 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:34.501639 2566 generic.go:356] "Generic (PLEG): container finished" podID="1ccc3e7f-f900-431b-8310-8945c6fb00ca" containerID="122efd39712bfa12c1f0c051c25d1b79041bc225a8b74ff2bee5146eb83387e6" exitCode=0 Jul 09 19:08:34.501749 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:34.501697 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfmnp" event={"ID":"1ccc3e7f-f900-431b-8310-8945c6fb00ca","Type":"ContainerDied","Data":"122efd39712bfa12c1f0c051c25d1b79041bc225a8b74ff2bee5146eb83387e6"} Jul 09 19:08:34.635090 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:34.635068 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dt6dn" Jul 09 19:08:34.670355 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:34.670327 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab362fb3-ec00-4d48-b5ae-494b00c79e84-bundle\") pod \"ab362fb3-ec00-4d48-b5ae-494b00c79e84\" (UID: \"ab362fb3-ec00-4d48-b5ae-494b00c79e84\") " Jul 09 19:08:34.670491 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:34.670376 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vpxd2\" (UniqueName: \"kubernetes.io/projected/ab362fb3-ec00-4d48-b5ae-494b00c79e84-kube-api-access-vpxd2\") pod \"ab362fb3-ec00-4d48-b5ae-494b00c79e84\" (UID: \"ab362fb3-ec00-4d48-b5ae-494b00c79e84\") " Jul 09 19:08:34.670491 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:34.670412 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab362fb3-ec00-4d48-b5ae-494b00c79e84-util\") pod \"ab362fb3-ec00-4d48-b5ae-494b00c79e84\" (UID: \"ab362fb3-ec00-4d48-b5ae-494b00c79e84\") " Jul 09 19:08:34.670941 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:34.670915 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab362fb3-ec00-4d48-b5ae-494b00c79e84-bundle" (OuterVolumeSpecName: "bundle") pod "ab362fb3-ec00-4d48-b5ae-494b00c79e84" (UID: "ab362fb3-ec00-4d48-b5ae-494b00c79e84"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:08:34.672509 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:34.672487 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab362fb3-ec00-4d48-b5ae-494b00c79e84-kube-api-access-vpxd2" (OuterVolumeSpecName: "kube-api-access-vpxd2") pod "ab362fb3-ec00-4d48-b5ae-494b00c79e84" (UID: "ab362fb3-ec00-4d48-b5ae-494b00c79e84"). InnerVolumeSpecName "kube-api-access-vpxd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:08:34.676220 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:34.676188 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab362fb3-ec00-4d48-b5ae-494b00c79e84-util" (OuterVolumeSpecName: "util") pod "ab362fb3-ec00-4d48-b5ae-494b00c79e84" (UID: "ab362fb3-ec00-4d48-b5ae-494b00c79e84"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:08:34.772159 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:34.772087 2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab362fb3-ec00-4d48-b5ae-494b00c79e84-bundle\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:08:34.772159 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:34.772113 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpxd2\" (UniqueName: \"kubernetes.io/projected/ab362fb3-ec00-4d48-b5ae-494b00c79e84-kube-api-access-vpxd2\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:08:34.772159 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:34.772123 2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab362fb3-ec00-4d48-b5ae-494b00c79e84-util\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:08:35.508671 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:35.508645 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dt6dn" Jul 09 19:08:35.509089 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:35.508644 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dt6dn" event={"ID":"ab362fb3-ec00-4d48-b5ae-494b00c79e84","Type":"ContainerDied","Data":"ebb0aa04466be309a5fa4e630533944b1abe576eb1c28bb813e1bb7e9743e24e"} Jul 09 19:08:35.509089 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:35.508760 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebb0aa04466be309a5fa4e630533944b1abe576eb1c28bb813e1bb7e9743e24e" Jul 09 19:08:35.511163 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:35.511139 2566 generic.go:356] "Generic (PLEG): container finished" podID="53a4270c-599b-4314-95f4-e0ce293d6568" containerID="a6254f421a4b177cd561ad49d8214d5e4954be86fc6a1e66612c1544c13d8b75" exitCode=0 Jul 09 19:08:35.511279 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:35.511191 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04qjbb" event={"ID":"53a4270c-599b-4314-95f4-e0ce293d6568","Type":"ContainerDied","Data":"a6254f421a4b177cd561ad49d8214d5e4954be86fc6a1e66612c1544c13d8b75"} Jul 09 19:08:35.643465 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:35.643443 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfmnp" Jul 09 19:08:35.680092 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:35.680066 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jvpl\" (UniqueName: \"kubernetes.io/projected/1ccc3e7f-f900-431b-8310-8945c6fb00ca-kube-api-access-9jvpl\") pod \"1ccc3e7f-f900-431b-8310-8945c6fb00ca\" (UID: \"1ccc3e7f-f900-431b-8310-8945c6fb00ca\") " Jul 09 19:08:35.680265 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:35.680121 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ccc3e7f-f900-431b-8310-8945c6fb00ca-util\") pod \"1ccc3e7f-f900-431b-8310-8945c6fb00ca\" (UID: \"1ccc3e7f-f900-431b-8310-8945c6fb00ca\") " Jul 09 19:08:35.680265 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:35.680170 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ccc3e7f-f900-431b-8310-8945c6fb00ca-bundle\") pod \"1ccc3e7f-f900-431b-8310-8945c6fb00ca\" (UID: \"1ccc3e7f-f900-431b-8310-8945c6fb00ca\") " Jul 09 19:08:35.680753 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:35.680723 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ccc3e7f-f900-431b-8310-8945c6fb00ca-bundle" (OuterVolumeSpecName: "bundle") pod "1ccc3e7f-f900-431b-8310-8945c6fb00ca" (UID: "1ccc3e7f-f900-431b-8310-8945c6fb00ca"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:08:35.682166 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:35.682141 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ccc3e7f-f900-431b-8310-8945c6fb00ca-kube-api-access-9jvpl" (OuterVolumeSpecName: "kube-api-access-9jvpl") pod "1ccc3e7f-f900-431b-8310-8945c6fb00ca" (UID: "1ccc3e7f-f900-431b-8310-8945c6fb00ca"). InnerVolumeSpecName "kube-api-access-9jvpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:08:35.688102 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:35.688071 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ccc3e7f-f900-431b-8310-8945c6fb00ca-util" (OuterVolumeSpecName: "util") pod "1ccc3e7f-f900-431b-8310-8945c6fb00ca" (UID: "1ccc3e7f-f900-431b-8310-8945c6fb00ca"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:08:35.781530 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:35.781436 2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ccc3e7f-f900-431b-8310-8945c6fb00ca-util\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:08:35.781530 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:35.781477 2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ccc3e7f-f900-431b-8310-8945c6fb00ca-bundle\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:08:35.781530 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:35.781487 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jvpl\" (UniqueName: \"kubernetes.io/projected/1ccc3e7f-f900-431b-8310-8945c6fb00ca-kube-api-access-9jvpl\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:08:36.517768 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:36.517739 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfmnp" Jul 09 19:08:36.518171 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:36.517776 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfmnp" event={"ID":"1ccc3e7f-f900-431b-8310-8945c6fb00ca","Type":"ContainerDied","Data":"03782534e0a66c757c28ad31af55eb5bc31ee3bf8535250674564877c260feff"} Jul 09 19:08:36.518171 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:36.517821 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03782534e0a66c757c28ad31af55eb5bc31ee3bf8535250674564877c260feff" Jul 09 19:08:36.649156 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:36.649134 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04qjbb" Jul 09 19:08:36.690980 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:36.690953 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53a4270c-599b-4314-95f4-e0ce293d6568-util\") pod \"53a4270c-599b-4314-95f4-e0ce293d6568\" (UID: \"53a4270c-599b-4314-95f4-e0ce293d6568\") " Jul 09 19:08:36.691118 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:36.690994 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9vvpg\" (UniqueName: \"kubernetes.io/projected/53a4270c-599b-4314-95f4-e0ce293d6568-kube-api-access-9vvpg\") pod \"53a4270c-599b-4314-95f4-e0ce293d6568\" (UID: \"53a4270c-599b-4314-95f4-e0ce293d6568\") " Jul 09 19:08:36.691118 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:36.691030 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53a4270c-599b-4314-95f4-e0ce293d6568-bundle\") pod \"53a4270c-599b-4314-95f4-e0ce293d6568\" (UID: \"53a4270c-599b-4314-95f4-e0ce293d6568\") " Jul 09 19:08:36.691597 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:36.691561 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53a4270c-599b-4314-95f4-e0ce293d6568-bundle" (OuterVolumeSpecName: "bundle") pod "53a4270c-599b-4314-95f4-e0ce293d6568" (UID: "53a4270c-599b-4314-95f4-e0ce293d6568"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:08:36.693051 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:36.693028 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53a4270c-599b-4314-95f4-e0ce293d6568-kube-api-access-9vvpg" (OuterVolumeSpecName: "kube-api-access-9vvpg") pod "53a4270c-599b-4314-95f4-e0ce293d6568" (UID: "53a4270c-599b-4314-95f4-e0ce293d6568"). InnerVolumeSpecName "kube-api-access-9vvpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:08:36.700021 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:36.699991 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53a4270c-599b-4314-95f4-e0ce293d6568-util" (OuterVolumeSpecName: "util") pod "53a4270c-599b-4314-95f4-e0ce293d6568" (UID: "53a4270c-599b-4314-95f4-e0ce293d6568"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:08:36.792301 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:36.792225 2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53a4270c-599b-4314-95f4-e0ce293d6568-bundle\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:08:36.792301 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:36.792249 2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53a4270c-599b-4314-95f4-e0ce293d6568-util\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:08:36.792301 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:36.792259 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vvpg\" (UniqueName: \"kubernetes.io/projected/53a4270c-599b-4314-95f4-e0ce293d6568-kube-api-access-9vvpg\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:08:37.524552 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:37.524527 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04qjbb" Jul 09 19:08:37.524994 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:37.524555 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04qjbb" event={"ID":"53a4270c-599b-4314-95f4-e0ce293d6568","Type":"ContainerDied","Data":"133a228c6c2f8f9fe6a897b3b2c7a5c27056bfc4a640ce1b1481498c0113eba5"} Jul 09 19:08:37.524994 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:37.524664 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="133a228c6c2f8f9fe6a897b3b2c7a5c27056bfc4a640ce1b1481498c0113eba5" Jul 09 19:08:48.818576 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:48.818539 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qm2t6"] Jul 09 19:08:48.818962 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:48.818927 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53a4270c-599b-4314-95f4-e0ce293d6568" containerName="pull" Jul 09 19:08:48.818962 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:48.818939 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="53a4270c-599b-4314-95f4-e0ce293d6568" containerName="pull" Jul 09 19:08:48.818962 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:48.818951 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ccc3e7f-f900-431b-8310-8945c6fb00ca" containerName="extract" Jul 09 19:08:48.818962 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:48.818957 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ccc3e7f-f900-431b-8310-8945c6fb00ca" containerName="extract" Jul 09 19:08:48.818962 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:48.818962 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab362fb3-ec00-4d48-b5ae-494b00c79e84" containerName="pull" Jul 09 19:08:48.819121 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:48.818968 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab362fb3-ec00-4d48-b5ae-494b00c79e84" containerName="pull" Jul 09 19:08:48.819121 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:48.818977 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ccc3e7f-f900-431b-8310-8945c6fb00ca" containerName="pull" Jul 09 19:08:48.819121 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:48.818982 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ccc3e7f-f900-431b-8310-8945c6fb00ca" containerName="pull" Jul 09 19:08:48.819121 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:48.818994 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbfbf153-e930-4709-bb2a-f5d20eb37cc5" containerName="extract" Jul 09 19:08:48.819121 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:48.819000 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbfbf153-e930-4709-bb2a-f5d20eb37cc5" containerName="extract" Jul 09 19:08:48.819121 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:48.819006 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab362fb3-ec00-4d48-b5ae-494b00c79e84" containerName="extract" Jul 09 19:08:48.819121 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:48.819011 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab362fb3-ec00-4d48-b5ae-494b00c79e84" containerName="extract" Jul 09 19:08:48.819121 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:48.819017 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53a4270c-599b-4314-95f4-e0ce293d6568" containerName="extract" Jul 09 19:08:48.819121 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:48.819022 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="53a4270c-599b-4314-95f4-e0ce293d6568" containerName="extract" Jul 09 19:08:48.819121 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:48.819030 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ccc3e7f-f900-431b-8310-8945c6fb00ca" containerName="util" Jul 09 19:08:48.819121 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:48.819035 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ccc3e7f-f900-431b-8310-8945c6fb00ca" containerName="util" Jul 09 19:08:48.819121 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:48.819042 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbfbf153-e930-4709-bb2a-f5d20eb37cc5" containerName="util" Jul 09 19:08:48.819121 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:48.819046 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbfbf153-e930-4709-bb2a-f5d20eb37cc5" containerName="util" Jul 09 19:08:48.819121 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:48.819052 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53a4270c-599b-4314-95f4-e0ce293d6568" containerName="util" Jul 09 19:08:48.819121 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:48.819056 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="53a4270c-599b-4314-95f4-e0ce293d6568" containerName="util" Jul 09 19:08:48.819121 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:48.819062 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbfbf153-e930-4709-bb2a-f5d20eb37cc5" containerName="pull" Jul 09 19:08:48.819121 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:48.819066 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbfbf153-e930-4709-bb2a-f5d20eb37cc5" containerName="pull" Jul 09 19:08:48.819121 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:48.819074 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab362fb3-ec00-4d48-b5ae-494b00c79e84" containerName="util" Jul 09 19:08:48.819121 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:48.819078 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab362fb3-ec00-4d48-b5ae-494b00c79e84" containerName="util" Jul 09 19:08:48.819625 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:48.819134 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ccc3e7f-f900-431b-8310-8945c6fb00ca" containerName="extract" Jul 09 19:08:48.819625 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:48.819141 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbfbf153-e930-4709-bb2a-f5d20eb37cc5" containerName="extract" Jul 09 19:08:48.819625 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:48.819147 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab362fb3-ec00-4d48-b5ae-494b00c79e84" containerName="extract" Jul 09 19:08:48.819625 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:48.819155 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="53a4270c-599b-4314-95f4-e0ce293d6568" containerName="extract" Jul 09 19:08:48.823598 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:48.823582 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qm2t6" Jul 09 19:08:48.827021 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:48.826998 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-nj4td\"" Jul 09 19:08:48.838642 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:48.838617 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qm2t6"] Jul 09 19:08:48.889702 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:48.889668 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xpbx\" (UniqueName: \"kubernetes.io/projected/45e77115-4fe0-41df-ba82-c0a94b6e66ac-kube-api-access-6xpbx\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qm2t6\" (UID: \"45e77115-4fe0-41df-ba82-c0a94b6e66ac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qm2t6" Jul 09 19:08:48.889842 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:48.889738 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/45e77115-4fe0-41df-ba82-c0a94b6e66ac-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qm2t6\" (UID: \"45e77115-4fe0-41df-ba82-c0a94b6e66ac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qm2t6" Jul 09 19:08:48.990771 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:48.990740 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xpbx\" (UniqueName: \"kubernetes.io/projected/45e77115-4fe0-41df-ba82-c0a94b6e66ac-kube-api-access-6xpbx\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qm2t6\" (UID: \"45e77115-4fe0-41df-ba82-c0a94b6e66ac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qm2t6" Jul 09 19:08:48.990913 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:48.990804 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/45e77115-4fe0-41df-ba82-c0a94b6e66ac-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qm2t6\" (UID: \"45e77115-4fe0-41df-ba82-c0a94b6e66ac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qm2t6" Jul 09 19:08:48.991134 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:48.991113 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/45e77115-4fe0-41df-ba82-c0a94b6e66ac-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qm2t6\" (UID: \"45e77115-4fe0-41df-ba82-c0a94b6e66ac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qm2t6" Jul 09 19:08:49.002696 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:49.002672 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xpbx\" (UniqueName: \"kubernetes.io/projected/45e77115-4fe0-41df-ba82-c0a94b6e66ac-kube-api-access-6xpbx\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qm2t6\" (UID: \"45e77115-4fe0-41df-ba82-c0a94b6e66ac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qm2t6" Jul 09 19:08:49.133368 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:49.133287 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qm2t6" Jul 09 19:08:49.275134 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:49.275109 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qm2t6"] Jul 09 19:08:49.277219 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:08:49.277195 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45e77115_4fe0_41df_ba82_c0a94b6e66ac.slice/crio-5ff9ce4eb8cef768837979be0b916731906a29929eba538513f68b381e543eb4 WatchSource:0}: Error finding container 5ff9ce4eb8cef768837979be0b916731906a29929eba538513f68b381e543eb4: Status 404 returned error can't find the container with id 5ff9ce4eb8cef768837979be0b916731906a29929eba538513f68b381e543eb4 Jul 09 19:08:49.584539 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:49.584506 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qm2t6" event={"ID":"45e77115-4fe0-41df-ba82-c0a94b6e66ac","Type":"ContainerStarted","Data":"5ff9ce4eb8cef768837979be0b916731906a29929eba538513f68b381e543eb4"} Jul 09 19:08:50.950580 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:50.950546 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-vws4c_7cf4af17-5218-4b92-bceb-addbefcc03cb/console-operator/1.log" Jul 09 19:08:50.951042 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:50.950668 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-vws4c_7cf4af17-5218-4b92-bceb-addbefcc03cb/console-operator/1.log" Jul 09 19:08:50.957406 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:50.957370 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6dh7g_db233460-134d-4d9d-8da9-31d89366a4e0/ovn-acl-logging/0.log" Jul 09 19:08:50.957623 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:50.957578 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6dh7g_db233460-134d-4d9d-8da9-31d89366a4e0/ovn-acl-logging/0.log" Jul 09 19:08:52.152276 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:52.152239 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7wmjm"] Jul 09 19:08:52.155651 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:52.155628 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7wmjm" Jul 09 19:08:52.161441 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:52.161419 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-vdr55\"" Jul 09 19:08:52.185890 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:52.185863 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7wmjm"] Jul 09 19:08:52.215338 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:52.215308 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhzhq\" (UniqueName: \"kubernetes.io/projected/4ec44b16-e08c-4c92-a2ab-132c335f8e2a-kube-api-access-hhzhq\") pod \"limitador-operator-controller-manager-859d7b78bd-7wmjm\" (UID: \"4ec44b16-e08c-4c92-a2ab-132c335f8e2a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7wmjm" Jul 09 19:08:52.316853 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:52.316821 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhzhq\" (UniqueName: \"kubernetes.io/projected/4ec44b16-e08c-4c92-a2ab-132c335f8e2a-kube-api-access-hhzhq\") pod \"limitador-operator-controller-manager-859d7b78bd-7wmjm\" (UID: \"4ec44b16-e08c-4c92-a2ab-132c335f8e2a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7wmjm" Jul 09 19:08:52.328885 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:52.328858 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhzhq\" (UniqueName: \"kubernetes.io/projected/4ec44b16-e08c-4c92-a2ab-132c335f8e2a-kube-api-access-hhzhq\") pod \"limitador-operator-controller-manager-859d7b78bd-7wmjm\" (UID: \"4ec44b16-e08c-4c92-a2ab-132c335f8e2a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7wmjm" Jul 09 19:08:52.467171 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:52.467131 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7wmjm" Jul 09 19:08:52.602380 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:52.602345 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7wmjm"] Jul 09 19:08:52.603750 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:08:52.603725 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ec44b16_e08c_4c92_a2ab_132c335f8e2a.slice/crio-a0f2674027b3de3a93ef88f04c49d46e6c952b37a26fc5d0d5e2930c90322371 WatchSource:0}: Error finding container a0f2674027b3de3a93ef88f04c49d46e6c952b37a26fc5d0d5e2930c90322371: Status 404 returned error can't find the container with id a0f2674027b3de3a93ef88f04c49d46e6c952b37a26fc5d0d5e2930c90322371 Jul 09 19:08:53.610900 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:53.610860 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7wmjm" event={"ID":"4ec44b16-e08c-4c92-a2ab-132c335f8e2a","Type":"ContainerStarted","Data":"a0f2674027b3de3a93ef88f04c49d46e6c952b37a26fc5d0d5e2930c90322371"} Jul 09 19:08:55.623666 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:55.623623 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qm2t6" event={"ID":"45e77115-4fe0-41df-ba82-c0a94b6e66ac","Type":"ContainerStarted","Data":"037196ef749a028e95680dcb620a183d57fdb38dfb9d1900b9db15fb2ea47fd7"} Jul 09 19:08:55.624063 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:55.623697 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qm2t6" Jul 09 19:08:55.647155 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:55.647107 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qm2t6" podStartSLOduration=2.086332383 podStartE2EDuration="7.64709288s" totalImagesPullingTime="5.560760497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:08:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:08:55.643512911 +0000 UTC m=+605.151292725" watchObservedRunningTime="2026-07-09 19:08:55.64709288 +0000 UTC m=+605.154872653" Jul 09 19:08:56.613370 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:56.613331 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-2jcmm"] Jul 09 19:08:56.616398 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:56.616377 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2jcmm" Jul 09 19:08:56.619331 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:56.619305 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-2nbnw\"" Jul 09 19:08:56.627891 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:56.627859 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-2jcmm"] Jul 09 19:08:56.632256 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:56.632230 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7wmjm" event={"ID":"4ec44b16-e08c-4c92-a2ab-132c335f8e2a","Type":"ContainerStarted","Data":"0d3fa9b33316f7453ef40749e73e5dbbe1c9af859fba53e292d41203a0932a96"} Jul 09 19:08:56.632387 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:56.632334 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7wmjm" Jul 09 19:08:56.659395 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:56.659348 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7wmjm" podStartSLOduration=0.82951891 podStartE2EDuration="4.659332049s" totalImagesPullingTime="3.829813139s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:08:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:08:56.658584419 +0000 UTC m=+606.166364194" watchObservedRunningTime="2026-07-09 19:08:56.659332049 +0000 UTC m=+606.167111823" Jul 09 19:08:56.759700 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:56.759671 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrf2x\" (UniqueName: \"kubernetes.io/projected/e91380bf-c93b-499e-9b14-78f4e6d1df18-kube-api-access-mrf2x\") pod \"authorino-operator-75cc8d5cb7-2jcmm\" (UID: \"e91380bf-c93b-499e-9b14-78f4e6d1df18\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-2jcmm" Jul 09 19:08:56.860481 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:56.860449 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrf2x\" (UniqueName: \"kubernetes.io/projected/e91380bf-c93b-499e-9b14-78f4e6d1df18-kube-api-access-mrf2x\") pod \"authorino-operator-75cc8d5cb7-2jcmm\" (UID: \"e91380bf-c93b-499e-9b14-78f4e6d1df18\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-2jcmm" Jul 09 19:08:56.872854 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:56.872819 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrf2x\" (UniqueName: \"kubernetes.io/projected/e91380bf-c93b-499e-9b14-78f4e6d1df18-kube-api-access-mrf2x\") pod \"authorino-operator-75cc8d5cb7-2jcmm\" (UID: \"e91380bf-c93b-499e-9b14-78f4e6d1df18\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-2jcmm" Jul 09 19:08:56.928727 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:56.928695 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2jcmm" Jul 09 19:08:57.065715 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:57.065653 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-2jcmm"] Jul 09 19:08:57.067870 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:08:57.067843 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode91380bf_c93b_499e_9b14_78f4e6d1df18.slice/crio-a5556bba1ee594ba7b774a027de24a394d8f45c9adbcd451f089810fa899b522 WatchSource:0}: Error finding container a5556bba1ee594ba7b774a027de24a394d8f45c9adbcd451f089810fa899b522: Status 404 returned error can't find the container with id a5556bba1ee594ba7b774a027de24a394d8f45c9adbcd451f089810fa899b522 Jul 09 19:08:57.638472 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:57.638429 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2jcmm" event={"ID":"e91380bf-c93b-499e-9b14-78f4e6d1df18","Type":"ContainerStarted","Data":"a5556bba1ee594ba7b774a027de24a394d8f45c9adbcd451f089810fa899b522"} Jul 09 19:08:59.657033 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:59.656936 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2jcmm" event={"ID":"e91380bf-c93b-499e-9b14-78f4e6d1df18","Type":"ContainerStarted","Data":"2b530ecf75ba680e42c74fcaf7f8b78fcba6abae513fa0d71d460d256b1060b1"} Jul 09 19:08:59.657448 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:59.657048 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2jcmm" Jul 09 19:08:59.674171 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:08:59.674128 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2jcmm" podStartSLOduration=1.459343633 podStartE2EDuration="3.674115851s" totalImagesPullingTime="2.214772218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:08:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:08:59.673672009 +0000 UTC m=+609.181451779" watchObservedRunningTime="2026-07-09 19:08:59.674115851 +0000 UTC m=+609.181895626" Jul 09 19:09:06.635327 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:06.635283 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qm2t6" Jul 09 19:09:07.641033 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:07.641005 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7wmjm" Jul 09 19:09:08.414705 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:08.414667 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qm2t6"] Jul 09 19:09:08.414898 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:08.414839 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qm2t6" podUID="45e77115-4fe0-41df-ba82-c0a94b6e66ac" containerName="manager" containerID="cri-o://037196ef749a028e95680dcb620a183d57fdb38dfb9d1900b9db15fb2ea47fd7" gracePeriod=2 Jul 09 19:09:08.423760 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:08.423731 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qm2t6"] Jul 09 19:09:08.444372 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:08.444344 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7wmjm"] Jul 09 19:09:08.444601 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:08.444573 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7wmjm" podUID="4ec44b16-e08c-4c92-a2ab-132c335f8e2a" containerName="manager" containerID="cri-o://0d3fa9b33316f7453ef40749e73e5dbbe1c9af859fba53e292d41203a0932a96" gracePeriod=2 Jul 09 19:09:08.471224 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:08.471194 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7wmjm"] Jul 09 19:09:08.487278 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:08.487252 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrplf"] Jul 09 19:09:08.487592 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:08.487580 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45e77115-4fe0-41df-ba82-c0a94b6e66ac" containerName="manager" Jul 09 19:09:08.487644 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:08.487594 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="45e77115-4fe0-41df-ba82-c0a94b6e66ac" containerName="manager" Jul 09 19:09:08.487681 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:08.487647 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ec44b16-e08c-4c92-a2ab-132c335f8e2a" containerName="manager" Jul 09 19:09:08.487681 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:08.487654 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ec44b16-e08c-4c92-a2ab-132c335f8e2a" containerName="manager" Jul 09 19:09:08.487746 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:08.487709 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ec44b16-e08c-4c92-a2ab-132c335f8e2a" containerName="manager" Jul 09 19:09:08.487746 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:08.487719 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="45e77115-4fe0-41df-ba82-c0a94b6e66ac" containerName="manager" Jul 09 19:09:08.491009 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:08.490987 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrplf" Jul 09 19:09:08.500548 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:09:08.500517 2566 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"4ec44b16-e08c-4c92-a2ab-132c335f8e2a\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-7wmjm\": pods \"limitador-operator-controller-manager-859d7b78bd-7wmjm\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7wmjm" Jul 09 19:09:08.502925 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:08.502904 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k4ws6"] Jul 09 19:09:08.503920 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:09:08.503888 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qm2t6\" is forbidden: User \"system:node:ip-10-0-140-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-153.ec2.internal' and this object" podUID="45e77115-4fe0-41df-ba82-c0a94b6e66ac" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qm2t6" Jul 09 19:09:08.506327 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:08.506311 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k4ws6" Jul 09 19:09:08.507927 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:09:08.507902 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-7wmjm\" is forbidden: User \"system:node:ip-10-0-140-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-153.ec2.internal' and this object" podUID="4ec44b16-e08c-4c92-a2ab-132c335f8e2a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7wmjm" Jul 09 19:09:08.511973 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:09:08.511951 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qm2t6\" is forbidden: User \"system:node:ip-10-0-140-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-153.ec2.internal' and this object" podUID="45e77115-4fe0-41df-ba82-c0a94b6e66ac" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qm2t6" Jul 09 19:09:08.525026 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:08.524998 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrplf"] Jul 09 19:09:08.527279 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:08.527257 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k4ws6"] Jul 09 19:09:08.565353 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:08.565324 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b75c3ca-687d-4665-8b00-9dca0801eaff-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nrplf\" (UID: \"1b75c3ca-687d-4665-8b00-9dca0801eaff\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrplf" Jul 09 19:09:08.565483 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:08.565425 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btpns\" (UniqueName: \"kubernetes.io/projected/1b75c3ca-687d-4665-8b00-9dca0801eaff-kube-api-access-btpns\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nrplf\" (UID: \"1b75c3ca-687d-4665-8b00-9dca0801eaff\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrplf" Jul 09 19:09:08.593799 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:09:08.593764 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-7wmjm\" is forbidden: User \"system:node:ip-10-0-140-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-153.ec2.internal' and this object" podUID="4ec44b16-e08c-4c92-a2ab-132c335f8e2a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7wmjm" Jul 09 19:09:08.596804 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:09:08.596776 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qm2t6\" is forbidden: User \"system:node:ip-10-0-140-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-153.ec2.internal' and this object" podUID="45e77115-4fe0-41df-ba82-c0a94b6e66ac" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qm2t6" Jul 09 19:09:08.666094 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:08.666000 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b75c3ca-687d-4665-8b00-9dca0801eaff-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nrplf\" (UID: \"1b75c3ca-687d-4665-8b00-9dca0801eaff\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrplf" Jul 09 19:09:08.666522 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:08.666108 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj678\" (UniqueName: \"kubernetes.io/projected/d8cbf6a2-9f8a-45ef-b075-cd0830fd285e-kube-api-access-nj678\") pod \"limitador-operator-controller-manager-859d7b78bd-k4ws6\" (UID: \"d8cbf6a2-9f8a-45ef-b075-cd0830fd285e\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k4ws6" Jul 09 19:09:08.666522 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:08.666137 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btpns\" (UniqueName: \"kubernetes.io/projected/1b75c3ca-687d-4665-8b00-9dca0801eaff-kube-api-access-btpns\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nrplf\" (UID: \"1b75c3ca-687d-4665-8b00-9dca0801eaff\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrplf" Jul 09 19:09:08.666522 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:08.666484 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b75c3ca-687d-4665-8b00-9dca0801eaff-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nrplf\" (UID: \"1b75c3ca-687d-4665-8b00-9dca0801eaff\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrplf" Jul 09 19:09:08.687834 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:08.687811 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btpns\" (UniqueName: \"kubernetes.io/projected/1b75c3ca-687d-4665-8b00-9dca0801eaff-kube-api-access-btpns\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nrplf\" (UID: \"1b75c3ca-687d-4665-8b00-9dca0801eaff\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrplf" Jul 09 19:09:08.694642 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:08.694600 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qm2t6" Jul 09 19:09:08.696703 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:08.696682 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7wmjm" Jul 09 19:09:08.699058 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:09:08.699025 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-7wmjm\" is forbidden: User \"system:node:ip-10-0-140-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-153.ec2.internal' and this object" podUID="4ec44b16-e08c-4c92-a2ab-132c335f8e2a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7wmjm" Jul 09 19:09:08.702025 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:09:08.701990 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qm2t6\" is forbidden: User \"system:node:ip-10-0-140-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-153.ec2.internal' and this object" podUID="45e77115-4fe0-41df-ba82-c0a94b6e66ac" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qm2t6" Jul 09 19:09:08.704286 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:08.704260 2566 generic.go:356] "Generic (PLEG): container finished" podID="45e77115-4fe0-41df-ba82-c0a94b6e66ac" containerID="037196ef749a028e95680dcb620a183d57fdb38dfb9d1900b9db15fb2ea47fd7" exitCode=0 Jul 09 19:09:08.704367 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:08.704294 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qm2t6" Jul 09 19:09:08.704367 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:08.704320 2566 scope.go:117] "RemoveContainer" containerID="037196ef749a028e95680dcb620a183d57fdb38dfb9d1900b9db15fb2ea47fd7" Jul 09 19:09:08.704882 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:09:08.704852 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-7wmjm\" is forbidden: User \"system:node:ip-10-0-140-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-153.ec2.internal' and this object" podUID="4ec44b16-e08c-4c92-a2ab-132c335f8e2a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7wmjm" Jul 09 19:09:08.705824 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:08.705794 2566 generic.go:356] "Generic (PLEG): container finished" podID="4ec44b16-e08c-4c92-a2ab-132c335f8e2a" containerID="0d3fa9b33316f7453ef40749e73e5dbbe1c9af859fba53e292d41203a0932a96" exitCode=0 Jul 09 19:09:08.705824 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:08.705820 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7wmjm" Jul 09 19:09:08.707797 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:09:08.707771 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qm2t6\" is forbidden: User \"system:node:ip-10-0-140-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-153.ec2.internal' and this object" podUID="45e77115-4fe0-41df-ba82-c0a94b6e66ac" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qm2t6" Jul 09 19:09:08.710822 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:09:08.710796 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-7wmjm\" is forbidden: User \"system:node:ip-10-0-140-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-153.ec2.internal' and this object" podUID="4ec44b16-e08c-4c92-a2ab-132c335f8e2a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7wmjm" Jul 09 19:09:08.713458 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:08.713442 2566 scope.go:117] "RemoveContainer" containerID="037196ef749a028e95680dcb620a183d57fdb38dfb9d1900b9db15fb2ea47fd7" Jul 09 19:09:08.713685 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:09:08.713664 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qm2t6\" is forbidden: User \"system:node:ip-10-0-140-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-153.ec2.internal' and this object" podUID="45e77115-4fe0-41df-ba82-c0a94b6e66ac" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qm2t6" Jul 09 19:09:08.713751 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:09:08.713735 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"037196ef749a028e95680dcb620a183d57fdb38dfb9d1900b9db15fb2ea47fd7\": container with ID starting with 037196ef749a028e95680dcb620a183d57fdb38dfb9d1900b9db15fb2ea47fd7 not found: ID does not exist" containerID="037196ef749a028e95680dcb620a183d57fdb38dfb9d1900b9db15fb2ea47fd7" Jul 09 19:09:08.713792 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:08.713758 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"037196ef749a028e95680dcb620a183d57fdb38dfb9d1900b9db15fb2ea47fd7"} err="failed to get container status \"037196ef749a028e95680dcb620a183d57fdb38dfb9d1900b9db15fb2ea47fd7\": rpc error: code = NotFound desc = could not find container \"037196ef749a028e95680dcb620a183d57fdb38dfb9d1900b9db15fb2ea47fd7\": container with ID starting with 037196ef749a028e95680dcb620a183d57fdb38dfb9d1900b9db15fb2ea47fd7 not found: ID does not exist" Jul 09 19:09:08.713792 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:08.713775 2566 scope.go:117] "RemoveContainer" containerID="0d3fa9b33316f7453ef40749e73e5dbbe1c9af859fba53e292d41203a0932a96" Jul 09 19:09:08.721732 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:08.721717 2566 scope.go:117] "RemoveContainer" containerID="0d3fa9b33316f7453ef40749e73e5dbbe1c9af859fba53e292d41203a0932a96" Jul 09 19:09:08.721991 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:09:08.721970 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d3fa9b33316f7453ef40749e73e5dbbe1c9af859fba53e292d41203a0932a96\": container with ID starting with 0d3fa9b33316f7453ef40749e73e5dbbe1c9af859fba53e292d41203a0932a96 not found: ID does not exist" containerID="0d3fa9b33316f7453ef40749e73e5dbbe1c9af859fba53e292d41203a0932a96" Jul 09 19:09:08.722048 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:08.721989 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d3fa9b33316f7453ef40749e73e5dbbe1c9af859fba53e292d41203a0932a96"} err="failed to get container status \"0d3fa9b33316f7453ef40749e73e5dbbe1c9af859fba53e292d41203a0932a96\": rpc error: code = NotFound desc = could not find container \"0d3fa9b33316f7453ef40749e73e5dbbe1c9af859fba53e292d41203a0932a96\": container with ID starting with 0d3fa9b33316f7453ef40749e73e5dbbe1c9af859fba53e292d41203a0932a96 not found: ID does not exist" Jul 09 19:09:08.767367 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:08.767343 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xpbx\" (UniqueName: \"kubernetes.io/projected/45e77115-4fe0-41df-ba82-c0a94b6e66ac-kube-api-access-6xpbx\") pod \"45e77115-4fe0-41df-ba82-c0a94b6e66ac\" (UID: \"45e77115-4fe0-41df-ba82-c0a94b6e66ac\") " Jul 09 19:09:08.767463 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:08.767392 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/45e77115-4fe0-41df-ba82-c0a94b6e66ac-extensions-socket-volume\") pod \"45e77115-4fe0-41df-ba82-c0a94b6e66ac\" (UID: \"45e77115-4fe0-41df-ba82-c0a94b6e66ac\") " Jul 09 19:09:08.767568 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:08.767550 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj678\" (UniqueName: \"kubernetes.io/projected/d8cbf6a2-9f8a-45ef-b075-cd0830fd285e-kube-api-access-nj678\") pod \"limitador-operator-controller-manager-859d7b78bd-k4ws6\" (UID: \"d8cbf6a2-9f8a-45ef-b075-cd0830fd285e\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k4ws6" Jul 09 19:09:08.767918 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:08.767886 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45e77115-4fe0-41df-ba82-c0a94b6e66ac-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "45e77115-4fe0-41df-ba82-c0a94b6e66ac" (UID: "45e77115-4fe0-41df-ba82-c0a94b6e66ac"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:09:08.769307 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:08.769289 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45e77115-4fe0-41df-ba82-c0a94b6e66ac-kube-api-access-6xpbx" (OuterVolumeSpecName: "kube-api-access-6xpbx") pod "45e77115-4fe0-41df-ba82-c0a94b6e66ac" (UID: "45e77115-4fe0-41df-ba82-c0a94b6e66ac"). InnerVolumeSpecName "kube-api-access-6xpbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:09:08.790735 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:08.790714 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj678\" (UniqueName: \"kubernetes.io/projected/d8cbf6a2-9f8a-45ef-b075-cd0830fd285e-kube-api-access-nj678\") pod \"limitador-operator-controller-manager-859d7b78bd-k4ws6\" (UID: \"d8cbf6a2-9f8a-45ef-b075-cd0830fd285e\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k4ws6" Jul 09 19:09:08.847513 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:08.847487 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrplf" Jul 09 19:09:08.854854 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:08.854835 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k4ws6" Jul 09 19:09:08.868795 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:08.868770 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhzhq\" (UniqueName: \"kubernetes.io/projected/4ec44b16-e08c-4c92-a2ab-132c335f8e2a-kube-api-access-hhzhq\") pod \"4ec44b16-e08c-4c92-a2ab-132c335f8e2a\" (UID: \"4ec44b16-e08c-4c92-a2ab-132c335f8e2a\") " Jul 09 19:09:08.869146 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:08.869125 2566 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/45e77115-4fe0-41df-ba82-c0a94b6e66ac-extensions-socket-volume\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:09:08.869202 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:08.869153 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xpbx\" (UniqueName: \"kubernetes.io/projected/45e77115-4fe0-41df-ba82-c0a94b6e66ac-kube-api-access-6xpbx\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:09:08.870788 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:08.870766 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ec44b16-e08c-4c92-a2ab-132c335f8e2a-kube-api-access-hhzhq" (OuterVolumeSpecName: "kube-api-access-hhzhq") pod "4ec44b16-e08c-4c92-a2ab-132c335f8e2a" (UID: "4ec44b16-e08c-4c92-a2ab-132c335f8e2a"). InnerVolumeSpecName "kube-api-access-hhzhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:09:08.970336 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:08.970303 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhzhq\" (UniqueName: \"kubernetes.io/projected/4ec44b16-e08c-4c92-a2ab-132c335f8e2a-kube-api-access-hhzhq\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:09:09.018134 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:09:09.018103 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-7wmjm\" is forbidden: User \"system:node:ip-10-0-140-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-153.ec2.internal' and this object" podUID="4ec44b16-e08c-4c92-a2ab-132c335f8e2a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7wmjm" Jul 09 19:09:09.021180 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:09:09.021156 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qm2t6\" is forbidden: User \"system:node:ip-10-0-140-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-153.ec2.internal' and this object" podUID="45e77115-4fe0-41df-ba82-c0a94b6e66ac" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qm2t6" Jul 09 19:09:09.023890 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:09:09.023850 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-7wmjm\" is forbidden: User \"system:node:ip-10-0-140-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-153.ec2.internal' and this object" podUID="4ec44b16-e08c-4c92-a2ab-132c335f8e2a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7wmjm" Jul 09 19:09:09.026698 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:09:09.026654 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qm2t6\" is forbidden: User \"system:node:ip-10-0-140-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-153.ec2.internal' and this object" podUID="45e77115-4fe0-41df-ba82-c0a94b6e66ac" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qm2t6" Jul 09 19:09:09.026918 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:09.026898 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrplf"] Jul 09 19:09:09.027958 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:09:09.027936 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b75c3ca_687d_4665_8b00_9dca0801eaff.slice/crio-c0b8ff6d2a2af2767702639eaaffcf1dedbfa887d26bbc40d655fee4d61eade4 WatchSource:0}: Error finding container c0b8ff6d2a2af2767702639eaaffcf1dedbfa887d26bbc40d655fee4d61eade4: Status 404 returned error can't find the container with id c0b8ff6d2a2af2767702639eaaffcf1dedbfa887d26bbc40d655fee4d61eade4 Jul 09 19:09:09.047008 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:09.046985 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k4ws6"] Jul 09 19:09:09.049430 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:09:09.049390 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8cbf6a2_9f8a_45ef_b075_cd0830fd285e.slice/crio-804fb63d517db9a4426fde73e2f0908dfc777aae08b6b1866fbee6ca86aacd3c WatchSource:0}: Error finding container 804fb63d517db9a4426fde73e2f0908dfc777aae08b6b1866fbee6ca86aacd3c: Status 404 returned error can't find the container with id 804fb63d517db9a4426fde73e2f0908dfc777aae08b6b1866fbee6ca86aacd3c Jul 09 19:09:09.057296 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:09.057268 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45e77115-4fe0-41df-ba82-c0a94b6e66ac" path="/var/lib/kubelet/pods/45e77115-4fe0-41df-ba82-c0a94b6e66ac/volumes" Jul 09 19:09:09.057579 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:09.057564 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ec44b16-e08c-4c92-a2ab-132c335f8e2a" path="/var/lib/kubelet/pods/4ec44b16-e08c-4c92-a2ab-132c335f8e2a/volumes" Jul 09 19:09:09.715481 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:09.715385 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k4ws6" event={"ID":"d8cbf6a2-9f8a-45ef-b075-cd0830fd285e","Type":"ContainerStarted","Data":"cd44f081f991e06e90e430cba79b4ff4895461bad5983f712bddc94beac0e0dd"} Jul 09 19:09:09.715481 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:09.715461 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k4ws6" Jul 09 19:09:09.715937 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:09.715487 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k4ws6" event={"ID":"d8cbf6a2-9f8a-45ef-b075-cd0830fd285e","Type":"ContainerStarted","Data":"804fb63d517db9a4426fde73e2f0908dfc777aae08b6b1866fbee6ca86aacd3c"} Jul 09 19:09:09.717772 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:09.717744 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrplf" event={"ID":"1b75c3ca-687d-4665-8b00-9dca0801eaff","Type":"ContainerStarted","Data":"783de835f234408a262f65fa6955712617817113a08e3a08c58990fc6d38f446"} Jul 09 19:09:09.717881 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:09.717788 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrplf" event={"ID":"1b75c3ca-687d-4665-8b00-9dca0801eaff","Type":"ContainerStarted","Data":"c0b8ff6d2a2af2767702639eaaffcf1dedbfa887d26bbc40d655fee4d61eade4"} Jul 09 19:09:09.717881 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:09.717816 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrplf" Jul 09 19:09:09.748662 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:09.748596 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k4ws6" podStartSLOduration=1.748582012 podStartE2EDuration="1.748582012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:09:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:09:09.748228889 +0000 UTC m=+619.256008662" watchObservedRunningTime="2026-07-09 19:09:09.748582012 +0000 UTC m=+619.256361764" Jul 09 19:09:09.788060 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:09.788011 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrplf" podStartSLOduration=1.7879946759999998 podStartE2EDuration="1.787994676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:09:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:09:09.787423314 +0000 UTC m=+619.295203092" watchObservedRunningTime="2026-07-09 19:09:09.787994676 +0000 UTC m=+619.295774449" Jul 09 19:09:10.401875 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:10.401843 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ndcrm"] Jul 09 19:09:10.405481 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:10.405459 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ndcrm" Jul 09 19:09:10.433228 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:10.433200 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ndcrm"] Jul 09 19:09:10.483732 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:10.483700 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5dsj\" (UniqueName: \"kubernetes.io/projected/b51e710e-4986-4534-85c6-4f695e01df88-kube-api-access-d5dsj\") pod \"kuadrant-operator-controller-manager-86878c769f-ndcrm\" (UID: \"b51e710e-4986-4534-85c6-4f695e01df88\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ndcrm" Jul 09 19:09:10.483888 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:10.483735 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b51e710e-4986-4534-85c6-4f695e01df88-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-ndcrm\" (UID: \"b51e710e-4986-4534-85c6-4f695e01df88\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ndcrm" Jul 09 19:09:10.585015 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:10.584977 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5dsj\" (UniqueName: \"kubernetes.io/projected/b51e710e-4986-4534-85c6-4f695e01df88-kube-api-access-d5dsj\") pod \"kuadrant-operator-controller-manager-86878c769f-ndcrm\" (UID: \"b51e710e-4986-4534-85c6-4f695e01df88\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ndcrm" Jul 09 19:09:10.585015 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:10.585020 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b51e710e-4986-4534-85c6-4f695e01df88-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-ndcrm\" (UID: \"b51e710e-4986-4534-85c6-4f695e01df88\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ndcrm" Jul 09 19:09:10.585386 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:10.585371 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b51e710e-4986-4534-85c6-4f695e01df88-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-ndcrm\" (UID: \"b51e710e-4986-4534-85c6-4f695e01df88\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ndcrm" Jul 09 19:09:10.597624 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:10.597588 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5dsj\" (UniqueName: \"kubernetes.io/projected/b51e710e-4986-4534-85c6-4f695e01df88-kube-api-access-d5dsj\") pod \"kuadrant-operator-controller-manager-86878c769f-ndcrm\" (UID: \"b51e710e-4986-4534-85c6-4f695e01df88\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ndcrm" Jul 09 19:09:10.664744 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:10.664666 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2jcmm" Jul 09 19:09:10.715816 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:10.715773 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ndcrm" Jul 09 19:09:10.867836 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:10.867814 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ndcrm"] Jul 09 19:09:10.869308 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:09:10.869283 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb51e710e_4986_4534_85c6_4f695e01df88.slice/crio-d75a76f3ff4a2096e8b9cb8b811a285fc5b7ce67e4d1cc58032413f76661a20c WatchSource:0}: Error finding container d75a76f3ff4a2096e8b9cb8b811a285fc5b7ce67e4d1cc58032413f76661a20c: Status 404 returned error can't find the container with id d75a76f3ff4a2096e8b9cb8b811a285fc5b7ce67e4d1cc58032413f76661a20c Jul 09 19:09:11.731161 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:11.731124 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ndcrm" event={"ID":"b51e710e-4986-4534-85c6-4f695e01df88","Type":"ContainerStarted","Data":"ce76555027785a617ee55e6828ec0300073de97b758ad40b072a187c045f3008"} Jul 09 19:09:11.731580 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:11.731182 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ndcrm" event={"ID":"b51e710e-4986-4534-85c6-4f695e01df88","Type":"ContainerStarted","Data":"d75a76f3ff4a2096e8b9cb8b811a285fc5b7ce67e4d1cc58032413f76661a20c"} Jul 09 19:09:11.731580 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:11.731339 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ndcrm" Jul 09 19:09:11.756791 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:11.756737 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ndcrm" podStartSLOduration=1.756719569 podStartE2EDuration="1.756719569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:09:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:09:11.754871943 +0000 UTC m=+621.262651716" watchObservedRunningTime="2026-07-09 19:09:11.756719569 +0000 UTC m=+621.264499344" Jul 09 19:09:20.725936 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:20.725865 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrplf" Jul 09 19:09:20.726310 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:20.725976 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-k4ws6" Jul 09 19:09:22.739012 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:22.738979 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ndcrm" Jul 09 19:09:22.790834 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:22.790799 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrplf"] Jul 09 19:09:22.791042 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:22.791001 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrplf" podUID="1b75c3ca-687d-4665-8b00-9dca0801eaff" containerName="manager" containerID="cri-o://783de835f234408a262f65fa6955712617817113a08e3a08c58990fc6d38f446" gracePeriod=10 Jul 09 19:09:23.052205 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:23.052186 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrplf" Jul 09 19:09:23.095243 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:23.095217 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btpns\" (UniqueName: \"kubernetes.io/projected/1b75c3ca-687d-4665-8b00-9dca0801eaff-kube-api-access-btpns\") pod \"1b75c3ca-687d-4665-8b00-9dca0801eaff\" (UID: \"1b75c3ca-687d-4665-8b00-9dca0801eaff\") " Jul 09 19:09:23.095384 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:23.095260 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b75c3ca-687d-4665-8b00-9dca0801eaff-extensions-socket-volume\") pod \"1b75c3ca-687d-4665-8b00-9dca0801eaff\" (UID: \"1b75c3ca-687d-4665-8b00-9dca0801eaff\") " Jul 09 19:09:23.095710 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:23.095681 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b75c3ca-687d-4665-8b00-9dca0801eaff-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "1b75c3ca-687d-4665-8b00-9dca0801eaff" (UID: "1b75c3ca-687d-4665-8b00-9dca0801eaff"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:09:23.097203 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:23.097182 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b75c3ca-687d-4665-8b00-9dca0801eaff-kube-api-access-btpns" (OuterVolumeSpecName: "kube-api-access-btpns") pod "1b75c3ca-687d-4665-8b00-9dca0801eaff" (UID: "1b75c3ca-687d-4665-8b00-9dca0801eaff"). InnerVolumeSpecName "kube-api-access-btpns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:09:23.195755 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:23.195721 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btpns\" (UniqueName: \"kubernetes.io/projected/1b75c3ca-687d-4665-8b00-9dca0801eaff-kube-api-access-btpns\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:09:23.195755 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:23.195749 2566 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b75c3ca-687d-4665-8b00-9dca0801eaff-extensions-socket-volume\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:09:23.799057 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:23.799023 2566 generic.go:356] "Generic (PLEG): container finished" podID="1b75c3ca-687d-4665-8b00-9dca0801eaff" containerID="783de835f234408a262f65fa6955712617817113a08e3a08c58990fc6d38f446" exitCode=0 Jul 09 19:09:23.799445 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:23.799068 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrplf" Jul 09 19:09:23.799445 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:23.799117 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrplf" event={"ID":"1b75c3ca-687d-4665-8b00-9dca0801eaff","Type":"ContainerDied","Data":"783de835f234408a262f65fa6955712617817113a08e3a08c58990fc6d38f446"} Jul 09 19:09:23.799445 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:23.799176 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrplf" event={"ID":"1b75c3ca-687d-4665-8b00-9dca0801eaff","Type":"ContainerDied","Data":"c0b8ff6d2a2af2767702639eaaffcf1dedbfa887d26bbc40d655fee4d61eade4"} Jul 09 19:09:23.799445 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:23.799200 2566 scope.go:117] "RemoveContainer" containerID="783de835f234408a262f65fa6955712617817113a08e3a08c58990fc6d38f446" Jul 09 19:09:23.808441 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:23.808426 2566 scope.go:117] "RemoveContainer" containerID="783de835f234408a262f65fa6955712617817113a08e3a08c58990fc6d38f446" Jul 09 19:09:23.808725 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:09:23.808706 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"783de835f234408a262f65fa6955712617817113a08e3a08c58990fc6d38f446\": container with ID starting with 783de835f234408a262f65fa6955712617817113a08e3a08c58990fc6d38f446 not found: ID does not exist" containerID="783de835f234408a262f65fa6955712617817113a08e3a08c58990fc6d38f446" Jul 09 19:09:23.808812 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:23.808738 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"783de835f234408a262f65fa6955712617817113a08e3a08c58990fc6d38f446"} err="failed to get container status \"783de835f234408a262f65fa6955712617817113a08e3a08c58990fc6d38f446\": rpc error: code = NotFound desc = could not find container \"783de835f234408a262f65fa6955712617817113a08e3a08c58990fc6d38f446\": container with ID starting with 783de835f234408a262f65fa6955712617817113a08e3a08c58990fc6d38f446 not found: ID does not exist" Jul 09 19:09:23.826357 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:23.826329 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrplf"] Jul 09 19:09:23.829584 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:23.829552 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nrplf"] Jul 09 19:09:25.057504 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:25.057469 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b75c3ca-687d-4665-8b00-9dca0801eaff" path="/var/lib/kubelet/pods/1b75c3ca-687d-4665-8b00-9dca0801eaff/volumes" Jul 09 19:09:39.414953 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:39.414917 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qwmxj"] Jul 09 19:09:39.415824 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:39.415405 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b75c3ca-687d-4665-8b00-9dca0801eaff" containerName="manager" Jul 09 19:09:39.415824 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:39.415425 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b75c3ca-687d-4665-8b00-9dca0801eaff" containerName="manager" Jul 09 19:09:39.415824 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:39.415516 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b75c3ca-687d-4665-8b00-9dca0801eaff" containerName="manager" Jul 09 19:09:39.420379 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:39.420360 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qwmxj" Jul 09 19:09:39.423299 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:39.423282 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-fg6jz\"" Jul 09 19:09:39.430780 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:39.430757 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qwmxj"] Jul 09 19:09:39.519054 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:39.519022 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5df48a3-3a56-4ff6-967e-984b325a7a2e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-qwmxj\" (UID: \"e5df48a3-3a56-4ff6-967e-984b325a7a2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qwmxj" Jul 09 19:09:39.519054 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:39.519056 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mb9x\" (UniqueName: \"kubernetes.io/projected/e5df48a3-3a56-4ff6-967e-984b325a7a2e-kube-api-access-5mb9x\") pod \"maas-default-gateway-openshift-default-56654dcf7-qwmxj\" (UID: \"e5df48a3-3a56-4ff6-967e-984b325a7a2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qwmxj" Jul 09 19:09:39.519273 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:39.519084 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5df48a3-3a56-4ff6-967e-984b325a7a2e-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-qwmxj\" (UID: \"e5df48a3-3a56-4ff6-967e-984b325a7a2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qwmxj" Jul 09 19:09:39.519273 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:39.519111 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5df48a3-3a56-4ff6-967e-984b325a7a2e-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-qwmxj\" (UID: \"e5df48a3-3a56-4ff6-967e-984b325a7a2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qwmxj" Jul 09 19:09:39.519273 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:39.519148 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5df48a3-3a56-4ff6-967e-984b325a7a2e-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-qwmxj\" (UID: \"e5df48a3-3a56-4ff6-967e-984b325a7a2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qwmxj" Jul 09 19:09:39.519273 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:39.519203 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5df48a3-3a56-4ff6-967e-984b325a7a2e-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-qwmxj\" (UID: \"e5df48a3-3a56-4ff6-967e-984b325a7a2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qwmxj" Jul 09 19:09:39.519273 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:39.519242 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5df48a3-3a56-4ff6-967e-984b325a7a2e-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-qwmxj\" (UID: \"e5df48a3-3a56-4ff6-967e-984b325a7a2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qwmxj" Jul 09 19:09:39.519273 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:39.519260 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5df48a3-3a56-4ff6-967e-984b325a7a2e-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-qwmxj\" (UID: \"e5df48a3-3a56-4ff6-967e-984b325a7a2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qwmxj" Jul 09 19:09:39.519460 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:39.519303 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5df48a3-3a56-4ff6-967e-984b325a7a2e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-qwmxj\" (UID: \"e5df48a3-3a56-4ff6-967e-984b325a7a2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qwmxj" Jul 09 19:09:39.620044 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:39.620008 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5df48a3-3a56-4ff6-967e-984b325a7a2e-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-qwmxj\" (UID: \"e5df48a3-3a56-4ff6-967e-984b325a7a2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qwmxj" Jul 09 19:09:39.620044 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:39.620049 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5df48a3-3a56-4ff6-967e-984b325a7a2e-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-qwmxj\" (UID: \"e5df48a3-3a56-4ff6-967e-984b325a7a2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qwmxj" Jul 09 19:09:39.620296 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:39.620104 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5df48a3-3a56-4ff6-967e-984b325a7a2e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-qwmxj\" (UID: \"e5df48a3-3a56-4ff6-967e-984b325a7a2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qwmxj" Jul 09 19:09:39.620296 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:39.620249 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5df48a3-3a56-4ff6-967e-984b325a7a2e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-qwmxj\" (UID: \"e5df48a3-3a56-4ff6-967e-984b325a7a2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qwmxj" Jul 09 19:09:39.620296 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:39.620281 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mb9x\" (UniqueName: \"kubernetes.io/projected/e5df48a3-3a56-4ff6-967e-984b325a7a2e-kube-api-access-5mb9x\") pod \"maas-default-gateway-openshift-default-56654dcf7-qwmxj\" (UID: \"e5df48a3-3a56-4ff6-967e-984b325a7a2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qwmxj" Jul 09 19:09:39.620460 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:39.620329 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5df48a3-3a56-4ff6-967e-984b325a7a2e-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-qwmxj\" (UID: \"e5df48a3-3a56-4ff6-967e-984b325a7a2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qwmxj" Jul 09 19:09:39.620460 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:39.620351 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5df48a3-3a56-4ff6-967e-984b325a7a2e-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-qwmxj\" (UID: \"e5df48a3-3a56-4ff6-967e-984b325a7a2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qwmxj" Jul 09 19:09:39.620460 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:39.620366 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5df48a3-3a56-4ff6-967e-984b325a7a2e-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-qwmxj\" (UID: \"e5df48a3-3a56-4ff6-967e-984b325a7a2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qwmxj" Jul 09 19:09:39.620460 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:39.620395 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5df48a3-3a56-4ff6-967e-984b325a7a2e-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-qwmxj\" (UID: \"e5df48a3-3a56-4ff6-967e-984b325a7a2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qwmxj" Jul 09 19:09:39.620460 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:39.620441 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5df48a3-3a56-4ff6-967e-984b325a7a2e-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-qwmxj\" (UID: \"e5df48a3-3a56-4ff6-967e-984b325a7a2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qwmxj" Jul 09 19:09:39.620750 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:39.620735 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5df48a3-3a56-4ff6-967e-984b325a7a2e-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-qwmxj\" (UID: \"e5df48a3-3a56-4ff6-967e-984b325a7a2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qwmxj" Jul 09 19:09:39.621047 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:39.621023 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5df48a3-3a56-4ff6-967e-984b325a7a2e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-qwmxj\" (UID: \"e5df48a3-3a56-4ff6-967e-984b325a7a2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qwmxj" Jul 09 19:09:39.621119 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:39.621053 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5df48a3-3a56-4ff6-967e-984b325a7a2e-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-qwmxj\" (UID: \"e5df48a3-3a56-4ff6-967e-984b325a7a2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qwmxj" Jul 09 19:09:39.621119 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:39.621100 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5df48a3-3a56-4ff6-967e-984b325a7a2e-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-qwmxj\" (UID: \"e5df48a3-3a56-4ff6-967e-984b325a7a2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qwmxj" Jul 09 19:09:39.622498 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:39.622477 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5df48a3-3a56-4ff6-967e-984b325a7a2e-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-qwmxj\" (UID: \"e5df48a3-3a56-4ff6-967e-984b325a7a2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qwmxj" Jul 09 19:09:39.622655 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:39.622640 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5df48a3-3a56-4ff6-967e-984b325a7a2e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-qwmxj\" (UID: \"e5df48a3-3a56-4ff6-967e-984b325a7a2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qwmxj" Jul 09 19:09:39.629231 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:39.629204 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5df48a3-3a56-4ff6-967e-984b325a7a2e-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-qwmxj\" (UID: \"e5df48a3-3a56-4ff6-967e-984b325a7a2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qwmxj" Jul 09 19:09:39.629665 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:39.629646 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mb9x\" (UniqueName: \"kubernetes.io/projected/e5df48a3-3a56-4ff6-967e-984b325a7a2e-kube-api-access-5mb9x\") pod \"maas-default-gateway-openshift-default-56654dcf7-qwmxj\" (UID: \"e5df48a3-3a56-4ff6-967e-984b325a7a2e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qwmxj" Jul 09 19:09:39.731791 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:39.731757 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qwmxj" Jul 09 19:09:39.875593 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:39.875566 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qwmxj"] Jul 09 19:09:39.877445 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:09:39.877412 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5df48a3_3a56_4ff6_967e_984b325a7a2e.slice/crio-ab5319fc68e76d9d26f8bc61e234edf58a8bbef920581488f93aeef3bcda9af2 WatchSource:0}: Error finding container ab5319fc68e76d9d26f8bc61e234edf58a8bbef920581488f93aeef3bcda9af2: Status 404 returned error can't find the container with id ab5319fc68e76d9d26f8bc61e234edf58a8bbef920581488f93aeef3bcda9af2 Jul 09 19:09:39.879776 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:39.879742 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:09:39.879852 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:39.879833 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:09:39.879913 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:39.879896 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:09:39.889198 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:39.889178 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qwmxj" event={"ID":"e5df48a3-3a56-4ff6-967e-984b325a7a2e","Type":"ContainerStarted","Data":"ab5319fc68e76d9d26f8bc61e234edf58a8bbef920581488f93aeef3bcda9af2"} Jul 09 19:09:40.902848 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:40.902815 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qwmxj" event={"ID":"e5df48a3-3a56-4ff6-967e-984b325a7a2e","Type":"ContainerStarted","Data":"20f7f312507571222d51bb52f46348e4a9b8c5ef79d6984ed14f4cf90908a300"} Jul 09 19:09:40.926651 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:40.926586 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qwmxj" podStartSLOduration=1.926569692 podStartE2EDuration="1.926569692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:09:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:09:40.92329252 +0000 UTC m=+650.431072295" watchObservedRunningTime="2026-07-09 19:09:40.926569692 +0000 UTC m=+650.434349544" Jul 09 19:09:41.732944 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:41.732903 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qwmxj" Jul 09 19:09:41.737805 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:41.737778 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qwmxj" Jul 09 19:09:41.908597 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:41.908571 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qwmxj" Jul 09 19:09:41.909508 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:41.909491 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qwmxj" Jul 09 19:09:42.472903 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:42.472867 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:09:42.475315 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:42.475296 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-mtw72" Jul 09 19:09:42.478352 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:42.478333 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-4wjtj\"" Jul 09 19:09:42.478457 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:42.478386 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 09 19:09:42.484816 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:42.484793 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:09:42.513290 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:42.513254 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:09:42.544079 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:42.544016 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a755661f-8b26-45ab-845a-c746557b80bc-config-file\") pod \"limitador-limitador-54d6d988d5-mtw72\" (UID: \"a755661f-8b26-45ab-845a-c746557b80bc\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-mtw72" Jul 09 19:09:42.544246 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:42.544091 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkfvg\" (UniqueName: \"kubernetes.io/projected/a755661f-8b26-45ab-845a-c746557b80bc-kube-api-access-hkfvg\") pod \"limitador-limitador-54d6d988d5-mtw72\" (UID: \"a755661f-8b26-45ab-845a-c746557b80bc\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-mtw72" Jul 09 19:09:42.644563 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:42.644525 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a755661f-8b26-45ab-845a-c746557b80bc-config-file\") pod \"limitador-limitador-54d6d988d5-mtw72\" (UID: \"a755661f-8b26-45ab-845a-c746557b80bc\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-mtw72" Jul 09 19:09:42.644797 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:42.644581 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkfvg\" (UniqueName: \"kubernetes.io/projected/a755661f-8b26-45ab-845a-c746557b80bc-kube-api-access-hkfvg\") pod \"limitador-limitador-54d6d988d5-mtw72\" (UID: \"a755661f-8b26-45ab-845a-c746557b80bc\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-mtw72" Jul 09 19:09:42.645214 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:42.645192 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a755661f-8b26-45ab-845a-c746557b80bc-config-file\") pod \"limitador-limitador-54d6d988d5-mtw72\" (UID: \"a755661f-8b26-45ab-845a-c746557b80bc\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-mtw72" Jul 09 19:09:42.652916 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:42.652888 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkfvg\" (UniqueName: \"kubernetes.io/projected/a755661f-8b26-45ab-845a-c746557b80bc-kube-api-access-hkfvg\") pod \"limitador-limitador-54d6d988d5-mtw72\" (UID: \"a755661f-8b26-45ab-845a-c746557b80bc\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-mtw72" Jul 09 19:09:42.787787 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:42.787709 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-mtw72" Jul 09 19:09:42.924106 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:42.924083 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:09:42.925770 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:09:42.925749 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda755661f_8b26_45ab_845a_c746557b80bc.slice/crio-c02c11692a82f1ef26f7792988d914c213518a507531f58882a26774d462bf14 WatchSource:0}: Error finding container c02c11692a82f1ef26f7792988d914c213518a507531f58882a26774d462bf14: Status 404 returned error can't find the container with id c02c11692a82f1ef26f7792988d914c213518a507531f58882a26774d462bf14 Jul 09 19:09:42.958379 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:42.958346 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-2s6fx"] Jul 09 19:09:42.961528 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:42.961512 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-2s6fx" Jul 09 19:09:42.964576 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:42.964556 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vfbr4\"" Jul 09 19:09:42.969054 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:42.969033 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-2s6fx"] Jul 09 19:09:43.048029 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:43.047946 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k48fl\" (UniqueName: \"kubernetes.io/projected/45ce6967-c17d-47d3-98f6-781bf3be4d47-kube-api-access-k48fl\") pod \"authorino-58cd955675-2s6fx\" (UID: \"45ce6967-c17d-47d3-98f6-781bf3be4d47\") " pod="kuadrant-system/authorino-58cd955675-2s6fx" Jul 09 19:09:43.148810 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:43.148780 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k48fl\" (UniqueName: \"kubernetes.io/projected/45ce6967-c17d-47d3-98f6-781bf3be4d47-kube-api-access-k48fl\") pod \"authorino-58cd955675-2s6fx\" (UID: \"45ce6967-c17d-47d3-98f6-781bf3be4d47\") " pod="kuadrant-system/authorino-58cd955675-2s6fx" Jul 09 19:09:43.157705 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:43.157671 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k48fl\" (UniqueName: \"kubernetes.io/projected/45ce6967-c17d-47d3-98f6-781bf3be4d47-kube-api-access-k48fl\") pod \"authorino-58cd955675-2s6fx\" (UID: \"45ce6967-c17d-47d3-98f6-781bf3be4d47\") " pod="kuadrant-system/authorino-58cd955675-2s6fx" Jul 09 19:09:43.273342 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:43.273303 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-2s6fx" Jul 09 19:09:43.409385 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:43.409346 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-2s6fx"] Jul 09 19:09:43.410832 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:09:43.410803 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod45ce6967_c17d_47d3_98f6_781bf3be4d47.slice/crio-b4e8a171213fdba193514b67af85477487a4ba013397706d700332e9f7a616f9 WatchSource:0}: Error finding container b4e8a171213fdba193514b67af85477487a4ba013397706d700332e9f7a616f9: Status 404 returned error can't find the container with id b4e8a171213fdba193514b67af85477487a4ba013397706d700332e9f7a616f9 Jul 09 19:09:43.922883 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:43.922846 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-mtw72" event={"ID":"a755661f-8b26-45ab-845a-c746557b80bc","Type":"ContainerStarted","Data":"c02c11692a82f1ef26f7792988d914c213518a507531f58882a26774d462bf14"} Jul 09 19:09:43.924480 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:43.924450 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-2s6fx" event={"ID":"45ce6967-c17d-47d3-98f6-781bf3be4d47","Type":"ContainerStarted","Data":"b4e8a171213fdba193514b67af85477487a4ba013397706d700332e9f7a616f9"} Jul 09 19:09:47.959447 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:47.959408 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-mtw72" event={"ID":"a755661f-8b26-45ab-845a-c746557b80bc","Type":"ContainerStarted","Data":"dad69e52f042444c23819cd7860318f633076ab4cdda58c37dbba9d0ddbbf128"} Jul 09 19:09:47.959961 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:47.959479 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-mtw72" Jul 09 19:09:47.960987 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:47.960961 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-2s6fx" event={"ID":"45ce6967-c17d-47d3-98f6-781bf3be4d47","Type":"ContainerStarted","Data":"16fad7432be90058969aec4280e2c2cf08ae702f4ead4434456176a03ec08a4a"} Jul 09 19:09:47.979909 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:47.979844 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-mtw72" podStartSLOduration=1.541746402 podStartE2EDuration="5.97982744s" totalImagesPullingTime="4.438081038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:09:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:09:47.978285753 +0000 UTC m=+657.486065528" watchObservedRunningTime="2026-07-09 19:09:47.97982744 +0000 UTC m=+657.487607256" Jul 09 19:09:47.994745 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:47.994702 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-2s6fx" podStartSLOduration=2.052341904 podStartE2EDuration="5.994686005s" totalImagesPullingTime="3.942344101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:09:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:09:47.993970179 +0000 UTC m=+657.501749967" watchObservedRunningTime="2026-07-09 19:09:47.994686005 +0000 UTC m=+657.502465778" Jul 09 19:09:49.908878 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:49.908848 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-2s6fx"] Jul 09 19:09:49.971921 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:49.971879 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-2s6fx" podUID="45ce6967-c17d-47d3-98f6-781bf3be4d47" containerName="authorino" containerID="cri-o://16fad7432be90058969aec4280e2c2cf08ae702f4ead4434456176a03ec08a4a" gracePeriod=30 Jul 09 19:09:50.226922 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:50.226895 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-2s6fx" Jul 09 19:09:50.317023 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:50.316989 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k48fl\" (UniqueName: \"kubernetes.io/projected/45ce6967-c17d-47d3-98f6-781bf3be4d47-kube-api-access-k48fl\") pod \"45ce6967-c17d-47d3-98f6-781bf3be4d47\" (UID: \"45ce6967-c17d-47d3-98f6-781bf3be4d47\") " Jul 09 19:09:50.319077 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:50.319046 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45ce6967-c17d-47d3-98f6-781bf3be4d47-kube-api-access-k48fl" (OuterVolumeSpecName: "kube-api-access-k48fl") pod "45ce6967-c17d-47d3-98f6-781bf3be4d47" (UID: "45ce6967-c17d-47d3-98f6-781bf3be4d47"). InnerVolumeSpecName "kube-api-access-k48fl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:09:50.418111 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:50.418077 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k48fl\" (UniqueName: \"kubernetes.io/projected/45ce6967-c17d-47d3-98f6-781bf3be4d47-kube-api-access-k48fl\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:09:50.978225 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:50.978198 2566 generic.go:356] "Generic (PLEG): container finished" podID="45ce6967-c17d-47d3-98f6-781bf3be4d47" containerID="16fad7432be90058969aec4280e2c2cf08ae702f4ead4434456176a03ec08a4a" exitCode=0 Jul 09 19:09:50.978504 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:50.978242 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-2s6fx" Jul 09 19:09:50.978504 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:50.978282 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-2s6fx" event={"ID":"45ce6967-c17d-47d3-98f6-781bf3be4d47","Type":"ContainerDied","Data":"16fad7432be90058969aec4280e2c2cf08ae702f4ead4434456176a03ec08a4a"} Jul 09 19:09:50.978504 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:50.978322 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-2s6fx" event={"ID":"45ce6967-c17d-47d3-98f6-781bf3be4d47","Type":"ContainerDied","Data":"b4e8a171213fdba193514b67af85477487a4ba013397706d700332e9f7a616f9"} Jul 09 19:09:50.978504 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:50.978345 2566 scope.go:117] "RemoveContainer" containerID="16fad7432be90058969aec4280e2c2cf08ae702f4ead4434456176a03ec08a4a" Jul 09 19:09:50.986753 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:50.986735 2566 scope.go:117] "RemoveContainer" containerID="16fad7432be90058969aec4280e2c2cf08ae702f4ead4434456176a03ec08a4a" Jul 09 19:09:50.987029 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:09:50.987006 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16fad7432be90058969aec4280e2c2cf08ae702f4ead4434456176a03ec08a4a\": container with ID starting with 16fad7432be90058969aec4280e2c2cf08ae702f4ead4434456176a03ec08a4a not found: ID does not exist" containerID="16fad7432be90058969aec4280e2c2cf08ae702f4ead4434456176a03ec08a4a" Jul 09 19:09:50.987094 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:50.987036 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16fad7432be90058969aec4280e2c2cf08ae702f4ead4434456176a03ec08a4a"} err="failed to get container status \"16fad7432be90058969aec4280e2c2cf08ae702f4ead4434456176a03ec08a4a\": rpc error: code = NotFound desc = could not find container \"16fad7432be90058969aec4280e2c2cf08ae702f4ead4434456176a03ec08a4a\": container with ID starting with 16fad7432be90058969aec4280e2c2cf08ae702f4ead4434456176a03ec08a4a not found: ID does not exist" Jul 09 19:09:50.999492 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:50.999466 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-2s6fx"] Jul 09 19:09:51.005760 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:51.005743 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-2s6fx"] Jul 09 19:09:51.057455 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:51.057425 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45ce6967-c17d-47d3-98f6-781bf3be4d47" path="/var/lib/kubelet/pods/45ce6967-c17d-47d3-98f6-781bf3be4d47/volumes" Jul 09 19:09:58.967555 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:09:58.967527 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-mtw72" Jul 09 19:10:16.753615 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:10:16.753569 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76877994-rnz8l"] Jul 09 19:10:16.754093 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:10:16.753957 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45ce6967-c17d-47d3-98f6-781bf3be4d47" containerName="authorino" Jul 09 19:10:16.754093 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:10:16.753968 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="45ce6967-c17d-47d3-98f6-781bf3be4d47" containerName="authorino" Jul 09 19:10:16.754093 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:10:16.754030 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="45ce6967-c17d-47d3-98f6-781bf3be4d47" containerName="authorino" Jul 09 19:10:16.757073 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:10:16.757055 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-rnz8l" Jul 09 19:10:16.760169 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:10:16.760152 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 09 19:10:16.761572 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:10:16.761554 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vfbr4\"" Jul 09 19:10:16.766518 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:10:16.766498 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76877994-rnz8l"] Jul 09 19:10:16.794106 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:10:16.794081 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-rnz8l"] Jul 09 19:10:16.794334 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:10:16.794316 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-jmz26 tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-76877994-rnz8l" podUID="9458b1f6-9b17-45ce-a7f4-ea5c91ea1926" Jul 09 19:10:16.825292 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:10:16.825269 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d6c4588cf-qkmkg"] Jul 09 19:10:16.828532 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:10:16.828516 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d6c4588cf-qkmkg" Jul 09 19:10:16.835525 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:10:16.835501 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d6c4588cf-qkmkg"] Jul 09 19:10:16.846966 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:10:16.846938 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmz26\" (UniqueName: \"kubernetes.io/projected/9458b1f6-9b17-45ce-a7f4-ea5c91ea1926-kube-api-access-jmz26\") pod \"authorino-76877994-rnz8l\" (UID: \"9458b1f6-9b17-45ce-a7f4-ea5c91ea1926\") " pod="kuadrant-system/authorino-76877994-rnz8l" Jul 09 19:10:16.847126 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:10:16.846973 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9458b1f6-9b17-45ce-a7f4-ea5c91ea1926-tls-cert\") pod \"authorino-76877994-rnz8l\" (UID: \"9458b1f6-9b17-45ce-a7f4-ea5c91ea1926\") " pod="kuadrant-system/authorino-76877994-rnz8l" Jul 09 19:10:16.947627 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:10:16.947573 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9458b1f6-9b17-45ce-a7f4-ea5c91ea1926-tls-cert\") pod \"authorino-76877994-rnz8l\" (UID: \"9458b1f6-9b17-45ce-a7f4-ea5c91ea1926\") " pod="kuadrant-system/authorino-76877994-rnz8l" Jul 09 19:10:16.947797 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:10:16.947691 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bhrj\" (UniqueName: \"kubernetes.io/projected/52685683-f0b1-47a5-96c6-385078c98413-kube-api-access-8bhrj\") pod \"authorino-6d6c4588cf-qkmkg\" (UID: \"52685683-f0b1-47a5-96c6-385078c98413\") " pod="kuadrant-system/authorino-6d6c4588cf-qkmkg" Jul 09 19:10:16.947797 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:10:16.947712 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/52685683-f0b1-47a5-96c6-385078c98413-tls-cert\") pod \"authorino-6d6c4588cf-qkmkg\" (UID: \"52685683-f0b1-47a5-96c6-385078c98413\") " pod="kuadrant-system/authorino-6d6c4588cf-qkmkg" Jul 09 19:10:16.947797 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:10:16.947749 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmz26\" (UniqueName: \"kubernetes.io/projected/9458b1f6-9b17-45ce-a7f4-ea5c91ea1926-kube-api-access-jmz26\") pod \"authorino-76877994-rnz8l\" (UID: \"9458b1f6-9b17-45ce-a7f4-ea5c91ea1926\") " pod="kuadrant-system/authorino-76877994-rnz8l" Jul 09 19:10:16.949952 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:10:16.949924 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9458b1f6-9b17-45ce-a7f4-ea5c91ea1926-tls-cert\") pod \"authorino-76877994-rnz8l\" (UID: \"9458b1f6-9b17-45ce-a7f4-ea5c91ea1926\") " pod="kuadrant-system/authorino-76877994-rnz8l" Jul 09 19:10:16.957176 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:10:16.957152 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmz26\" (UniqueName: \"kubernetes.io/projected/9458b1f6-9b17-45ce-a7f4-ea5c91ea1926-kube-api-access-jmz26\") pod \"authorino-76877994-rnz8l\" (UID: \"9458b1f6-9b17-45ce-a7f4-ea5c91ea1926\") " pod="kuadrant-system/authorino-76877994-rnz8l" Jul 09 19:10:17.048231 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:10:17.048159 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bhrj\" (UniqueName: \"kubernetes.io/projected/52685683-f0b1-47a5-96c6-385078c98413-kube-api-access-8bhrj\") pod \"authorino-6d6c4588cf-qkmkg\" (UID: \"52685683-f0b1-47a5-96c6-385078c98413\") " pod="kuadrant-system/authorino-6d6c4588cf-qkmkg" Jul 09 19:10:17.048231 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:10:17.048193 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/52685683-f0b1-47a5-96c6-385078c98413-tls-cert\") pod \"authorino-6d6c4588cf-qkmkg\" (UID: \"52685683-f0b1-47a5-96c6-385078c98413\") " pod="kuadrant-system/authorino-6d6c4588cf-qkmkg" Jul 09 19:10:17.050598 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:10:17.050578 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/52685683-f0b1-47a5-96c6-385078c98413-tls-cert\") pod \"authorino-6d6c4588cf-qkmkg\" (UID: \"52685683-f0b1-47a5-96c6-385078c98413\") " pod="kuadrant-system/authorino-6d6c4588cf-qkmkg" Jul 09 19:10:17.057260 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:10:17.057234 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bhrj\" (UniqueName: \"kubernetes.io/projected/52685683-f0b1-47a5-96c6-385078c98413-kube-api-access-8bhrj\") pod \"authorino-6d6c4588cf-qkmkg\" (UID: \"52685683-f0b1-47a5-96c6-385078c98413\") " pod="kuadrant-system/authorino-6d6c4588cf-qkmkg" Jul 09 19:10:17.115686 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:10:17.115663 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-rnz8l" Jul 09 19:10:17.122985 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:10:17.122960 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-rnz8l" Jul 09 19:10:17.138132 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:10:17.138112 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d6c4588cf-qkmkg" Jul 09 19:10:17.250105 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:10:17.250077 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmz26\" (UniqueName: \"kubernetes.io/projected/9458b1f6-9b17-45ce-a7f4-ea5c91ea1926-kube-api-access-jmz26\") pod \"9458b1f6-9b17-45ce-a7f4-ea5c91ea1926\" (UID: \"9458b1f6-9b17-45ce-a7f4-ea5c91ea1926\") " Jul 09 19:10:17.250268 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:10:17.250113 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9458b1f6-9b17-45ce-a7f4-ea5c91ea1926-tls-cert\") pod \"9458b1f6-9b17-45ce-a7f4-ea5c91ea1926\" (UID: \"9458b1f6-9b17-45ce-a7f4-ea5c91ea1926\") " Jul 09 19:10:17.252206 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:10:17.252176 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9458b1f6-9b17-45ce-a7f4-ea5c91ea1926-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "9458b1f6-9b17-45ce-a7f4-ea5c91ea1926" (UID: "9458b1f6-9b17-45ce-a7f4-ea5c91ea1926"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 19:10:17.252318 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:10:17.252218 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9458b1f6-9b17-45ce-a7f4-ea5c91ea1926-kube-api-access-jmz26" (OuterVolumeSpecName: "kube-api-access-jmz26") pod "9458b1f6-9b17-45ce-a7f4-ea5c91ea1926" (UID: "9458b1f6-9b17-45ce-a7f4-ea5c91ea1926"). InnerVolumeSpecName "kube-api-access-jmz26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:10:17.273251 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:10:17.271809 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d6c4588cf-qkmkg"] Jul 09 19:10:17.275851 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:10:17.275824 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod52685683_f0b1_47a5_96c6_385078c98413.slice/crio-2b86f900dd666b0dfd45585c5a11c3e981b4e36936c664f35133d30ab2c86ef7 WatchSource:0}: Error finding container 2b86f900dd666b0dfd45585c5a11c3e981b4e36936c664f35133d30ab2c86ef7: Status 404 returned error can't find the container with id 2b86f900dd666b0dfd45585c5a11c3e981b4e36936c664f35133d30ab2c86ef7 Jul 09 19:10:17.351480 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:10:17.351413 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmz26\" (UniqueName: \"kubernetes.io/projected/9458b1f6-9b17-45ce-a7f4-ea5c91ea1926-kube-api-access-jmz26\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:10:17.351480 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:10:17.351441 2566 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9458b1f6-9b17-45ce-a7f4-ea5c91ea1926-tls-cert\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:10:18.122465 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:10:18.122424 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d6c4588cf-qkmkg" event={"ID":"52685683-f0b1-47a5-96c6-385078c98413","Type":"ContainerStarted","Data":"d4427fdd2b444ed9221b25fe9dbddfa5c54fd6a85cccf2674b86f8841acba8d7"} Jul 09 19:10:18.122465 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:10:18.122445 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-rnz8l" Jul 09 19:10:18.122465 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:10:18.122468 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d6c4588cf-qkmkg" event={"ID":"52685683-f0b1-47a5-96c6-385078c98413","Type":"ContainerStarted","Data":"2b86f900dd666b0dfd45585c5a11c3e981b4e36936c664f35133d30ab2c86ef7"} Jul 09 19:10:18.142948 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:10:18.142902 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6d6c4588cf-qkmkg" podStartSLOduration=1.716184396 podStartE2EDuration="2.142886419s" totalImagesPullingTime="426.702023ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:10:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:10:18.14040583 +0000 UTC m=+687.648185624" watchObservedRunningTime="2026-07-09 19:10:18.142886419 +0000 UTC m=+687.650666193" Jul 09 19:10:18.164441 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:10:18.164414 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-rnz8l"] Jul 09 19:10:18.174133 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:10:18.174100 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76877994-rnz8l"] Jul 09 19:10:19.058521 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:10:19.058487 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9458b1f6-9b17-45ce-a7f4-ea5c91ea1926" path="/var/lib/kubelet/pods/9458b1f6-9b17-45ce-a7f4-ea5c91ea1926/volumes" Jul 09 19:11:00.608415 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:11:00.608377 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:11:08.388987 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:11:08.388953 2566 generic.go:356] "Generic (PLEG): container finished" podID="e5df48a3-3a56-4ff6-967e-984b325a7a2e" containerID="20f7f312507571222d51bb52f46348e4a9b8c5ef79d6984ed14f4cf90908a300" exitCode=0 Jul 09 19:11:08.389386 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:11:08.389028 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qwmxj" event={"ID":"e5df48a3-3a56-4ff6-967e-984b325a7a2e","Type":"ContainerDied","Data":"20f7f312507571222d51bb52f46348e4a9b8c5ef79d6984ed14f4cf90908a300"} Jul 09 19:11:08.389451 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:11:08.389435 2566 scope.go:117] "RemoveContainer" containerID="20f7f312507571222d51bb52f46348e4a9b8c5ef79d6984ed14f4cf90908a300" Jul 09 19:11:08.390429 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:11:08.390405 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:11:08.390492 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:11:08.390469 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:11:08.390538 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:11:08.390500 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:11:09.395670 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:11:09.395638 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qwmxj" event={"ID":"e5df48a3-3a56-4ff6-967e-984b325a7a2e","Type":"ContainerStarted","Data":"5109ddec3b151224603fd32731a21fb7ac3d17ea8a31d259e8e5810216322422"} Jul 09 19:11:09.731947 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:11:09.731915 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qwmxj" Jul 09 19:11:09.733289 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:11:09.733257 2566 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-qwmxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 09 19:11:09.733439 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:11:09.733307 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qwmxj" podUID="e5df48a3-3a56-4ff6-967e-984b325a7a2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 09 19:11:10.732730 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:11:10.732685 2566 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-qwmxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 09 19:11:10.733242 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:11:10.732742 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qwmxj" podUID="e5df48a3-3a56-4ff6-967e-984b325a7a2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 09 19:11:11.732785 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:11:11.732744 2566 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-qwmxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 09 19:11:11.733281 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:11:11.732809 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qwmxj" podUID="e5df48a3-3a56-4ff6-967e-984b325a7a2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 09 19:11:11.908794 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:11:11.908761 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qwmxj" Jul 09 19:11:12.732777 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:11:12.732734 2566 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-qwmxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 09 19:11:12.732989 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:11:12.732797 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qwmxj" podUID="e5df48a3-3a56-4ff6-967e-984b325a7a2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 09 19:11:13.736444 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:11:13.736412 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qwmxj" Jul 09 19:11:13.737378 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:11:13.737360 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-qwmxj" Jul 09 19:11:24.364177 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:11:24.364137 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:11:42.757681 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:11:42.757643 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:11:56.562625 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:11:56.562576 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:12:03.565354 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:12:03.565318 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:12:09.859192 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:12:09.859144 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:12:21.568280 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:12:21.568200 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:13:03.433249 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:13:03.433205 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-9b647dc98-67ckh"] Jul 09 19:13:03.435921 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:13:03.435902 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-9b647dc98-67ckh" Jul 09 19:13:03.446846 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:13:03.446824 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-9b647dc98-67ckh"] Jul 09 19:13:03.488991 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:13:03.488953 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d2c6d1ea-6d72-479d-a5f8-0406b68616f9-tls-cert\") pod \"authorino-9b647dc98-67ckh\" (UID: \"d2c6d1ea-6d72-479d-a5f8-0406b68616f9\") " pod="kuadrant-system/authorino-9b647dc98-67ckh" Jul 09 19:13:03.488991 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:13:03.488990 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glm26\" (UniqueName: \"kubernetes.io/projected/d2c6d1ea-6d72-479d-a5f8-0406b68616f9-kube-api-access-glm26\") pod \"authorino-9b647dc98-67ckh\" (UID: \"d2c6d1ea-6d72-479d-a5f8-0406b68616f9\") " pod="kuadrant-system/authorino-9b647dc98-67ckh" Jul 09 19:13:03.589484 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:13:03.589452 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d2c6d1ea-6d72-479d-a5f8-0406b68616f9-tls-cert\") pod \"authorino-9b647dc98-67ckh\" (UID: \"d2c6d1ea-6d72-479d-a5f8-0406b68616f9\") " pod="kuadrant-system/authorino-9b647dc98-67ckh" Jul 09 19:13:03.589680 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:13:03.589497 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glm26\" (UniqueName: \"kubernetes.io/projected/d2c6d1ea-6d72-479d-a5f8-0406b68616f9-kube-api-access-glm26\") pod \"authorino-9b647dc98-67ckh\" (UID: \"d2c6d1ea-6d72-479d-a5f8-0406b68616f9\") " pod="kuadrant-system/authorino-9b647dc98-67ckh" Jul 09 19:13:03.591954 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:13:03.591935 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d2c6d1ea-6d72-479d-a5f8-0406b68616f9-tls-cert\") pod \"authorino-9b647dc98-67ckh\" (UID: \"d2c6d1ea-6d72-479d-a5f8-0406b68616f9\") " pod="kuadrant-system/authorino-9b647dc98-67ckh" Jul 09 19:13:03.598175 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:13:03.598143 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glm26\" (UniqueName: \"kubernetes.io/projected/d2c6d1ea-6d72-479d-a5f8-0406b68616f9-kube-api-access-glm26\") pod \"authorino-9b647dc98-67ckh\" (UID: \"d2c6d1ea-6d72-479d-a5f8-0406b68616f9\") " pod="kuadrant-system/authorino-9b647dc98-67ckh" Jul 09 19:13:03.745314 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:13:03.745280 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-9b647dc98-67ckh" Jul 09 19:13:03.889088 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:13:03.889059 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-9b647dc98-67ckh"] Jul 09 19:13:03.891046 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:13:03.891016 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2c6d1ea_6d72_479d_a5f8_0406b68616f9.slice/crio-7e0187b931d4573bb25565874362a86a2e6bf0a5767f83b616cb763f0c02ce37 WatchSource:0}: Error finding container 7e0187b931d4573bb25565874362a86a2e6bf0a5767f83b616cb763f0c02ce37: Status 404 returned error can't find the container with id 7e0187b931d4573bb25565874362a86a2e6bf0a5767f83b616cb763f0c02ce37 Jul 09 19:13:03.892488 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:13:03.892465 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 19:13:04.013286 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:13:04.013203 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-9b647dc98-67ckh" event={"ID":"d2c6d1ea-6d72-479d-a5f8-0406b68616f9","Type":"ContainerStarted","Data":"7e0187b931d4573bb25565874362a86a2e6bf0a5767f83b616cb763f0c02ce37"} Jul 09 19:13:05.021896 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:13:05.021859 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-9b647dc98-67ckh" event={"ID":"d2c6d1ea-6d72-479d-a5f8-0406b68616f9","Type":"ContainerStarted","Data":"eb7417eba92bd5ea2fd6cd363169e95f8e05761c681d1205092e2e3d3f80c8c8"} Jul 09 19:13:05.051096 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:13:05.051028 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-9b647dc98-67ckh" podStartSLOduration=1.40455634 podStartE2EDuration="2.0510063s" totalImagesPullingTime="646.44996ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:13:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:13:05.046806351 +0000 UTC m=+854.554586125" watchObservedRunningTime="2026-07-09 19:13:05.0510063 +0000 UTC m=+854.558786075" Jul 09 19:13:05.081257 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:13:05.081223 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d6c4588cf-qkmkg"] Jul 09 19:13:05.081710 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:13:05.081655 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6d6c4588cf-qkmkg" podUID="52685683-f0b1-47a5-96c6-385078c98413" containerName="authorino" containerID="cri-o://d4427fdd2b444ed9221b25fe9dbddfa5c54fd6a85cccf2674b86f8841acba8d7" gracePeriod=30 Jul 09 19:13:05.346340 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:13:05.346316 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d6c4588cf-qkmkg" Jul 09 19:13:05.404200 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:13:05.404166 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8bhrj\" (UniqueName: \"kubernetes.io/projected/52685683-f0b1-47a5-96c6-385078c98413-kube-api-access-8bhrj\") pod \"52685683-f0b1-47a5-96c6-385078c98413\" (UID: \"52685683-f0b1-47a5-96c6-385078c98413\") " Jul 09 19:13:05.404343 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:13:05.404224 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/52685683-f0b1-47a5-96c6-385078c98413-tls-cert\") pod \"52685683-f0b1-47a5-96c6-385078c98413\" (UID: \"52685683-f0b1-47a5-96c6-385078c98413\") " Jul 09 19:13:05.406828 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:13:05.406793 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52685683-f0b1-47a5-96c6-385078c98413-kube-api-access-8bhrj" (OuterVolumeSpecName: "kube-api-access-8bhrj") pod "52685683-f0b1-47a5-96c6-385078c98413" (UID: "52685683-f0b1-47a5-96c6-385078c98413"). InnerVolumeSpecName "kube-api-access-8bhrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:13:05.419797 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:13:05.419755 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52685683-f0b1-47a5-96c6-385078c98413-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "52685683-f0b1-47a5-96c6-385078c98413" (UID: "52685683-f0b1-47a5-96c6-385078c98413"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 19:13:05.505116 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:13:05.505084 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bhrj\" (UniqueName: \"kubernetes.io/projected/52685683-f0b1-47a5-96c6-385078c98413-kube-api-access-8bhrj\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:13:05.505116 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:13:05.505110 2566 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/52685683-f0b1-47a5-96c6-385078c98413-tls-cert\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:13:06.028994 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:13:06.028955 2566 generic.go:356] "Generic (PLEG): container finished" podID="52685683-f0b1-47a5-96c6-385078c98413" containerID="d4427fdd2b444ed9221b25fe9dbddfa5c54fd6a85cccf2674b86f8841acba8d7" exitCode=0 Jul 09 19:13:06.028994 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:13:06.028997 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d6c4588cf-qkmkg" Jul 09 19:13:06.029556 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:13:06.029041 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d6c4588cf-qkmkg" event={"ID":"52685683-f0b1-47a5-96c6-385078c98413","Type":"ContainerDied","Data":"d4427fdd2b444ed9221b25fe9dbddfa5c54fd6a85cccf2674b86f8841acba8d7"} Jul 09 19:13:06.029556 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:13:06.029079 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d6c4588cf-qkmkg" event={"ID":"52685683-f0b1-47a5-96c6-385078c98413","Type":"ContainerDied","Data":"2b86f900dd666b0dfd45585c5a11c3e981b4e36936c664f35133d30ab2c86ef7"} Jul 09 19:13:06.029556 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:13:06.029101 2566 scope.go:117] "RemoveContainer" containerID="d4427fdd2b444ed9221b25fe9dbddfa5c54fd6a85cccf2674b86f8841acba8d7" Jul 09 19:13:06.038502 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:13:06.038481 2566 scope.go:117] "RemoveContainer" containerID="d4427fdd2b444ed9221b25fe9dbddfa5c54fd6a85cccf2674b86f8841acba8d7" Jul 09 19:13:06.038781 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:13:06.038761 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4427fdd2b444ed9221b25fe9dbddfa5c54fd6a85cccf2674b86f8841acba8d7\": container with ID starting with d4427fdd2b444ed9221b25fe9dbddfa5c54fd6a85cccf2674b86f8841acba8d7 not found: ID does not exist" containerID="d4427fdd2b444ed9221b25fe9dbddfa5c54fd6a85cccf2674b86f8841acba8d7" Jul 09 19:13:06.038829 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:13:06.038790 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4427fdd2b444ed9221b25fe9dbddfa5c54fd6a85cccf2674b86f8841acba8d7"} err="failed to get container status \"d4427fdd2b444ed9221b25fe9dbddfa5c54fd6a85cccf2674b86f8841acba8d7\": rpc error: code = NotFound desc = could not find container \"d4427fdd2b444ed9221b25fe9dbddfa5c54fd6a85cccf2674b86f8841acba8d7\": container with ID starting with d4427fdd2b444ed9221b25fe9dbddfa5c54fd6a85cccf2674b86f8841acba8d7 not found: ID does not exist" Jul 09 19:13:06.053423 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:13:06.053390 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d6c4588cf-qkmkg"] Jul 09 19:13:06.059759 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:13:06.059731 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d6c4588cf-qkmkg"] Jul 09 19:13:07.056535 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:13:07.056497 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52685683-f0b1-47a5-96c6-385078c98413" path="/var/lib/kubelet/pods/52685683-f0b1-47a5-96c6-385078c98413/volumes" Jul 09 19:13:50.990071 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:13:50.990041 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-vws4c_7cf4af17-5218-4b92-bceb-addbefcc03cb/console-operator/1.log" Jul 09 19:13:50.992170 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:13:50.992148 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-vws4c_7cf4af17-5218-4b92-bceb-addbefcc03cb/console-operator/1.log" Jul 09 19:13:50.994451 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:13:50.994434 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6dh7g_db233460-134d-4d9d-8da9-31d89366a4e0/ovn-acl-logging/0.log" Jul 09 19:13:50.996461 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:13:50.996444 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6dh7g_db233460-134d-4d9d-8da9-31d89366a4e0/ovn-acl-logging/0.log" Jul 09 19:14:05.062885 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:14:05.062848 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:14:15.766191 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:14:15.766157 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:14:26.360785 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:14:26.360745 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:14:35.956622 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:14:35.956577 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:14:47.573412 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:14:47.573374 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:15:00.148272 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:15:00.148229 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727075-rjqg7"] Jul 09 19:15:00.149126 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:15:00.149096 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52685683-f0b1-47a5-96c6-385078c98413" containerName="authorino" Jul 09 19:15:00.149126 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:15:00.149128 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="52685683-f0b1-47a5-96c6-385078c98413" containerName="authorino" Jul 09 19:15:00.149416 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:15:00.149399 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="52685683-f0b1-47a5-96c6-385078c98413" containerName="authorino" Jul 09 19:15:00.154305 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:15:00.153773 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727075-rjqg7" Jul 09 19:15:00.156693 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:15:00.156669 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9pdt2\"" Jul 09 19:15:00.156813 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:15:00.156680 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 09 19:15:00.157324 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:15:00.157303 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727075-rjqg7"] Jul 09 19:15:00.158253 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:15:00.158234 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 09 19:15:00.254593 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:15:00.254558 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4btq\" (UniqueName: \"kubernetes.io/projected/a1e0dd5e-fc68-4fe9-a247-b60271d20832-kube-api-access-g4btq\") pod \"maas-api-key-cleanup-29727075-rjqg7\" (UID: \"a1e0dd5e-fc68-4fe9-a247-b60271d20832\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727075-rjqg7" Jul 09 19:15:00.355984 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:15:00.355952 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4btq\" (UniqueName: \"kubernetes.io/projected/a1e0dd5e-fc68-4fe9-a247-b60271d20832-kube-api-access-g4btq\") pod \"maas-api-key-cleanup-29727075-rjqg7\" (UID: \"a1e0dd5e-fc68-4fe9-a247-b60271d20832\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727075-rjqg7" Jul 09 19:15:00.365263 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:15:00.365231 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4btq\" (UniqueName: \"kubernetes.io/projected/a1e0dd5e-fc68-4fe9-a247-b60271d20832-kube-api-access-g4btq\") pod \"maas-api-key-cleanup-29727075-rjqg7\" (UID: \"a1e0dd5e-fc68-4fe9-a247-b60271d20832\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727075-rjqg7" Jul 09 19:15:00.465546 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:15:00.465519 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727075-rjqg7" Jul 09 19:15:00.598895 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:15:00.598873 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727075-rjqg7"] Jul 09 19:15:00.600454 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:15:00.600423 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1e0dd5e_fc68_4fe9_a247_b60271d20832.slice/crio-2ce77d9df72b9d220045524a0a261c39275888a6e5032667c4714f2bc0b83112 WatchSource:0}: Error finding container 2ce77d9df72b9d220045524a0a261c39275888a6e5032667c4714f2bc0b83112: Status 404 returned error can't find the container with id 2ce77d9df72b9d220045524a0a261c39275888a6e5032667c4714f2bc0b83112 Jul 09 19:15:00.667473 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:15:00.667436 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727075-rjqg7" event={"ID":"a1e0dd5e-fc68-4fe9-a247-b60271d20832","Type":"ContainerStarted","Data":"2ce77d9df72b9d220045524a0a261c39275888a6e5032667c4714f2bc0b83112"} Jul 09 19:15:03.687434 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:15:03.687340 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727075-rjqg7" event={"ID":"a1e0dd5e-fc68-4fe9-a247-b60271d20832","Type":"ContainerStarted","Data":"3e7d6edca69e38d90d35b02c5ebb0e5d38853aae78f14f405515ee2c079d4677"} Jul 09 19:15:03.704015 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:15:03.703966 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727075-rjqg7" podStartSLOduration=1.159260819 podStartE2EDuration="3.703949836s" totalImagesPullingTime="2.544689017s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:15:03.703005311 +0000 UTC m=+973.210785099" watchObservedRunningTime="2026-07-09 19:15:03.703949836 +0000 UTC m=+973.211729611" Jul 09 19:15:13.664378 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:15:13.664293 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:15:26.764669 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:15:26.764639 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:16:00.012416 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:16:00.012383 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727075-rjqg7"] Jul 09 19:16:00.012936 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:16:00.012594 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727075-rjqg7" podUID="a1e0dd5e-fc68-4fe9-a247-b60271d20832" containerName="cleanup" containerID="cri-o://3e7d6edca69e38d90d35b02c5ebb0e5d38853aae78f14f405515ee2c079d4677" gracePeriod=30 Jul 09 19:16:01.561384 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:16:01.561349 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:16:09.857761 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:16:09.857722 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:16:20.666980 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:16:20.666942 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:16:28.066678 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:16:28.066636 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:16:30.159501 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:16:30.159466 2566 generic.go:356] "Generic (PLEG): container finished" podID="a1e0dd5e-fc68-4fe9-a247-b60271d20832" containerID="3e7d6edca69e38d90d35b02c5ebb0e5d38853aae78f14f405515ee2c079d4677" exitCode=137 Jul 09 19:16:30.159930 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:16:30.159541 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727075-rjqg7" event={"ID":"a1e0dd5e-fc68-4fe9-a247-b60271d20832","Type":"ContainerDied","Data":"3e7d6edca69e38d90d35b02c5ebb0e5d38853aae78f14f405515ee2c079d4677"} Jul 09 19:16:30.208324 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:16:30.208293 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727075-rjqg7" Jul 09 19:16:30.229543 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:16:30.229167 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4btq\" (UniqueName: \"kubernetes.io/projected/a1e0dd5e-fc68-4fe9-a247-b60271d20832-kube-api-access-g4btq\") pod \"a1e0dd5e-fc68-4fe9-a247-b60271d20832\" (UID: \"a1e0dd5e-fc68-4fe9-a247-b60271d20832\") " Jul 09 19:16:30.231912 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:16:30.231879 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1e0dd5e-fc68-4fe9-a247-b60271d20832-kube-api-access-g4btq" (OuterVolumeSpecName: "kube-api-access-g4btq") pod "a1e0dd5e-fc68-4fe9-a247-b60271d20832" (UID: "a1e0dd5e-fc68-4fe9-a247-b60271d20832"). InnerVolumeSpecName "kube-api-access-g4btq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:16:30.330366 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:16:30.330281 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4btq\" (UniqueName: \"kubernetes.io/projected/a1e0dd5e-fc68-4fe9-a247-b60271d20832-kube-api-access-g4btq\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:16:31.166665 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:16:31.166639 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727075-rjqg7" Jul 09 19:16:31.167062 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:16:31.166638 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727075-rjqg7" event={"ID":"a1e0dd5e-fc68-4fe9-a247-b60271d20832","Type":"ContainerDied","Data":"2ce77d9df72b9d220045524a0a261c39275888a6e5032667c4714f2bc0b83112"} Jul 09 19:16:31.167062 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:16:31.166764 2566 scope.go:117] "RemoveContainer" containerID="3e7d6edca69e38d90d35b02c5ebb0e5d38853aae78f14f405515ee2c079d4677" Jul 09 19:16:31.185339 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:16:31.185310 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727075-rjqg7"] Jul 09 19:16:31.187508 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:16:31.187487 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727075-rjqg7"] Jul 09 19:16:33.060533 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:16:33.060495 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1e0dd5e-fc68-4fe9-a247-b60271d20832" path="/var/lib/kubelet/pods/a1e0dd5e-fc68-4fe9-a247-b60271d20832/volumes" Jul 09 19:16:39.568632 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:16:39.568575 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:16:49.655431 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:16:49.655344 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:17:00.355087 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:17:00.355054 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:17:10.368822 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:17:10.368788 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:17:35.763136 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:17:35.763105 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:17:46.361569 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:17:46.361535 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:18:09.363396 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:18:09.363367 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:18:20.864284 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:18:20.864202 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:18:36.066255 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:18:36.066219 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:18:45.759099 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:18:45.759063 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:18:51.027858 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:18:51.027830 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-vws4c_7cf4af17-5218-4b92-bceb-addbefcc03cb/console-operator/1.log" Jul 09 19:18:51.032524 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:18:51.032493 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-vws4c_7cf4af17-5218-4b92-bceb-addbefcc03cb/console-operator/1.log" Jul 09 19:18:51.033126 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:18:51.033092 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6dh7g_db233460-134d-4d9d-8da9-31d89366a4e0/ovn-acl-logging/0.log" Jul 09 19:18:51.037852 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:18:51.037832 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6dh7g_db233460-134d-4d9d-8da9-31d89366a4e0/ovn-acl-logging/0.log" Jul 09 19:18:55.567138 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:18:55.567093 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:19:05.457752 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:19:05.457722 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:19:15.362061 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:19:15.362026 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:19:40.946041 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:19:40.946003 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ndcrm"] Jul 09 19:19:40.946813 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:19:40.946785 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ndcrm" podUID="b51e710e-4986-4534-85c6-4f695e01df88" containerName="manager" containerID="cri-o://ce76555027785a617ee55e6828ec0300073de97b758ad40b072a187c045f3008" gracePeriod=10 Jul 09 19:19:45.100915 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:19:45.100890 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ndcrm" Jul 09 19:19:45.241020 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:19:45.240935 2566 generic.go:356] "Generic (PLEG): container finished" podID="b51e710e-4986-4534-85c6-4f695e01df88" containerID="ce76555027785a617ee55e6828ec0300073de97b758ad40b072a187c045f3008" exitCode=0 Jul 09 19:19:45.241020 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:19:45.240981 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ndcrm" Jul 09 19:19:45.241197 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:19:45.241016 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ndcrm" event={"ID":"b51e710e-4986-4534-85c6-4f695e01df88","Type":"ContainerDied","Data":"ce76555027785a617ee55e6828ec0300073de97b758ad40b072a187c045f3008"} Jul 09 19:19:45.241197 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:19:45.241070 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ndcrm" event={"ID":"b51e710e-4986-4534-85c6-4f695e01df88","Type":"ContainerDied","Data":"d75a76f3ff4a2096e8b9cb8b811a285fc5b7ce67e4d1cc58032413f76661a20c"} Jul 09 19:19:45.241197 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:19:45.241092 2566 scope.go:117] "RemoveContainer" containerID="ce76555027785a617ee55e6828ec0300073de97b758ad40b072a187c045f3008" Jul 09 19:19:45.250898 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:19:45.250878 2566 scope.go:117] "RemoveContainer" containerID="ce76555027785a617ee55e6828ec0300073de97b758ad40b072a187c045f3008" Jul 09 19:19:45.251137 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:19:45.251119 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce76555027785a617ee55e6828ec0300073de97b758ad40b072a187c045f3008\": container with ID starting with ce76555027785a617ee55e6828ec0300073de97b758ad40b072a187c045f3008 not found: ID does not exist" containerID="ce76555027785a617ee55e6828ec0300073de97b758ad40b072a187c045f3008" Jul 09 19:19:45.251186 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:19:45.251145 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce76555027785a617ee55e6828ec0300073de97b758ad40b072a187c045f3008"} err="failed to get container status \"ce76555027785a617ee55e6828ec0300073de97b758ad40b072a187c045f3008\": rpc error: code = NotFound desc = could not find container \"ce76555027785a617ee55e6828ec0300073de97b758ad40b072a187c045f3008\": container with ID starting with ce76555027785a617ee55e6828ec0300073de97b758ad40b072a187c045f3008 not found: ID does not exist" Jul 09 19:19:45.266593 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:19:45.266572 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d5dsj\" (UniqueName: \"kubernetes.io/projected/b51e710e-4986-4534-85c6-4f695e01df88-kube-api-access-d5dsj\") pod \"b51e710e-4986-4534-85c6-4f695e01df88\" (UID: \"b51e710e-4986-4534-85c6-4f695e01df88\") " Jul 09 19:19:45.266693 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:19:45.266626 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b51e710e-4986-4534-85c6-4f695e01df88-extensions-socket-volume\") pod \"b51e710e-4986-4534-85c6-4f695e01df88\" (UID: \"b51e710e-4986-4534-85c6-4f695e01df88\") " Jul 09 19:19:45.267037 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:19:45.267013 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b51e710e-4986-4534-85c6-4f695e01df88-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b51e710e-4986-4534-85c6-4f695e01df88" (UID: "b51e710e-4986-4534-85c6-4f695e01df88"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:19:45.268569 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:19:45.268548 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b51e710e-4986-4534-85c6-4f695e01df88-kube-api-access-d5dsj" (OuterVolumeSpecName: "kube-api-access-d5dsj") pod "b51e710e-4986-4534-85c6-4f695e01df88" (UID: "b51e710e-4986-4534-85c6-4f695e01df88"). InnerVolumeSpecName "kube-api-access-d5dsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:19:45.367866 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:19:45.367838 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5dsj\" (UniqueName: \"kubernetes.io/projected/b51e710e-4986-4534-85c6-4f695e01df88-kube-api-access-d5dsj\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:19:45.367866 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:19:45.367862 2566 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b51e710e-4986-4534-85c6-4f695e01df88-extensions-socket-volume\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:19:45.570216 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:19:45.570182 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ndcrm"] Jul 09 19:19:45.573836 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:19:45.573811 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ndcrm"] Jul 09 19:19:47.057056 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:19:47.057023 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b51e710e-4986-4534-85c6-4f695e01df88" path="/var/lib/kubelet/pods/b51e710e-4986-4534-85c6-4f695e01df88/volumes" Jul 09 19:20:42.048325 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:20:42.048294 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6ckvt"] Jul 09 19:20:42.048840 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:20:42.048691 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b51e710e-4986-4534-85c6-4f695e01df88" containerName="manager" Jul 09 19:20:42.048840 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:20:42.048701 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b51e710e-4986-4534-85c6-4f695e01df88" containerName="manager" Jul 09 19:20:42.048840 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:20:42.048724 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1e0dd5e-fc68-4fe9-a247-b60271d20832" containerName="cleanup" Jul 09 19:20:42.048840 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:20:42.048730 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1e0dd5e-fc68-4fe9-a247-b60271d20832" containerName="cleanup" Jul 09 19:20:42.048840 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:20:42.048787 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1e0dd5e-fc68-4fe9-a247-b60271d20832" containerName="cleanup" Jul 09 19:20:42.048840 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:20:42.048796 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b51e710e-4986-4534-85c6-4f695e01df88" containerName="manager" Jul 09 19:20:42.051705 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:20:42.051688 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6ckvt" Jul 09 19:20:42.054669 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:20:42.054653 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-nj4td\"" Jul 09 19:20:42.065773 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:20:42.065746 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6ckvt"] Jul 09 19:20:42.244737 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:20:42.244701 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hmhg\" (UniqueName: \"kubernetes.io/projected/e7d5db49-6211-44f9-98a3-9ca7946d714d-kube-api-access-6hmhg\") pod \"kuadrant-operator-controller-manager-86878c769f-6ckvt\" (UID: \"e7d5db49-6211-44f9-98a3-9ca7946d714d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6ckvt" Jul 09 19:20:42.244934 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:20:42.244770 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7d5db49-6211-44f9-98a3-9ca7946d714d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-6ckvt\" (UID: \"e7d5db49-6211-44f9-98a3-9ca7946d714d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6ckvt" Jul 09 19:20:42.346268 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:20:42.346151 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7d5db49-6211-44f9-98a3-9ca7946d714d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-6ckvt\" (UID: \"e7d5db49-6211-44f9-98a3-9ca7946d714d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6ckvt" Jul 09 19:20:42.346268 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:20:42.346264 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hmhg\" (UniqueName: \"kubernetes.io/projected/e7d5db49-6211-44f9-98a3-9ca7946d714d-kube-api-access-6hmhg\") pod \"kuadrant-operator-controller-manager-86878c769f-6ckvt\" (UID: \"e7d5db49-6211-44f9-98a3-9ca7946d714d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6ckvt" Jul 09 19:20:42.346544 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:20:42.346522 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7d5db49-6211-44f9-98a3-9ca7946d714d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-6ckvt\" (UID: \"e7d5db49-6211-44f9-98a3-9ca7946d714d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6ckvt" Jul 09 19:20:42.365045 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:20:42.365016 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hmhg\" (UniqueName: \"kubernetes.io/projected/e7d5db49-6211-44f9-98a3-9ca7946d714d-kube-api-access-6hmhg\") pod \"kuadrant-operator-controller-manager-86878c769f-6ckvt\" (UID: \"e7d5db49-6211-44f9-98a3-9ca7946d714d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6ckvt" Jul 09 19:20:42.662804 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:20:42.662695 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6ckvt" Jul 09 19:20:42.810597 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:20:42.810568 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6ckvt"] Jul 09 19:20:42.811562 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:20:42.811533 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7d5db49_6211_44f9_98a3_9ca7946d714d.slice/crio-dc6c47af86a9d131d44e3fdca2e3231aa3c66b68e8d87efeb1808f5da3563612 WatchSource:0}: Error finding container dc6c47af86a9d131d44e3fdca2e3231aa3c66b68e8d87efeb1808f5da3563612: Status 404 returned error can't find the container with id dc6c47af86a9d131d44e3fdca2e3231aa3c66b68e8d87efeb1808f5da3563612 Jul 09 19:20:43.547455 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:20:43.547419 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6ckvt" event={"ID":"e7d5db49-6211-44f9-98a3-9ca7946d714d","Type":"ContainerStarted","Data":"663fa06868e0742464b29241a744f634559de8d885150a2fa55cc83ccdd6a33a"} Jul 09 19:20:43.547892 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:20:43.547470 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6ckvt" event={"ID":"e7d5db49-6211-44f9-98a3-9ca7946d714d","Type":"ContainerStarted","Data":"dc6c47af86a9d131d44e3fdca2e3231aa3c66b68e8d87efeb1808f5da3563612"} Jul 09 19:20:43.547892 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:20:43.547534 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6ckvt" Jul 09 19:20:43.569713 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:20:43.569661 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6ckvt" podStartSLOduration=1.569640664 podStartE2EDuration="1.569640664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:20:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:20:43.567984803 +0000 UTC m=+1313.075764578" watchObservedRunningTime="2026-07-09 19:20:43.569640664 +0000 UTC m=+1313.077420438" Jul 09 19:20:54.555361 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:20:54.555330 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6ckvt" Jul 09 19:21:02.516167 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:21:02.516130 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:21:14.971418 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:21:14.971340 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:21:23.769157 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:21:23.769123 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:21:35.261667 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:21:35.261628 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:21:47.561689 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:21:47.561653 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:21:57.569985 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:21:57.569951 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:22:05.563181 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:22:05.563144 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:22:16.460582 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:22:16.460543 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:22:27.465378 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:22:27.465339 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:22:36.967080 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:22:36.967043 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:22:50.673570 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:22:50.673491 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:23:08.658542 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:23:08.658506 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:23:18.163407 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:23:18.163372 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:23:28.562969 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:23:28.562932 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:23:37.967259 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:23:37.967223 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:23:44.758264 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:23:44.758227 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:23:51.064542 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:23:51.064507 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-vws4c_7cf4af17-5218-4b92-bceb-addbefcc03cb/console-operator/1.log" Jul 09 19:23:51.069221 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:23:51.069200 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6dh7g_db233460-134d-4d9d-8da9-31d89366a4e0/ovn-acl-logging/0.log" Jul 09 19:23:51.072052 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:23:51.072006 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-vws4c_7cf4af17-5218-4b92-bceb-addbefcc03cb/console-operator/1.log" Jul 09 19:23:51.076305 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:23:51.076290 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6dh7g_db233460-134d-4d9d-8da9-31d89366a4e0/ovn-acl-logging/0.log" Jul 09 19:24:00.561577 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:24:00.561540 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:24:10.065579 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:24:10.065547 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:24:19.476977 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:24:19.476895 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:24:27.270890 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:24:27.270855 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:24:38.867181 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:24:38.867140 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:24:48.655673 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:24:48.655637 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:24:58.772575 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:24:58.772537 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:25:10.672865 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:25:10.672830 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:25:25.665818 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:25:25.665779 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:25:34.669862 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:25:34.669825 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:25:47.065523 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:25:47.065446 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:26:03.474427 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:26:03.474388 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:26:15.368962 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:26:15.368925 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:26:28.767356 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:26:28.767320 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:26:42.471347 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:26:42.471310 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:26:51.467972 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:26:51.467936 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:27:00.364741 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:00.364708 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:27:02.003933 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:02.003903 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64"] Jul 09 19:27:02.008120 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:02.008098 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64" Jul 09 19:27:02.011251 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:02.011221 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-639f19cb-openshift-default-dockercfg-6rwds\"" Jul 09 19:27:02.028136 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:02.028112 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64"] Jul 09 19:27:02.072073 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:02.072045 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f96a1d-0f80-4071-b3e5-795476fccd50-workload-socket\") pod \"e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64\" (UID: \"a5f96a1d-0f80-4071-b3e5-795476fccd50\") " pod="openshift-ingress/e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64" Jul 09 19:27:02.072237 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:02.072086 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5f96a1d-0f80-4071-b3e5-795476fccd50-istio-envoy\") pod \"e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64\" (UID: \"a5f96a1d-0f80-4071-b3e5-795476fccd50\") " pod="openshift-ingress/e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64" Jul 09 19:27:02.072237 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:02.072107 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5f96a1d-0f80-4071-b3e5-795476fccd50-istiod-ca-cert\") pod \"e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64\" (UID: \"a5f96a1d-0f80-4071-b3e5-795476fccd50\") " pod="openshift-ingress/e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64" Jul 09 19:27:02.072237 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:02.072171 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5f96a1d-0f80-4071-b3e5-795476fccd50-istio-data\") pod \"e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64\" (UID: \"a5f96a1d-0f80-4071-b3e5-795476fccd50\") " pod="openshift-ingress/e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64" Jul 09 19:27:02.072237 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:02.072210 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5f96a1d-0f80-4071-b3e5-795476fccd50-istio-podinfo\") pod \"e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64\" (UID: \"a5f96a1d-0f80-4071-b3e5-795476fccd50\") " pod="openshift-ingress/e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64" Jul 09 19:27:02.072237 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:02.072231 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5f96a1d-0f80-4071-b3e5-795476fccd50-istio-token\") pod \"e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64\" (UID: \"a5f96a1d-0f80-4071-b3e5-795476fccd50\") " pod="openshift-ingress/e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64" Jul 09 19:27:02.072406 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:02.072268 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spldt\" (UniqueName: \"kubernetes.io/projected/a5f96a1d-0f80-4071-b3e5-795476fccd50-kube-api-access-spldt\") pod \"e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64\" (UID: \"a5f96a1d-0f80-4071-b3e5-795476fccd50\") " pod="openshift-ingress/e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64" Jul 09 19:27:02.072406 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:02.072335 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f96a1d-0f80-4071-b3e5-795476fccd50-credential-socket\") pod \"e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64\" (UID: \"a5f96a1d-0f80-4071-b3e5-795476fccd50\") " pod="openshift-ingress/e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64" Jul 09 19:27:02.072406 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:02.072372 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5f96a1d-0f80-4071-b3e5-795476fccd50-workload-certs\") pod \"e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64\" (UID: \"a5f96a1d-0f80-4071-b3e5-795476fccd50\") " pod="openshift-ingress/e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64" Jul 09 19:27:02.173717 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:02.173686 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5f96a1d-0f80-4071-b3e5-795476fccd50-istio-data\") pod \"e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64\" (UID: \"a5f96a1d-0f80-4071-b3e5-795476fccd50\") " pod="openshift-ingress/e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64" Jul 09 19:27:02.173910 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:02.173724 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5f96a1d-0f80-4071-b3e5-795476fccd50-istio-podinfo\") pod \"e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64\" (UID: \"a5f96a1d-0f80-4071-b3e5-795476fccd50\") " pod="openshift-ingress/e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64" Jul 09 19:27:02.173910 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:02.173745 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5f96a1d-0f80-4071-b3e5-795476fccd50-istio-token\") pod \"e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64\" (UID: \"a5f96a1d-0f80-4071-b3e5-795476fccd50\") " pod="openshift-ingress/e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64" Jul 09 19:27:02.173910 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:02.173795 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spldt\" (UniqueName: \"kubernetes.io/projected/a5f96a1d-0f80-4071-b3e5-795476fccd50-kube-api-access-spldt\") pod \"e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64\" (UID: \"a5f96a1d-0f80-4071-b3e5-795476fccd50\") " pod="openshift-ingress/e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64" Jul 09 19:27:02.173910 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:02.173880 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f96a1d-0f80-4071-b3e5-795476fccd50-credential-socket\") pod \"e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64\" (UID: \"a5f96a1d-0f80-4071-b3e5-795476fccd50\") " pod="openshift-ingress/e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64" Jul 09 19:27:02.174140 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:02.173937 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5f96a1d-0f80-4071-b3e5-795476fccd50-workload-certs\") pod \"e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64\" (UID: \"a5f96a1d-0f80-4071-b3e5-795476fccd50\") " pod="openshift-ingress/e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64" Jul 09 19:27:02.174140 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:02.173965 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f96a1d-0f80-4071-b3e5-795476fccd50-workload-socket\") pod \"e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64\" (UID: \"a5f96a1d-0f80-4071-b3e5-795476fccd50\") " pod="openshift-ingress/e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64" Jul 09 19:27:02.174140 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:02.174009 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5f96a1d-0f80-4071-b3e5-795476fccd50-istio-envoy\") pod \"e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64\" (UID: \"a5f96a1d-0f80-4071-b3e5-795476fccd50\") " pod="openshift-ingress/e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64" Jul 09 19:27:02.174140 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:02.174043 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5f96a1d-0f80-4071-b3e5-795476fccd50-istiod-ca-cert\") pod \"e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64\" (UID: \"a5f96a1d-0f80-4071-b3e5-795476fccd50\") " pod="openshift-ingress/e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64" Jul 09 19:27:02.174140 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:02.174130 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5f96a1d-0f80-4071-b3e5-795476fccd50-istio-data\") pod \"e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64\" (UID: \"a5f96a1d-0f80-4071-b3e5-795476fccd50\") " pod="openshift-ingress/e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64" Jul 09 19:27:02.174374 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:02.174289 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f96a1d-0f80-4071-b3e5-795476fccd50-credential-socket\") pod \"e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64\" (UID: \"a5f96a1d-0f80-4071-b3e5-795476fccd50\") " pod="openshift-ingress/e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64" Jul 09 19:27:02.174374 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:02.174327 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5f96a1d-0f80-4071-b3e5-795476fccd50-workload-certs\") pod \"e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64\" (UID: \"a5f96a1d-0f80-4071-b3e5-795476fccd50\") " pod="openshift-ingress/e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64" Jul 09 19:27:02.174374 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:02.174360 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f96a1d-0f80-4071-b3e5-795476fccd50-workload-socket\") pod \"e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64\" (UID: \"a5f96a1d-0f80-4071-b3e5-795476fccd50\") " pod="openshift-ingress/e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64" Jul 09 19:27:02.174807 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:02.174784 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5f96a1d-0f80-4071-b3e5-795476fccd50-istiod-ca-cert\") pod \"e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64\" (UID: \"a5f96a1d-0f80-4071-b3e5-795476fccd50\") " pod="openshift-ingress/e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64" Jul 09 19:27:02.176320 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:02.176292 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5f96a1d-0f80-4071-b3e5-795476fccd50-istio-envoy\") pod \"e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64\" (UID: \"a5f96a1d-0f80-4071-b3e5-795476fccd50\") " pod="openshift-ingress/e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64" Jul 09 19:27:02.176437 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:02.176420 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5f96a1d-0f80-4071-b3e5-795476fccd50-istio-podinfo\") pod \"e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64\" (UID: \"a5f96a1d-0f80-4071-b3e5-795476fccd50\") " pod="openshift-ingress/e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64" Jul 09 19:27:02.183149 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:02.183117 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5f96a1d-0f80-4071-b3e5-795476fccd50-istio-token\") pod \"e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64\" (UID: \"a5f96a1d-0f80-4071-b3e5-795476fccd50\") " pod="openshift-ingress/e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64" Jul 09 19:27:02.183259 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:02.183243 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spldt\" (UniqueName: \"kubernetes.io/projected/a5f96a1d-0f80-4071-b3e5-795476fccd50-kube-api-access-spldt\") pod \"e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64\" (UID: \"a5f96a1d-0f80-4071-b3e5-795476fccd50\") " pod="openshift-ingress/e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64" Jul 09 19:27:02.319254 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:02.319167 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64" Jul 09 19:27:02.473134 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:02.473109 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64"] Jul 09 19:27:02.474902 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:27:02.474877 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5f96a1d_0f80_4071_b3e5_795476fccd50.slice/crio-f40b1b791c73f54a88f007c3dde781d8edd4bdb04fc95f8d0567d6dc5ff1e104 WatchSource:0}: Error finding container f40b1b791c73f54a88f007c3dde781d8edd4bdb04fc95f8d0567d6dc5ff1e104: Status 404 returned error can't find the container with id f40b1b791c73f54a88f007c3dde781d8edd4bdb04fc95f8d0567d6dc5ff1e104 Jul 09 19:27:02.476943 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:02.476912 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:27:02.477031 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:02.476976 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:27:02.477031 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:02.477005 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:27:02.618990 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:02.618892 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64" event={"ID":"a5f96a1d-0f80-4071-b3e5-795476fccd50","Type":"ContainerStarted","Data":"b711ab9409dbcc6421cdd87402108331f2a8f35e736af195fbb555e70df4b70a"} Jul 09 19:27:02.618990 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:02.618949 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64" event={"ID":"a5f96a1d-0f80-4071-b3e5-795476fccd50","Type":"ContainerStarted","Data":"f40b1b791c73f54a88f007c3dde781d8edd4bdb04fc95f8d0567d6dc5ff1e104"} Jul 09 19:27:02.641775 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:02.641712 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64" podStartSLOduration=1.641689094 podStartE2EDuration="1.641689094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:27:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:27:02.641019623 +0000 UTC m=+1692.148799398" watchObservedRunningTime="2026-07-09 19:27:02.641689094 +0000 UTC m=+1692.149468869" Jul 09 19:27:03.319772 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:03.319732 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64" Jul 09 19:27:03.325024 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:03.324998 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64" Jul 09 19:27:03.625810 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:03.625724 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64" Jul 09 19:27:03.626711 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:03.626689 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64" Jul 09 19:27:08.390378 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:08.390333 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64"] Jul 09 19:27:08.390873 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:08.390518 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64" podUID="a5f96a1d-0f80-4071-b3e5-795476fccd50" containerName="istio-proxy" containerID="cri-o://b711ab9409dbcc6421cdd87402108331f2a8f35e736af195fbb555e70df4b70a" gracePeriod=30 Jul 09 19:27:13.662233 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:13.662169 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64" Jul 09 19:27:13.681940 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:13.681907 2566 generic.go:356] "Generic (PLEG): container finished" podID="a5f96a1d-0f80-4071-b3e5-795476fccd50" containerID="b711ab9409dbcc6421cdd87402108331f2a8f35e736af195fbb555e70df4b70a" exitCode=0 Jul 09 19:27:13.682118 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:13.681949 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64" Jul 09 19:27:13.682118 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:13.681957 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64" event={"ID":"a5f96a1d-0f80-4071-b3e5-795476fccd50","Type":"ContainerDied","Data":"b711ab9409dbcc6421cdd87402108331f2a8f35e736af195fbb555e70df4b70a"} Jul 09 19:27:13.682118 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:13.682001 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64" event={"ID":"a5f96a1d-0f80-4071-b3e5-795476fccd50","Type":"ContainerDied","Data":"f40b1b791c73f54a88f007c3dde781d8edd4bdb04fc95f8d0567d6dc5ff1e104"} Jul 09 19:27:13.682118 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:13.682029 2566 scope.go:117] "RemoveContainer" containerID="b711ab9409dbcc6421cdd87402108331f2a8f35e736af195fbb555e70df4b70a" Jul 09 19:27:13.692132 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:13.692110 2566 scope.go:117] "RemoveContainer" containerID="b711ab9409dbcc6421cdd87402108331f2a8f35e736af195fbb555e70df4b70a" Jul 09 19:27:13.692415 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:27:13.692398 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b711ab9409dbcc6421cdd87402108331f2a8f35e736af195fbb555e70df4b70a\": container with ID starting with b711ab9409dbcc6421cdd87402108331f2a8f35e736af195fbb555e70df4b70a not found: ID does not exist" containerID="b711ab9409dbcc6421cdd87402108331f2a8f35e736af195fbb555e70df4b70a" Jul 09 19:27:13.692491 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:13.692423 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b711ab9409dbcc6421cdd87402108331f2a8f35e736af195fbb555e70df4b70a"} err="failed to get container status \"b711ab9409dbcc6421cdd87402108331f2a8f35e736af195fbb555e70df4b70a\": rpc error: code = NotFound desc = could not find container \"b711ab9409dbcc6421cdd87402108331f2a8f35e736af195fbb555e70df4b70a\": container with ID starting with b711ab9409dbcc6421cdd87402108331f2a8f35e736af195fbb555e70df4b70a not found: ID does not exist" Jul 09 19:27:13.777935 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:13.777901 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5f96a1d-0f80-4071-b3e5-795476fccd50-istio-data\") pod \"a5f96a1d-0f80-4071-b3e5-795476fccd50\" (UID: \"a5f96a1d-0f80-4071-b3e5-795476fccd50\") " Jul 09 19:27:13.778112 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:13.777947 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-spldt\" (UniqueName: \"kubernetes.io/projected/a5f96a1d-0f80-4071-b3e5-795476fccd50-kube-api-access-spldt\") pod \"a5f96a1d-0f80-4071-b3e5-795476fccd50\" (UID: \"a5f96a1d-0f80-4071-b3e5-795476fccd50\") " Jul 09 19:27:13.778112 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:13.777998 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5f96a1d-0f80-4071-b3e5-795476fccd50-istiod-ca-cert\") pod \"a5f96a1d-0f80-4071-b3e5-795476fccd50\" (UID: \"a5f96a1d-0f80-4071-b3e5-795476fccd50\") " Jul 09 19:27:13.778112 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:13.778034 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5f96a1d-0f80-4071-b3e5-795476fccd50-istio-token\") pod \"a5f96a1d-0f80-4071-b3e5-795476fccd50\" (UID: \"a5f96a1d-0f80-4071-b3e5-795476fccd50\") " Jul 09 19:27:13.778112 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:13.778081 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5f96a1d-0f80-4071-b3e5-795476fccd50-istio-envoy\") pod \"a5f96a1d-0f80-4071-b3e5-795476fccd50\" (UID: \"a5f96a1d-0f80-4071-b3e5-795476fccd50\") " Jul 09 19:27:13.778112 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:13.778111 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f96a1d-0f80-4071-b3e5-795476fccd50-credential-socket\") pod \"a5f96a1d-0f80-4071-b3e5-795476fccd50\" (UID: \"a5f96a1d-0f80-4071-b3e5-795476fccd50\") " Jul 09 19:27:13.778368 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:13.778166 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5f96a1d-0f80-4071-b3e5-795476fccd50-workload-certs\") pod \"a5f96a1d-0f80-4071-b3e5-795476fccd50\" (UID: \"a5f96a1d-0f80-4071-b3e5-795476fccd50\") " Jul 09 19:27:13.778368 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:13.778210 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f96a1d-0f80-4071-b3e5-795476fccd50-workload-socket\") pod \"a5f96a1d-0f80-4071-b3e5-795476fccd50\" (UID: \"a5f96a1d-0f80-4071-b3e5-795476fccd50\") " Jul 09 19:27:13.778368 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:13.778236 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5f96a1d-0f80-4071-b3e5-795476fccd50-istio-podinfo\") pod \"a5f96a1d-0f80-4071-b3e5-795476fccd50\" (UID: \"a5f96a1d-0f80-4071-b3e5-795476fccd50\") " Jul 09 19:27:13.778528 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:13.778458 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5f96a1d-0f80-4071-b3e5-795476fccd50-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a5f96a1d-0f80-4071-b3e5-795476fccd50" (UID: "a5f96a1d-0f80-4071-b3e5-795476fccd50"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 19:27:13.778835 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:13.778787 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f96a1d-0f80-4071-b3e5-795476fccd50-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a5f96a1d-0f80-4071-b3e5-795476fccd50" (UID: "a5f96a1d-0f80-4071-b3e5-795476fccd50"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:27:13.779059 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:13.779036 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f96a1d-0f80-4071-b3e5-795476fccd50-istio-data" (OuterVolumeSpecName: "istio-data") pod "a5f96a1d-0f80-4071-b3e5-795476fccd50" (UID: "a5f96a1d-0f80-4071-b3e5-795476fccd50"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:27:13.779211 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:13.779060 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f96a1d-0f80-4071-b3e5-795476fccd50-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a5f96a1d-0f80-4071-b3e5-795476fccd50" (UID: "a5f96a1d-0f80-4071-b3e5-795476fccd50"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:27:13.779312 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:13.778945 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f96a1d-0f80-4071-b3e5-795476fccd50-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a5f96a1d-0f80-4071-b3e5-795476fccd50" (UID: "a5f96a1d-0f80-4071-b3e5-795476fccd50"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:27:13.780452 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:13.780421 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5f96a1d-0f80-4071-b3e5-795476fccd50-kube-api-access-spldt" (OuterVolumeSpecName: "kube-api-access-spldt") pod "a5f96a1d-0f80-4071-b3e5-795476fccd50" (UID: "a5f96a1d-0f80-4071-b3e5-795476fccd50"). InnerVolumeSpecName "kube-api-access-spldt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:27:13.780581 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:13.780568 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5f96a1d-0f80-4071-b3e5-795476fccd50-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a5f96a1d-0f80-4071-b3e5-795476fccd50" (UID: "a5f96a1d-0f80-4071-b3e5-795476fccd50"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 19:27:13.780745 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:13.780725 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5f96a1d-0f80-4071-b3e5-795476fccd50-istio-token" (OuterVolumeSpecName: "istio-token") pod "a5f96a1d-0f80-4071-b3e5-795476fccd50" (UID: "a5f96a1d-0f80-4071-b3e5-795476fccd50"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:27:13.780822 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:13.780746 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f96a1d-0f80-4071-b3e5-795476fccd50-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a5f96a1d-0f80-4071-b3e5-795476fccd50" (UID: "a5f96a1d-0f80-4071-b3e5-795476fccd50"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:27:13.879300 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:13.879266 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5f96a1d-0f80-4071-b3e5-795476fccd50-istio-data\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:27:13.879300 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:13.879292 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spldt\" (UniqueName: \"kubernetes.io/projected/a5f96a1d-0f80-4071-b3e5-795476fccd50-kube-api-access-spldt\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:27:13.879300 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:13.879302 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5f96a1d-0f80-4071-b3e5-795476fccd50-istiod-ca-cert\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:27:13.879300 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:13.879311 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5f96a1d-0f80-4071-b3e5-795476fccd50-istio-token\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:27:13.879668 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:13.879320 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5f96a1d-0f80-4071-b3e5-795476fccd50-istio-envoy\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:27:13.879668 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:13.879330 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f96a1d-0f80-4071-b3e5-795476fccd50-credential-socket\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:27:13.879668 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:13.879338 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5f96a1d-0f80-4071-b3e5-795476fccd50-workload-certs\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:27:13.879668 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:13.879347 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f96a1d-0f80-4071-b3e5-795476fccd50-workload-socket\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:27:13.879668 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:13.879354 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5f96a1d-0f80-4071-b3e5-795476fccd50-istio-podinfo\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:27:14.004983 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:14.004942 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64"] Jul 09 19:27:14.009125 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:14.009098 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-639f19cb-openshift-default-84fb55fb57-4vt64"] Jul 09 19:27:15.058658 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:15.058596 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5f96a1d-0f80-4071-b3e5-795476fccd50" path="/var/lib/kubelet/pods/a5f96a1d-0f80-4071-b3e5-795476fccd50/volumes" Jul 09 19:27:15.155882 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:15.155855 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf"] Jul 09 19:27:15.156229 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:15.156218 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5f96a1d-0f80-4071-b3e5-795476fccd50" containerName="istio-proxy" Jul 09 19:27:15.156270 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:15.156231 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5f96a1d-0f80-4071-b3e5-795476fccd50" containerName="istio-proxy" Jul 09 19:27:15.156313 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:15.156304 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5f96a1d-0f80-4071-b3e5-795476fccd50" containerName="istio-proxy" Jul 09 19:27:15.158963 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:15.158944 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf" Jul 09 19:27:15.161937 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:15.161914 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-8aa344f6-openshift-default-dockercfg-9rvz9\"" Jul 09 19:27:15.171864 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:15.171839 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf"] Jul 09 19:27:15.191588 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:15.191560 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhxzl\" (UniqueName: \"kubernetes.io/projected/54c583cc-5ee8-4acd-8928-d96bdff8f4a8-kube-api-access-xhxzl\") pod \"e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf\" (UID: \"54c583cc-5ee8-4acd-8928-d96bdff8f4a8\") " pod="openshift-ingress/e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf" Jul 09 19:27:15.191764 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:15.191599 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c583cc-5ee8-4acd-8928-d96bdff8f4a8-credential-socket\") pod \"e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf\" (UID: \"54c583cc-5ee8-4acd-8928-d96bdff8f4a8\") " pod="openshift-ingress/e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf" Jul 09 19:27:15.191764 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:15.191720 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54c583cc-5ee8-4acd-8928-d96bdff8f4a8-istio-podinfo\") pod \"e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf\" (UID: \"54c583cc-5ee8-4acd-8928-d96bdff8f4a8\") " pod="openshift-ingress/e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf" Jul 09 19:27:15.191764 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:15.191753 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c583cc-5ee8-4acd-8928-d96bdff8f4a8-workload-socket\") pod \"e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf\" (UID: \"54c583cc-5ee8-4acd-8928-d96bdff8f4a8\") " pod="openshift-ingress/e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf" Jul 09 19:27:15.191890 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:15.191772 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54c583cc-5ee8-4acd-8928-d96bdff8f4a8-workload-certs\") pod \"e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf\" (UID: \"54c583cc-5ee8-4acd-8928-d96bdff8f4a8\") " pod="openshift-ingress/e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf" Jul 09 19:27:15.191890 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:15.191788 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54c583cc-5ee8-4acd-8928-d96bdff8f4a8-istio-envoy\") pod \"e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf\" (UID: \"54c583cc-5ee8-4acd-8928-d96bdff8f4a8\") " pod="openshift-ingress/e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf" Jul 09 19:27:15.191890 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:15.191803 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54c583cc-5ee8-4acd-8928-d96bdff8f4a8-istio-token\") pod \"e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf\" (UID: \"54c583cc-5ee8-4acd-8928-d96bdff8f4a8\") " pod="openshift-ingress/e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf" Jul 09 19:27:15.191890 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:15.191835 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54c583cc-5ee8-4acd-8928-d96bdff8f4a8-istio-data\") pod \"e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf\" (UID: \"54c583cc-5ee8-4acd-8928-d96bdff8f4a8\") " pod="openshift-ingress/e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf" Jul 09 19:27:15.192042 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:15.191897 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54c583cc-5ee8-4acd-8928-d96bdff8f4a8-istiod-ca-cert\") pod \"e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf\" (UID: \"54c583cc-5ee8-4acd-8928-d96bdff8f4a8\") " pod="openshift-ingress/e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf" Jul 09 19:27:15.292984 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:15.292948 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54c583cc-5ee8-4acd-8928-d96bdff8f4a8-workload-certs\") pod \"e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf\" (UID: \"54c583cc-5ee8-4acd-8928-d96bdff8f4a8\") " pod="openshift-ingress/e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf" Jul 09 19:27:15.292984 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:15.292983 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54c583cc-5ee8-4acd-8928-d96bdff8f4a8-istio-envoy\") pod \"e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf\" (UID: \"54c583cc-5ee8-4acd-8928-d96bdff8f4a8\") " pod="openshift-ingress/e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf" Jul 09 19:27:15.293259 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:15.292998 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54c583cc-5ee8-4acd-8928-d96bdff8f4a8-istio-token\") pod \"e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf\" (UID: \"54c583cc-5ee8-4acd-8928-d96bdff8f4a8\") " pod="openshift-ingress/e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf" Jul 09 19:27:15.293259 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:15.293017 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54c583cc-5ee8-4acd-8928-d96bdff8f4a8-istio-data\") pod \"e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf\" (UID: \"54c583cc-5ee8-4acd-8928-d96bdff8f4a8\") " pod="openshift-ingress/e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf" Jul 09 19:27:15.293259 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:15.293053 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54c583cc-5ee8-4acd-8928-d96bdff8f4a8-istiod-ca-cert\") pod \"e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf\" (UID: \"54c583cc-5ee8-4acd-8928-d96bdff8f4a8\") " pod="openshift-ingress/e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf" Jul 09 19:27:15.293259 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:15.293098 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhxzl\" (UniqueName: \"kubernetes.io/projected/54c583cc-5ee8-4acd-8928-d96bdff8f4a8-kube-api-access-xhxzl\") pod \"e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf\" (UID: \"54c583cc-5ee8-4acd-8928-d96bdff8f4a8\") " pod="openshift-ingress/e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf" Jul 09 19:27:15.293259 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:15.293135 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c583cc-5ee8-4acd-8928-d96bdff8f4a8-credential-socket\") pod \"e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf\" (UID: \"54c583cc-5ee8-4acd-8928-d96bdff8f4a8\") " pod="openshift-ingress/e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf" Jul 09 19:27:15.293259 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:15.293198 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54c583cc-5ee8-4acd-8928-d96bdff8f4a8-istio-podinfo\") pod \"e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf\" (UID: \"54c583cc-5ee8-4acd-8928-d96bdff8f4a8\") " pod="openshift-ingress/e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf" Jul 09 19:27:15.293259 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:15.293235 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c583cc-5ee8-4acd-8928-d96bdff8f4a8-workload-socket\") pod \"e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf\" (UID: \"54c583cc-5ee8-4acd-8928-d96bdff8f4a8\") " pod="openshift-ingress/e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf" Jul 09 19:27:15.293659 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:15.293410 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54c583cc-5ee8-4acd-8928-d96bdff8f4a8-workload-certs\") pod \"e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf\" (UID: \"54c583cc-5ee8-4acd-8928-d96bdff8f4a8\") " pod="openshift-ingress/e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf" Jul 09 19:27:15.293659 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:15.293495 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54c583cc-5ee8-4acd-8928-d96bdff8f4a8-istio-data\") pod \"e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf\" (UID: \"54c583cc-5ee8-4acd-8928-d96bdff8f4a8\") " pod="openshift-ingress/e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf" Jul 09 19:27:15.293795 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:15.293774 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c583cc-5ee8-4acd-8928-d96bdff8f4a8-credential-socket\") pod \"e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf\" (UID: \"54c583cc-5ee8-4acd-8928-d96bdff8f4a8\") " pod="openshift-ingress/e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf" Jul 09 19:27:15.293855 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:15.293775 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54c583cc-5ee8-4acd-8928-d96bdff8f4a8-istiod-ca-cert\") pod \"e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf\" (UID: \"54c583cc-5ee8-4acd-8928-d96bdff8f4a8\") " pod="openshift-ingress/e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf" Jul 09 19:27:15.293855 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:15.293778 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c583cc-5ee8-4acd-8928-d96bdff8f4a8-workload-socket\") pod \"e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf\" (UID: \"54c583cc-5ee8-4acd-8928-d96bdff8f4a8\") " pod="openshift-ingress/e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf" Jul 09 19:27:15.295454 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:15.295426 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54c583cc-5ee8-4acd-8928-d96bdff8f4a8-istio-envoy\") pod \"e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf\" (UID: \"54c583cc-5ee8-4acd-8928-d96bdff8f4a8\") " pod="openshift-ingress/e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf" Jul 09 19:27:15.295572 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:15.295525 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54c583cc-5ee8-4acd-8928-d96bdff8f4a8-istio-podinfo\") pod \"e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf\" (UID: \"54c583cc-5ee8-4acd-8928-d96bdff8f4a8\") " pod="openshift-ingress/e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf" Jul 09 19:27:15.301916 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:15.301891 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54c583cc-5ee8-4acd-8928-d96bdff8f4a8-istio-token\") pod \"e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf\" (UID: \"54c583cc-5ee8-4acd-8928-d96bdff8f4a8\") " pod="openshift-ingress/e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf" Jul 09 19:27:15.302058 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:15.302042 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhxzl\" (UniqueName: \"kubernetes.io/projected/54c583cc-5ee8-4acd-8928-d96bdff8f4a8-kube-api-access-xhxzl\") pod \"e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf\" (UID: \"54c583cc-5ee8-4acd-8928-d96bdff8f4a8\") " pod="openshift-ingress/e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf" Jul 09 19:27:15.471353 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:15.471317 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf" Jul 09 19:27:15.646812 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:15.646788 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf"] Jul 09 19:27:15.648864 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:27:15.648829 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54c583cc_5ee8_4acd_8928_d96bdff8f4a8.slice/crio-29e43b4bd5886e3cc8bbcff4c8ebfd26cbc2baa8945519c0b4c2583bcee38fd2 WatchSource:0}: Error finding container 29e43b4bd5886e3cc8bbcff4c8ebfd26cbc2baa8945519c0b4c2583bcee38fd2: Status 404 returned error can't find the container with id 29e43b4bd5886e3cc8bbcff4c8ebfd26cbc2baa8945519c0b4c2583bcee38fd2 Jul 09 19:27:15.650984 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:15.650953 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:27:15.651143 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:15.651034 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:27:15.651143 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:15.651082 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:27:15.695205 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:15.695182 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf" event={"ID":"54c583cc-5ee8-4acd-8928-d96bdff8f4a8","Type":"ContainerStarted","Data":"29e43b4bd5886e3cc8bbcff4c8ebfd26cbc2baa8945519c0b4c2583bcee38fd2"} Jul 09 19:27:16.702102 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:16.702062 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf" event={"ID":"54c583cc-5ee8-4acd-8928-d96bdff8f4a8","Type":"ContainerStarted","Data":"090c9137aa60621f350b9a4f44171cbd8da8c7e798609250d3eed04efba6c046"} Jul 09 19:27:16.734881 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:16.734830 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf" podStartSLOduration=1.7348133730000002 podStartE2EDuration="1.734813373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:27:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:27:16.731794009 +0000 UTC m=+1706.239573783" watchObservedRunningTime="2026-07-09 19:27:16.734813373 +0000 UTC m=+1706.242593147" Jul 09 19:27:17.472390 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:17.472358 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf" Jul 09 19:27:17.477457 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:17.477428 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf" Jul 09 19:27:17.707933 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:17.707904 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf" Jul 09 19:27:17.709011 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:17.708987 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf" Jul 09 19:27:19.363861 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:19.363828 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:27:27.484779 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:27.484739 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf"] Jul 09 19:27:27.485288 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:27.484947 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf" podUID="54c583cc-5ee8-4acd-8928-d96bdff8f4a8" containerName="istio-proxy" containerID="cri-o://090c9137aa60621f350b9a4f44171cbd8da8c7e798609250d3eed04efba6c046" gracePeriod=30 Jul 09 19:27:27.645345 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:27.645319 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf" Jul 09 19:27:27.702886 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:27.702853 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54c583cc-5ee8-4acd-8928-d96bdff8f4a8-istio-podinfo\") pod \"54c583cc-5ee8-4acd-8928-d96bdff8f4a8\" (UID: \"54c583cc-5ee8-4acd-8928-d96bdff8f4a8\") " Jul 09 19:27:27.702886 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:27.702889 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54c583cc-5ee8-4acd-8928-d96bdff8f4a8-workload-certs\") pod \"54c583cc-5ee8-4acd-8928-d96bdff8f4a8\" (UID: \"54c583cc-5ee8-4acd-8928-d96bdff8f4a8\") " Jul 09 19:27:27.703124 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:27.702934 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c583cc-5ee8-4acd-8928-d96bdff8f4a8-workload-socket\") pod \"54c583cc-5ee8-4acd-8928-d96bdff8f4a8\" (UID: \"54c583cc-5ee8-4acd-8928-d96bdff8f4a8\") " Jul 09 19:27:27.703124 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:27.702959 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c583cc-5ee8-4acd-8928-d96bdff8f4a8-credential-socket\") pod \"54c583cc-5ee8-4acd-8928-d96bdff8f4a8\" (UID: \"54c583cc-5ee8-4acd-8928-d96bdff8f4a8\") " Jul 09 19:27:27.703124 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:27.702980 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54c583cc-5ee8-4acd-8928-d96bdff8f4a8-istio-token\") pod \"54c583cc-5ee8-4acd-8928-d96bdff8f4a8\" (UID: \"54c583cc-5ee8-4acd-8928-d96bdff8f4a8\") " Jul 09 19:27:27.703124 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:27.703002 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54c583cc-5ee8-4acd-8928-d96bdff8f4a8-istio-data\") pod \"54c583cc-5ee8-4acd-8928-d96bdff8f4a8\" (UID: \"54c583cc-5ee8-4acd-8928-d96bdff8f4a8\") " Jul 09 19:27:27.703124 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:27.703030 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xhxzl\" (UniqueName: \"kubernetes.io/projected/54c583cc-5ee8-4acd-8928-d96bdff8f4a8-kube-api-access-xhxzl\") pod \"54c583cc-5ee8-4acd-8928-d96bdff8f4a8\" (UID: \"54c583cc-5ee8-4acd-8928-d96bdff8f4a8\") " Jul 09 19:27:27.703124 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:27.703111 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54c583cc-5ee8-4acd-8928-d96bdff8f4a8-istiod-ca-cert\") pod \"54c583cc-5ee8-4acd-8928-d96bdff8f4a8\" (UID: \"54c583cc-5ee8-4acd-8928-d96bdff8f4a8\") " Jul 09 19:27:27.703439 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:27.703151 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54c583cc-5ee8-4acd-8928-d96bdff8f4a8-istio-envoy\") pod \"54c583cc-5ee8-4acd-8928-d96bdff8f4a8\" (UID: \"54c583cc-5ee8-4acd-8928-d96bdff8f4a8\") " Jul 09 19:27:27.703439 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:27.703187 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c583cc-5ee8-4acd-8928-d96bdff8f4a8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "54c583cc-5ee8-4acd-8928-d96bdff8f4a8" (UID: "54c583cc-5ee8-4acd-8928-d96bdff8f4a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:27:27.703439 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:27.703238 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c583cc-5ee8-4acd-8928-d96bdff8f4a8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "54c583cc-5ee8-4acd-8928-d96bdff8f4a8" (UID: "54c583cc-5ee8-4acd-8928-d96bdff8f4a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:27:27.703602 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:27.703475 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54c583cc-5ee8-4acd-8928-d96bdff8f4a8-workload-certs\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:27:27.703602 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:27.703480 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54c583cc-5ee8-4acd-8928-d96bdff8f4a8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "54c583cc-5ee8-4acd-8928-d96bdff8f4a8" (UID: "54c583cc-5ee8-4acd-8928-d96bdff8f4a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 19:27:27.703602 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:27.703495 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c583cc-5ee8-4acd-8928-d96bdff8f4a8-workload-socket\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:27:27.703778 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:27.703656 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c583cc-5ee8-4acd-8928-d96bdff8f4a8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "54c583cc-5ee8-4acd-8928-d96bdff8f4a8" (UID: "54c583cc-5ee8-4acd-8928-d96bdff8f4a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:27:27.704368 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:27.704316 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c583cc-5ee8-4acd-8928-d96bdff8f4a8-istio-data" (OuterVolumeSpecName: "istio-data") pod "54c583cc-5ee8-4acd-8928-d96bdff8f4a8" (UID: "54c583cc-5ee8-4acd-8928-d96bdff8f4a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:27:27.705845 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:27.705809 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c583cc-5ee8-4acd-8928-d96bdff8f4a8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "54c583cc-5ee8-4acd-8928-d96bdff8f4a8" (UID: "54c583cc-5ee8-4acd-8928-d96bdff8f4a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:27:27.706655 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:27.706573 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54c583cc-5ee8-4acd-8928-d96bdff8f4a8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "54c583cc-5ee8-4acd-8928-d96bdff8f4a8" (UID: "54c583cc-5ee8-4acd-8928-d96bdff8f4a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 19:27:27.707205 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:27.707160 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54c583cc-5ee8-4acd-8928-d96bdff8f4a8-kube-api-access-xhxzl" (OuterVolumeSpecName: "kube-api-access-xhxzl") pod "54c583cc-5ee8-4acd-8928-d96bdff8f4a8" (UID: "54c583cc-5ee8-4acd-8928-d96bdff8f4a8"). InnerVolumeSpecName "kube-api-access-xhxzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:27:27.707477 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:27.707453 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54c583cc-5ee8-4acd-8928-d96bdff8f4a8-istio-token" (OuterVolumeSpecName: "istio-token") pod "54c583cc-5ee8-4acd-8928-d96bdff8f4a8" (UID: "54c583cc-5ee8-4acd-8928-d96bdff8f4a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:27:27.769081 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:27.769051 2566 generic.go:356] "Generic (PLEG): container finished" podID="54c583cc-5ee8-4acd-8928-d96bdff8f4a8" containerID="090c9137aa60621f350b9a4f44171cbd8da8c7e798609250d3eed04efba6c046" exitCode=0 Jul 09 19:27:27.769247 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:27.769098 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf" Jul 09 19:27:27.769247 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:27.769098 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf" event={"ID":"54c583cc-5ee8-4acd-8928-d96bdff8f4a8","Type":"ContainerDied","Data":"090c9137aa60621f350b9a4f44171cbd8da8c7e798609250d3eed04efba6c046"} Jul 09 19:27:27.769247 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:27.769222 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf" event={"ID":"54c583cc-5ee8-4acd-8928-d96bdff8f4a8","Type":"ContainerDied","Data":"29e43b4bd5886e3cc8bbcff4c8ebfd26cbc2baa8945519c0b4c2583bcee38fd2"} Jul 09 19:27:27.769358 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:27.769251 2566 scope.go:117] "RemoveContainer" containerID="090c9137aa60621f350b9a4f44171cbd8da8c7e798609250d3eed04efba6c046" Jul 09 19:27:27.779582 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:27.779565 2566 scope.go:117] "RemoveContainer" containerID="090c9137aa60621f350b9a4f44171cbd8da8c7e798609250d3eed04efba6c046" Jul 09 19:27:27.779902 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:27:27.779881 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"090c9137aa60621f350b9a4f44171cbd8da8c7e798609250d3eed04efba6c046\": container with ID starting with 090c9137aa60621f350b9a4f44171cbd8da8c7e798609250d3eed04efba6c046 not found: ID does not exist" containerID="090c9137aa60621f350b9a4f44171cbd8da8c7e798609250d3eed04efba6c046" Jul 09 19:27:27.779983 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:27.779912 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"090c9137aa60621f350b9a4f44171cbd8da8c7e798609250d3eed04efba6c046"} err="failed to get container status \"090c9137aa60621f350b9a4f44171cbd8da8c7e798609250d3eed04efba6c046\": rpc error: code = NotFound desc = could not find container \"090c9137aa60621f350b9a4f44171cbd8da8c7e798609250d3eed04efba6c046\": container with ID starting with 090c9137aa60621f350b9a4f44171cbd8da8c7e798609250d3eed04efba6c046 not found: ID does not exist" Jul 09 19:27:27.793821 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:27.793792 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf"] Jul 09 19:27:27.797799 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:27.797775 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-8aa344f6-openshift-default-7885cf4f57-dvhnf"] Jul 09 19:27:27.804957 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:27.804935 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54c583cc-5ee8-4acd-8928-d96bdff8f4a8-istio-podinfo\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:27:27.804957 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:27.804956 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c583cc-5ee8-4acd-8928-d96bdff8f4a8-credential-socket\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:27:27.805136 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:27.804966 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54c583cc-5ee8-4acd-8928-d96bdff8f4a8-istio-token\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:27:27.805136 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:27.804979 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54c583cc-5ee8-4acd-8928-d96bdff8f4a8-istio-data\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:27:27.805136 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:27.804999 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhxzl\" (UniqueName: \"kubernetes.io/projected/54c583cc-5ee8-4acd-8928-d96bdff8f4a8-kube-api-access-xhxzl\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:27:27.805136 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:27.805008 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54c583cc-5ee8-4acd-8928-d96bdff8f4a8-istiod-ca-cert\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:27:27.805136 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:27.805016 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54c583cc-5ee8-4acd-8928-d96bdff8f4a8-istio-envoy\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:27:29.061984 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:29.061948 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54c583cc-5ee8-4acd-8928-d96bdff8f4a8" path="/var/lib/kubelet/pods/54c583cc-5ee8-4acd-8928-d96bdff8f4a8/volumes" Jul 09 19:27:31.365909 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:31.365874 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:27:33.735429 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:33.735166 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l"] Jul 09 19:27:33.735674 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:33.735561 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54c583cc-5ee8-4acd-8928-d96bdff8f4a8" containerName="istio-proxy" Jul 09 19:27:33.735674 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:33.735572 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="54c583cc-5ee8-4acd-8928-d96bdff8f4a8" containerName="istio-proxy" Jul 09 19:27:33.735674 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:33.735665 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="54c583cc-5ee8-4acd-8928-d96bdff8f4a8" containerName="istio-proxy" Jul 09 19:27:33.740198 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:33.740175 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l" Jul 09 19:27:33.743104 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:33.743083 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-0153dcdd-openshift-default-dockercfg-rtqjj\"" Jul 09 19:27:33.752677 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:33.752646 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l"] Jul 09 19:27:33.858645 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:33.858582 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f300db84-958e-49f2-89a5-41a0aa00cf28-workload-certs\") pod \"e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l\" (UID: \"f300db84-958e-49f2-89a5-41a0aa00cf28\") " pod="openshift-ingress/e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l" Jul 09 19:27:33.858821 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:33.858663 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdjnv\" (UniqueName: \"kubernetes.io/projected/f300db84-958e-49f2-89a5-41a0aa00cf28-kube-api-access-hdjnv\") pod \"e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l\" (UID: \"f300db84-958e-49f2-89a5-41a0aa00cf28\") " pod="openshift-ingress/e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l" Jul 09 19:27:33.858821 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:33.858693 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f300db84-958e-49f2-89a5-41a0aa00cf28-istio-podinfo\") pod \"e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l\" (UID: \"f300db84-958e-49f2-89a5-41a0aa00cf28\") " pod="openshift-ingress/e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l" Jul 09 19:27:33.858821 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:33.858720 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f300db84-958e-49f2-89a5-41a0aa00cf28-istiod-ca-cert\") pod \"e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l\" (UID: \"f300db84-958e-49f2-89a5-41a0aa00cf28\") " pod="openshift-ingress/e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l" Jul 09 19:27:33.858821 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:33.858795 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f300db84-958e-49f2-89a5-41a0aa00cf28-istio-envoy\") pod \"e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l\" (UID: \"f300db84-958e-49f2-89a5-41a0aa00cf28\") " pod="openshift-ingress/e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l" Jul 09 19:27:33.858821 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:33.858810 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f300db84-958e-49f2-89a5-41a0aa00cf28-istio-token\") pod \"e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l\" (UID: \"f300db84-958e-49f2-89a5-41a0aa00cf28\") " pod="openshift-ingress/e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l" Jul 09 19:27:33.858992 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:33.858826 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f300db84-958e-49f2-89a5-41a0aa00cf28-istio-data\") pod \"e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l\" (UID: \"f300db84-958e-49f2-89a5-41a0aa00cf28\") " pod="openshift-ingress/e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l" Jul 09 19:27:33.858992 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:33.858910 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f300db84-958e-49f2-89a5-41a0aa00cf28-workload-socket\") pod \"e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l\" (UID: \"f300db84-958e-49f2-89a5-41a0aa00cf28\") " pod="openshift-ingress/e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l" Jul 09 19:27:33.858992 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:33.858933 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f300db84-958e-49f2-89a5-41a0aa00cf28-credential-socket\") pod \"e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l\" (UID: \"f300db84-958e-49f2-89a5-41a0aa00cf28\") " pod="openshift-ingress/e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l" Jul 09 19:27:33.959408 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:33.959366 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdjnv\" (UniqueName: \"kubernetes.io/projected/f300db84-958e-49f2-89a5-41a0aa00cf28-kube-api-access-hdjnv\") pod \"e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l\" (UID: \"f300db84-958e-49f2-89a5-41a0aa00cf28\") " pod="openshift-ingress/e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l" Jul 09 19:27:33.959408 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:33.959414 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f300db84-958e-49f2-89a5-41a0aa00cf28-istio-podinfo\") pod \"e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l\" (UID: \"f300db84-958e-49f2-89a5-41a0aa00cf28\") " pod="openshift-ingress/e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l" Jul 09 19:27:33.959733 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:33.959454 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f300db84-958e-49f2-89a5-41a0aa00cf28-istiod-ca-cert\") pod \"e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l\" (UID: \"f300db84-958e-49f2-89a5-41a0aa00cf28\") " pod="openshift-ingress/e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l" Jul 09 19:27:33.959733 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:33.959502 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f300db84-958e-49f2-89a5-41a0aa00cf28-istio-envoy\") pod \"e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l\" (UID: \"f300db84-958e-49f2-89a5-41a0aa00cf28\") " pod="openshift-ingress/e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l" Jul 09 19:27:33.959733 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:33.959523 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f300db84-958e-49f2-89a5-41a0aa00cf28-istio-token\") pod \"e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l\" (UID: \"f300db84-958e-49f2-89a5-41a0aa00cf28\") " pod="openshift-ingress/e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l" Jul 09 19:27:33.959733 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:33.959657 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f300db84-958e-49f2-89a5-41a0aa00cf28-istio-data\") pod \"e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l\" (UID: \"f300db84-958e-49f2-89a5-41a0aa00cf28\") " pod="openshift-ingress/e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l" Jul 09 19:27:33.959985 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:33.959760 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f300db84-958e-49f2-89a5-41a0aa00cf28-workload-socket\") pod \"e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l\" (UID: \"f300db84-958e-49f2-89a5-41a0aa00cf28\") " pod="openshift-ingress/e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l" Jul 09 19:27:33.959985 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:33.959788 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f300db84-958e-49f2-89a5-41a0aa00cf28-credential-socket\") pod \"e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l\" (UID: \"f300db84-958e-49f2-89a5-41a0aa00cf28\") " pod="openshift-ingress/e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l" Jul 09 19:27:33.959985 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:33.959836 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f300db84-958e-49f2-89a5-41a0aa00cf28-workload-certs\") pod \"e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l\" (UID: \"f300db84-958e-49f2-89a5-41a0aa00cf28\") " pod="openshift-ingress/e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l" Jul 09 19:27:33.960147 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:33.960078 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f300db84-958e-49f2-89a5-41a0aa00cf28-istio-data\") pod \"e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l\" (UID: \"f300db84-958e-49f2-89a5-41a0aa00cf28\") " pod="openshift-ingress/e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l" Jul 09 19:27:33.960231 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:33.960201 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f300db84-958e-49f2-89a5-41a0aa00cf28-workload-socket\") pod \"e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l\" (UID: \"f300db84-958e-49f2-89a5-41a0aa00cf28\") " pod="openshift-ingress/e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l" Jul 09 19:27:33.960281 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:33.960228 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f300db84-958e-49f2-89a5-41a0aa00cf28-credential-socket\") pod \"e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l\" (UID: \"f300db84-958e-49f2-89a5-41a0aa00cf28\") " pod="openshift-ingress/e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l" Jul 09 19:27:33.960281 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:33.960261 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f300db84-958e-49f2-89a5-41a0aa00cf28-workload-certs\") pod \"e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l\" (UID: \"f300db84-958e-49f2-89a5-41a0aa00cf28\") " pod="openshift-ingress/e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l" Jul 09 19:27:33.960419 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:33.960399 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f300db84-958e-49f2-89a5-41a0aa00cf28-istiod-ca-cert\") pod \"e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l\" (UID: \"f300db84-958e-49f2-89a5-41a0aa00cf28\") " pod="openshift-ingress/e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l" Jul 09 19:27:33.961899 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:33.961877 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f300db84-958e-49f2-89a5-41a0aa00cf28-istio-envoy\") pod \"e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l\" (UID: \"f300db84-958e-49f2-89a5-41a0aa00cf28\") " pod="openshift-ingress/e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l" Jul 09 19:27:33.962115 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:33.962094 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f300db84-958e-49f2-89a5-41a0aa00cf28-istio-podinfo\") pod \"e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l\" (UID: \"f300db84-958e-49f2-89a5-41a0aa00cf28\") " pod="openshift-ingress/e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l" Jul 09 19:27:33.969497 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:33.969470 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f300db84-958e-49f2-89a5-41a0aa00cf28-istio-token\") pod \"e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l\" (UID: \"f300db84-958e-49f2-89a5-41a0aa00cf28\") " pod="openshift-ingress/e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l" Jul 09 19:27:33.969626 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:33.969563 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdjnv\" (UniqueName: \"kubernetes.io/projected/f300db84-958e-49f2-89a5-41a0aa00cf28-kube-api-access-hdjnv\") pod \"e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l\" (UID: \"f300db84-958e-49f2-89a5-41a0aa00cf28\") " pod="openshift-ingress/e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l" Jul 09 19:27:34.052396 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:34.052300 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l" Jul 09 19:27:34.205390 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:34.205358 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l"] Jul 09 19:27:34.207330 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:27:34.207305 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf300db84_958e_49f2_89a5_41a0aa00cf28.slice/crio-9d8ea3a2e00e47118596b7f013850e5a51b01171cd96016a12a932ffb45e3b4e WatchSource:0}: Error finding container 9d8ea3a2e00e47118596b7f013850e5a51b01171cd96016a12a932ffb45e3b4e: Status 404 returned error can't find the container with id 9d8ea3a2e00e47118596b7f013850e5a51b01171cd96016a12a932ffb45e3b4e Jul 09 19:27:34.210021 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:34.209973 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:27:34.210136 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:34.210076 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:27:34.210136 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:34.210110 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:27:34.809014 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:34.808978 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l" event={"ID":"f300db84-958e-49f2-89a5-41a0aa00cf28","Type":"ContainerStarted","Data":"2adbb2bf69ef21e36df917b1a392657b912e556ea291ab915b9949bcb2fa0e53"} Jul 09 19:27:34.809447 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:34.809046 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l" event={"ID":"f300db84-958e-49f2-89a5-41a0aa00cf28","Type":"ContainerStarted","Data":"9d8ea3a2e00e47118596b7f013850e5a51b01171cd96016a12a932ffb45e3b4e"} Jul 09 19:27:34.836004 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:34.835948 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l" podStartSLOduration=1.835931889 podStartE2EDuration="1.835931889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:27:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:27:34.834458564 +0000 UTC m=+1724.342238338" watchObservedRunningTime="2026-07-09 19:27:34.835931889 +0000 UTC m=+1724.343711662" Jul 09 19:27:35.072193 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:35.072106 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l" Jul 09 19:27:35.084452 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:35.084419 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l" Jul 09 19:27:35.815106 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:35.815074 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l" Jul 09 19:27:35.816155 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:35.816135 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l" Jul 09 19:27:39.513371 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:39.513336 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l"] Jul 09 19:27:39.513951 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:39.513544 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l" podUID="f300db84-958e-49f2-89a5-41a0aa00cf28" containerName="istio-proxy" containerID="cri-o://2adbb2bf69ef21e36df917b1a392657b912e556ea291ab915b9949bcb2fa0e53" gracePeriod=30 Jul 09 19:27:39.673396 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:39.673362 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:27:44.774919 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:44.774858 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l" Jul 09 19:27:44.858686 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:44.858654 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f300db84-958e-49f2-89a5-41a0aa00cf28-credential-socket\") pod \"f300db84-958e-49f2-89a5-41a0aa00cf28\" (UID: \"f300db84-958e-49f2-89a5-41a0aa00cf28\") " Jul 09 19:27:44.858857 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:44.858721 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f300db84-958e-49f2-89a5-41a0aa00cf28-workload-certs\") pod \"f300db84-958e-49f2-89a5-41a0aa00cf28\" (UID: \"f300db84-958e-49f2-89a5-41a0aa00cf28\") " Jul 09 19:27:44.858857 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:44.858747 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hdjnv\" (UniqueName: \"kubernetes.io/projected/f300db84-958e-49f2-89a5-41a0aa00cf28-kube-api-access-hdjnv\") pod \"f300db84-958e-49f2-89a5-41a0aa00cf28\" (UID: \"f300db84-958e-49f2-89a5-41a0aa00cf28\") " Jul 09 19:27:44.858857 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:44.858791 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f300db84-958e-49f2-89a5-41a0aa00cf28-istio-podinfo\") pod \"f300db84-958e-49f2-89a5-41a0aa00cf28\" (UID: \"f300db84-958e-49f2-89a5-41a0aa00cf28\") " Jul 09 19:27:44.858857 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:44.858825 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f300db84-958e-49f2-89a5-41a0aa00cf28-istio-data\") pod \"f300db84-958e-49f2-89a5-41a0aa00cf28\" (UID: \"f300db84-958e-49f2-89a5-41a0aa00cf28\") " Jul 09 19:27:44.858857 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:44.858856 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f300db84-958e-49f2-89a5-41a0aa00cf28-istio-envoy\") pod \"f300db84-958e-49f2-89a5-41a0aa00cf28\" (UID: \"f300db84-958e-49f2-89a5-41a0aa00cf28\") " Jul 09 19:27:44.859182 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:44.858886 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f300db84-958e-49f2-89a5-41a0aa00cf28-istiod-ca-cert\") pod \"f300db84-958e-49f2-89a5-41a0aa00cf28\" (UID: \"f300db84-958e-49f2-89a5-41a0aa00cf28\") " Jul 09 19:27:44.859182 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:44.858936 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f300db84-958e-49f2-89a5-41a0aa00cf28-workload-socket\") pod \"f300db84-958e-49f2-89a5-41a0aa00cf28\" (UID: \"f300db84-958e-49f2-89a5-41a0aa00cf28\") " Jul 09 19:27:44.859182 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:44.858968 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f300db84-958e-49f2-89a5-41a0aa00cf28-istio-token\") pod \"f300db84-958e-49f2-89a5-41a0aa00cf28\" (UID: \"f300db84-958e-49f2-89a5-41a0aa00cf28\") " Jul 09 19:27:44.859182 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:44.858968 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f300db84-958e-49f2-89a5-41a0aa00cf28-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f300db84-958e-49f2-89a5-41a0aa00cf28" (UID: "f300db84-958e-49f2-89a5-41a0aa00cf28"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:27:44.859182 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:44.859026 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f300db84-958e-49f2-89a5-41a0aa00cf28-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f300db84-958e-49f2-89a5-41a0aa00cf28" (UID: "f300db84-958e-49f2-89a5-41a0aa00cf28"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:27:44.859182 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:44.859150 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f300db84-958e-49f2-89a5-41a0aa00cf28-istio-data" (OuterVolumeSpecName: "istio-data") pod "f300db84-958e-49f2-89a5-41a0aa00cf28" (UID: "f300db84-958e-49f2-89a5-41a0aa00cf28"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:27:44.859510 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:44.859246 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f300db84-958e-49f2-89a5-41a0aa00cf28-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f300db84-958e-49f2-89a5-41a0aa00cf28" (UID: "f300db84-958e-49f2-89a5-41a0aa00cf28"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:27:44.859510 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:44.859262 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f300db84-958e-49f2-89a5-41a0aa00cf28-istio-data\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:27:44.859510 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:44.859281 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f300db84-958e-49f2-89a5-41a0aa00cf28-credential-socket\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:27:44.859510 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:44.859284 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f300db84-958e-49f2-89a5-41a0aa00cf28-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f300db84-958e-49f2-89a5-41a0aa00cf28" (UID: "f300db84-958e-49f2-89a5-41a0aa00cf28"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 19:27:44.859510 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:44.859295 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f300db84-958e-49f2-89a5-41a0aa00cf28-workload-certs\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:27:44.861497 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:44.861467 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f300db84-958e-49f2-89a5-41a0aa00cf28-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f300db84-958e-49f2-89a5-41a0aa00cf28" (UID: "f300db84-958e-49f2-89a5-41a0aa00cf28"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 19:27:44.861497 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:44.861473 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f300db84-958e-49f2-89a5-41a0aa00cf28-kube-api-access-hdjnv" (OuterVolumeSpecName: "kube-api-access-hdjnv") pod "f300db84-958e-49f2-89a5-41a0aa00cf28" (UID: "f300db84-958e-49f2-89a5-41a0aa00cf28"). InnerVolumeSpecName "kube-api-access-hdjnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:27:44.861705 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:44.861632 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f300db84-958e-49f2-89a5-41a0aa00cf28-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f300db84-958e-49f2-89a5-41a0aa00cf28" (UID: "f300db84-958e-49f2-89a5-41a0aa00cf28"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:27:44.861759 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:44.861707 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f300db84-958e-49f2-89a5-41a0aa00cf28-istio-token" (OuterVolumeSpecName: "istio-token") pod "f300db84-958e-49f2-89a5-41a0aa00cf28" (UID: "f300db84-958e-49f2-89a5-41a0aa00cf28"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:27:44.865051 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:44.865026 2566 generic.go:356] "Generic (PLEG): container finished" podID="f300db84-958e-49f2-89a5-41a0aa00cf28" containerID="2adbb2bf69ef21e36df917b1a392657b912e556ea291ab915b9949bcb2fa0e53" exitCode=0 Jul 09 19:27:44.865163 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:44.865069 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l" Jul 09 19:27:44.865163 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:44.865111 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l" event={"ID":"f300db84-958e-49f2-89a5-41a0aa00cf28","Type":"ContainerDied","Data":"2adbb2bf69ef21e36df917b1a392657b912e556ea291ab915b9949bcb2fa0e53"} Jul 09 19:27:44.865237 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:44.865168 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l" event={"ID":"f300db84-958e-49f2-89a5-41a0aa00cf28","Type":"ContainerDied","Data":"9d8ea3a2e00e47118596b7f013850e5a51b01171cd96016a12a932ffb45e3b4e"} Jul 09 19:27:44.865237 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:44.865198 2566 scope.go:117] "RemoveContainer" containerID="2adbb2bf69ef21e36df917b1a392657b912e556ea291ab915b9949bcb2fa0e53" Jul 09 19:27:44.882128 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:44.881804 2566 scope.go:117] "RemoveContainer" containerID="2adbb2bf69ef21e36df917b1a392657b912e556ea291ab915b9949bcb2fa0e53" Jul 09 19:27:44.882423 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:27:44.882368 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2adbb2bf69ef21e36df917b1a392657b912e556ea291ab915b9949bcb2fa0e53\": container with ID starting with 2adbb2bf69ef21e36df917b1a392657b912e556ea291ab915b9949bcb2fa0e53 not found: ID does not exist" containerID="2adbb2bf69ef21e36df917b1a392657b912e556ea291ab915b9949bcb2fa0e53" Jul 09 19:27:44.882423 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:44.882399 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2adbb2bf69ef21e36df917b1a392657b912e556ea291ab915b9949bcb2fa0e53"} err="failed to get container status \"2adbb2bf69ef21e36df917b1a392657b912e556ea291ab915b9949bcb2fa0e53\": rpc error: code = NotFound desc = could not find container \"2adbb2bf69ef21e36df917b1a392657b912e556ea291ab915b9949bcb2fa0e53\": container with ID starting with 2adbb2bf69ef21e36df917b1a392657b912e556ea291ab915b9949bcb2fa0e53 not found: ID does not exist" Jul 09 19:27:44.894396 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:44.894357 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l"] Jul 09 19:27:44.898429 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:44.898408 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-0153dcdd-openshift-default-667b9d4b4c-7wd7l"] Jul 09 19:27:44.960726 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:44.960704 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdjnv\" (UniqueName: \"kubernetes.io/projected/f300db84-958e-49f2-89a5-41a0aa00cf28-kube-api-access-hdjnv\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:27:44.960726 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:44.960726 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f300db84-958e-49f2-89a5-41a0aa00cf28-istio-podinfo\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:27:44.960726 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:44.960738 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f300db84-958e-49f2-89a5-41a0aa00cf28-istio-envoy\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:27:44.960920 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:44.960746 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f300db84-958e-49f2-89a5-41a0aa00cf28-istiod-ca-cert\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:27:44.960920 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:44.960755 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f300db84-958e-49f2-89a5-41a0aa00cf28-workload-socket\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:27:44.960920 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:44.960764 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f300db84-958e-49f2-89a5-41a0aa00cf28-istio-token\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:27:45.057790 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:45.057725 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f300db84-958e-49f2-89a5-41a0aa00cf28" path="/var/lib/kubelet/pods/f300db84-958e-49f2-89a5-41a0aa00cf28/volumes" Jul 09 19:27:50.162577 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:27:50.162544 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:28:51.099236 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:28:51.099207 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-vws4c_7cf4af17-5218-4b92-bceb-addbefcc03cb/console-operator/1.log" Jul 09 19:28:51.104192 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:28:51.104168 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6dh7g_db233460-134d-4d9d-8da9-31d89366a4e0/ovn-acl-logging/0.log" Jul 09 19:28:51.108470 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:28:51.108450 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-vws4c_7cf4af17-5218-4b92-bceb-addbefcc03cb/console-operator/1.log" Jul 09 19:28:51.113146 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:28:51.113126 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6dh7g_db233460-134d-4d9d-8da9-31d89366a4e0/ovn-acl-logging/0.log" Jul 09 19:29:08.665064 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:08.665032 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:29:16.365400 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:16.365367 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:29:22.481159 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:22.481127 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb"] Jul 09 19:29:22.481754 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:22.481735 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f300db84-958e-49f2-89a5-41a0aa00cf28" containerName="istio-proxy" Jul 09 19:29:22.481806 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:22.481758 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f300db84-958e-49f2-89a5-41a0aa00cf28" containerName="istio-proxy" Jul 09 19:29:22.481874 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:22.481863 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f300db84-958e-49f2-89a5-41a0aa00cf28" containerName="istio-proxy" Jul 09 19:29:22.484818 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:22.484801 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb" Jul 09 19:29:22.487576 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:22.487559 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-5ec314ef-openshift-default-dockercfg-cscln\"" Jul 09 19:29:22.501881 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:22.501854 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb"] Jul 09 19:29:22.595135 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:22.595097 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35ca30e4-21ca-49f0-bf38-17e0395c7b15-istio-data\") pod \"e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb\" (UID: \"35ca30e4-21ca-49f0-bf38-17e0395c7b15\") " pod="openshift-ingress/e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb" Jul 09 19:29:22.595364 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:22.595204 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35ca30e4-21ca-49f0-bf38-17e0395c7b15-credential-socket\") pod \"e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb\" (UID: \"35ca30e4-21ca-49f0-bf38-17e0395c7b15\") " pod="openshift-ingress/e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb" Jul 09 19:29:22.595364 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:22.595229 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35ca30e4-21ca-49f0-bf38-17e0395c7b15-workload-certs\") pod \"e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb\" (UID: \"35ca30e4-21ca-49f0-bf38-17e0395c7b15\") " pod="openshift-ingress/e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb" Jul 09 19:29:22.595364 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:22.595244 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35ca30e4-21ca-49f0-bf38-17e0395c7b15-istio-podinfo\") pod \"e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb\" (UID: \"35ca30e4-21ca-49f0-bf38-17e0395c7b15\") " pod="openshift-ingress/e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb" Jul 09 19:29:22.595364 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:22.595318 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35ca30e4-21ca-49f0-bf38-17e0395c7b15-istiod-ca-cert\") pod \"e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb\" (UID: \"35ca30e4-21ca-49f0-bf38-17e0395c7b15\") " pod="openshift-ingress/e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb" Jul 09 19:29:22.595549 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:22.595379 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35ca30e4-21ca-49f0-bf38-17e0395c7b15-istio-token\") pod \"e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb\" (UID: \"35ca30e4-21ca-49f0-bf38-17e0395c7b15\") " pod="openshift-ingress/e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb" Jul 09 19:29:22.595549 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:22.595400 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjt8f\" (UniqueName: \"kubernetes.io/projected/35ca30e4-21ca-49f0-bf38-17e0395c7b15-kube-api-access-sjt8f\") pod \"e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb\" (UID: \"35ca30e4-21ca-49f0-bf38-17e0395c7b15\") " pod="openshift-ingress/e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb" Jul 09 19:29:22.595549 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:22.595426 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35ca30e4-21ca-49f0-bf38-17e0395c7b15-istio-envoy\") pod \"e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb\" (UID: \"35ca30e4-21ca-49f0-bf38-17e0395c7b15\") " pod="openshift-ingress/e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb" Jul 09 19:29:22.595549 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:22.595480 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35ca30e4-21ca-49f0-bf38-17e0395c7b15-workload-socket\") pod \"e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb\" (UID: \"35ca30e4-21ca-49f0-bf38-17e0395c7b15\") " pod="openshift-ingress/e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb" Jul 09 19:29:22.696257 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:22.696222 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35ca30e4-21ca-49f0-bf38-17e0395c7b15-workload-socket\") pod \"e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb\" (UID: \"35ca30e4-21ca-49f0-bf38-17e0395c7b15\") " pod="openshift-ingress/e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb" Jul 09 19:29:22.696426 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:22.696268 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35ca30e4-21ca-49f0-bf38-17e0395c7b15-istio-data\") pod \"e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb\" (UID: \"35ca30e4-21ca-49f0-bf38-17e0395c7b15\") " pod="openshift-ingress/e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb" Jul 09 19:29:22.696426 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:22.696362 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35ca30e4-21ca-49f0-bf38-17e0395c7b15-credential-socket\") pod \"e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb\" (UID: \"35ca30e4-21ca-49f0-bf38-17e0395c7b15\") " pod="openshift-ingress/e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb" Jul 09 19:29:22.696426 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:22.696406 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35ca30e4-21ca-49f0-bf38-17e0395c7b15-workload-certs\") pod \"e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb\" (UID: \"35ca30e4-21ca-49f0-bf38-17e0395c7b15\") " pod="openshift-ingress/e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb" Jul 09 19:29:22.696558 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:22.696535 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35ca30e4-21ca-49f0-bf38-17e0395c7b15-istio-podinfo\") pod \"e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb\" (UID: \"35ca30e4-21ca-49f0-bf38-17e0395c7b15\") " pod="openshift-ingress/e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb" Jul 09 19:29:22.696601 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:22.696586 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35ca30e4-21ca-49f0-bf38-17e0395c7b15-istiod-ca-cert\") pod \"e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb\" (UID: \"35ca30e4-21ca-49f0-bf38-17e0395c7b15\") " pod="openshift-ingress/e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb" Jul 09 19:29:22.696696 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:22.696648 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35ca30e4-21ca-49f0-bf38-17e0395c7b15-istio-data\") pod \"e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb\" (UID: \"35ca30e4-21ca-49f0-bf38-17e0395c7b15\") " pod="openshift-ingress/e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb" Jul 09 19:29:22.696696 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:22.696662 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35ca30e4-21ca-49f0-bf38-17e0395c7b15-istio-token\") pod \"e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb\" (UID: \"35ca30e4-21ca-49f0-bf38-17e0395c7b15\") " pod="openshift-ingress/e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb" Jul 09 19:29:22.696696 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:22.696692 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjt8f\" (UniqueName: \"kubernetes.io/projected/35ca30e4-21ca-49f0-bf38-17e0395c7b15-kube-api-access-sjt8f\") pod \"e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb\" (UID: \"35ca30e4-21ca-49f0-bf38-17e0395c7b15\") " pod="openshift-ingress/e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb" Jul 09 19:29:22.696853 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:22.696699 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35ca30e4-21ca-49f0-bf38-17e0395c7b15-workload-socket\") pod \"e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb\" (UID: \"35ca30e4-21ca-49f0-bf38-17e0395c7b15\") " pod="openshift-ingress/e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb" Jul 09 19:29:22.696853 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:22.696714 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35ca30e4-21ca-49f0-bf38-17e0395c7b15-istio-envoy\") pod \"e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb\" (UID: \"35ca30e4-21ca-49f0-bf38-17e0395c7b15\") " pod="openshift-ingress/e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb" Jul 09 19:29:22.696853 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:22.696735 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35ca30e4-21ca-49f0-bf38-17e0395c7b15-workload-certs\") pod \"e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb\" (UID: \"35ca30e4-21ca-49f0-bf38-17e0395c7b15\") " pod="openshift-ingress/e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb" Jul 09 19:29:22.696853 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:22.696798 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35ca30e4-21ca-49f0-bf38-17e0395c7b15-credential-socket\") pod \"e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb\" (UID: \"35ca30e4-21ca-49f0-bf38-17e0395c7b15\") " pod="openshift-ingress/e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb" Jul 09 19:29:22.697292 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:22.697268 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35ca30e4-21ca-49f0-bf38-17e0395c7b15-istiod-ca-cert\") pod \"e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb\" (UID: \"35ca30e4-21ca-49f0-bf38-17e0395c7b15\") " pod="openshift-ingress/e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb" Jul 09 19:29:22.699004 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:22.698981 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35ca30e4-21ca-49f0-bf38-17e0395c7b15-istio-envoy\") pod \"e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb\" (UID: \"35ca30e4-21ca-49f0-bf38-17e0395c7b15\") " pod="openshift-ingress/e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb" Jul 09 19:29:22.699149 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:22.699127 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35ca30e4-21ca-49f0-bf38-17e0395c7b15-istio-podinfo\") pod \"e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb\" (UID: \"35ca30e4-21ca-49f0-bf38-17e0395c7b15\") " pod="openshift-ingress/e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb" Jul 09 19:29:22.706187 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:22.706162 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35ca30e4-21ca-49f0-bf38-17e0395c7b15-istio-token\") pod \"e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb\" (UID: \"35ca30e4-21ca-49f0-bf38-17e0395c7b15\") " pod="openshift-ingress/e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb" Jul 09 19:29:22.708330 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:22.708299 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjt8f\" (UniqueName: \"kubernetes.io/projected/35ca30e4-21ca-49f0-bf38-17e0395c7b15-kube-api-access-sjt8f\") pod \"e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb\" (UID: \"35ca30e4-21ca-49f0-bf38-17e0395c7b15\") " pod="openshift-ingress/e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb" Jul 09 19:29:22.795066 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:22.794992 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb" Jul 09 19:29:22.943719 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:22.943690 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb"] Jul 09 19:29:22.944439 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:29:22.944408 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35ca30e4_21ca_49f0_bf38_17e0395c7b15.slice/crio-7f0e64da870de36c59a54d89906d8f945c506cba6e0c5debd8f2343b33b424ef WatchSource:0}: Error finding container 7f0e64da870de36c59a54d89906d8f945c506cba6e0c5debd8f2343b33b424ef: Status 404 returned error can't find the container with id 7f0e64da870de36c59a54d89906d8f945c506cba6e0c5debd8f2343b33b424ef Jul 09 19:29:22.946667 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:22.946625 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:29:22.946746 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:22.946714 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:29:22.946784 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:22.946757 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:29:23.388969 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:23.388933 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb" event={"ID":"35ca30e4-21ca-49f0-bf38-17e0395c7b15","Type":"ContainerStarted","Data":"ae1580186aad9d08d5f6d434193b2cad368e365988108df776832bd7610a5578"} Jul 09 19:29:23.389151 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:23.388994 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb" event={"ID":"35ca30e4-21ca-49f0-bf38-17e0395c7b15","Type":"ContainerStarted","Data":"7f0e64da870de36c59a54d89906d8f945c506cba6e0c5debd8f2343b33b424ef"} Jul 09 19:29:23.411577 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:23.411524 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb" podStartSLOduration=1.411508459 podStartE2EDuration="1.411508459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:29:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:29:23.409974646 +0000 UTC m=+1832.917754421" watchObservedRunningTime="2026-07-09 19:29:23.411508459 +0000 UTC m=+1832.919288233" Jul 09 19:29:23.795697 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:23.795665 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb" Jul 09 19:29:24.801143 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:24.801115 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb" Jul 09 19:29:25.405339 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:25.405308 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb" Jul 09 19:29:25.406503 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:25.406475 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb" Jul 09 19:29:25.583897 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:25.583862 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:29:37.069575 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:37.069494 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:29:40.407242 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:40.407204 2566 patch_prober.go:28] interesting pod/e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 09 19:29:40.407716 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:40.407260 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb" podUID="35ca30e4-21ca-49f0-bf38-17e0395c7b15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 09 19:29:40.489845 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:40.489812 2566 generic.go:356] "Generic (PLEG): container finished" podID="35ca30e4-21ca-49f0-bf38-17e0395c7b15" containerID="ae1580186aad9d08d5f6d434193b2cad368e365988108df776832bd7610a5578" exitCode=0 Jul 09 19:29:40.490006 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:40.489853 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb" event={"ID":"35ca30e4-21ca-49f0-bf38-17e0395c7b15","Type":"ContainerDied","Data":"ae1580186aad9d08d5f6d434193b2cad368e365988108df776832bd7610a5578"} Jul 09 19:29:40.490276 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:40.490251 2566 scope.go:117] "RemoveContainer" containerID="ae1580186aad9d08d5f6d434193b2cad368e365988108df776832bd7610a5578" Jul 09 19:29:40.491232 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:40.491205 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:29:40.491290 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:40.491271 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:29:40.491326 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:40.491302 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:29:41.502525 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:41.502484 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb" event={"ID":"35ca30e4-21ca-49f0-bf38-17e0395c7b15","Type":"ContainerStarted","Data":"3d027baab6ca00f6c0bc3b1a40361dbcd677461e67bd8a1cef1e8aa1e0a8b383"} Jul 09 19:29:41.795823 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:41.795738 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb" Jul 09 19:29:41.796007 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:41.795953 2566 patch_prober.go:28] interesting pod/e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 09 19:29:41.796080 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:41.796010 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb" podUID="35ca30e4-21ca-49f0-bf38-17e0395c7b15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 09 19:29:42.796158 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:42.796126 2566 patch_prober.go:28] interesting pod/e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 09 19:29:42.796536 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:42.796178 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb" podUID="35ca30e4-21ca-49f0-bf38-17e0395c7b15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 09 19:29:43.597949 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:43.597905 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb"] Jul 09 19:29:43.598188 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:43.598160 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb" podUID="35ca30e4-21ca-49f0-bf38-17e0395c7b15" containerName="istio-proxy" containerID="cri-o://3d027baab6ca00f6c0bc3b1a40361dbcd677461e67bd8a1cef1e8aa1e0a8b383" gracePeriod=30 Jul 09 19:29:46.763209 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:46.763172 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:29:50.861352 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:50.861328 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb" Jul 09 19:29:50.961755 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:50.961733 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35ca30e4-21ca-49f0-bf38-17e0395c7b15-workload-certs\") pod \"35ca30e4-21ca-49f0-bf38-17e0395c7b15\" (UID: \"35ca30e4-21ca-49f0-bf38-17e0395c7b15\") " Jul 09 19:29:50.961915 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:50.961765 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35ca30e4-21ca-49f0-bf38-17e0395c7b15-istio-data\") pod \"35ca30e4-21ca-49f0-bf38-17e0395c7b15\" (UID: \"35ca30e4-21ca-49f0-bf38-17e0395c7b15\") " Jul 09 19:29:50.961915 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:50.961786 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35ca30e4-21ca-49f0-bf38-17e0395c7b15-istiod-ca-cert\") pod \"35ca30e4-21ca-49f0-bf38-17e0395c7b15\" (UID: \"35ca30e4-21ca-49f0-bf38-17e0395c7b15\") " Jul 09 19:29:50.961915 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:50.961813 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sjt8f\" (UniqueName: \"kubernetes.io/projected/35ca30e4-21ca-49f0-bf38-17e0395c7b15-kube-api-access-sjt8f\") pod \"35ca30e4-21ca-49f0-bf38-17e0395c7b15\" (UID: \"35ca30e4-21ca-49f0-bf38-17e0395c7b15\") " Jul 09 19:29:50.961915 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:50.961857 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35ca30e4-21ca-49f0-bf38-17e0395c7b15-istio-envoy\") pod \"35ca30e4-21ca-49f0-bf38-17e0395c7b15\" (UID: \"35ca30e4-21ca-49f0-bf38-17e0395c7b15\") " Jul 09 19:29:50.961915 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:50.961908 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35ca30e4-21ca-49f0-bf38-17e0395c7b15-istio-podinfo\") pod \"35ca30e4-21ca-49f0-bf38-17e0395c7b15\" (UID: \"35ca30e4-21ca-49f0-bf38-17e0395c7b15\") " Jul 09 19:29:50.962201 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:50.962041 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35ca30e4-21ca-49f0-bf38-17e0395c7b15-workload-socket\") pod \"35ca30e4-21ca-49f0-bf38-17e0395c7b15\" (UID: \"35ca30e4-21ca-49f0-bf38-17e0395c7b15\") " Jul 09 19:29:50.962201 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:50.962087 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35ca30e4-21ca-49f0-bf38-17e0395c7b15-istio-token\") pod \"35ca30e4-21ca-49f0-bf38-17e0395c7b15\" (UID: \"35ca30e4-21ca-49f0-bf38-17e0395c7b15\") " Jul 09 19:29:50.962201 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:50.962124 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35ca30e4-21ca-49f0-bf38-17e0395c7b15-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "35ca30e4-21ca-49f0-bf38-17e0395c7b15" (UID: "35ca30e4-21ca-49f0-bf38-17e0395c7b15"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:29:50.962201 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:50.962147 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35ca30e4-21ca-49f0-bf38-17e0395c7b15-credential-socket\") pod \"35ca30e4-21ca-49f0-bf38-17e0395c7b15\" (UID: \"35ca30e4-21ca-49f0-bf38-17e0395c7b15\") " Jul 09 19:29:50.962201 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:50.962181 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35ca30e4-21ca-49f0-bf38-17e0395c7b15-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "35ca30e4-21ca-49f0-bf38-17e0395c7b15" (UID: "35ca30e4-21ca-49f0-bf38-17e0395c7b15"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 19:29:50.962491 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:50.962329 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35ca30e4-21ca-49f0-bf38-17e0395c7b15-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "35ca30e4-21ca-49f0-bf38-17e0395c7b15" (UID: "35ca30e4-21ca-49f0-bf38-17e0395c7b15"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:29:50.962545 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:50.962498 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35ca30e4-21ca-49f0-bf38-17e0395c7b15-workload-socket\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:29:50.962545 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:50.962518 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35ca30e4-21ca-49f0-bf38-17e0395c7b15-workload-certs\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:29:50.962545 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:50.962519 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35ca30e4-21ca-49f0-bf38-17e0395c7b15-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "35ca30e4-21ca-49f0-bf38-17e0395c7b15" (UID: "35ca30e4-21ca-49f0-bf38-17e0395c7b15"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:29:50.962744 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:50.962531 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35ca30e4-21ca-49f0-bf38-17e0395c7b15-istiod-ca-cert\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:29:50.962804 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:50.962766 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35ca30e4-21ca-49f0-bf38-17e0395c7b15-istio-data" (OuterVolumeSpecName: "istio-data") pod "35ca30e4-21ca-49f0-bf38-17e0395c7b15" (UID: "35ca30e4-21ca-49f0-bf38-17e0395c7b15"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:29:50.964182 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:50.964153 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35ca30e4-21ca-49f0-bf38-17e0395c7b15-istio-token" (OuterVolumeSpecName: "istio-token") pod "35ca30e4-21ca-49f0-bf38-17e0395c7b15" (UID: "35ca30e4-21ca-49f0-bf38-17e0395c7b15"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:29:50.964293 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:50.964197 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35ca30e4-21ca-49f0-bf38-17e0395c7b15-kube-api-access-sjt8f" (OuterVolumeSpecName: "kube-api-access-sjt8f") pod "35ca30e4-21ca-49f0-bf38-17e0395c7b15" (UID: "35ca30e4-21ca-49f0-bf38-17e0395c7b15"). InnerVolumeSpecName "kube-api-access-sjt8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:29:50.964293 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:50.964218 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35ca30e4-21ca-49f0-bf38-17e0395c7b15-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "35ca30e4-21ca-49f0-bf38-17e0395c7b15" (UID: "35ca30e4-21ca-49f0-bf38-17e0395c7b15"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 19:29:50.964293 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:50.964217 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35ca30e4-21ca-49f0-bf38-17e0395c7b15-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "35ca30e4-21ca-49f0-bf38-17e0395c7b15" (UID: "35ca30e4-21ca-49f0-bf38-17e0395c7b15"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:29:51.063107 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:51.063082 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35ca30e4-21ca-49f0-bf38-17e0395c7b15-credential-socket\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:29:51.063107 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:51.063104 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35ca30e4-21ca-49f0-bf38-17e0395c7b15-istio-data\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:29:51.063107 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:51.063113 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjt8f\" (UniqueName: \"kubernetes.io/projected/35ca30e4-21ca-49f0-bf38-17e0395c7b15-kube-api-access-sjt8f\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:29:51.063337 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:51.063122 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35ca30e4-21ca-49f0-bf38-17e0395c7b15-istio-envoy\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:29:51.063337 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:51.063131 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35ca30e4-21ca-49f0-bf38-17e0395c7b15-istio-podinfo\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:29:51.063337 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:51.063141 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35ca30e4-21ca-49f0-bf38-17e0395c7b15-istio-token\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:29:51.569222 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:51.569185 2566 generic.go:356] "Generic (PLEG): container finished" podID="35ca30e4-21ca-49f0-bf38-17e0395c7b15" containerID="3d027baab6ca00f6c0bc3b1a40361dbcd677461e67bd8a1cef1e8aa1e0a8b383" exitCode=0 Jul 09 19:29:51.569397 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:51.569261 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb" Jul 09 19:29:51.569397 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:51.569274 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb" event={"ID":"35ca30e4-21ca-49f0-bf38-17e0395c7b15","Type":"ContainerDied","Data":"3d027baab6ca00f6c0bc3b1a40361dbcd677461e67bd8a1cef1e8aa1e0a8b383"} Jul 09 19:29:51.569397 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:51.569371 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb" event={"ID":"35ca30e4-21ca-49f0-bf38-17e0395c7b15","Type":"ContainerDied","Data":"7f0e64da870de36c59a54d89906d8f945c506cba6e0c5debd8f2343b33b424ef"} Jul 09 19:29:51.569508 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:51.569401 2566 scope.go:117] "RemoveContainer" containerID="3d027baab6ca00f6c0bc3b1a40361dbcd677461e67bd8a1cef1e8aa1e0a8b383" Jul 09 19:29:51.578726 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:51.578708 2566 scope.go:117] "RemoveContainer" containerID="ae1580186aad9d08d5f6d434193b2cad368e365988108df776832bd7610a5578" Jul 09 19:29:51.586454 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:51.586436 2566 scope.go:117] "RemoveContainer" containerID="3d027baab6ca00f6c0bc3b1a40361dbcd677461e67bd8a1cef1e8aa1e0a8b383" Jul 09 19:29:51.586719 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:29:51.586700 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d027baab6ca00f6c0bc3b1a40361dbcd677461e67bd8a1cef1e8aa1e0a8b383\": container with ID starting with 3d027baab6ca00f6c0bc3b1a40361dbcd677461e67bd8a1cef1e8aa1e0a8b383 not found: ID does not exist" containerID="3d027baab6ca00f6c0bc3b1a40361dbcd677461e67bd8a1cef1e8aa1e0a8b383" Jul 09 19:29:51.586788 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:51.586726 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d027baab6ca00f6c0bc3b1a40361dbcd677461e67bd8a1cef1e8aa1e0a8b383"} err="failed to get container status \"3d027baab6ca00f6c0bc3b1a40361dbcd677461e67bd8a1cef1e8aa1e0a8b383\": rpc error: code = NotFound desc = could not find container \"3d027baab6ca00f6c0bc3b1a40361dbcd677461e67bd8a1cef1e8aa1e0a8b383\": container with ID starting with 3d027baab6ca00f6c0bc3b1a40361dbcd677461e67bd8a1cef1e8aa1e0a8b383 not found: ID does not exist" Jul 09 19:29:51.586788 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:51.586743 2566 scope.go:117] "RemoveContainer" containerID="ae1580186aad9d08d5f6d434193b2cad368e365988108df776832bd7610a5578" Jul 09 19:29:51.586962 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:29:51.586947 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae1580186aad9d08d5f6d434193b2cad368e365988108df776832bd7610a5578\": container with ID starting with ae1580186aad9d08d5f6d434193b2cad368e365988108df776832bd7610a5578 not found: ID does not exist" containerID="ae1580186aad9d08d5f6d434193b2cad368e365988108df776832bd7610a5578" Jul 09 19:29:51.587005 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:51.586965 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae1580186aad9d08d5f6d434193b2cad368e365988108df776832bd7610a5578"} err="failed to get container status \"ae1580186aad9d08d5f6d434193b2cad368e365988108df776832bd7610a5578\": rpc error: code = NotFound desc = could not find container \"ae1580186aad9d08d5f6d434193b2cad368e365988108df776832bd7610a5578\": container with ID starting with ae1580186aad9d08d5f6d434193b2cad368e365988108df776832bd7610a5578 not found: ID does not exist" Jul 09 19:29:51.590935 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:51.590916 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb"] Jul 09 19:29:51.597373 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:51.597353 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5ec314ef-openshift-default-5bc7b8949f-pl6nb"] Jul 09 19:29:53.057574 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:29:53.057540 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35ca30e4-21ca-49f0-bf38-17e0395c7b15" path="/var/lib/kubelet/pods/35ca30e4-21ca-49f0-bf38-17e0395c7b15/volumes" Jul 09 19:30:00.135789 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:00.135758 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727090-w8b2d"] Jul 09 19:30:00.136295 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:00.136174 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35ca30e4-21ca-49f0-bf38-17e0395c7b15" containerName="istio-proxy" Jul 09 19:30:00.136295 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:00.136187 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="35ca30e4-21ca-49f0-bf38-17e0395c7b15" containerName="istio-proxy" Jul 09 19:30:00.136295 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:00.136198 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35ca30e4-21ca-49f0-bf38-17e0395c7b15" containerName="istio-proxy" Jul 09 19:30:00.136295 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:00.136204 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="35ca30e4-21ca-49f0-bf38-17e0395c7b15" containerName="istio-proxy" Jul 09 19:30:00.136295 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:00.136271 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="35ca30e4-21ca-49f0-bf38-17e0395c7b15" containerName="istio-proxy" Jul 09 19:30:00.136295 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:00.136280 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="35ca30e4-21ca-49f0-bf38-17e0395c7b15" containerName="istio-proxy" Jul 09 19:30:00.139662 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:00.139640 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727090-w8b2d" Jul 09 19:30:00.142728 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:00.142707 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 09 19:30:00.144136 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:00.144112 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 09 19:30:00.144439 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:00.144138 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9pdt2\"" Jul 09 19:30:00.147374 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:00.147356 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727090-w8b2d"] Jul 09 19:30:00.238139 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:00.238105 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fn68\" (UniqueName: \"kubernetes.io/projected/52d37539-16f6-4561-bce0-a6f43fc1fac3-kube-api-access-2fn68\") pod \"maas-api-key-cleanup-29727090-w8b2d\" (UID: \"52d37539-16f6-4561-bce0-a6f43fc1fac3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727090-w8b2d" Jul 09 19:30:00.338710 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:00.338669 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fn68\" (UniqueName: \"kubernetes.io/projected/52d37539-16f6-4561-bce0-a6f43fc1fac3-kube-api-access-2fn68\") pod \"maas-api-key-cleanup-29727090-w8b2d\" (UID: \"52d37539-16f6-4561-bce0-a6f43fc1fac3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727090-w8b2d" Jul 09 19:30:00.350712 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:00.350682 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fn68\" (UniqueName: \"kubernetes.io/projected/52d37539-16f6-4561-bce0-a6f43fc1fac3-kube-api-access-2fn68\") pod \"maas-api-key-cleanup-29727090-w8b2d\" (UID: \"52d37539-16f6-4561-bce0-a6f43fc1fac3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727090-w8b2d" Jul 09 19:30:00.451304 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:00.451278 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727090-w8b2d" Jul 09 19:30:00.591011 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:00.590892 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727090-w8b2d"] Jul 09 19:30:00.593455 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:30:00.593429 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52d37539_16f6_4561_bce0_a6f43fc1fac3.slice/crio-9ccaec6c80290440bb1568a8065487698a69c5aa88d202a19d0a75bc177f5579 WatchSource:0}: Error finding container 9ccaec6c80290440bb1568a8065487698a69c5aa88d202a19d0a75bc177f5579: Status 404 returned error can't find the container with id 9ccaec6c80290440bb1568a8065487698a69c5aa88d202a19d0a75bc177f5579 Jul 09 19:30:00.621898 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:00.621870 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727090-w8b2d" event={"ID":"52d37539-16f6-4561-bce0-a6f43fc1fac3","Type":"ContainerStarted","Data":"9ccaec6c80290440bb1568a8065487698a69c5aa88d202a19d0a75bc177f5579"} Jul 09 19:30:01.629222 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:01.629184 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727090-w8b2d" event={"ID":"52d37539-16f6-4561-bce0-a6f43fc1fac3","Type":"ContainerStarted","Data":"2d73d092e0fbbee5654a2fc19e1fd13db942d7018e8b6e4ca8a0e5140dff4606"} Jul 09 19:30:01.646419 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:01.646368 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727090-w8b2d" podStartSLOduration=1.646352216 podStartE2EDuration="1.646352216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:30:01.644856355 +0000 UTC m=+1871.152636129" watchObservedRunningTime="2026-07-09 19:30:01.646352216 +0000 UTC m=+1871.154131990" Jul 09 19:30:06.637574 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:06.637543 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd"] Jul 09 19:30:06.641683 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:06.641660 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd" Jul 09 19:30:06.645024 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:06.645007 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-8d9f6bca-openshift-default-dockercfg-qh9mb\"" Jul 09 19:30:06.657460 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:06.657438 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd"] Jul 09 19:30:06.694893 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:06.694868 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dc5475d-c365-49b9-b6c2-489a35002863-workload-socket\") pod \"e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd\" (UID: \"9dc5475d-c365-49b9-b6c2-489a35002863\") " pod="openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd" Jul 09 19:30:06.695043 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:06.694913 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dc5475d-c365-49b9-b6c2-489a35002863-workload-certs\") pod \"e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd\" (UID: \"9dc5475d-c365-49b9-b6c2-489a35002863\") " pod="openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd" Jul 09 19:30:06.695043 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:06.694979 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr9r4\" (UniqueName: \"kubernetes.io/projected/9dc5475d-c365-49b9-b6c2-489a35002863-kube-api-access-vr9r4\") pod \"e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd\" (UID: \"9dc5475d-c365-49b9-b6c2-489a35002863\") " pod="openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd" Jul 09 19:30:06.695043 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:06.695009 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dc5475d-c365-49b9-b6c2-489a35002863-istio-data\") pod \"e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd\" (UID: \"9dc5475d-c365-49b9-b6c2-489a35002863\") " pod="openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd" Jul 09 19:30:06.695043 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:06.695032 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dc5475d-c365-49b9-b6c2-489a35002863-istiod-ca-cert\") pod \"e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd\" (UID: \"9dc5475d-c365-49b9-b6c2-489a35002863\") " pod="openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd" Jul 09 19:30:06.695189 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:06.695079 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dc5475d-c365-49b9-b6c2-489a35002863-istio-token\") pod \"e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd\" (UID: \"9dc5475d-c365-49b9-b6c2-489a35002863\") " pod="openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd" Jul 09 19:30:06.695189 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:06.695105 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dc5475d-c365-49b9-b6c2-489a35002863-credential-socket\") pod \"e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd\" (UID: \"9dc5475d-c365-49b9-b6c2-489a35002863\") " pod="openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd" Jul 09 19:30:06.695189 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:06.695121 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dc5475d-c365-49b9-b6c2-489a35002863-istio-podinfo\") pod \"e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd\" (UID: \"9dc5475d-c365-49b9-b6c2-489a35002863\") " pod="openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd" Jul 09 19:30:06.695189 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:06.695152 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dc5475d-c365-49b9-b6c2-489a35002863-istio-envoy\") pod \"e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd\" (UID: \"9dc5475d-c365-49b9-b6c2-489a35002863\") " pod="openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd" Jul 09 19:30:06.796114 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:06.796078 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr9r4\" (UniqueName: \"kubernetes.io/projected/9dc5475d-c365-49b9-b6c2-489a35002863-kube-api-access-vr9r4\") pod \"e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd\" (UID: \"9dc5475d-c365-49b9-b6c2-489a35002863\") " pod="openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd" Jul 09 19:30:06.796114 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:06.796116 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dc5475d-c365-49b9-b6c2-489a35002863-istio-data\") pod \"e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd\" (UID: \"9dc5475d-c365-49b9-b6c2-489a35002863\") " pod="openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd" Jul 09 19:30:06.796382 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:06.796139 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dc5475d-c365-49b9-b6c2-489a35002863-istiod-ca-cert\") pod \"e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd\" (UID: \"9dc5475d-c365-49b9-b6c2-489a35002863\") " pod="openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd" Jul 09 19:30:06.796382 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:06.796202 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dc5475d-c365-49b9-b6c2-489a35002863-istio-token\") pod \"e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd\" (UID: \"9dc5475d-c365-49b9-b6c2-489a35002863\") " pod="openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd" Jul 09 19:30:06.796382 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:06.796233 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dc5475d-c365-49b9-b6c2-489a35002863-credential-socket\") pod \"e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd\" (UID: \"9dc5475d-c365-49b9-b6c2-489a35002863\") " pod="openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd" Jul 09 19:30:06.796382 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:06.796254 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dc5475d-c365-49b9-b6c2-489a35002863-istio-podinfo\") pod \"e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd\" (UID: \"9dc5475d-c365-49b9-b6c2-489a35002863\") " pod="openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd" Jul 09 19:30:06.796382 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:06.796291 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dc5475d-c365-49b9-b6c2-489a35002863-istio-envoy\") pod \"e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd\" (UID: \"9dc5475d-c365-49b9-b6c2-489a35002863\") " pod="openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd" Jul 09 19:30:06.796382 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:06.796329 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dc5475d-c365-49b9-b6c2-489a35002863-workload-socket\") pod \"e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd\" (UID: \"9dc5475d-c365-49b9-b6c2-489a35002863\") " pod="openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd" Jul 09 19:30:06.796382 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:06.796380 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dc5475d-c365-49b9-b6c2-489a35002863-workload-certs\") pod \"e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd\" (UID: \"9dc5475d-c365-49b9-b6c2-489a35002863\") " pod="openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd" Jul 09 19:30:06.796788 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:06.796640 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dc5475d-c365-49b9-b6c2-489a35002863-credential-socket\") pod \"e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd\" (UID: \"9dc5475d-c365-49b9-b6c2-489a35002863\") " pod="openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd" Jul 09 19:30:06.796788 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:06.796733 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dc5475d-c365-49b9-b6c2-489a35002863-workload-certs\") pod \"e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd\" (UID: \"9dc5475d-c365-49b9-b6c2-489a35002863\") " pod="openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd" Jul 09 19:30:06.796788 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:06.796766 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dc5475d-c365-49b9-b6c2-489a35002863-workload-socket\") pod \"e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd\" (UID: \"9dc5475d-c365-49b9-b6c2-489a35002863\") " pod="openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd" Jul 09 19:30:06.797006 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:06.796906 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dc5475d-c365-49b9-b6c2-489a35002863-istio-data\") pod \"e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd\" (UID: \"9dc5475d-c365-49b9-b6c2-489a35002863\") " pod="openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd" Jul 09 19:30:06.797006 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:06.796915 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dc5475d-c365-49b9-b6c2-489a35002863-istiod-ca-cert\") pod \"e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd\" (UID: \"9dc5475d-c365-49b9-b6c2-489a35002863\") " pod="openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd" Jul 09 19:30:06.798721 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:06.798702 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dc5475d-c365-49b9-b6c2-489a35002863-istio-envoy\") pod \"e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd\" (UID: \"9dc5475d-c365-49b9-b6c2-489a35002863\") " pod="openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd" Jul 09 19:30:06.798829 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:06.798801 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dc5475d-c365-49b9-b6c2-489a35002863-istio-podinfo\") pod \"e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd\" (UID: \"9dc5475d-c365-49b9-b6c2-489a35002863\") " pod="openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd" Jul 09 19:30:06.806808 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:06.806755 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dc5475d-c365-49b9-b6c2-489a35002863-istio-token\") pod \"e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd\" (UID: \"9dc5475d-c365-49b9-b6c2-489a35002863\") " pod="openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd" Jul 09 19:30:06.807688 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:06.807663 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr9r4\" (UniqueName: \"kubernetes.io/projected/9dc5475d-c365-49b9-b6c2-489a35002863-kube-api-access-vr9r4\") pod \"e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd\" (UID: \"9dc5475d-c365-49b9-b6c2-489a35002863\") " pod="openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd" Jul 09 19:30:06.951296 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:06.951265 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd" Jul 09 19:30:07.095312 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:07.095284 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd"] Jul 09 19:30:07.096926 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:30:07.096885 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dc5475d_c365_49b9_b6c2_489a35002863.slice/crio-b5d5b794681ca34db360f4244cf99452938a2dc22117bc3e736adb9480e91ed0 WatchSource:0}: Error finding container b5d5b794681ca34db360f4244cf99452938a2dc22117bc3e736adb9480e91ed0: Status 404 returned error can't find the container with id b5d5b794681ca34db360f4244cf99452938a2dc22117bc3e736adb9480e91ed0 Jul 09 19:30:07.098940 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:07.098911 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:30:07.099031 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:07.098971 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:30:07.099031 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:07.098998 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:30:07.670592 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:07.670557 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd" event={"ID":"9dc5475d-c365-49b9-b6c2-489a35002863","Type":"ContainerStarted","Data":"c7ab34fb1b38c6722ac663870819369815b3d9a9c4f020c6c4f58b7adf277beb"} Jul 09 19:30:07.671001 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:07.670626 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd" event={"ID":"9dc5475d-c365-49b9-b6c2-489a35002863","Type":"ContainerStarted","Data":"b5d5b794681ca34db360f4244cf99452938a2dc22117bc3e736adb9480e91ed0"} Jul 09 19:30:07.695468 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:07.695420 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd" podStartSLOduration=1.695404833 podStartE2EDuration="1.695404833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:30:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:30:07.692970822 +0000 UTC m=+1877.200750606" watchObservedRunningTime="2026-07-09 19:30:07.695404833 +0000 UTC m=+1877.203184606" Jul 09 19:30:07.951866 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:07.951832 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd" Jul 09 19:30:08.957206 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:08.957175 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd" Jul 09 19:30:09.683948 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:09.683915 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd" Jul 09 19:30:09.685023 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:09.684997 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd" Jul 09 19:30:17.368052 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:17.368018 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:30:25.130687 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:25.130649 2566 patch_prober.go:28] interesting pod/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": read tcp 10.134.0.2:46196->10.134.0.59:15021: read: connection reset by peer" start-of-body= Jul 09 19:30:25.131108 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:25.130724 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd" podUID="9dc5475d-c365-49b9-b6c2-489a35002863" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": read tcp 10.134.0.2:46196->10.134.0.59:15021: read: connection reset by peer" Jul 09 19:30:25.772801 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:25.772769 2566 generic.go:356] "Generic (PLEG): container finished" podID="9dc5475d-c365-49b9-b6c2-489a35002863" containerID="c7ab34fb1b38c6722ac663870819369815b3d9a9c4f020c6c4f58b7adf277beb" exitCode=0 Jul 09 19:30:25.772981 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:25.772853 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd" event={"ID":"9dc5475d-c365-49b9-b6c2-489a35002863","Type":"ContainerDied","Data":"c7ab34fb1b38c6722ac663870819369815b3d9a9c4f020c6c4f58b7adf277beb"} Jul 09 19:30:25.773356 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:25.773337 2566 scope.go:117] "RemoveContainer" containerID="c7ab34fb1b38c6722ac663870819369815b3d9a9c4f020c6c4f58b7adf277beb" Jul 09 19:30:25.774384 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:25.774359 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:30:25.774440 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:25.774426 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:30:25.774479 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:25.774460 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:30:26.779789 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:26.779755 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd" event={"ID":"9dc5475d-c365-49b9-b6c2-489a35002863","Type":"ContainerStarted","Data":"c639f3745fb2b2873412b3486faea1dd094d85351414d34dc7fddf8e71b2f4be"} Jul 09 19:30:26.952161 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:26.952124 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd" Jul 09 19:30:26.952353 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:26.952310 2566 patch_prober.go:28] interesting pod/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 09 19:30:26.952434 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:26.952367 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd" podUID="9dc5475d-c365-49b9-b6c2-489a35002863" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 09 19:30:27.951762 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:27.951725 2566 patch_prober.go:28] interesting pod/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 09 19:30:27.952227 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:27.951781 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd" podUID="9dc5475d-c365-49b9-b6c2-489a35002863" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 09 19:30:28.270559 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:28.270474 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:30:28.952071 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:28.952030 2566 patch_prober.go:28] interesting pod/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 09 19:30:28.952481 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:28.952103 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd" podUID="9dc5475d-c365-49b9-b6c2-489a35002863" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 09 19:30:29.951809 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:29.951771 2566 patch_prober.go:28] interesting pod/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 09 19:30:29.951994 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:29.951825 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd" podUID="9dc5475d-c365-49b9-b6c2-489a35002863" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 09 19:30:30.955322 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:30.955293 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd" Jul 09 19:30:30.955792 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:30.955575 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd" Jul 09 19:30:30.956405 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:30.956387 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd" Jul 09 19:30:44.179999 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:44.179960 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:30:57.774537 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:30:57.774497 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:31:00.010502 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:00.010460 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727090-w8b2d"] Jul 09 19:31:00.011030 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:00.010722 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727090-w8b2d" podUID="52d37539-16f6-4561-bce0-a6f43fc1fac3" containerName="cleanup" containerID="cri-o://2d73d092e0fbbee5654a2fc19e1fd13db942d7018e8b6e4ca8a0e5140dff4606" gracePeriod=30 Jul 09 19:31:30.139615 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:30.139569 2566 generic.go:356] "Generic (PLEG): container finished" podID="52d37539-16f6-4561-bce0-a6f43fc1fac3" containerID="2d73d092e0fbbee5654a2fc19e1fd13db942d7018e8b6e4ca8a0e5140dff4606" exitCode=137 Jul 09 19:31:30.139996 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:30.139635 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727090-w8b2d" event={"ID":"52d37539-16f6-4561-bce0-a6f43fc1fac3","Type":"ContainerDied","Data":"2d73d092e0fbbee5654a2fc19e1fd13db942d7018e8b6e4ca8a0e5140dff4606"} Jul 09 19:31:30.206939 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:30.206916 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727090-w8b2d" Jul 09 19:31:30.346331 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:30.346236 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2fn68\" (UniqueName: \"kubernetes.io/projected/52d37539-16f6-4561-bce0-a6f43fc1fac3-kube-api-access-2fn68\") pod \"52d37539-16f6-4561-bce0-a6f43fc1fac3\" (UID: \"52d37539-16f6-4561-bce0-a6f43fc1fac3\") " Jul 09 19:31:30.348267 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:30.348236 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52d37539-16f6-4561-bce0-a6f43fc1fac3-kube-api-access-2fn68" (OuterVolumeSpecName: "kube-api-access-2fn68") pod "52d37539-16f6-4561-bce0-a6f43fc1fac3" (UID: "52d37539-16f6-4561-bce0-a6f43fc1fac3"). InnerVolumeSpecName "kube-api-access-2fn68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:31:30.447448 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:30.447415 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fn68\" (UniqueName: \"kubernetes.io/projected/52d37539-16f6-4561-bce0-a6f43fc1fac3-kube-api-access-2fn68\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:31:31.146979 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:31.146951 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727090-w8b2d" Jul 09 19:31:31.147403 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:31.146950 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727090-w8b2d" event={"ID":"52d37539-16f6-4561-bce0-a6f43fc1fac3","Type":"ContainerDied","Data":"9ccaec6c80290440bb1568a8065487698a69c5aa88d202a19d0a75bc177f5579"} Jul 09 19:31:31.147403 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:31.147079 2566 scope.go:117] "RemoveContainer" containerID="2d73d092e0fbbee5654a2fc19e1fd13db942d7018e8b6e4ca8a0e5140dff4606" Jul 09 19:31:31.166123 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:31.166082 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727090-w8b2d"] Jul 09 19:31:31.171911 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:31.171886 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727090-w8b2d"] Jul 09 19:31:33.057325 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:33.057285 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52d37539-16f6-4561-bce0-a6f43fc1fac3" path="/var/lib/kubelet/pods/52d37539-16f6-4561-bce0-a6f43fc1fac3/volumes" Jul 09 19:31:37.364174 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:37.364144 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:31:40.880164 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:40.880115 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd"] Jul 09 19:31:40.880963 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:40.880374 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd" podUID="9dc5475d-c365-49b9-b6c2-489a35002863" containerName="istio-proxy" containerID="cri-o://c639f3745fb2b2873412b3486faea1dd094d85351414d34dc7fddf8e71b2f4be" gracePeriod=30 Jul 09 19:31:45.956925 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:45.956894 2566 patch_prober.go:28] interesting pod/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 09 19:31:45.957277 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:45.956961 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd" podUID="9dc5475d-c365-49b9-b6c2-489a35002863" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 09 19:31:46.136301 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:46.136278 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd" Jul 09 19:31:46.189289 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:46.189265 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dc5475d-c365-49b9-b6c2-489a35002863-istio-token\") pod \"9dc5475d-c365-49b9-b6c2-489a35002863\" (UID: \"9dc5475d-c365-49b9-b6c2-489a35002863\") " Jul 09 19:31:46.189464 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:46.189295 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dc5475d-c365-49b9-b6c2-489a35002863-istio-podinfo\") pod \"9dc5475d-c365-49b9-b6c2-489a35002863\" (UID: \"9dc5475d-c365-49b9-b6c2-489a35002863\") " Jul 09 19:31:46.189464 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:46.189331 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dc5475d-c365-49b9-b6c2-489a35002863-credential-socket\") pod \"9dc5475d-c365-49b9-b6c2-489a35002863\" (UID: \"9dc5475d-c365-49b9-b6c2-489a35002863\") " Jul 09 19:31:46.189464 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:46.189352 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dc5475d-c365-49b9-b6c2-489a35002863-istio-envoy\") pod \"9dc5475d-c365-49b9-b6c2-489a35002863\" (UID: \"9dc5475d-c365-49b9-b6c2-489a35002863\") " Jul 09 19:31:46.189464 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:46.189383 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dc5475d-c365-49b9-b6c2-489a35002863-istio-data\") pod \"9dc5475d-c365-49b9-b6c2-489a35002863\" (UID: \"9dc5475d-c365-49b9-b6c2-489a35002863\") " Jul 09 19:31:46.189464 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:46.189416 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dc5475d-c365-49b9-b6c2-489a35002863-istiod-ca-cert\") pod \"9dc5475d-c365-49b9-b6c2-489a35002863\" (UID: \"9dc5475d-c365-49b9-b6c2-489a35002863\") " Jul 09 19:31:46.189764 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:46.189471 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dc5475d-c365-49b9-b6c2-489a35002863-workload-certs\") pod \"9dc5475d-c365-49b9-b6c2-489a35002863\" (UID: \"9dc5475d-c365-49b9-b6c2-489a35002863\") " Jul 09 19:31:46.189764 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:46.189509 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vr9r4\" (UniqueName: \"kubernetes.io/projected/9dc5475d-c365-49b9-b6c2-489a35002863-kube-api-access-vr9r4\") pod \"9dc5475d-c365-49b9-b6c2-489a35002863\" (UID: \"9dc5475d-c365-49b9-b6c2-489a35002863\") " Jul 09 19:31:46.189764 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:46.189583 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dc5475d-c365-49b9-b6c2-489a35002863-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9dc5475d-c365-49b9-b6c2-489a35002863" (UID: "9dc5475d-c365-49b9-b6c2-489a35002863"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:31:46.189764 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:46.189651 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dc5475d-c365-49b9-b6c2-489a35002863-workload-socket\") pod \"9dc5475d-c365-49b9-b6c2-489a35002863\" (UID: \"9dc5475d-c365-49b9-b6c2-489a35002863\") " Jul 09 19:31:46.189764 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:46.189744 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dc5475d-c365-49b9-b6c2-489a35002863-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9dc5475d-c365-49b9-b6c2-489a35002863" (UID: "9dc5475d-c365-49b9-b6c2-489a35002863"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:31:46.190095 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:46.189864 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dc5475d-c365-49b9-b6c2-489a35002863-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9dc5475d-c365-49b9-b6c2-489a35002863" (UID: "9dc5475d-c365-49b9-b6c2-489a35002863"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 19:31:46.190095 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:46.189992 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dc5475d-c365-49b9-b6c2-489a35002863-credential-socket\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:31:46.190095 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:46.190011 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dc5475d-c365-49b9-b6c2-489a35002863-istiod-ca-cert\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:31:46.190095 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:46.190021 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dc5475d-c365-49b9-b6c2-489a35002863-workload-certs\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:31:46.190095 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:46.190049 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dc5475d-c365-49b9-b6c2-489a35002863-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9dc5475d-c365-49b9-b6c2-489a35002863" (UID: "9dc5475d-c365-49b9-b6c2-489a35002863"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:31:46.190852 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:46.190820 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dc5475d-c365-49b9-b6c2-489a35002863-istio-data" (OuterVolumeSpecName: "istio-data") pod "9dc5475d-c365-49b9-b6c2-489a35002863" (UID: "9dc5475d-c365-49b9-b6c2-489a35002863"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:31:46.191701 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:46.191670 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dc5475d-c365-49b9-b6c2-489a35002863-istio-token" (OuterVolumeSpecName: "istio-token") pod "9dc5475d-c365-49b9-b6c2-489a35002863" (UID: "9dc5475d-c365-49b9-b6c2-489a35002863"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:31:46.191788 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:46.191761 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dc5475d-c365-49b9-b6c2-489a35002863-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9dc5475d-c365-49b9-b6c2-489a35002863" (UID: "9dc5475d-c365-49b9-b6c2-489a35002863"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:31:46.191987 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:46.191954 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9dc5475d-c365-49b9-b6c2-489a35002863-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9dc5475d-c365-49b9-b6c2-489a35002863" (UID: "9dc5475d-c365-49b9-b6c2-489a35002863"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 19:31:46.192099 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:46.192083 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dc5475d-c365-49b9-b6c2-489a35002863-kube-api-access-vr9r4" (OuterVolumeSpecName: "kube-api-access-vr9r4") pod "9dc5475d-c365-49b9-b6c2-489a35002863" (UID: "9dc5475d-c365-49b9-b6c2-489a35002863"). InnerVolumeSpecName "kube-api-access-vr9r4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:31:46.239541 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:46.239508 2566 generic.go:356] "Generic (PLEG): container finished" podID="9dc5475d-c365-49b9-b6c2-489a35002863" containerID="c639f3745fb2b2873412b3486faea1dd094d85351414d34dc7fddf8e71b2f4be" exitCode=0 Jul 09 19:31:46.239716 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:46.239561 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd" Jul 09 19:31:46.239716 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:46.239591 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd" event={"ID":"9dc5475d-c365-49b9-b6c2-489a35002863","Type":"ContainerDied","Data":"c639f3745fb2b2873412b3486faea1dd094d85351414d34dc7fddf8e71b2f4be"} Jul 09 19:31:46.239716 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:46.239701 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd" event={"ID":"9dc5475d-c365-49b9-b6c2-489a35002863","Type":"ContainerDied","Data":"b5d5b794681ca34db360f4244cf99452938a2dc22117bc3e736adb9480e91ed0"} Jul 09 19:31:46.239827 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:46.239730 2566 scope.go:117] "RemoveContainer" containerID="c639f3745fb2b2873412b3486faea1dd094d85351414d34dc7fddf8e71b2f4be" Jul 09 19:31:46.249532 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:46.249508 2566 scope.go:117] "RemoveContainer" containerID="c7ab34fb1b38c6722ac663870819369815b3d9a9c4f020c6c4f58b7adf277beb" Jul 09 19:31:46.257769 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:46.257749 2566 scope.go:117] "RemoveContainer" containerID="c639f3745fb2b2873412b3486faea1dd094d85351414d34dc7fddf8e71b2f4be" Jul 09 19:31:46.257988 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:31:46.257971 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c639f3745fb2b2873412b3486faea1dd094d85351414d34dc7fddf8e71b2f4be\": container with ID starting with c639f3745fb2b2873412b3486faea1dd094d85351414d34dc7fddf8e71b2f4be not found: ID does not exist" containerID="c639f3745fb2b2873412b3486faea1dd094d85351414d34dc7fddf8e71b2f4be" Jul 09 19:31:46.258027 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:46.257999 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c639f3745fb2b2873412b3486faea1dd094d85351414d34dc7fddf8e71b2f4be"} err="failed to get container status \"c639f3745fb2b2873412b3486faea1dd094d85351414d34dc7fddf8e71b2f4be\": rpc error: code = NotFound desc = could not find container \"c639f3745fb2b2873412b3486faea1dd094d85351414d34dc7fddf8e71b2f4be\": container with ID starting with c639f3745fb2b2873412b3486faea1dd094d85351414d34dc7fddf8e71b2f4be not found: ID does not exist" Jul 09 19:31:46.258027 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:46.258016 2566 scope.go:117] "RemoveContainer" containerID="c7ab34fb1b38c6722ac663870819369815b3d9a9c4f020c6c4f58b7adf277beb" Jul 09 19:31:46.258269 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:31:46.258254 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7ab34fb1b38c6722ac663870819369815b3d9a9c4f020c6c4f58b7adf277beb\": container with ID starting with c7ab34fb1b38c6722ac663870819369815b3d9a9c4f020c6c4f58b7adf277beb not found: ID does not exist" containerID="c7ab34fb1b38c6722ac663870819369815b3d9a9c4f020c6c4f58b7adf277beb" Jul 09 19:31:46.258310 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:46.258273 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7ab34fb1b38c6722ac663870819369815b3d9a9c4f020c6c4f58b7adf277beb"} err="failed to get container status \"c7ab34fb1b38c6722ac663870819369815b3d9a9c4f020c6c4f58b7adf277beb\": rpc error: code = NotFound desc = could not find container \"c7ab34fb1b38c6722ac663870819369815b3d9a9c4f020c6c4f58b7adf277beb\": container with ID starting with c7ab34fb1b38c6722ac663870819369815b3d9a9c4f020c6c4f58b7adf277beb not found: ID does not exist" Jul 09 19:31:46.262674 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:46.262652 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd"] Jul 09 19:31:46.267695 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:46.267669 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-8d9f6bca-openshift-default-6f76b64d47-vqlqd"] Jul 09 19:31:46.290760 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:46.290712 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr9r4\" (UniqueName: \"kubernetes.io/projected/9dc5475d-c365-49b9-b6c2-489a35002863-kube-api-access-vr9r4\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:31:46.290760 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:46.290755 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dc5475d-c365-49b9-b6c2-489a35002863-workload-socket\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:31:46.290760 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:46.290769 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dc5475d-c365-49b9-b6c2-489a35002863-istio-token\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:31:46.290987 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:46.290781 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dc5475d-c365-49b9-b6c2-489a35002863-istio-podinfo\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:31:46.290987 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:46.290792 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dc5475d-c365-49b9-b6c2-489a35002863-istio-envoy\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:31:46.290987 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:46.290802 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dc5475d-c365-49b9-b6c2-489a35002863-istio-data\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:31:47.058530 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:47.058500 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9dc5475d-c365-49b9-b6c2-489a35002863" path="/var/lib/kubelet/pods/9dc5475d-c365-49b9-b6c2-489a35002863/volumes" Jul 09 19:31:47.272323 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:47.272286 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:31:58.562371 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:31:58.562338 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:32:36.163968 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:36.163937 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb"] Jul 09 19:32:36.164391 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:36.164302 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dc5475d-c365-49b9-b6c2-489a35002863" containerName="istio-proxy" Jul 09 19:32:36.164391 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:36.164312 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dc5475d-c365-49b9-b6c2-489a35002863" containerName="istio-proxy" Jul 09 19:32:36.164391 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:36.164327 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52d37539-16f6-4561-bce0-a6f43fc1fac3" containerName="cleanup" Jul 09 19:32:36.164391 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:36.164333 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="52d37539-16f6-4561-bce0-a6f43fc1fac3" containerName="cleanup" Jul 09 19:32:36.164391 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:36.164340 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dc5475d-c365-49b9-b6c2-489a35002863" containerName="istio-proxy" Jul 09 19:32:36.164391 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:36.164345 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dc5475d-c365-49b9-b6c2-489a35002863" containerName="istio-proxy" Jul 09 19:32:36.164583 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:36.164420 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9dc5475d-c365-49b9-b6c2-489a35002863" containerName="istio-proxy" Jul 09 19:32:36.164583 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:36.164428 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="52d37539-16f6-4561-bce0-a6f43fc1fac3" containerName="cleanup" Jul 09 19:32:36.164583 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:36.164435 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9dc5475d-c365-49b9-b6c2-489a35002863" containerName="istio-proxy" Jul 09 19:32:36.167562 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:36.167546 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb" Jul 09 19:32:36.170589 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:36.170569 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a7212d-openshift-default-dockercfg-b8q6v\"" Jul 09 19:32:36.181222 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:36.181193 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb"] Jul 09 19:32:36.326275 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:36.326238 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c6c42c-2013-4284-bf52-fec8f6d7d5a5-workload-socket\") pod \"e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb\" (UID: \"11c6c42c-2013-4284-bf52-fec8f6d7d5a5\") " pod="openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb" Jul 09 19:32:36.326442 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:36.326308 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11c6c42c-2013-4284-bf52-fec8f6d7d5a5-istio-envoy\") pod \"e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb\" (UID: \"11c6c42c-2013-4284-bf52-fec8f6d7d5a5\") " pod="openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb" Jul 09 19:32:36.326442 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:36.326365 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11c6c42c-2013-4284-bf52-fec8f6d7d5a5-istio-token\") pod \"e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb\" (UID: \"11c6c42c-2013-4284-bf52-fec8f6d7d5a5\") " pod="openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb" Jul 09 19:32:36.326442 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:36.326393 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vx9g\" (UniqueName: \"kubernetes.io/projected/11c6c42c-2013-4284-bf52-fec8f6d7d5a5-kube-api-access-4vx9g\") pod \"e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb\" (UID: \"11c6c42c-2013-4284-bf52-fec8f6d7d5a5\") " pod="openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb" Jul 09 19:32:36.326442 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:36.326428 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11c6c42c-2013-4284-bf52-fec8f6d7d5a5-workload-certs\") pod \"e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb\" (UID: \"11c6c42c-2013-4284-bf52-fec8f6d7d5a5\") " pod="openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb" Jul 09 19:32:36.326588 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:36.326453 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11c6c42c-2013-4284-bf52-fec8f6d7d5a5-istio-data\") pod \"e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb\" (UID: \"11c6c42c-2013-4284-bf52-fec8f6d7d5a5\") " pod="openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb" Jul 09 19:32:36.326588 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:36.326468 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11c6c42c-2013-4284-bf52-fec8f6d7d5a5-istio-podinfo\") pod \"e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb\" (UID: \"11c6c42c-2013-4284-bf52-fec8f6d7d5a5\") " pod="openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb" Jul 09 19:32:36.326588 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:36.326485 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11c6c42c-2013-4284-bf52-fec8f6d7d5a5-istiod-ca-cert\") pod \"e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb\" (UID: \"11c6c42c-2013-4284-bf52-fec8f6d7d5a5\") " pod="openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb" Jul 09 19:32:36.326588 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:36.326558 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c6c42c-2013-4284-bf52-fec8f6d7d5a5-credential-socket\") pod \"e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb\" (UID: \"11c6c42c-2013-4284-bf52-fec8f6d7d5a5\") " pod="openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb" Jul 09 19:32:36.427377 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:36.427296 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11c6c42c-2013-4284-bf52-fec8f6d7d5a5-istio-envoy\") pod \"e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb\" (UID: \"11c6c42c-2013-4284-bf52-fec8f6d7d5a5\") " pod="openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb" Jul 09 19:32:36.427377 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:36.427337 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11c6c42c-2013-4284-bf52-fec8f6d7d5a5-istio-token\") pod \"e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb\" (UID: \"11c6c42c-2013-4284-bf52-fec8f6d7d5a5\") " pod="openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb" Jul 09 19:32:36.427377 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:36.427357 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vx9g\" (UniqueName: \"kubernetes.io/projected/11c6c42c-2013-4284-bf52-fec8f6d7d5a5-kube-api-access-4vx9g\") pod \"e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb\" (UID: \"11c6c42c-2013-4284-bf52-fec8f6d7d5a5\") " pod="openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb" Jul 09 19:32:36.427377 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:36.427378 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11c6c42c-2013-4284-bf52-fec8f6d7d5a5-workload-certs\") pod \"e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb\" (UID: \"11c6c42c-2013-4284-bf52-fec8f6d7d5a5\") " pod="openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb" Jul 09 19:32:36.427739 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:36.427402 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11c6c42c-2013-4284-bf52-fec8f6d7d5a5-istio-data\") pod \"e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb\" (UID: \"11c6c42c-2013-4284-bf52-fec8f6d7d5a5\") " pod="openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb" Jul 09 19:32:36.427739 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:36.427493 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11c6c42c-2013-4284-bf52-fec8f6d7d5a5-istio-podinfo\") pod \"e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb\" (UID: \"11c6c42c-2013-4284-bf52-fec8f6d7d5a5\") " pod="openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb" Jul 09 19:32:36.427739 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:36.427535 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11c6c42c-2013-4284-bf52-fec8f6d7d5a5-istiod-ca-cert\") pod \"e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb\" (UID: \"11c6c42c-2013-4284-bf52-fec8f6d7d5a5\") " pod="openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb" Jul 09 19:32:36.427739 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:36.427585 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c6c42c-2013-4284-bf52-fec8f6d7d5a5-credential-socket\") pod \"e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb\" (UID: \"11c6c42c-2013-4284-bf52-fec8f6d7d5a5\") " pod="openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb" Jul 09 19:32:36.427739 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:36.427655 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c6c42c-2013-4284-bf52-fec8f6d7d5a5-workload-socket\") pod \"e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb\" (UID: \"11c6c42c-2013-4284-bf52-fec8f6d7d5a5\") " pod="openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb" Jul 09 19:32:36.427998 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:36.427789 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11c6c42c-2013-4284-bf52-fec8f6d7d5a5-workload-certs\") pod \"e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb\" (UID: \"11c6c42c-2013-4284-bf52-fec8f6d7d5a5\") " pod="openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb" Jul 09 19:32:36.427998 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:36.427918 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11c6c42c-2013-4284-bf52-fec8f6d7d5a5-istio-data\") pod \"e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb\" (UID: \"11c6c42c-2013-4284-bf52-fec8f6d7d5a5\") " pod="openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb" Jul 09 19:32:36.427998 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:36.427991 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c6c42c-2013-4284-bf52-fec8f6d7d5a5-workload-socket\") pod \"e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb\" (UID: \"11c6c42c-2013-4284-bf52-fec8f6d7d5a5\") " pod="openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb" Jul 09 19:32:36.428154 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:36.428065 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c6c42c-2013-4284-bf52-fec8f6d7d5a5-credential-socket\") pod \"e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb\" (UID: \"11c6c42c-2013-4284-bf52-fec8f6d7d5a5\") " pod="openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb" Jul 09 19:32:36.428339 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:36.428321 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11c6c42c-2013-4284-bf52-fec8f6d7d5a5-istiod-ca-cert\") pod \"e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb\" (UID: \"11c6c42c-2013-4284-bf52-fec8f6d7d5a5\") " pod="openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb" Jul 09 19:32:36.429680 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:36.429657 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11c6c42c-2013-4284-bf52-fec8f6d7d5a5-istio-envoy\") pod \"e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb\" (UID: \"11c6c42c-2013-4284-bf52-fec8f6d7d5a5\") " pod="openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb" Jul 09 19:32:36.429938 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:36.429921 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11c6c42c-2013-4284-bf52-fec8f6d7d5a5-istio-podinfo\") pod \"e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb\" (UID: \"11c6c42c-2013-4284-bf52-fec8f6d7d5a5\") " pod="openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb" Jul 09 19:32:36.436049 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:36.436025 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11c6c42c-2013-4284-bf52-fec8f6d7d5a5-istio-token\") pod \"e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb\" (UID: \"11c6c42c-2013-4284-bf52-fec8f6d7d5a5\") " pod="openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb" Jul 09 19:32:36.436252 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:36.436235 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vx9g\" (UniqueName: \"kubernetes.io/projected/11c6c42c-2013-4284-bf52-fec8f6d7d5a5-kube-api-access-4vx9g\") pod \"e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb\" (UID: \"11c6c42c-2013-4284-bf52-fec8f6d7d5a5\") " pod="openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb" Jul 09 19:32:36.478431 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:36.478100 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb" Jul 09 19:32:36.622401 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:36.622376 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb"] Jul 09 19:32:36.624999 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:32:36.624976 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c6c42c_2013_4284_bf52_fec8f6d7d5a5.slice/crio-a91595f14c5a8764112a84769bb95327736f660f88cd63a2c55ae73e1b31ee39 WatchSource:0}: Error finding container a91595f14c5a8764112a84769bb95327736f660f88cd63a2c55ae73e1b31ee39: Status 404 returned error can't find the container with id a91595f14c5a8764112a84769bb95327736f660f88cd63a2c55ae73e1b31ee39 Jul 09 19:32:36.627383 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:36.627312 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:32:36.627505 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:36.627477 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:32:36.627658 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:36.627634 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:32:37.533436 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:37.533399 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb" event={"ID":"11c6c42c-2013-4284-bf52-fec8f6d7d5a5","Type":"ContainerStarted","Data":"8ebc5236971316cecd387c96c62346d35215137cb5a610d546d95e817aeb71c5"} Jul 09 19:32:37.534628 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:37.534612 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb" event={"ID":"11c6c42c-2013-4284-bf52-fec8f6d7d5a5","Type":"ContainerStarted","Data":"a91595f14c5a8764112a84769bb95327736f660f88cd63a2c55ae73e1b31ee39"} Jul 09 19:32:37.558102 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:37.558033 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb" podStartSLOduration=1.558013827 podStartE2EDuration="1.558013827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:32:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:32:37.556542919 +0000 UTC m=+2027.064322703" watchObservedRunningTime="2026-07-09 19:32:37.558013827 +0000 UTC m=+2027.065793600" Jul 09 19:32:38.479567 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:38.479527 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb" Jul 09 19:32:38.484469 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:38.484446 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb" Jul 09 19:32:38.538935 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:38.538905 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb" Jul 09 19:32:38.539835 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:38.539816 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb" Jul 09 19:32:44.558881 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:44.558849 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch"] Jul 09 19:32:44.562433 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:44.562393 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch" Jul 09 19:32:44.565793 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:44.565771 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-bc9ef2-openshift-default-dockercfg-rjmfc\"" Jul 09 19:32:44.575561 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:44.575527 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch"] Jul 09 19:32:44.592318 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:44.592297 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a7a0652-e4d0-485f-919e-6bed7a8e981c-credential-socket\") pod \"e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch\" (UID: \"1a7a0652-e4d0-485f-919e-6bed7a8e981c\") " pod="openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch" Jul 09 19:32:44.592435 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:44.592351 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a7a0652-e4d0-485f-919e-6bed7a8e981c-istio-envoy\") pod \"e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch\" (UID: \"1a7a0652-e4d0-485f-919e-6bed7a8e981c\") " pod="openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch" Jul 09 19:32:44.592435 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:44.592407 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a7a0652-e4d0-485f-919e-6bed7a8e981c-istio-data\") pod \"e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch\" (UID: \"1a7a0652-e4d0-485f-919e-6bed7a8e981c\") " pod="openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch" Jul 09 19:32:44.592435 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:44.592425 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a7a0652-e4d0-485f-919e-6bed7a8e981c-istio-token\") pod \"e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch\" (UID: \"1a7a0652-e4d0-485f-919e-6bed7a8e981c\") " pod="openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch" Jul 09 19:32:44.592546 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:44.592449 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-467t4\" (UniqueName: \"kubernetes.io/projected/1a7a0652-e4d0-485f-919e-6bed7a8e981c-kube-api-access-467t4\") pod \"e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch\" (UID: \"1a7a0652-e4d0-485f-919e-6bed7a8e981c\") " pod="openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch" Jul 09 19:32:44.592546 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:44.592482 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a7a0652-e4d0-485f-919e-6bed7a8e981c-istiod-ca-cert\") pod \"e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch\" (UID: \"1a7a0652-e4d0-485f-919e-6bed7a8e981c\") " pod="openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch" Jul 09 19:32:44.592546 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:44.592512 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a7a0652-e4d0-485f-919e-6bed7a8e981c-istio-podinfo\") pod \"e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch\" (UID: \"1a7a0652-e4d0-485f-919e-6bed7a8e981c\") " pod="openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch" Jul 09 19:32:44.592661 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:44.592565 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a7a0652-e4d0-485f-919e-6bed7a8e981c-workload-socket\") pod \"e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch\" (UID: \"1a7a0652-e4d0-485f-919e-6bed7a8e981c\") " pod="openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch" Jul 09 19:32:44.592661 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:44.592597 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a7a0652-e4d0-485f-919e-6bed7a8e981c-workload-certs\") pod \"e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch\" (UID: \"1a7a0652-e4d0-485f-919e-6bed7a8e981c\") " pod="openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch" Jul 09 19:32:44.693540 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:44.693507 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a7a0652-e4d0-485f-919e-6bed7a8e981c-istio-envoy\") pod \"e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch\" (UID: \"1a7a0652-e4d0-485f-919e-6bed7a8e981c\") " pod="openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch" Jul 09 19:32:44.693742 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:44.693548 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a7a0652-e4d0-485f-919e-6bed7a8e981c-istio-data\") pod \"e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch\" (UID: \"1a7a0652-e4d0-485f-919e-6bed7a8e981c\") " pod="openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch" Jul 09 19:32:44.693742 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:44.693567 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a7a0652-e4d0-485f-919e-6bed7a8e981c-istio-token\") pod \"e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch\" (UID: \"1a7a0652-e4d0-485f-919e-6bed7a8e981c\") " pod="openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch" Jul 09 19:32:44.693742 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:44.693593 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-467t4\" (UniqueName: \"kubernetes.io/projected/1a7a0652-e4d0-485f-919e-6bed7a8e981c-kube-api-access-467t4\") pod \"e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch\" (UID: \"1a7a0652-e4d0-485f-919e-6bed7a8e981c\") " pod="openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch" Jul 09 19:32:44.693742 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:44.693641 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a7a0652-e4d0-485f-919e-6bed7a8e981c-istiod-ca-cert\") pod \"e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch\" (UID: \"1a7a0652-e4d0-485f-919e-6bed7a8e981c\") " pod="openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch" Jul 09 19:32:44.693742 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:44.693670 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a7a0652-e4d0-485f-919e-6bed7a8e981c-istio-podinfo\") pod \"e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch\" (UID: \"1a7a0652-e4d0-485f-919e-6bed7a8e981c\") " pod="openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch" Jul 09 19:32:44.693742 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:44.693686 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a7a0652-e4d0-485f-919e-6bed7a8e981c-workload-socket\") pod \"e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch\" (UID: \"1a7a0652-e4d0-485f-919e-6bed7a8e981c\") " pod="openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch" Jul 09 19:32:44.693742 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:44.693701 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a7a0652-e4d0-485f-919e-6bed7a8e981c-workload-certs\") pod \"e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch\" (UID: \"1a7a0652-e4d0-485f-919e-6bed7a8e981c\") " pod="openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch" Jul 09 19:32:44.693742 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:44.693729 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a7a0652-e4d0-485f-919e-6bed7a8e981c-credential-socket\") pod \"e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch\" (UID: \"1a7a0652-e4d0-485f-919e-6bed7a8e981c\") " pod="openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch" Jul 09 19:32:44.694215 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:44.694141 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a7a0652-e4d0-485f-919e-6bed7a8e981c-credential-socket\") pod \"e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch\" (UID: \"1a7a0652-e4d0-485f-919e-6bed7a8e981c\") " pod="openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch" Jul 09 19:32:44.694215 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:44.694186 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a7a0652-e4d0-485f-919e-6bed7a8e981c-istio-data\") pod \"e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch\" (UID: \"1a7a0652-e4d0-485f-919e-6bed7a8e981c\") " pod="openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch" Jul 09 19:32:44.694345 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:44.694329 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a7a0652-e4d0-485f-919e-6bed7a8e981c-workload-socket\") pod \"e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch\" (UID: \"1a7a0652-e4d0-485f-919e-6bed7a8e981c\") " pod="openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch" Jul 09 19:32:44.694469 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:44.694405 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a7a0652-e4d0-485f-919e-6bed7a8e981c-workload-certs\") pod \"e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch\" (UID: \"1a7a0652-e4d0-485f-919e-6bed7a8e981c\") " pod="openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch" Jul 09 19:32:44.694534 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:44.694471 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a7a0652-e4d0-485f-919e-6bed7a8e981c-istiod-ca-cert\") pod \"e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch\" (UID: \"1a7a0652-e4d0-485f-919e-6bed7a8e981c\") " pod="openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch" Jul 09 19:32:44.696377 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:44.696348 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a7a0652-e4d0-485f-919e-6bed7a8e981c-istio-envoy\") pod \"e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch\" (UID: \"1a7a0652-e4d0-485f-919e-6bed7a8e981c\") " pod="openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch" Jul 09 19:32:44.696496 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:44.696477 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a7a0652-e4d0-485f-919e-6bed7a8e981c-istio-podinfo\") pod \"e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch\" (UID: \"1a7a0652-e4d0-485f-919e-6bed7a8e981c\") " pod="openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch" Jul 09 19:32:44.702256 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:44.702231 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-467t4\" (UniqueName: \"kubernetes.io/projected/1a7a0652-e4d0-485f-919e-6bed7a8e981c-kube-api-access-467t4\") pod \"e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch\" (UID: \"1a7a0652-e4d0-485f-919e-6bed7a8e981c\") " pod="openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch" Jul 09 19:32:44.702675 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:44.702654 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a7a0652-e4d0-485f-919e-6bed7a8e981c-istio-token\") pod \"e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch\" (UID: \"1a7a0652-e4d0-485f-919e-6bed7a8e981c\") " pod="openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch" Jul 09 19:32:44.874210 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:44.874127 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch" Jul 09 19:32:45.023679 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:45.023651 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch"] Jul 09 19:32:45.025030 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:32:45.025003 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a7a0652_e4d0_485f_919e_6bed7a8e981c.slice/crio-95e4c5372cb56b276022731991d81498a46c07da552635c24d055cf5c5cd51af WatchSource:0}: Error finding container 95e4c5372cb56b276022731991d81498a46c07da552635c24d055cf5c5cd51af: Status 404 returned error can't find the container with id 95e4c5372cb56b276022731991d81498a46c07da552635c24d055cf5c5cd51af Jul 09 19:32:45.027156 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:45.027126 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:32:45.027219 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:45.027192 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:32:45.027256 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:45.027220 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:32:45.583672 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:45.583634 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch" event={"ID":"1a7a0652-e4d0-485f-919e-6bed7a8e981c","Type":"ContainerStarted","Data":"ee14bcc7052a856acd90816d026d3b21a6822d54ba535a400b0c1b72bac1d293"} Jul 09 19:32:45.584130 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:45.583704 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch" event={"ID":"1a7a0652-e4d0-485f-919e-6bed7a8e981c","Type":"ContainerStarted","Data":"95e4c5372cb56b276022731991d81498a46c07da552635c24d055cf5c5cd51af"} Jul 09 19:32:45.609939 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:45.609892 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch" podStartSLOduration=1.6098774790000001 podStartE2EDuration="1.609877479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:32:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:32:45.607035236 +0000 UTC m=+2035.114815013" watchObservedRunningTime="2026-07-09 19:32:45.609877479 +0000 UTC m=+2035.117657258" Jul 09 19:32:45.875045 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:45.874957 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch" Jul 09 19:32:46.879057 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:46.879027 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch" Jul 09 19:32:47.595135 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:47.595102 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch" Jul 09 19:32:47.595980 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:47.595962 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch" Jul 09 19:32:47.972918 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:47.972889 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:32:54.580819 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:54.580778 2566 patch_prober.go:28] interesting pod/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 09 19:32:54.581210 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:54.580847 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb" podUID="11c6c42c-2013-4284-bf52-fec8f6d7d5a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 09 19:32:56.657340 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:56.657310 2566 generic.go:356] "Generic (PLEG): container finished" podID="11c6c42c-2013-4284-bf52-fec8f6d7d5a5" containerID="8ebc5236971316cecd387c96c62346d35215137cb5a610d546d95e817aeb71c5" exitCode=0 Jul 09 19:32:56.657805 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:56.657385 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb" event={"ID":"11c6c42c-2013-4284-bf52-fec8f6d7d5a5","Type":"ContainerDied","Data":"8ebc5236971316cecd387c96c62346d35215137cb5a610d546d95e817aeb71c5"} Jul 09 19:32:56.657877 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:56.657857 2566 scope.go:117] "RemoveContainer" containerID="8ebc5236971316cecd387c96c62346d35215137cb5a610d546d95e817aeb71c5" Jul 09 19:32:56.658865 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:56.658837 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:32:56.658965 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:56.658902 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:32:56.658965 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:56.658932 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:32:57.664777 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:57.664741 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb" event={"ID":"11c6c42c-2013-4284-bf52-fec8f6d7d5a5","Type":"ContainerStarted","Data":"e63acdf14eed908ad539cc6c45398282c0cf9246dd6789b381868c2804fe9148"} Jul 09 19:32:58.479874 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:58.479258 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb" Jul 09 19:32:58.479874 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:58.479628 2566 patch_prober.go:28] interesting pod/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 09 19:32:58.479874 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:58.479667 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb" podUID="11c6c42c-2013-4284-bf52-fec8f6d7d5a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 09 19:32:59.479515 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:59.479480 2566 patch_prober.go:28] interesting pod/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 09 19:32:59.479926 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:32:59.479549 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb" podUID="11c6c42c-2013-4284-bf52-fec8f6d7d5a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 09 19:33:00.479190 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:33:00.479150 2566 patch_prober.go:28] interesting pod/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 09 19:33:00.479380 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:33:00.479207 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb" podUID="11c6c42c-2013-4284-bf52-fec8f6d7d5a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 09 19:33:01.478955 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:33:01.478923 2566 patch_prober.go:28] interesting pod/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 09 19:33:01.479355 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:33:01.478974 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb" podUID="11c6c42c-2013-4284-bf52-fec8f6d7d5a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 09 19:33:02.483284 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:33:02.483249 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb" Jul 09 19:33:02.483687 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:33:02.483642 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb" Jul 09 19:33:02.484343 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:33:02.484321 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb" Jul 09 19:33:03.636783 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:33:03.636750 2566 patch_prober.go:28] interesting pod/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 09 19:33:03.637293 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:33:03.636821 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch" podUID="1a7a0652-e4d0-485f-919e-6bed7a8e981c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 09 19:33:05.712572 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:33:05.712542 2566 generic.go:356] "Generic (PLEG): container finished" podID="1a7a0652-e4d0-485f-919e-6bed7a8e981c" containerID="ee14bcc7052a856acd90816d026d3b21a6822d54ba535a400b0c1b72bac1d293" exitCode=0 Jul 09 19:33:05.713024 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:33:05.712592 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch" event={"ID":"1a7a0652-e4d0-485f-919e-6bed7a8e981c","Type":"ContainerDied","Data":"ee14bcc7052a856acd90816d026d3b21a6822d54ba535a400b0c1b72bac1d293"} Jul 09 19:33:05.713024 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:33:05.712982 2566 scope.go:117] "RemoveContainer" containerID="ee14bcc7052a856acd90816d026d3b21a6822d54ba535a400b0c1b72bac1d293" Jul 09 19:33:05.714033 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:33:05.714007 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:33:05.714115 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:33:05.714080 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:33:05.714158 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:33:05.714115 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:33:06.161627 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:33:06.161537 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:33:06.734230 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:33:06.734184 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch" event={"ID":"1a7a0652-e4d0-485f-919e-6bed7a8e981c","Type":"ContainerStarted","Data":"b4cdf138cec63b4df2b8fd2079a3292f7442e8644bcfe11f05dc1dbd5d2b6e63"} Jul 09 19:33:06.874708 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:33:06.874666 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch" Jul 09 19:33:06.874955 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:33:06.874932 2566 patch_prober.go:28] interesting pod/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 09 19:33:06.875002 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:33:06.874986 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch" podUID="1a7a0652-e4d0-485f-919e-6bed7a8e981c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 09 19:33:07.875294 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:33:07.875248 2566 patch_prober.go:28] interesting pod/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 09 19:33:07.875842 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:33:07.875306 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch" podUID="1a7a0652-e4d0-485f-919e-6bed7a8e981c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 09 19:33:08.874970 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:33:08.874926 2566 patch_prober.go:28] interesting pod/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 09 19:33:08.875152 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:33:08.874988 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch" podUID="1a7a0652-e4d0-485f-919e-6bed7a8e981c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 09 19:33:09.875193 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:33:09.875161 2566 patch_prober.go:28] interesting pod/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 09 19:33:09.875629 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:33:09.875216 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch" podUID="1a7a0652-e4d0-485f-919e-6bed7a8e981c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 09 19:33:10.878012 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:33:10.877982 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch" Jul 09 19:33:10.878391 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:33:10.878224 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch" Jul 09 19:33:10.878818 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:33:10.878802 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch" Jul 09 19:33:27.494225 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:33:27.494184 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:33:51.138075 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:33:51.138042 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-vws4c_7cf4af17-5218-4b92-bceb-addbefcc03cb/console-operator/1.log" Jul 09 19:33:51.142928 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:33:51.142904 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6dh7g_db233460-134d-4d9d-8da9-31d89366a4e0/ovn-acl-logging/0.log" Jul 09 19:33:51.146654 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:33:51.146636 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-vws4c_7cf4af17-5218-4b92-bceb-addbefcc03cb/console-operator/1.log" Jul 09 19:33:51.150895 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:33:51.150877 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6dh7g_db233460-134d-4d9d-8da9-31d89366a4e0/ovn-acl-logging/0.log" Jul 09 19:33:51.464623 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:33:51.464577 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:34:28.929594 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:28.929561 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb"] Jul 09 19:34:28.930505 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:28.929785 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb" podUID="11c6c42c-2013-4284-bf52-fec8f6d7d5a5" containerName="istio-proxy" containerID="cri-o://e63acdf14eed908ad539cc6c45398282c0cf9246dd6789b381868c2804fe9148" gracePeriod=30 Jul 09 19:34:32.485986 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:32.485939 2566 patch_prober.go:28] interesting pod/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 09 19:34:32.486386 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:32.486026 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb" podUID="11c6c42c-2013-4284-bf52-fec8f6d7d5a5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 19:34:33.261296 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:33.261259 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:34:34.189988 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:34.189965 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb" Jul 09 19:34:34.229768 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:34.229687 2566 generic.go:356] "Generic (PLEG): container finished" podID="11c6c42c-2013-4284-bf52-fec8f6d7d5a5" containerID="e63acdf14eed908ad539cc6c45398282c0cf9246dd6789b381868c2804fe9148" exitCode=0 Jul 09 19:34:34.229768 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:34.229743 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb" event={"ID":"11c6c42c-2013-4284-bf52-fec8f6d7d5a5","Type":"ContainerDied","Data":"e63acdf14eed908ad539cc6c45398282c0cf9246dd6789b381868c2804fe9148"} Jul 09 19:34:34.229768 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:34.229747 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb" Jul 09 19:34:34.229985 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:34.229786 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb" event={"ID":"11c6c42c-2013-4284-bf52-fec8f6d7d5a5","Type":"ContainerDied","Data":"a91595f14c5a8764112a84769bb95327736f660f88cd63a2c55ae73e1b31ee39"} Jul 09 19:34:34.229985 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:34.229808 2566 scope.go:117] "RemoveContainer" containerID="e63acdf14eed908ad539cc6c45398282c0cf9246dd6789b381868c2804fe9148" Jul 09 19:34:34.239474 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:34.239455 2566 scope.go:117] "RemoveContainer" containerID="8ebc5236971316cecd387c96c62346d35215137cb5a610d546d95e817aeb71c5" Jul 09 19:34:34.247927 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:34.247907 2566 scope.go:117] "RemoveContainer" containerID="e63acdf14eed908ad539cc6c45398282c0cf9246dd6789b381868c2804fe9148" Jul 09 19:34:34.248203 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:34:34.248178 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e63acdf14eed908ad539cc6c45398282c0cf9246dd6789b381868c2804fe9148\": container with ID starting with e63acdf14eed908ad539cc6c45398282c0cf9246dd6789b381868c2804fe9148 not found: ID does not exist" containerID="e63acdf14eed908ad539cc6c45398282c0cf9246dd6789b381868c2804fe9148" Jul 09 19:34:34.248271 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:34.248216 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e63acdf14eed908ad539cc6c45398282c0cf9246dd6789b381868c2804fe9148"} err="failed to get container status \"e63acdf14eed908ad539cc6c45398282c0cf9246dd6789b381868c2804fe9148\": rpc error: code = NotFound desc = could not find container \"e63acdf14eed908ad539cc6c45398282c0cf9246dd6789b381868c2804fe9148\": container with ID starting with e63acdf14eed908ad539cc6c45398282c0cf9246dd6789b381868c2804fe9148 not found: ID does not exist" Jul 09 19:34:34.248271 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:34.248241 2566 scope.go:117] "RemoveContainer" containerID="8ebc5236971316cecd387c96c62346d35215137cb5a610d546d95e817aeb71c5" Jul 09 19:34:34.248510 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:34:34.248490 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ebc5236971316cecd387c96c62346d35215137cb5a610d546d95e817aeb71c5\": container with ID starting with 8ebc5236971316cecd387c96c62346d35215137cb5a610d546d95e817aeb71c5 not found: ID does not exist" containerID="8ebc5236971316cecd387c96c62346d35215137cb5a610d546d95e817aeb71c5" Jul 09 19:34:34.248625 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:34.248514 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ebc5236971316cecd387c96c62346d35215137cb5a610d546d95e817aeb71c5"} err="failed to get container status \"8ebc5236971316cecd387c96c62346d35215137cb5a610d546d95e817aeb71c5\": rpc error: code = NotFound desc = could not find container \"8ebc5236971316cecd387c96c62346d35215137cb5a610d546d95e817aeb71c5\": container with ID starting with 8ebc5236971316cecd387c96c62346d35215137cb5a610d546d95e817aeb71c5 not found: ID does not exist" Jul 09 19:34:34.323365 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:34.323337 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vx9g\" (UniqueName: \"kubernetes.io/projected/11c6c42c-2013-4284-bf52-fec8f6d7d5a5-kube-api-access-4vx9g\") pod \"11c6c42c-2013-4284-bf52-fec8f6d7d5a5\" (UID: \"11c6c42c-2013-4284-bf52-fec8f6d7d5a5\") " Jul 09 19:34:34.323529 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:34.323391 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11c6c42c-2013-4284-bf52-fec8f6d7d5a5-istiod-ca-cert\") pod \"11c6c42c-2013-4284-bf52-fec8f6d7d5a5\" (UID: \"11c6c42c-2013-4284-bf52-fec8f6d7d5a5\") " Jul 09 19:34:34.323529 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:34.323416 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c6c42c-2013-4284-bf52-fec8f6d7d5a5-credential-socket\") pod \"11c6c42c-2013-4284-bf52-fec8f6d7d5a5\" (UID: \"11c6c42c-2013-4284-bf52-fec8f6d7d5a5\") " Jul 09 19:34:34.323529 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:34.323446 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11c6c42c-2013-4284-bf52-fec8f6d7d5a5-istio-data\") pod \"11c6c42c-2013-4284-bf52-fec8f6d7d5a5\" (UID: \"11c6c42c-2013-4284-bf52-fec8f6d7d5a5\") " Jul 09 19:34:34.323529 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:34.323479 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11c6c42c-2013-4284-bf52-fec8f6d7d5a5-istio-token\") pod \"11c6c42c-2013-4284-bf52-fec8f6d7d5a5\" (UID: \"11c6c42c-2013-4284-bf52-fec8f6d7d5a5\") " Jul 09 19:34:34.323529 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:34.323515 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c6c42c-2013-4284-bf52-fec8f6d7d5a5-workload-socket\") pod \"11c6c42c-2013-4284-bf52-fec8f6d7d5a5\" (UID: \"11c6c42c-2013-4284-bf52-fec8f6d7d5a5\") " Jul 09 19:34:34.323848 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:34.323554 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11c6c42c-2013-4284-bf52-fec8f6d7d5a5-istio-podinfo\") pod \"11c6c42c-2013-4284-bf52-fec8f6d7d5a5\" (UID: \"11c6c42c-2013-4284-bf52-fec8f6d7d5a5\") " Jul 09 19:34:34.323848 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:34.323594 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11c6c42c-2013-4284-bf52-fec8f6d7d5a5-workload-certs\") pod \"11c6c42c-2013-4284-bf52-fec8f6d7d5a5\" (UID: \"11c6c42c-2013-4284-bf52-fec8f6d7d5a5\") " Jul 09 19:34:34.323848 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:34.323662 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11c6c42c-2013-4284-bf52-fec8f6d7d5a5-istio-envoy\") pod \"11c6c42c-2013-4284-bf52-fec8f6d7d5a5\" (UID: \"11c6c42c-2013-4284-bf52-fec8f6d7d5a5\") " Jul 09 19:34:34.323848 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:34.323762 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11c6c42c-2013-4284-bf52-fec8f6d7d5a5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "11c6c42c-2013-4284-bf52-fec8f6d7d5a5" (UID: "11c6c42c-2013-4284-bf52-fec8f6d7d5a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:34:34.324125 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:34.323864 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11c6c42c-2013-4284-bf52-fec8f6d7d5a5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "11c6c42c-2013-4284-bf52-fec8f6d7d5a5" (UID: "11c6c42c-2013-4284-bf52-fec8f6d7d5a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:34:34.324125 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:34.323886 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11c6c42c-2013-4284-bf52-fec8f6d7d5a5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "11c6c42c-2013-4284-bf52-fec8f6d7d5a5" (UID: "11c6c42c-2013-4284-bf52-fec8f6d7d5a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 19:34:34.324125 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:34.324058 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11c6c42c-2013-4284-bf52-fec8f6d7d5a5-istiod-ca-cert\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:34:34.324125 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:34.324078 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c6c42c-2013-4284-bf52-fec8f6d7d5a5-credential-socket\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:34:34.324125 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:34.324091 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11c6c42c-2013-4284-bf52-fec8f6d7d5a5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "11c6c42c-2013-4284-bf52-fec8f6d7d5a5" (UID: "11c6c42c-2013-4284-bf52-fec8f6d7d5a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:34:34.324125 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:34.324097 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c6c42c-2013-4284-bf52-fec8f6d7d5a5-workload-socket\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:34:34.324526 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:34.324499 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11c6c42c-2013-4284-bf52-fec8f6d7d5a5-istio-data" (OuterVolumeSpecName: "istio-data") pod "11c6c42c-2013-4284-bf52-fec8f6d7d5a5" (UID: "11c6c42c-2013-4284-bf52-fec8f6d7d5a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:34:34.325685 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:34.325646 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11c6c42c-2013-4284-bf52-fec8f6d7d5a5-kube-api-access-4vx9g" (OuterVolumeSpecName: "kube-api-access-4vx9g") pod "11c6c42c-2013-4284-bf52-fec8f6d7d5a5" (UID: "11c6c42c-2013-4284-bf52-fec8f6d7d5a5"). InnerVolumeSpecName "kube-api-access-4vx9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:34:34.325808 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:34.325729 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11c6c42c-2013-4284-bf52-fec8f6d7d5a5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "11c6c42c-2013-4284-bf52-fec8f6d7d5a5" (UID: "11c6c42c-2013-4284-bf52-fec8f6d7d5a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:34:34.325873 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:34.325820 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11c6c42c-2013-4284-bf52-fec8f6d7d5a5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "11c6c42c-2013-4284-bf52-fec8f6d7d5a5" (UID: "11c6c42c-2013-4284-bf52-fec8f6d7d5a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 19:34:34.325873 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:34.325834 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11c6c42c-2013-4284-bf52-fec8f6d7d5a5-istio-token" (OuterVolumeSpecName: "istio-token") pod "11c6c42c-2013-4284-bf52-fec8f6d7d5a5" (UID: "11c6c42c-2013-4284-bf52-fec8f6d7d5a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:34:34.425256 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:34.425222 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11c6c42c-2013-4284-bf52-fec8f6d7d5a5-istio-envoy\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:34:34.425256 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:34.425250 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vx9g\" (UniqueName: \"kubernetes.io/projected/11c6c42c-2013-4284-bf52-fec8f6d7d5a5-kube-api-access-4vx9g\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:34:34.425256 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:34.425260 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11c6c42c-2013-4284-bf52-fec8f6d7d5a5-istio-data\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:34:34.425483 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:34.425269 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11c6c42c-2013-4284-bf52-fec8f6d7d5a5-istio-token\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:34:34.425483 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:34.425277 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11c6c42c-2013-4284-bf52-fec8f6d7d5a5-istio-podinfo\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:34:34.425483 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:34.425285 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11c6c42c-2013-4284-bf52-fec8f6d7d5a5-workload-certs\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:34:34.553131 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:34.553099 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb"] Jul 09 19:34:34.557233 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:34.557198 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a7212d-openshift-default-75bcb9b947-bbrbb"] Jul 09 19:34:35.058209 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:35.058170 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11c6c42c-2013-4284-bf52-fec8f6d7d5a5" path="/var/lib/kubelet/pods/11c6c42c-2013-4284-bf52-fec8f6d7d5a5/volumes" Jul 09 19:34:35.966161 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:35.966126 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch"] Jul 09 19:34:35.966598 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:35.966382 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch" podUID="1a7a0652-e4d0-485f-919e-6bed7a8e981c" containerName="istio-proxy" containerID="cri-o://b4cdf138cec63b4df2b8fd2079a3292f7442e8644bcfe11f05dc1dbd5d2b6e63" gracePeriod=30 Jul 09 19:34:36.846207 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:36.846166 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5"] Jul 09 19:34:36.846631 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:36.846599 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11c6c42c-2013-4284-bf52-fec8f6d7d5a5" containerName="istio-proxy" Jul 09 19:34:36.846696 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:36.846632 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="11c6c42c-2013-4284-bf52-fec8f6d7d5a5" containerName="istio-proxy" Jul 09 19:34:36.846696 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:36.846661 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11c6c42c-2013-4284-bf52-fec8f6d7d5a5" containerName="istio-proxy" Jul 09 19:34:36.846696 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:36.846666 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="11c6c42c-2013-4284-bf52-fec8f6d7d5a5" containerName="istio-proxy" Jul 09 19:34:36.846796 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:36.846741 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="11c6c42c-2013-4284-bf52-fec8f6d7d5a5" containerName="istio-proxy" Jul 09 19:34:36.846796 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:36.846749 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="11c6c42c-2013-4284-bf52-fec8f6d7d5a5" containerName="istio-proxy" Jul 09 19:34:36.851203 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:36.851187 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5" Jul 09 19:34:36.854344 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:36.854324 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-47c272-openshift-default-dockercfg-lcsff\"" Jul 09 19:34:36.863487 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:36.863445 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5"] Jul 09 19:34:36.948697 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:36.948654 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db-credential-socket\") pod \"e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5\" (UID: \"5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db\") " pod="openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5" Jul 09 19:34:36.948697 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:36.948703 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db-istio-podinfo\") pod \"e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5\" (UID: \"5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db\") " pod="openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5" Jul 09 19:34:36.948921 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:36.948726 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db-istiod-ca-cert\") pod \"e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5\" (UID: \"5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db\") " pod="openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5" Jul 09 19:34:36.948921 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:36.948804 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db-workload-certs\") pod \"e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5\" (UID: \"5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db\") " pod="openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5" Jul 09 19:34:36.948921 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:36.948821 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db-istio-token\") pod \"e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5\" (UID: \"5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db\") " pod="openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5" Jul 09 19:34:36.948921 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:36.948905 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db-workload-socket\") pod \"e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5\" (UID: \"5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db\") " pod="openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5" Jul 09 19:34:36.949075 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:36.948938 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db-istio-data\") pod \"e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5\" (UID: \"5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db\") " pod="openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5" Jul 09 19:34:36.949075 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:36.948970 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db-istio-envoy\") pod \"e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5\" (UID: \"5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db\") " pod="openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5" Jul 09 19:34:36.949075 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:36.948988 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grnlg\" (UniqueName: \"kubernetes.io/projected/5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db-kube-api-access-grnlg\") pod \"e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5\" (UID: \"5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db\") " pod="openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5" Jul 09 19:34:37.050066 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:37.050028 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db-credential-socket\") pod \"e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5\" (UID: \"5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db\") " pod="openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5" Jul 09 19:34:37.050066 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:37.050077 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db-istio-podinfo\") pod \"e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5\" (UID: \"5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db\") " pod="openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5" Jul 09 19:34:37.050574 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:37.050118 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db-istiod-ca-cert\") pod \"e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5\" (UID: \"5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db\") " pod="openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5" Jul 09 19:34:37.050574 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:37.050144 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db-workload-certs\") pod \"e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5\" (UID: \"5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db\") " pod="openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5" Jul 09 19:34:37.050574 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:37.050161 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db-istio-token\") pod \"e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5\" (UID: \"5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db\") " pod="openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5" Jul 09 19:34:37.050574 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:37.050236 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db-workload-socket\") pod \"e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5\" (UID: \"5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db\") " pod="openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5" Jul 09 19:34:37.050574 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:37.050264 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db-istio-data\") pod \"e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5\" (UID: \"5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db\") " pod="openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5" Jul 09 19:34:37.050574 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:37.050409 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db-istio-envoy\") pod \"e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5\" (UID: \"5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db\") " pod="openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5" Jul 09 19:34:37.050574 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:37.050459 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grnlg\" (UniqueName: \"kubernetes.io/projected/5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db-kube-api-access-grnlg\") pod \"e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5\" (UID: \"5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db\") " pod="openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5" Jul 09 19:34:37.050950 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:37.050885 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db-credential-socket\") pod \"e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5\" (UID: \"5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db\") " pod="openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5" Jul 09 19:34:37.050950 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:37.050930 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db-workload-certs\") pod \"e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5\" (UID: \"5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db\") " pod="openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5" Jul 09 19:34:37.050950 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:37.050939 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db-workload-socket\") pod \"e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5\" (UID: \"5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db\") " pod="openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5" Jul 09 19:34:37.051112 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:37.051032 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db-istio-data\") pod \"e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5\" (UID: \"5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db\") " pod="openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5" Jul 09 19:34:37.051322 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:37.051296 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db-istiod-ca-cert\") pod \"e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5\" (UID: \"5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db\") " pod="openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5" Jul 09 19:34:37.052755 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:37.052737 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db-istio-envoy\") pod \"e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5\" (UID: \"5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db\") " pod="openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5" Jul 09 19:34:37.053174 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:37.053156 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db-istio-podinfo\") pod \"e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5\" (UID: \"5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db\") " pod="openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5" Jul 09 19:34:37.058730 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:37.058708 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db-istio-token\") pod \"e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5\" (UID: \"5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db\") " pod="openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5" Jul 09 19:34:37.058981 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:37.058964 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grnlg\" (UniqueName: \"kubernetes.io/projected/5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db-kube-api-access-grnlg\") pod \"e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5\" (UID: \"5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db\") " pod="openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5" Jul 09 19:34:37.169012 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:37.168929 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5" Jul 09 19:34:37.318615 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:37.318579 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5"] Jul 09 19:34:37.320254 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:34:37.320229 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e6ccf66_cddf_4928_bdf9_9ed2b1bd42db.slice/crio-692c05c36176306d8d150959b9602cb503f212081e693b4f776212b107a6bf3e WatchSource:0}: Error finding container 692c05c36176306d8d150959b9602cb503f212081e693b4f776212b107a6bf3e: Status 404 returned error can't find the container with id 692c05c36176306d8d150959b9602cb503f212081e693b4f776212b107a6bf3e Jul 09 19:34:37.322395 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:37.322363 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:34:37.322494 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:37.322445 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:34:37.322563 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:37.322491 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:34:38.255505 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:38.255471 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5" event={"ID":"5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db","Type":"ContainerStarted","Data":"afb189f0872b9ae35d402f009495be412517bedf81846c9455d5c4cf1a63df3b"} Jul 09 19:34:38.255930 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:38.255522 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5" event={"ID":"5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db","Type":"ContainerStarted","Data":"692c05c36176306d8d150959b9602cb503f212081e693b4f776212b107a6bf3e"} Jul 09 19:34:38.279856 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:38.279809 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5" podStartSLOduration=2.279793057 podStartE2EDuration="2.279793057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:34:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:34:38.277598248 +0000 UTC m=+2147.785378027" watchObservedRunningTime="2026-07-09 19:34:38.279793057 +0000 UTC m=+2147.787572831" Jul 09 19:34:39.169331 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:39.169297 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5" Jul 09 19:34:39.174780 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:39.174753 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5" Jul 09 19:34:39.260565 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:39.260532 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5" Jul 09 19:34:39.261575 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:39.261558 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5" Jul 09 19:34:40.881471 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:40.881436 2566 patch_prober.go:28] interesting pod/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 09 19:34:40.881961 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:40.881641 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch" podUID="1a7a0652-e4d0-485f-919e-6bed7a8e981c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 19:34:41.240783 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:41.240757 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch" Jul 09 19:34:41.274786 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:41.274751 2566 generic.go:356] "Generic (PLEG): container finished" podID="1a7a0652-e4d0-485f-919e-6bed7a8e981c" containerID="b4cdf138cec63b4df2b8fd2079a3292f7442e8644bcfe11f05dc1dbd5d2b6e63" exitCode=0 Jul 09 19:34:41.274971 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:41.274799 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch" Jul 09 19:34:41.274971 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:41.274839 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch" event={"ID":"1a7a0652-e4d0-485f-919e-6bed7a8e981c","Type":"ContainerDied","Data":"b4cdf138cec63b4df2b8fd2079a3292f7442e8644bcfe11f05dc1dbd5d2b6e63"} Jul 09 19:34:41.274971 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:41.274929 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch" event={"ID":"1a7a0652-e4d0-485f-919e-6bed7a8e981c","Type":"ContainerDied","Data":"95e4c5372cb56b276022731991d81498a46c07da552635c24d055cf5c5cd51af"} Jul 09 19:34:41.274971 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:41.274961 2566 scope.go:117] "RemoveContainer" containerID="b4cdf138cec63b4df2b8fd2079a3292f7442e8644bcfe11f05dc1dbd5d2b6e63" Jul 09 19:34:41.284065 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:41.284045 2566 scope.go:117] "RemoveContainer" containerID="ee14bcc7052a856acd90816d026d3b21a6822d54ba535a400b0c1b72bac1d293" Jul 09 19:34:41.292827 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:41.292799 2566 scope.go:117] "RemoveContainer" containerID="b4cdf138cec63b4df2b8fd2079a3292f7442e8644bcfe11f05dc1dbd5d2b6e63" Jul 09 19:34:41.293102 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:34:41.293078 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4cdf138cec63b4df2b8fd2079a3292f7442e8644bcfe11f05dc1dbd5d2b6e63\": container with ID starting with b4cdf138cec63b4df2b8fd2079a3292f7442e8644bcfe11f05dc1dbd5d2b6e63 not found: ID does not exist" containerID="b4cdf138cec63b4df2b8fd2079a3292f7442e8644bcfe11f05dc1dbd5d2b6e63" Jul 09 19:34:41.293148 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:41.293118 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4cdf138cec63b4df2b8fd2079a3292f7442e8644bcfe11f05dc1dbd5d2b6e63"} err="failed to get container status \"b4cdf138cec63b4df2b8fd2079a3292f7442e8644bcfe11f05dc1dbd5d2b6e63\": rpc error: code = NotFound desc = could not find container \"b4cdf138cec63b4df2b8fd2079a3292f7442e8644bcfe11f05dc1dbd5d2b6e63\": container with ID starting with b4cdf138cec63b4df2b8fd2079a3292f7442e8644bcfe11f05dc1dbd5d2b6e63 not found: ID does not exist" Jul 09 19:34:41.293148 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:41.293143 2566 scope.go:117] "RemoveContainer" containerID="ee14bcc7052a856acd90816d026d3b21a6822d54ba535a400b0c1b72bac1d293" Jul 09 19:34:41.293394 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:34:41.293377 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee14bcc7052a856acd90816d026d3b21a6822d54ba535a400b0c1b72bac1d293\": container with ID starting with ee14bcc7052a856acd90816d026d3b21a6822d54ba535a400b0c1b72bac1d293 not found: ID does not exist" containerID="ee14bcc7052a856acd90816d026d3b21a6822d54ba535a400b0c1b72bac1d293" Jul 09 19:34:41.293434 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:41.293402 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee14bcc7052a856acd90816d026d3b21a6822d54ba535a400b0c1b72bac1d293"} err="failed to get container status \"ee14bcc7052a856acd90816d026d3b21a6822d54ba535a400b0c1b72bac1d293\": rpc error: code = NotFound desc = could not find container \"ee14bcc7052a856acd90816d026d3b21a6822d54ba535a400b0c1b72bac1d293\": container with ID starting with ee14bcc7052a856acd90816d026d3b21a6822d54ba535a400b0c1b72bac1d293 not found: ID does not exist" Jul 09 19:34:41.397321 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:41.397234 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a7a0652-e4d0-485f-919e-6bed7a8e981c-istio-token\") pod \"1a7a0652-e4d0-485f-919e-6bed7a8e981c\" (UID: \"1a7a0652-e4d0-485f-919e-6bed7a8e981c\") " Jul 09 19:34:41.397321 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:41.397279 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a7a0652-e4d0-485f-919e-6bed7a8e981c-workload-socket\") pod \"1a7a0652-e4d0-485f-919e-6bed7a8e981c\" (UID: \"1a7a0652-e4d0-485f-919e-6bed7a8e981c\") " Jul 09 19:34:41.397321 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:41.397304 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a7a0652-e4d0-485f-919e-6bed7a8e981c-workload-certs\") pod \"1a7a0652-e4d0-485f-919e-6bed7a8e981c\" (UID: \"1a7a0652-e4d0-485f-919e-6bed7a8e981c\") " Jul 09 19:34:41.397589 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:41.397342 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-467t4\" (UniqueName: \"kubernetes.io/projected/1a7a0652-e4d0-485f-919e-6bed7a8e981c-kube-api-access-467t4\") pod \"1a7a0652-e4d0-485f-919e-6bed7a8e981c\" (UID: \"1a7a0652-e4d0-485f-919e-6bed7a8e981c\") " Jul 09 19:34:41.397589 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:41.397366 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a7a0652-e4d0-485f-919e-6bed7a8e981c-istio-envoy\") pod \"1a7a0652-e4d0-485f-919e-6bed7a8e981c\" (UID: \"1a7a0652-e4d0-485f-919e-6bed7a8e981c\") " Jul 09 19:34:41.397589 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:41.397394 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a7a0652-e4d0-485f-919e-6bed7a8e981c-istiod-ca-cert\") pod \"1a7a0652-e4d0-485f-919e-6bed7a8e981c\" (UID: \"1a7a0652-e4d0-485f-919e-6bed7a8e981c\") " Jul 09 19:34:41.397589 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:41.397502 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a7a0652-e4d0-485f-919e-6bed7a8e981c-istio-data\") pod \"1a7a0652-e4d0-485f-919e-6bed7a8e981c\" (UID: \"1a7a0652-e4d0-485f-919e-6bed7a8e981c\") " Jul 09 19:34:41.397589 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:41.397566 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a7a0652-e4d0-485f-919e-6bed7a8e981c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1a7a0652-e4d0-485f-919e-6bed7a8e981c" (UID: "1a7a0652-e4d0-485f-919e-6bed7a8e981c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:34:41.397882 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:41.397788 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a7a0652-e4d0-485f-919e-6bed7a8e981c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1a7a0652-e4d0-485f-919e-6bed7a8e981c" (UID: "1a7a0652-e4d0-485f-919e-6bed7a8e981c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:34:41.397882 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:41.397834 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a7a0652-e4d0-485f-919e-6bed7a8e981c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1a7a0652-e4d0-485f-919e-6bed7a8e981c" (UID: "1a7a0652-e4d0-485f-919e-6bed7a8e981c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 19:34:41.398425 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:41.398397 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a7a0652-e4d0-485f-919e-6bed7a8e981c-istio-data" (OuterVolumeSpecName: "istio-data") pod "1a7a0652-e4d0-485f-919e-6bed7a8e981c" (UID: "1a7a0652-e4d0-485f-919e-6bed7a8e981c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:34:41.398510 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:41.398493 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a7a0652-e4d0-485f-919e-6bed7a8e981c-credential-socket\") pod \"1a7a0652-e4d0-485f-919e-6bed7a8e981c\" (UID: \"1a7a0652-e4d0-485f-919e-6bed7a8e981c\") " Jul 09 19:34:41.398555 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:41.398534 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a7a0652-e4d0-485f-919e-6bed7a8e981c-istio-podinfo\") pod \"1a7a0652-e4d0-485f-919e-6bed7a8e981c\" (UID: \"1a7a0652-e4d0-485f-919e-6bed7a8e981c\") " Jul 09 19:34:41.398727 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:41.398708 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a7a0652-e4d0-485f-919e-6bed7a8e981c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1a7a0652-e4d0-485f-919e-6bed7a8e981c" (UID: "1a7a0652-e4d0-485f-919e-6bed7a8e981c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:34:41.398945 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:41.398924 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a7a0652-e4d0-485f-919e-6bed7a8e981c-workload-socket\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:34:41.399009 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:41.398952 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a7a0652-e4d0-485f-919e-6bed7a8e981c-workload-certs\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:34:41.399009 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:41.398966 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a7a0652-e4d0-485f-919e-6bed7a8e981c-istiod-ca-cert\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:34:41.399009 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:41.398979 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a7a0652-e4d0-485f-919e-6bed7a8e981c-istio-data\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:34:41.399009 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:41.398992 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a7a0652-e4d0-485f-919e-6bed7a8e981c-credential-socket\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:34:41.399659 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:41.399595 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a7a0652-e4d0-485f-919e-6bed7a8e981c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1a7a0652-e4d0-485f-919e-6bed7a8e981c" (UID: "1a7a0652-e4d0-485f-919e-6bed7a8e981c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:34:41.399659 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:41.399647 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a7a0652-e4d0-485f-919e-6bed7a8e981c-istio-token" (OuterVolumeSpecName: "istio-token") pod "1a7a0652-e4d0-485f-919e-6bed7a8e981c" (UID: "1a7a0652-e4d0-485f-919e-6bed7a8e981c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:34:41.399842 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:41.399750 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a7a0652-e4d0-485f-919e-6bed7a8e981c-kube-api-access-467t4" (OuterVolumeSpecName: "kube-api-access-467t4") pod "1a7a0652-e4d0-485f-919e-6bed7a8e981c" (UID: "1a7a0652-e4d0-485f-919e-6bed7a8e981c"). InnerVolumeSpecName "kube-api-access-467t4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:34:41.400467 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:41.400448 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a7a0652-e4d0-485f-919e-6bed7a8e981c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1a7a0652-e4d0-485f-919e-6bed7a8e981c" (UID: "1a7a0652-e4d0-485f-919e-6bed7a8e981c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 19:34:41.499599 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:41.499570 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-467t4\" (UniqueName: \"kubernetes.io/projected/1a7a0652-e4d0-485f-919e-6bed7a8e981c-kube-api-access-467t4\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:34:41.499599 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:41.499594 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a7a0652-e4d0-485f-919e-6bed7a8e981c-istio-envoy\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:34:41.499599 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:41.499622 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a7a0652-e4d0-485f-919e-6bed7a8e981c-istio-podinfo\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:34:41.499837 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:41.499630 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a7a0652-e4d0-485f-919e-6bed7a8e981c-istio-token\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:34:41.598727 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:41.598700 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch"] Jul 09 19:34:41.600943 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:41.600914 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-bc9ef2-openshift-default-776c794cd5-gq4ch"] Jul 09 19:34:43.058501 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:43.058422 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a7a0652-e4d0-485f-919e-6bed7a8e981c" path="/var/lib/kubelet/pods/1a7a0652-e4d0-485f-919e-6bed7a8e981c/volumes" Jul 09 19:34:45.399732 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:45.399699 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz"] Jul 09 19:34:45.400195 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:45.400082 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a7a0652-e4d0-485f-919e-6bed7a8e981c" containerName="istio-proxy" Jul 09 19:34:45.400195 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:45.400093 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a7a0652-e4d0-485f-919e-6bed7a8e981c" containerName="istio-proxy" Jul 09 19:34:45.400195 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:45.400109 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a7a0652-e4d0-485f-919e-6bed7a8e981c" containerName="istio-proxy" Jul 09 19:34:45.400195 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:45.400115 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a7a0652-e4d0-485f-919e-6bed7a8e981c" containerName="istio-proxy" Jul 09 19:34:45.400195 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:45.400178 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a7a0652-e4d0-485f-919e-6bed7a8e981c" containerName="istio-proxy" Jul 09 19:34:45.400195 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:45.400185 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a7a0652-e4d0-485f-919e-6bed7a8e981c" containerName="istio-proxy" Jul 09 19:34:45.404474 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:45.404453 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz" Jul 09 19:34:45.407341 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:45.407317 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-a73219-openshift-default-dockercfg-5qx9v\"" Jul 09 19:34:45.414368 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:45.414345 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz"] Jul 09 19:34:45.437419 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:45.437393 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6815e3f1-22b0-472e-946a-6e6579cc901e-istio-data\") pod \"e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz\" (UID: \"6815e3f1-22b0-472e-946a-6e6579cc901e\") " pod="openshift-ingress/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz" Jul 09 19:34:45.437564 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:45.437422 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6815e3f1-22b0-472e-946a-6e6579cc901e-istio-token\") pod \"e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz\" (UID: \"6815e3f1-22b0-472e-946a-6e6579cc901e\") " pod="openshift-ingress/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz" Jul 09 19:34:45.437564 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:45.437441 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6815e3f1-22b0-472e-946a-6e6579cc901e-istio-envoy\") pod \"e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz\" (UID: \"6815e3f1-22b0-472e-946a-6e6579cc901e\") " pod="openshift-ingress/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz" Jul 09 19:34:45.437564 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:45.437482 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6815e3f1-22b0-472e-946a-6e6579cc901e-workload-socket\") pod \"e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz\" (UID: \"6815e3f1-22b0-472e-946a-6e6579cc901e\") " pod="openshift-ingress/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz" Jul 09 19:34:45.437564 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:45.437505 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvjpj\" (UniqueName: \"kubernetes.io/projected/6815e3f1-22b0-472e-946a-6e6579cc901e-kube-api-access-kvjpj\") pod \"e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz\" (UID: \"6815e3f1-22b0-472e-946a-6e6579cc901e\") " pod="openshift-ingress/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz" Jul 09 19:34:45.437564 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:45.437529 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6815e3f1-22b0-472e-946a-6e6579cc901e-istiod-ca-cert\") pod \"e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz\" (UID: \"6815e3f1-22b0-472e-946a-6e6579cc901e\") " pod="openshift-ingress/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz" Jul 09 19:34:45.437564 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:45.437547 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6815e3f1-22b0-472e-946a-6e6579cc901e-credential-socket\") pod \"e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz\" (UID: \"6815e3f1-22b0-472e-946a-6e6579cc901e\") " pod="openshift-ingress/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz" Jul 09 19:34:45.437819 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:45.437571 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6815e3f1-22b0-472e-946a-6e6579cc901e-workload-certs\") pod \"e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz\" (UID: \"6815e3f1-22b0-472e-946a-6e6579cc901e\") " pod="openshift-ingress/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz" Jul 09 19:34:45.437819 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:45.437593 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6815e3f1-22b0-472e-946a-6e6579cc901e-istio-podinfo\") pod \"e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz\" (UID: \"6815e3f1-22b0-472e-946a-6e6579cc901e\") " pod="openshift-ingress/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz" Jul 09 19:34:45.538983 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:45.538955 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6815e3f1-22b0-472e-946a-6e6579cc901e-istio-data\") pod \"e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz\" (UID: \"6815e3f1-22b0-472e-946a-6e6579cc901e\") " pod="openshift-ingress/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz" Jul 09 19:34:45.538983 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:45.538986 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6815e3f1-22b0-472e-946a-6e6579cc901e-istio-token\") pod \"e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz\" (UID: \"6815e3f1-22b0-472e-946a-6e6579cc901e\") " pod="openshift-ingress/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz" Jul 09 19:34:45.539260 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:45.539007 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6815e3f1-22b0-472e-946a-6e6579cc901e-istio-envoy\") pod \"e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz\" (UID: \"6815e3f1-22b0-472e-946a-6e6579cc901e\") " pod="openshift-ingress/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz" Jul 09 19:34:45.539260 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:45.539044 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6815e3f1-22b0-472e-946a-6e6579cc901e-workload-socket\") pod \"e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz\" (UID: \"6815e3f1-22b0-472e-946a-6e6579cc901e\") " pod="openshift-ingress/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz" Jul 09 19:34:45.539260 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:45.539070 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvjpj\" (UniqueName: \"kubernetes.io/projected/6815e3f1-22b0-472e-946a-6e6579cc901e-kube-api-access-kvjpj\") pod \"e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz\" (UID: \"6815e3f1-22b0-472e-946a-6e6579cc901e\") " pod="openshift-ingress/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz" Jul 09 19:34:45.539260 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:45.539097 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6815e3f1-22b0-472e-946a-6e6579cc901e-istiod-ca-cert\") pod \"e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz\" (UID: \"6815e3f1-22b0-472e-946a-6e6579cc901e\") " pod="openshift-ingress/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz" Jul 09 19:34:45.539260 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:45.539118 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6815e3f1-22b0-472e-946a-6e6579cc901e-credential-socket\") pod \"e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz\" (UID: \"6815e3f1-22b0-472e-946a-6e6579cc901e\") " pod="openshift-ingress/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz" Jul 09 19:34:45.539260 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:45.539146 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6815e3f1-22b0-472e-946a-6e6579cc901e-workload-certs\") pod \"e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz\" (UID: \"6815e3f1-22b0-472e-946a-6e6579cc901e\") " pod="openshift-ingress/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz" Jul 09 19:34:45.539260 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:45.539169 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6815e3f1-22b0-472e-946a-6e6579cc901e-istio-podinfo\") pod \"e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz\" (UID: \"6815e3f1-22b0-472e-946a-6e6579cc901e\") " pod="openshift-ingress/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz" Jul 09 19:34:45.539551 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:45.539400 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6815e3f1-22b0-472e-946a-6e6579cc901e-istio-data\") pod \"e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz\" (UID: \"6815e3f1-22b0-472e-946a-6e6579cc901e\") " pod="openshift-ingress/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz" Jul 09 19:34:45.539673 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:45.539646 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6815e3f1-22b0-472e-946a-6e6579cc901e-workload-socket\") pod \"e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz\" (UID: \"6815e3f1-22b0-472e-946a-6e6579cc901e\") " pod="openshift-ingress/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz" Jul 09 19:34:45.539811 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:45.539729 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6815e3f1-22b0-472e-946a-6e6579cc901e-credential-socket\") pod \"e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz\" (UID: \"6815e3f1-22b0-472e-946a-6e6579cc901e\") " pod="openshift-ingress/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz" Jul 09 19:34:45.539811 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:45.539754 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6815e3f1-22b0-472e-946a-6e6579cc901e-workload-certs\") pod \"e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz\" (UID: \"6815e3f1-22b0-472e-946a-6e6579cc901e\") " pod="openshift-ingress/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz" Jul 09 19:34:45.539983 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:45.539960 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6815e3f1-22b0-472e-946a-6e6579cc901e-istiod-ca-cert\") pod \"e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz\" (UID: \"6815e3f1-22b0-472e-946a-6e6579cc901e\") " pod="openshift-ingress/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz" Jul 09 19:34:45.541202 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:45.541184 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6815e3f1-22b0-472e-946a-6e6579cc901e-istio-envoy\") pod \"e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz\" (UID: \"6815e3f1-22b0-472e-946a-6e6579cc901e\") " pod="openshift-ingress/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz" Jul 09 19:34:45.541595 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:45.541578 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6815e3f1-22b0-472e-946a-6e6579cc901e-istio-podinfo\") pod \"e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz\" (UID: \"6815e3f1-22b0-472e-946a-6e6579cc901e\") " pod="openshift-ingress/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz" Jul 09 19:34:45.547287 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:45.547255 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6815e3f1-22b0-472e-946a-6e6579cc901e-istio-token\") pod \"e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz\" (UID: \"6815e3f1-22b0-472e-946a-6e6579cc901e\") " pod="openshift-ingress/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz" Jul 09 19:34:45.547824 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:45.547778 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvjpj\" (UniqueName: \"kubernetes.io/projected/6815e3f1-22b0-472e-946a-6e6579cc901e-kube-api-access-kvjpj\") pod \"e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz\" (UID: \"6815e3f1-22b0-472e-946a-6e6579cc901e\") " pod="openshift-ingress/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz" Jul 09 19:34:45.718109 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:45.718056 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz" Jul 09 19:34:45.864575 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:45.864546 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz"] Jul 09 19:34:45.865872 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:34:45.865845 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6815e3f1_22b0_472e_946a_6e6579cc901e.slice/crio-ed39743f91e980921e3591bc4b1725ea0d06990a50f80c0fd9d15b753731c0a8 WatchSource:0}: Error finding container ed39743f91e980921e3591bc4b1725ea0d06990a50f80c0fd9d15b753731c0a8: Status 404 returned error can't find the container with id ed39743f91e980921e3591bc4b1725ea0d06990a50f80c0fd9d15b753731c0a8 Jul 09 19:34:45.868359 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:45.868330 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:34:45.868460 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:45.868390 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:34:45.868460 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:45.868435 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:34:46.303151 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:46.303114 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz" event={"ID":"6815e3f1-22b0-472e-946a-6e6579cc901e","Type":"ContainerStarted","Data":"befd18598503afd85a62e3b96f7bb400d8d411ed2d3677b28d09fa48854bf535"} Jul 09 19:34:46.303343 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:46.303165 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz" event={"ID":"6815e3f1-22b0-472e-946a-6e6579cc901e","Type":"ContainerStarted","Data":"ed39743f91e980921e3591bc4b1725ea0d06990a50f80c0fd9d15b753731c0a8"} Jul 09 19:34:46.327967 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:46.327919 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz" podStartSLOduration=1.327900517 podStartE2EDuration="1.327900517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:34:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:34:46.326553324 +0000 UTC m=+2155.834333100" watchObservedRunningTime="2026-07-09 19:34:46.327900517 +0000 UTC m=+2155.835680293" Jul 09 19:34:46.719156 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:46.719122 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz" Jul 09 19:34:46.724182 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:46.724159 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz" Jul 09 19:34:47.310527 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:47.310497 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz" Jul 09 19:34:47.311527 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:47.311502 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz" Jul 09 19:34:48.668451 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:48.668413 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:34:55.302798 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:55.302752 2566 patch_prober.go:28] interesting pod/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 09 19:34:55.303306 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:55.302818 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5" podUID="5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 09 19:34:57.371235 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:57.371201 2566 generic.go:356] "Generic (PLEG): container finished" podID="5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db" containerID="afb189f0872b9ae35d402f009495be412517bedf81846c9455d5c4cf1a63df3b" exitCode=0 Jul 09 19:34:57.371749 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:57.371247 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5" event={"ID":"5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db","Type":"ContainerDied","Data":"afb189f0872b9ae35d402f009495be412517bedf81846c9455d5c4cf1a63df3b"} Jul 09 19:34:57.371749 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:57.371652 2566 scope.go:117] "RemoveContainer" containerID="afb189f0872b9ae35d402f009495be412517bedf81846c9455d5c4cf1a63df3b" Jul 09 19:34:57.372558 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:57.372532 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:34:57.372681 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:57.372596 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:34:57.372681 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:57.372649 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:34:58.379666 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:58.379631 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5" event={"ID":"5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db","Type":"ContainerStarted","Data":"f6fd1605ccb3a83164ccbf023f4a6ec47e10b880404321d1a65861253427a2cb"} Jul 09 19:34:59.169489 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:59.169457 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5" Jul 09 19:34:59.169681 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:59.169649 2566 patch_prober.go:28] interesting pod/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 09 19:34:59.169737 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:34:59.169704 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5" podUID="5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 09 19:35:00.169298 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:00.169267 2566 patch_prober.go:28] interesting pod/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 09 19:35:00.169790 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:00.169764 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5" podUID="5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 09 19:35:01.169748 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:01.169715 2566 patch_prober.go:28] interesting pod/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 09 19:35:01.170134 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:01.169771 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5" podUID="5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 09 19:35:02.170238 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:02.170198 2566 patch_prober.go:28] interesting pod/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 09 19:35:02.170667 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:02.170253 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5" podUID="5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 09 19:35:03.173175 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:03.173145 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5" Jul 09 19:35:03.173541 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:03.173492 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5" Jul 09 19:35:03.174132 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:03.174113 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5" Jul 09 19:35:05.353398 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:05.353362 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:35:06.431754 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:06.431721 2566 generic.go:356] "Generic (PLEG): container finished" podID="6815e3f1-22b0-472e-946a-6e6579cc901e" containerID="befd18598503afd85a62e3b96f7bb400d8d411ed2d3677b28d09fa48854bf535" exitCode=0 Jul 09 19:35:06.432167 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:06.431797 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz" event={"ID":"6815e3f1-22b0-472e-946a-6e6579cc901e","Type":"ContainerDied","Data":"befd18598503afd85a62e3b96f7bb400d8d411ed2d3677b28d09fa48854bf535"} Jul 09 19:35:06.432208 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:06.432196 2566 scope.go:117] "RemoveContainer" containerID="befd18598503afd85a62e3b96f7bb400d8d411ed2d3677b28d09fa48854bf535" Jul 09 19:35:06.433210 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:06.433181 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:35:06.433304 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:06.433252 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:35:06.433304 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:06.433299 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:35:07.439931 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:07.439891 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz" event={"ID":"6815e3f1-22b0-472e-946a-6e6579cc901e","Type":"ContainerStarted","Data":"81f71a475688438f2bf78d313ede18ae64bf03c5b53a12f1817b44a95afff53a"} Jul 09 19:35:07.652045 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:07.652015 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5"] Jul 09 19:35:07.652243 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:07.652220 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5" podUID="5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db" containerName="istio-proxy" containerID="cri-o://f6fd1605ccb3a83164ccbf023f4a6ec47e10b880404321d1a65861253427a2cb" gracePeriod=30 Jul 09 19:35:07.721693 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:07.721003 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz" Jul 09 19:35:07.721876 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:07.721702 2566 patch_prober.go:28] interesting pod/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Jul 09 19:35:07.721876 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:07.721774 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz" podUID="6815e3f1-22b0-472e-946a-6e6579cc901e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Jul 09 19:35:08.719061 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:08.719022 2566 patch_prober.go:28] interesting pod/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Jul 09 19:35:08.719708 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:08.719671 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz" podUID="6815e3f1-22b0-472e-946a-6e6579cc901e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Jul 09 19:35:09.718627 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:09.718573 2566 patch_prober.go:28] interesting pod/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Jul 09 19:35:09.718932 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:09.718903 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz" podUID="6815e3f1-22b0-472e-946a-6e6579cc901e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Jul 09 19:35:10.719328 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:10.719283 2566 patch_prober.go:28] interesting pod/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Jul 09 19:35:10.719866 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:10.719358 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz" podUID="6815e3f1-22b0-472e-946a-6e6579cc901e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Jul 09 19:35:11.721910 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:11.721880 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz" Jul 09 19:35:11.722325 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:11.722250 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz" Jul 09 19:35:11.723060 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:11.723041 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz" Jul 09 19:35:12.915078 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:12.915013 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5" Jul 09 19:35:12.996629 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:12.996561 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db-workload-certs\") pod \"5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db\" (UID: \"5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db\") " Jul 09 19:35:12.996823 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:12.996684 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db-istio-envoy\") pod \"5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db\" (UID: \"5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db\") " Jul 09 19:35:12.996823 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:12.996719 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db-istiod-ca-cert\") pod \"5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db\" (UID: \"5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db\") " Jul 09 19:35:12.996823 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:12.996795 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db-credential-socket\") pod \"5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db\" (UID: \"5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db\") " Jul 09 19:35:12.996823 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:12.996820 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db-istio-token\") pod \"5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db\" (UID: \"5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db\") " Jul 09 19:35:12.997051 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:12.996856 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-grnlg\" (UniqueName: \"kubernetes.io/projected/5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db-kube-api-access-grnlg\") pod \"5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db\" (UID: \"5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db\") " Jul 09 19:35:12.997051 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:12.996857 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db" (UID: "5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:35:12.997051 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:12.996890 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db-istio-podinfo\") pod \"5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db\" (UID: \"5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db\") " Jul 09 19:35:12.997051 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:12.996925 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db-istio-data\") pod \"5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db\" (UID: \"5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db\") " Jul 09 19:35:12.997051 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:12.996961 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db-workload-socket\") pod \"5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db\" (UID: \"5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db\") " Jul 09 19:35:12.997303 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:12.997209 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db-workload-certs\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:35:12.997361 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:12.997302 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db" (UID: "5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:35:12.997470 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:12.997450 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db" (UID: "5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:35:12.997707 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:12.997686 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db" (UID: "5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 19:35:12.998419 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:12.998375 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db-istio-data" (OuterVolumeSpecName: "istio-data") pod "5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db" (UID: "5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:35:12.999288 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:12.999266 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db-istio-token" (OuterVolumeSpecName: "istio-token") pod "5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db" (UID: "5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:35:12.999396 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:12.999369 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db" (UID: "5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:35:12.999876 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:12.999850 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db-kube-api-access-grnlg" (OuterVolumeSpecName: "kube-api-access-grnlg") pod "5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db" (UID: "5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db"). InnerVolumeSpecName "kube-api-access-grnlg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:35:12.999957 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:12.999857 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db" (UID: "5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 19:35:13.098166 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:13.098135 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db-istiod-ca-cert\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:35:13.098166 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:13.098163 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db-credential-socket\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:35:13.098337 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:13.098173 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db-istio-token\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:35:13.098337 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:13.098183 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grnlg\" (UniqueName: \"kubernetes.io/projected/5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db-kube-api-access-grnlg\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:35:13.098337 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:13.098191 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db-istio-podinfo\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:35:13.098337 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:13.098200 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db-istio-data\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:35:13.098337 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:13.098208 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db-workload-socket\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:35:13.098337 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:13.098218 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db-istio-envoy\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:35:13.474598 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:13.474560 2566 generic.go:356] "Generic (PLEG): container finished" podID="5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db" containerID="f6fd1605ccb3a83164ccbf023f4a6ec47e10b880404321d1a65861253427a2cb" exitCode=0 Jul 09 19:35:13.474598 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:13.474628 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5" Jul 09 19:35:13.474834 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:13.474645 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5" event={"ID":"5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db","Type":"ContainerDied","Data":"f6fd1605ccb3a83164ccbf023f4a6ec47e10b880404321d1a65861253427a2cb"} Jul 09 19:35:13.474834 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:13.474700 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5" event={"ID":"5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db","Type":"ContainerDied","Data":"692c05c36176306d8d150959b9602cb503f212081e693b4f776212b107a6bf3e"} Jul 09 19:35:13.474834 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:13.474725 2566 scope.go:117] "RemoveContainer" containerID="f6fd1605ccb3a83164ccbf023f4a6ec47e10b880404321d1a65861253427a2cb" Jul 09 19:35:13.485755 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:13.485730 2566 scope.go:117] "RemoveContainer" containerID="afb189f0872b9ae35d402f009495be412517bedf81846c9455d5c4cf1a63df3b" Jul 09 19:35:13.493783 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:13.493738 2566 scope.go:117] "RemoveContainer" containerID="f6fd1605ccb3a83164ccbf023f4a6ec47e10b880404321d1a65861253427a2cb" Jul 09 19:35:13.494038 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:35:13.494014 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6fd1605ccb3a83164ccbf023f4a6ec47e10b880404321d1a65861253427a2cb\": container with ID starting with f6fd1605ccb3a83164ccbf023f4a6ec47e10b880404321d1a65861253427a2cb not found: ID does not exist" containerID="f6fd1605ccb3a83164ccbf023f4a6ec47e10b880404321d1a65861253427a2cb" Jul 09 19:35:13.494106 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:13.494048 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6fd1605ccb3a83164ccbf023f4a6ec47e10b880404321d1a65861253427a2cb"} err="failed to get container status \"f6fd1605ccb3a83164ccbf023f4a6ec47e10b880404321d1a65861253427a2cb\": rpc error: code = NotFound desc = could not find container \"f6fd1605ccb3a83164ccbf023f4a6ec47e10b880404321d1a65861253427a2cb\": container with ID starting with f6fd1605ccb3a83164ccbf023f4a6ec47e10b880404321d1a65861253427a2cb not found: ID does not exist" Jul 09 19:35:13.494106 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:13.494075 2566 scope.go:117] "RemoveContainer" containerID="afb189f0872b9ae35d402f009495be412517bedf81846c9455d5c4cf1a63df3b" Jul 09 19:35:13.494332 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:35:13.494313 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afb189f0872b9ae35d402f009495be412517bedf81846c9455d5c4cf1a63df3b\": container with ID starting with afb189f0872b9ae35d402f009495be412517bedf81846c9455d5c4cf1a63df3b not found: ID does not exist" containerID="afb189f0872b9ae35d402f009495be412517bedf81846c9455d5c4cf1a63df3b" Jul 09 19:35:13.494450 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:13.494334 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afb189f0872b9ae35d402f009495be412517bedf81846c9455d5c4cf1a63df3b"} err="failed to get container status \"afb189f0872b9ae35d402f009495be412517bedf81846c9455d5c4cf1a63df3b\": rpc error: code = NotFound desc = could not find container \"afb189f0872b9ae35d402f009495be412517bedf81846c9455d5c4cf1a63df3b\": container with ID starting with afb189f0872b9ae35d402f009495be412517bedf81846c9455d5c4cf1a63df3b not found: ID does not exist" Jul 09 19:35:13.497276 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:13.497255 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5"] Jul 09 19:35:13.503988 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:13.503959 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-47c272-openshift-default-85cd45bc65-gs2b5"] Jul 09 19:35:14.595350 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:14.595317 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz"] Jul 09 19:35:14.595978 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:14.595534 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz" podUID="6815e3f1-22b0-472e-946a-6e6579cc901e" containerName="istio-proxy" containerID="cri-o://81f71a475688438f2bf78d313ede18ae64bf03c5b53a12f1817b44a95afff53a" gracePeriod=30 Jul 09 19:35:15.057075 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:15.057041 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db" path="/var/lib/kubelet/pods/5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db/volumes" Jul 09 19:35:15.461374 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:15.461341 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc"] Jul 09 19:35:15.461946 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:15.461929 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db" containerName="istio-proxy" Jul 09 19:35:15.462005 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:15.461949 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db" containerName="istio-proxy" Jul 09 19:35:15.462005 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:15.461981 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db" containerName="istio-proxy" Jul 09 19:35:15.462005 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:15.461990 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db" containerName="istio-proxy" Jul 09 19:35:15.462108 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:15.462089 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db" containerName="istio-proxy" Jul 09 19:35:15.462108 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:15.462102 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e6ccf66-cddf-4928-bdf9-9ed2b1bd42db" containerName="istio-proxy" Jul 09 19:35:15.465438 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:15.465419 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc" Jul 09 19:35:15.468308 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:15.468287 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-77ffac-openshift-default-dockercfg-hgh6v\"" Jul 09 19:35:15.476568 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:15.476547 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc"] Jul 09 19:35:15.623979 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:15.623945 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1afb71de-d888-46a9-ab0f-18e2ab8279b5-workload-certs\") pod \"e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc\" (UID: \"1afb71de-d888-46a9-ab0f-18e2ab8279b5\") " pod="openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc" Jul 09 19:35:15.624385 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:15.624001 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1afb71de-d888-46a9-ab0f-18e2ab8279b5-istio-data\") pod \"e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc\" (UID: \"1afb71de-d888-46a9-ab0f-18e2ab8279b5\") " pod="openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc" Jul 09 19:35:15.624385 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:15.624069 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1afb71de-d888-46a9-ab0f-18e2ab8279b5-workload-socket\") pod \"e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc\" (UID: \"1afb71de-d888-46a9-ab0f-18e2ab8279b5\") " pod="openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc" Jul 09 19:35:15.624385 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:15.624106 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1afb71de-d888-46a9-ab0f-18e2ab8279b5-credential-socket\") pod \"e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc\" (UID: \"1afb71de-d888-46a9-ab0f-18e2ab8279b5\") " pod="openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc" Jul 09 19:35:15.624385 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:15.624159 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1afb71de-d888-46a9-ab0f-18e2ab8279b5-istiod-ca-cert\") pod \"e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc\" (UID: \"1afb71de-d888-46a9-ab0f-18e2ab8279b5\") " pod="openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc" Jul 09 19:35:15.624385 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:15.624194 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvtwc\" (UniqueName: \"kubernetes.io/projected/1afb71de-d888-46a9-ab0f-18e2ab8279b5-kube-api-access-qvtwc\") pod \"e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc\" (UID: \"1afb71de-d888-46a9-ab0f-18e2ab8279b5\") " pod="openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc" Jul 09 19:35:15.624385 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:15.624226 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1afb71de-d888-46a9-ab0f-18e2ab8279b5-istio-podinfo\") pod \"e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc\" (UID: \"1afb71de-d888-46a9-ab0f-18e2ab8279b5\") " pod="openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc" Jul 09 19:35:15.624385 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:15.624243 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1afb71de-d888-46a9-ab0f-18e2ab8279b5-istio-envoy\") pod \"e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc\" (UID: \"1afb71de-d888-46a9-ab0f-18e2ab8279b5\") " pod="openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc" Jul 09 19:35:15.624385 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:15.624283 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1afb71de-d888-46a9-ab0f-18e2ab8279b5-istio-token\") pod \"e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc\" (UID: \"1afb71de-d888-46a9-ab0f-18e2ab8279b5\") " pod="openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc" Jul 09 19:35:15.725335 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:15.725247 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1afb71de-d888-46a9-ab0f-18e2ab8279b5-istio-data\") pod \"e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc\" (UID: \"1afb71de-d888-46a9-ab0f-18e2ab8279b5\") " pod="openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc" Jul 09 19:35:15.725335 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:15.725292 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1afb71de-d888-46a9-ab0f-18e2ab8279b5-workload-socket\") pod \"e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc\" (UID: \"1afb71de-d888-46a9-ab0f-18e2ab8279b5\") " pod="openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc" Jul 09 19:35:15.725335 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:15.725313 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1afb71de-d888-46a9-ab0f-18e2ab8279b5-credential-socket\") pod \"e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc\" (UID: \"1afb71de-d888-46a9-ab0f-18e2ab8279b5\") " pod="openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc" Jul 09 19:35:15.725655 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:15.725357 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1afb71de-d888-46a9-ab0f-18e2ab8279b5-istiod-ca-cert\") pod \"e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc\" (UID: \"1afb71de-d888-46a9-ab0f-18e2ab8279b5\") " pod="openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc" Jul 09 19:35:15.725655 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:15.725395 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvtwc\" (UniqueName: \"kubernetes.io/projected/1afb71de-d888-46a9-ab0f-18e2ab8279b5-kube-api-access-qvtwc\") pod \"e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc\" (UID: \"1afb71de-d888-46a9-ab0f-18e2ab8279b5\") " pod="openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc" Jul 09 19:35:15.725655 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:15.725429 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1afb71de-d888-46a9-ab0f-18e2ab8279b5-istio-podinfo\") pod \"e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc\" (UID: \"1afb71de-d888-46a9-ab0f-18e2ab8279b5\") " pod="openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc" Jul 09 19:35:15.725655 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:15.725453 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1afb71de-d888-46a9-ab0f-18e2ab8279b5-istio-envoy\") pod \"e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc\" (UID: \"1afb71de-d888-46a9-ab0f-18e2ab8279b5\") " pod="openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc" Jul 09 19:35:15.725867 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:15.725698 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1afb71de-d888-46a9-ab0f-18e2ab8279b5-istio-token\") pod \"e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc\" (UID: \"1afb71de-d888-46a9-ab0f-18e2ab8279b5\") " pod="openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc" Jul 09 19:35:15.725867 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:15.725758 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1afb71de-d888-46a9-ab0f-18e2ab8279b5-istio-data\") pod \"e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc\" (UID: \"1afb71de-d888-46a9-ab0f-18e2ab8279b5\") " pod="openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc" Jul 09 19:35:15.725867 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:15.725783 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1afb71de-d888-46a9-ab0f-18e2ab8279b5-workload-certs\") pod \"e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc\" (UID: \"1afb71de-d888-46a9-ab0f-18e2ab8279b5\") " pod="openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc" Jul 09 19:35:15.726031 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:15.725918 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1afb71de-d888-46a9-ab0f-18e2ab8279b5-credential-socket\") pod \"e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc\" (UID: \"1afb71de-d888-46a9-ab0f-18e2ab8279b5\") " pod="openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc" Jul 09 19:35:15.726169 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:15.726146 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1afb71de-d888-46a9-ab0f-18e2ab8279b5-workload-certs\") pod \"e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc\" (UID: \"1afb71de-d888-46a9-ab0f-18e2ab8279b5\") " pod="openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc" Jul 09 19:35:15.726240 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:15.726202 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1afb71de-d888-46a9-ab0f-18e2ab8279b5-workload-socket\") pod \"e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc\" (UID: \"1afb71de-d888-46a9-ab0f-18e2ab8279b5\") " pod="openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc" Jul 09 19:35:15.726293 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:15.726238 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1afb71de-d888-46a9-ab0f-18e2ab8279b5-istiod-ca-cert\") pod \"e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc\" (UID: \"1afb71de-d888-46a9-ab0f-18e2ab8279b5\") " pod="openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc" Jul 09 19:35:15.727924 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:15.727903 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1afb71de-d888-46a9-ab0f-18e2ab8279b5-istio-envoy\") pod \"e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc\" (UID: \"1afb71de-d888-46a9-ab0f-18e2ab8279b5\") " pod="openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc" Jul 09 19:35:15.728038 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:15.727954 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1afb71de-d888-46a9-ab0f-18e2ab8279b5-istio-podinfo\") pod \"e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc\" (UID: \"1afb71de-d888-46a9-ab0f-18e2ab8279b5\") " pod="openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc" Jul 09 19:35:15.739670 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:15.739643 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1afb71de-d888-46a9-ab0f-18e2ab8279b5-istio-token\") pod \"e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc\" (UID: \"1afb71de-d888-46a9-ab0f-18e2ab8279b5\") " pod="openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc" Jul 09 19:35:15.744199 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:15.744178 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvtwc\" (UniqueName: \"kubernetes.io/projected/1afb71de-d888-46a9-ab0f-18e2ab8279b5-kube-api-access-qvtwc\") pod \"e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc\" (UID: \"1afb71de-d888-46a9-ab0f-18e2ab8279b5\") " pod="openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc" Jul 09 19:35:15.777137 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:15.777112 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc" Jul 09 19:35:15.915341 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:15.915244 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc"] Jul 09 19:35:15.917828 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:35:15.917806 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1afb71de_d888_46a9_ab0f_18e2ab8279b5.slice/crio-bc015b1219409c88ee764a19fd9c64374cf0583380b95194ddd91c78c63856d3 WatchSource:0}: Error finding container bc015b1219409c88ee764a19fd9c64374cf0583380b95194ddd91c78c63856d3: Status 404 returned error can't find the container with id bc015b1219409c88ee764a19fd9c64374cf0583380b95194ddd91c78c63856d3 Jul 09 19:35:15.919972 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:15.919917 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:35:15.920074 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:15.920019 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:35:15.920074 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:15.920063 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:35:16.493142 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:16.493111 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc" event={"ID":"1afb71de-d888-46a9-ab0f-18e2ab8279b5","Type":"ContainerStarted","Data":"fee05fd4c465d7497dc12a707a81a4aee8e297c0d4b21909a14e1c59b58eee87"} Jul 09 19:35:16.493321 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:16.493159 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc" event={"ID":"1afb71de-d888-46a9-ab0f-18e2ab8279b5","Type":"ContainerStarted","Data":"bc015b1219409c88ee764a19fd9c64374cf0583380b95194ddd91c78c63856d3"} Jul 09 19:35:16.516083 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:16.516033 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc" podStartSLOduration=1.5160170549999998 podStartE2EDuration="1.516017055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:35:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:35:16.513808922 +0000 UTC m=+2186.021588695" watchObservedRunningTime="2026-07-09 19:35:16.516017055 +0000 UTC m=+2186.023796828" Jul 09 19:35:16.777841 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:16.777765 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc" Jul 09 19:35:16.782770 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:16.782747 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc" Jul 09 19:35:17.499120 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:17.499087 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc" Jul 09 19:35:17.500075 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:17.500052 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc" Jul 09 19:35:19.363962 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:19.363925 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:35:19.859033 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:19.859004 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz" Jul 09 19:35:19.964753 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:19.964728 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6815e3f1-22b0-472e-946a-6e6579cc901e-workload-certs\") pod \"6815e3f1-22b0-472e-946a-6e6579cc901e\" (UID: \"6815e3f1-22b0-472e-946a-6e6579cc901e\") " Jul 09 19:35:19.964915 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:19.964762 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6815e3f1-22b0-472e-946a-6e6579cc901e-istio-data\") pod \"6815e3f1-22b0-472e-946a-6e6579cc901e\" (UID: \"6815e3f1-22b0-472e-946a-6e6579cc901e\") " Jul 09 19:35:19.964915 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:19.964784 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6815e3f1-22b0-472e-946a-6e6579cc901e-istio-envoy\") pod \"6815e3f1-22b0-472e-946a-6e6579cc901e\" (UID: \"6815e3f1-22b0-472e-946a-6e6579cc901e\") " Jul 09 19:35:19.964915 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:19.964807 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvjpj\" (UniqueName: \"kubernetes.io/projected/6815e3f1-22b0-472e-946a-6e6579cc901e-kube-api-access-kvjpj\") pod \"6815e3f1-22b0-472e-946a-6e6579cc901e\" (UID: \"6815e3f1-22b0-472e-946a-6e6579cc901e\") " Jul 09 19:35:19.964915 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:19.964873 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6815e3f1-22b0-472e-946a-6e6579cc901e-istiod-ca-cert\") pod \"6815e3f1-22b0-472e-946a-6e6579cc901e\" (UID: \"6815e3f1-22b0-472e-946a-6e6579cc901e\") " Jul 09 19:35:19.965095 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:19.964917 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6815e3f1-22b0-472e-946a-6e6579cc901e-credential-socket\") pod \"6815e3f1-22b0-472e-946a-6e6579cc901e\" (UID: \"6815e3f1-22b0-472e-946a-6e6579cc901e\") " Jul 09 19:35:19.965095 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:19.965055 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6815e3f1-22b0-472e-946a-6e6579cc901e-istio-podinfo\") pod \"6815e3f1-22b0-472e-946a-6e6579cc901e\" (UID: \"6815e3f1-22b0-472e-946a-6e6579cc901e\") " Jul 09 19:35:19.965209 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:19.965132 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6815e3f1-22b0-472e-946a-6e6579cc901e-istio-token\") pod \"6815e3f1-22b0-472e-946a-6e6579cc901e\" (UID: \"6815e3f1-22b0-472e-946a-6e6579cc901e\") " Jul 09 19:35:19.965209 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:19.965171 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6815e3f1-22b0-472e-946a-6e6579cc901e-workload-socket\") pod \"6815e3f1-22b0-472e-946a-6e6579cc901e\" (UID: \"6815e3f1-22b0-472e-946a-6e6579cc901e\") " Jul 09 19:35:19.965209 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:19.965060 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6815e3f1-22b0-472e-946a-6e6579cc901e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6815e3f1-22b0-472e-946a-6e6579cc901e" (UID: "6815e3f1-22b0-472e-946a-6e6579cc901e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:35:19.965209 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:19.965168 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6815e3f1-22b0-472e-946a-6e6579cc901e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6815e3f1-22b0-472e-946a-6e6579cc901e" (UID: "6815e3f1-22b0-472e-946a-6e6579cc901e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:35:19.965460 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:19.965301 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6815e3f1-22b0-472e-946a-6e6579cc901e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6815e3f1-22b0-472e-946a-6e6579cc901e" (UID: "6815e3f1-22b0-472e-946a-6e6579cc901e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 19:35:19.965599 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:19.965564 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6815e3f1-22b0-472e-946a-6e6579cc901e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6815e3f1-22b0-472e-946a-6e6579cc901e" (UID: "6815e3f1-22b0-472e-946a-6e6579cc901e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:35:19.965699 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:19.965652 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6815e3f1-22b0-472e-946a-6e6579cc901e-workload-certs\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:35:19.965699 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:19.965674 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6815e3f1-22b0-472e-946a-6e6579cc901e-istiod-ca-cert\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:35:19.965699 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:19.965687 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6815e3f1-22b0-472e-946a-6e6579cc901e-credential-socket\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:35:19.965837 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:19.965812 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6815e3f1-22b0-472e-946a-6e6579cc901e-istio-data" (OuterVolumeSpecName: "istio-data") pod "6815e3f1-22b0-472e-946a-6e6579cc901e" (UID: "6815e3f1-22b0-472e-946a-6e6579cc901e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:35:19.967103 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:19.967078 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6815e3f1-22b0-472e-946a-6e6579cc901e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6815e3f1-22b0-472e-946a-6e6579cc901e" (UID: "6815e3f1-22b0-472e-946a-6e6579cc901e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 19:35:19.967200 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:19.967142 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6815e3f1-22b0-472e-946a-6e6579cc901e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6815e3f1-22b0-472e-946a-6e6579cc901e" (UID: "6815e3f1-22b0-472e-946a-6e6579cc901e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:35:19.967454 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:19.967437 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6815e3f1-22b0-472e-946a-6e6579cc901e-istio-token" (OuterVolumeSpecName: "istio-token") pod "6815e3f1-22b0-472e-946a-6e6579cc901e" (UID: "6815e3f1-22b0-472e-946a-6e6579cc901e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:35:19.967517 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:19.967458 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6815e3f1-22b0-472e-946a-6e6579cc901e-kube-api-access-kvjpj" (OuterVolumeSpecName: "kube-api-access-kvjpj") pod "6815e3f1-22b0-472e-946a-6e6579cc901e" (UID: "6815e3f1-22b0-472e-946a-6e6579cc901e"). InnerVolumeSpecName "kube-api-access-kvjpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:35:20.066251 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:20.066218 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6815e3f1-22b0-472e-946a-6e6579cc901e-istio-podinfo\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:35:20.066251 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:20.066247 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6815e3f1-22b0-472e-946a-6e6579cc901e-istio-token\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:35:20.066251 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:20.066260 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6815e3f1-22b0-472e-946a-6e6579cc901e-workload-socket\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:35:20.066477 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:20.066271 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6815e3f1-22b0-472e-946a-6e6579cc901e-istio-data\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:35:20.066477 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:20.066283 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6815e3f1-22b0-472e-946a-6e6579cc901e-istio-envoy\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:35:20.066477 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:20.066294 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvjpj\" (UniqueName: \"kubernetes.io/projected/6815e3f1-22b0-472e-946a-6e6579cc901e-kube-api-access-kvjpj\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:35:20.519389 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:20.519358 2566 generic.go:356] "Generic (PLEG): container finished" podID="6815e3f1-22b0-472e-946a-6e6579cc901e" containerID="81f71a475688438f2bf78d313ede18ae64bf03c5b53a12f1817b44a95afff53a" exitCode=0 Jul 09 19:35:20.519850 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:20.519403 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz" Jul 09 19:35:20.519850 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:20.519443 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz" event={"ID":"6815e3f1-22b0-472e-946a-6e6579cc901e","Type":"ContainerDied","Data":"81f71a475688438f2bf78d313ede18ae64bf03c5b53a12f1817b44a95afff53a"} Jul 09 19:35:20.519850 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:20.519509 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz" event={"ID":"6815e3f1-22b0-472e-946a-6e6579cc901e","Type":"ContainerDied","Data":"ed39743f91e980921e3591bc4b1725ea0d06990a50f80c0fd9d15b753731c0a8"} Jul 09 19:35:20.519850 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:20.519544 2566 scope.go:117] "RemoveContainer" containerID="81f71a475688438f2bf78d313ede18ae64bf03c5b53a12f1817b44a95afff53a" Jul 09 19:35:20.530396 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:20.530375 2566 scope.go:117] "RemoveContainer" containerID="befd18598503afd85a62e3b96f7bb400d8d411ed2d3677b28d09fa48854bf535" Jul 09 19:35:20.538533 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:20.538514 2566 scope.go:117] "RemoveContainer" containerID="81f71a475688438f2bf78d313ede18ae64bf03c5b53a12f1817b44a95afff53a" Jul 09 19:35:20.538795 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:35:20.538775 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81f71a475688438f2bf78d313ede18ae64bf03c5b53a12f1817b44a95afff53a\": container with ID starting with 81f71a475688438f2bf78d313ede18ae64bf03c5b53a12f1817b44a95afff53a not found: ID does not exist" containerID="81f71a475688438f2bf78d313ede18ae64bf03c5b53a12f1817b44a95afff53a" Jul 09 19:35:20.538856 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:20.538805 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81f71a475688438f2bf78d313ede18ae64bf03c5b53a12f1817b44a95afff53a"} err="failed to get container status \"81f71a475688438f2bf78d313ede18ae64bf03c5b53a12f1817b44a95afff53a\": rpc error: code = NotFound desc = could not find container \"81f71a475688438f2bf78d313ede18ae64bf03c5b53a12f1817b44a95afff53a\": container with ID starting with 81f71a475688438f2bf78d313ede18ae64bf03c5b53a12f1817b44a95afff53a not found: ID does not exist" Jul 09 19:35:20.538856 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:20.538821 2566 scope.go:117] "RemoveContainer" containerID="befd18598503afd85a62e3b96f7bb400d8d411ed2d3677b28d09fa48854bf535" Jul 09 19:35:20.539044 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:35:20.539027 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"befd18598503afd85a62e3b96f7bb400d8d411ed2d3677b28d09fa48854bf535\": container with ID starting with befd18598503afd85a62e3b96f7bb400d8d411ed2d3677b28d09fa48854bf535 not found: ID does not exist" containerID="befd18598503afd85a62e3b96f7bb400d8d411ed2d3677b28d09fa48854bf535" Jul 09 19:35:20.539099 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:20.539054 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"befd18598503afd85a62e3b96f7bb400d8d411ed2d3677b28d09fa48854bf535"} err="failed to get container status \"befd18598503afd85a62e3b96f7bb400d8d411ed2d3677b28d09fa48854bf535\": rpc error: code = NotFound desc = could not find container \"befd18598503afd85a62e3b96f7bb400d8d411ed2d3677b28d09fa48854bf535\": container with ID starting with befd18598503afd85a62e3b96f7bb400d8d411ed2d3677b28d09fa48854bf535 not found: ID does not exist" Jul 09 19:35:20.543127 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:20.543107 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz"] Jul 09 19:35:20.546580 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:20.546558 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-a73219-openshift-default-7f976ffd87-xznpz"] Jul 09 19:35:21.057436 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:21.057406 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6815e3f1-22b0-472e-946a-6e6579cc901e" path="/var/lib/kubelet/pods/6815e3f1-22b0-472e-946a-6e6579cc901e/volumes" Jul 09 19:35:23.529102 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:23.529059 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv"] Jul 09 19:35:23.529716 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:23.529588 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6815e3f1-22b0-472e-946a-6e6579cc901e" containerName="istio-proxy" Jul 09 19:35:23.529716 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:23.529622 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6815e3f1-22b0-472e-946a-6e6579cc901e" containerName="istio-proxy" Jul 09 19:35:23.529716 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:23.529661 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6815e3f1-22b0-472e-946a-6e6579cc901e" containerName="istio-proxy" Jul 09 19:35:23.529716 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:23.529669 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6815e3f1-22b0-472e-946a-6e6579cc901e" containerName="istio-proxy" Jul 09 19:35:23.529948 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:23.529793 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6815e3f1-22b0-472e-946a-6e6579cc901e" containerName="istio-proxy" Jul 09 19:35:23.529948 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:23.529808 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6815e3f1-22b0-472e-946a-6e6579cc901e" containerName="istio-proxy" Jul 09 19:35:23.532776 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:23.532755 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv" Jul 09 19:35:23.535751 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:23.535728 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-11fb58-openshift-default-dockercfg-p4lm7\"" Jul 09 19:35:23.545648 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:23.545625 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv"] Jul 09 19:35:23.696858 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:23.696823 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/513b8c1c-53cc-4a4e-9d16-f945ddde744b-istio-envoy\") pod \"e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv\" (UID: \"513b8c1c-53cc-4a4e-9d16-f945ddde744b\") " pod="openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv" Jul 09 19:35:23.697010 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:23.696868 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/513b8c1c-53cc-4a4e-9d16-f945ddde744b-istiod-ca-cert\") pod \"e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv\" (UID: \"513b8c1c-53cc-4a4e-9d16-f945ddde744b\") " pod="openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv" Jul 09 19:35:23.697010 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:23.696924 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28xb5\" (UniqueName: \"kubernetes.io/projected/513b8c1c-53cc-4a4e-9d16-f945ddde744b-kube-api-access-28xb5\") pod \"e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv\" (UID: \"513b8c1c-53cc-4a4e-9d16-f945ddde744b\") " pod="openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv" Jul 09 19:35:23.697010 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:23.696958 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/513b8c1c-53cc-4a4e-9d16-f945ddde744b-istio-podinfo\") pod \"e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv\" (UID: \"513b8c1c-53cc-4a4e-9d16-f945ddde744b\") " pod="openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv" Jul 09 19:35:23.697010 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:23.697000 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/513b8c1c-53cc-4a4e-9d16-f945ddde744b-istio-data\") pod \"e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv\" (UID: \"513b8c1c-53cc-4a4e-9d16-f945ddde744b\") " pod="openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv" Jul 09 19:35:23.697166 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:23.697047 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/513b8c1c-53cc-4a4e-9d16-f945ddde744b-workload-socket\") pod \"e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv\" (UID: \"513b8c1c-53cc-4a4e-9d16-f945ddde744b\") " pod="openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv" Jul 09 19:35:23.697166 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:23.697118 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/513b8c1c-53cc-4a4e-9d16-f945ddde744b-credential-socket\") pod \"e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv\" (UID: \"513b8c1c-53cc-4a4e-9d16-f945ddde744b\") " pod="openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv" Jul 09 19:35:23.697166 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:23.697150 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/513b8c1c-53cc-4a4e-9d16-f945ddde744b-workload-certs\") pod \"e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv\" (UID: \"513b8c1c-53cc-4a4e-9d16-f945ddde744b\") " pod="openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv" Jul 09 19:35:23.697262 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:23.697165 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/513b8c1c-53cc-4a4e-9d16-f945ddde744b-istio-token\") pod \"e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv\" (UID: \"513b8c1c-53cc-4a4e-9d16-f945ddde744b\") " pod="openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv" Jul 09 19:35:23.798453 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:23.798361 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/513b8c1c-53cc-4a4e-9d16-f945ddde744b-workload-certs\") pod \"e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv\" (UID: \"513b8c1c-53cc-4a4e-9d16-f945ddde744b\") " pod="openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv" Jul 09 19:35:23.798453 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:23.798402 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/513b8c1c-53cc-4a4e-9d16-f945ddde744b-istio-token\") pod \"e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv\" (UID: \"513b8c1c-53cc-4a4e-9d16-f945ddde744b\") " pod="openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv" Jul 09 19:35:23.798453 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:23.798420 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/513b8c1c-53cc-4a4e-9d16-f945ddde744b-istio-envoy\") pod \"e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv\" (UID: \"513b8c1c-53cc-4a4e-9d16-f945ddde744b\") " pod="openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv" Jul 09 19:35:23.798782 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:23.798455 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/513b8c1c-53cc-4a4e-9d16-f945ddde744b-istiod-ca-cert\") pod \"e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv\" (UID: \"513b8c1c-53cc-4a4e-9d16-f945ddde744b\") " pod="openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv" Jul 09 19:35:23.798782 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:23.798499 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28xb5\" (UniqueName: \"kubernetes.io/projected/513b8c1c-53cc-4a4e-9d16-f945ddde744b-kube-api-access-28xb5\") pod \"e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv\" (UID: \"513b8c1c-53cc-4a4e-9d16-f945ddde744b\") " pod="openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv" Jul 09 19:35:23.798782 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:23.798530 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/513b8c1c-53cc-4a4e-9d16-f945ddde744b-istio-podinfo\") pod \"e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv\" (UID: \"513b8c1c-53cc-4a4e-9d16-f945ddde744b\") " pod="openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv" Jul 09 19:35:23.798782 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:23.798568 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/513b8c1c-53cc-4a4e-9d16-f945ddde744b-istio-data\") pod \"e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv\" (UID: \"513b8c1c-53cc-4a4e-9d16-f945ddde744b\") " pod="openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv" Jul 09 19:35:23.798979 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:23.798851 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/513b8c1c-53cc-4a4e-9d16-f945ddde744b-workload-socket\") pod \"e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv\" (UID: \"513b8c1c-53cc-4a4e-9d16-f945ddde744b\") " pod="openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv" Jul 09 19:35:23.798979 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:23.798945 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/513b8c1c-53cc-4a4e-9d16-f945ddde744b-credential-socket\") pod \"e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv\" (UID: \"513b8c1c-53cc-4a4e-9d16-f945ddde744b\") " pod="openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv" Jul 09 19:35:23.798979 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:23.798958 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/513b8c1c-53cc-4a4e-9d16-f945ddde744b-workload-certs\") pod \"e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv\" (UID: \"513b8c1c-53cc-4a4e-9d16-f945ddde744b\") " pod="openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv" Jul 09 19:35:23.799124 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:23.799033 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/513b8c1c-53cc-4a4e-9d16-f945ddde744b-istio-data\") pod \"e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv\" (UID: \"513b8c1c-53cc-4a4e-9d16-f945ddde744b\") " pod="openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv" Jul 09 19:35:23.799186 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:23.799153 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/513b8c1c-53cc-4a4e-9d16-f945ddde744b-workload-socket\") pod \"e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv\" (UID: \"513b8c1c-53cc-4a4e-9d16-f945ddde744b\") " pod="openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv" Jul 09 19:35:23.799309 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:23.799288 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/513b8c1c-53cc-4a4e-9d16-f945ddde744b-credential-socket\") pod \"e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv\" (UID: \"513b8c1c-53cc-4a4e-9d16-f945ddde744b\") " pod="openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv" Jul 09 19:35:23.799396 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:23.799381 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/513b8c1c-53cc-4a4e-9d16-f945ddde744b-istiod-ca-cert\") pod \"e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv\" (UID: \"513b8c1c-53cc-4a4e-9d16-f945ddde744b\") " pod="openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv" Jul 09 19:35:23.801232 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:23.801205 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/513b8c1c-53cc-4a4e-9d16-f945ddde744b-istio-envoy\") pod \"e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv\" (UID: \"513b8c1c-53cc-4a4e-9d16-f945ddde744b\") " pod="openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv" Jul 09 19:35:23.801435 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:23.801415 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/513b8c1c-53cc-4a4e-9d16-f945ddde744b-istio-podinfo\") pod \"e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv\" (UID: \"513b8c1c-53cc-4a4e-9d16-f945ddde744b\") " pod="openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv" Jul 09 19:35:23.807779 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:23.807756 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28xb5\" (UniqueName: \"kubernetes.io/projected/513b8c1c-53cc-4a4e-9d16-f945ddde744b-kube-api-access-28xb5\") pod \"e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv\" (UID: \"513b8c1c-53cc-4a4e-9d16-f945ddde744b\") " pod="openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv" Jul 09 19:35:23.807779 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:23.807762 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/513b8c1c-53cc-4a4e-9d16-f945ddde744b-istio-token\") pod \"e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv\" (UID: \"513b8c1c-53cc-4a4e-9d16-f945ddde744b\") " pod="openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv" Jul 09 19:35:23.843732 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:23.843705 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv" Jul 09 19:35:23.985161 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:23.985135 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv"] Jul 09 19:35:23.986953 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:35:23.986928 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod513b8c1c_53cc_4a4e_9d16_f945ddde744b.slice/crio-1f3b2bda504b2f8085b96b455bb65b8f2b11df8471a0952b1fba921f6f667112 WatchSource:0}: Error finding container 1f3b2bda504b2f8085b96b455bb65b8f2b11df8471a0952b1fba921f6f667112: Status 404 returned error can't find the container with id 1f3b2bda504b2f8085b96b455bb65b8f2b11df8471a0952b1fba921f6f667112 Jul 09 19:35:23.989423 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:23.989395 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:35:23.989506 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:23.989476 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:35:23.989547 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:23.989507 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:35:24.544690 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:24.544644 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv" event={"ID":"513b8c1c-53cc-4a4e-9d16-f945ddde744b","Type":"ContainerStarted","Data":"834b795a74015e0fe2caf876b3830d884da79b742b8d2c4469ca76116193d9ce"} Jul 09 19:35:24.545091 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:24.544707 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv" event={"ID":"513b8c1c-53cc-4a4e-9d16-f945ddde744b","Type":"ContainerStarted","Data":"1f3b2bda504b2f8085b96b455bb65b8f2b11df8471a0952b1fba921f6f667112"} Jul 09 19:35:24.577211 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:24.577162 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv" podStartSLOduration=1.577144486 podStartE2EDuration="1.577144486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:35:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:35:24.574666716 +0000 UTC m=+2194.082446492" watchObservedRunningTime="2026-07-09 19:35:24.577144486 +0000 UTC m=+2194.084924260" Jul 09 19:35:24.843926 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:24.843835 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv" Jul 09 19:35:25.849716 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:25.849684 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv" Jul 09 19:35:26.555218 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:26.555186 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv" Jul 09 19:35:26.556114 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:26.556099 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv" Jul 09 19:35:33.541751 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:33.541713 2566 patch_prober.go:28] interesting pod/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 09 19:35:33.542301 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:33.541785 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc" podUID="1afb71de-d888-46a9-ab0f-18e2ab8279b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 09 19:35:33.567499 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:33.567458 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:35:34.601120 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:34.601086 2566 generic.go:356] "Generic (PLEG): container finished" podID="1afb71de-d888-46a9-ab0f-18e2ab8279b5" containerID="fee05fd4c465d7497dc12a707a81a4aee8e297c0d4b21909a14e1c59b58eee87" exitCode=0 Jul 09 19:35:34.601120 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:34.601124 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc" event={"ID":"1afb71de-d888-46a9-ab0f-18e2ab8279b5","Type":"ContainerDied","Data":"fee05fd4c465d7497dc12a707a81a4aee8e297c0d4b21909a14e1c59b58eee87"} Jul 09 19:35:34.601635 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:34.601492 2566 scope.go:117] "RemoveContainer" containerID="fee05fd4c465d7497dc12a707a81a4aee8e297c0d4b21909a14e1c59b58eee87" Jul 09 19:35:34.602618 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:34.602574 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:35:34.602721 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:34.602681 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:35:34.602762 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:34.602730 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:35:35.611200 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:35.611155 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc" event={"ID":"1afb71de-d888-46a9-ab0f-18e2ab8279b5","Type":"ContainerStarted","Data":"3a84663c54acce5eb4cc081cf7d73377e265936388e8b81c96d4a28d45d4328f"} Jul 09 19:35:35.777570 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:35.777532 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc" Jul 09 19:35:35.777817 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:35.777789 2566 patch_prober.go:28] interesting pod/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Jul 09 19:35:35.777871 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:35.777850 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc" podUID="1afb71de-d888-46a9-ab0f-18e2ab8279b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Jul 09 19:35:36.778403 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:36.778359 2566 patch_prober.go:28] interesting pod/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Jul 09 19:35:36.778805 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:36.778415 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc" podUID="1afb71de-d888-46a9-ab0f-18e2ab8279b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Jul 09 19:35:37.778113 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:37.778080 2566 patch_prober.go:28] interesting pod/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Jul 09 19:35:37.778377 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:37.778352 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc" podUID="1afb71de-d888-46a9-ab0f-18e2ab8279b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Jul 09 19:35:38.778432 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:38.778396 2566 patch_prober.go:28] interesting pod/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Jul 09 19:35:38.778862 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:38.778450 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc" podUID="1afb71de-d888-46a9-ab0f-18e2ab8279b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Jul 09 19:35:39.781838 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:39.781800 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc" Jul 09 19:35:39.782243 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:39.782189 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc" Jul 09 19:35:39.783092 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:39.783067 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc" Jul 09 19:35:41.579745 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:41.579717 2566 patch_prober.go:28] interesting pod/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": read tcp 10.134.0.2:59926->10.134.0.65:15021: read: connection reset by peer" start-of-body= Jul 09 19:35:41.580172 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:41.579792 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv" podUID="513b8c1c-53cc-4a4e-9d16-f945ddde744b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": read tcp 10.134.0.2:59926->10.134.0.65:15021: read: connection reset by peer" Jul 09 19:35:41.656433 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:41.656406 2566 generic.go:356] "Generic (PLEG): container finished" podID="513b8c1c-53cc-4a4e-9d16-f945ddde744b" containerID="834b795a74015e0fe2caf876b3830d884da79b742b8d2c4469ca76116193d9ce" exitCode=0 Jul 09 19:35:41.656597 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:41.656486 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv" event={"ID":"513b8c1c-53cc-4a4e-9d16-f945ddde744b","Type":"ContainerDied","Data":"834b795a74015e0fe2caf876b3830d884da79b742b8d2c4469ca76116193d9ce"} Jul 09 19:35:41.656998 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:41.656984 2566 scope.go:117] "RemoveContainer" containerID="834b795a74015e0fe2caf876b3830d884da79b742b8d2c4469ca76116193d9ce" Jul 09 19:35:41.658031 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:41.657997 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:35:41.658097 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:41.658079 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:35:41.658151 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:41.658135 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 19:35:42.663731 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:42.663695 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv" event={"ID":"513b8c1c-53cc-4a4e-9d16-f945ddde744b","Type":"ContainerStarted","Data":"1f610ce2eb398ecc7c071911a259642763495ddad43a9e58d23384b5512b95e3"} Jul 09 19:35:42.844230 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:42.844183 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv" Jul 09 19:35:42.844493 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:42.844468 2566 patch_prober.go:28] interesting pod/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Jul 09 19:35:42.844586 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:42.844524 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv" podUID="513b8c1c-53cc-4a4e-9d16-f945ddde744b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Jul 09 19:35:43.844599 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:43.844559 2566 patch_prober.go:28] interesting pod/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Jul 09 19:35:43.845012 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:43.844633 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv" podUID="513b8c1c-53cc-4a4e-9d16-f945ddde744b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Jul 09 19:35:44.844570 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:44.844520 2566 patch_prober.go:28] interesting pod/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Jul 09 19:35:44.844777 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:44.844583 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv" podUID="513b8c1c-53cc-4a4e-9d16-f945ddde744b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Jul 09 19:35:45.844797 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:45.844758 2566 patch_prober.go:28] interesting pod/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Jul 09 19:35:45.845211 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:45.844818 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv" podUID="513b8c1c-53cc-4a4e-9d16-f945ddde744b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Jul 09 19:35:46.847840 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:46.847811 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv" Jul 09 19:35:46.848327 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:46.848194 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv" Jul 09 19:35:46.848924 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:35:46.848905 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv" Jul 09 19:36:05.062876 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:36:05.062841 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:36:26.564141 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:36:26.564100 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:36:38.157160 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:36:38.157126 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:36:57.268499 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:36:57.268467 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:37:15.254080 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:37:15.254047 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:37:32.256912 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:37:32.256879 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:37:51.264506 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:37:51.264424 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:38:08.657220 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:38:08.657183 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:38:28.052923 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:38:28.052887 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:38:45.066809 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:38:45.066777 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:38:51.174340 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:38:51.174308 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-vws4c_7cf4af17-5218-4b92-bceb-addbefcc03cb/console-operator/1.log" Jul 09 19:38:51.179060 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:38:51.179040 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6dh7g_db233460-134d-4d9d-8da9-31d89366a4e0/ovn-acl-logging/0.log" Jul 09 19:38:51.184829 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:38:51.184809 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-vws4c_7cf4af17-5218-4b92-bceb-addbefcc03cb/console-operator/1.log" Jul 09 19:38:51.189408 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:38:51.189392 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6dh7g_db233460-134d-4d9d-8da9-31d89366a4e0/ovn-acl-logging/0.log" Jul 09 19:39:03.469428 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:39:03.469389 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:39:18.263679 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:39:18.263584 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:39:37.860305 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:39:37.860274 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:39:57.260730 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:39:57.260696 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:40:09.361883 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:40:09.361851 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:40:48.061038 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:40:48.060954 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:41:03.365383 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:03.365346 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:41:23.360461 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:23.360425 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:41:29.480834 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:29.480798 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7975-140086109579568"] Jul 09 19:41:29.484452 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:29.484429 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7975-140086109579568" Jul 09 19:41:29.487524 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:29.487499 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-mjr6x\"" Jul 09 19:41:29.489920 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:29.489897 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7975-140086109579568"] Jul 09 19:41:29.501693 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:29.501667 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tflvn\" (UniqueName: \"kubernetes.io/projected/0fa76514-7dae-4868-a1ba-c5912b48433b-kube-api-access-tflvn\") pod \"test-curl-7975-140086109579568\" (UID: \"0fa76514-7dae-4868-a1ba-c5912b48433b\") " pod="openshift-ingress/test-curl-7975-140086109579568" Jul 09 19:41:29.603157 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:29.603118 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tflvn\" (UniqueName: \"kubernetes.io/projected/0fa76514-7dae-4868-a1ba-c5912b48433b-kube-api-access-tflvn\") pod \"test-curl-7975-140086109579568\" (UID: \"0fa76514-7dae-4868-a1ba-c5912b48433b\") " pod="openshift-ingress/test-curl-7975-140086109579568" Jul 09 19:41:29.611896 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:29.611863 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tflvn\" (UniqueName: \"kubernetes.io/projected/0fa76514-7dae-4868-a1ba-c5912b48433b-kube-api-access-tflvn\") pod \"test-curl-7975-140086109579568\" (UID: \"0fa76514-7dae-4868-a1ba-c5912b48433b\") " pod="openshift-ingress/test-curl-7975-140086109579568" Jul 09 19:41:29.794799 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:29.794714 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7975-140086109579568" Jul 09 19:41:29.930481 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:29.930453 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7975-140086109579568"] Jul 09 19:41:29.932101 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:41:29.932074 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0fa76514_7dae_4868_a1ba_c5912b48433b.slice/crio-4a83db476fd218c83fef8baeb265447374d1d1e9ea282b6088bc1a472bada966 WatchSource:0}: Error finding container 4a83db476fd218c83fef8baeb265447374d1d1e9ea282b6088bc1a472bada966: Status 404 returned error can't find the container with id 4a83db476fd218c83fef8baeb265447374d1d1e9ea282b6088bc1a472bada966 Jul 09 19:41:29.933285 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:29.933268 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 19:41:30.620788 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:30.620756 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7975-140086109579568" event={"ID":"0fa76514-7dae-4868-a1ba-c5912b48433b","Type":"ContainerStarted","Data":"4a83db476fd218c83fef8baeb265447374d1d1e9ea282b6088bc1a472bada966"} Jul 09 19:41:31.627854 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:31.627820 2566 generic.go:356] "Generic (PLEG): container finished" podID="0fa76514-7dae-4868-a1ba-c5912b48433b" containerID="1b1f3d64cd6c669e24535c42799247e65e012ab1183879994b05250a7c4bd84f" exitCode=0 Jul 09 19:41:31.628227 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:31.627914 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7975-140086109579568" event={"ID":"0fa76514-7dae-4868-a1ba-c5912b48433b","Type":"ContainerDied","Data":"1b1f3d64cd6c669e24535c42799247e65e012ab1183879994b05250a7c4bd84f"} Jul 09 19:41:32.778120 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:32.778096 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7975-140086109579568" Jul 09 19:41:32.812938 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:32.812912 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7975-140086109579568_0fa76514-7dae-4868-a1ba-c5912b48433b/test-curl-7975-140086109579568/0.log" Jul 09 19:41:32.834532 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:32.834506 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tflvn\" (UniqueName: \"kubernetes.io/projected/0fa76514-7dae-4868-a1ba-c5912b48433b-kube-api-access-tflvn\") pod \"0fa76514-7dae-4868-a1ba-c5912b48433b\" (UID: \"0fa76514-7dae-4868-a1ba-c5912b48433b\") " Jul 09 19:41:32.836621 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:32.836582 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fa76514-7dae-4868-a1ba-c5912b48433b-kube-api-access-tflvn" (OuterVolumeSpecName: "kube-api-access-tflvn") pod "0fa76514-7dae-4868-a1ba-c5912b48433b" (UID: "0fa76514-7dae-4868-a1ba-c5912b48433b"). InnerVolumeSpecName "kube-api-access-tflvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:41:32.837016 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:32.836996 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7975-140086109579568"] Jul 09 19:41:32.842279 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:32.842258 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7975-140086109579568"] Jul 09 19:41:32.935268 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:32.935237 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tflvn\" (UniqueName: \"kubernetes.io/projected/0fa76514-7dae-4868-a1ba-c5912b48433b-kube-api-access-tflvn\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:41:32.963772 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:32.963736 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7975-140086109580080"] Jul 09 19:41:32.964139 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:32.964126 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fa76514-7dae-4868-a1ba-c5912b48433b" containerName="test-curl-7975-140086109579568" Jul 09 19:41:32.964184 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:32.964140 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fa76514-7dae-4868-a1ba-c5912b48433b" containerName="test-curl-7975-140086109579568" Jul 09 19:41:32.964224 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:32.964218 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fa76514-7dae-4868-a1ba-c5912b48433b" containerName="test-curl-7975-140086109579568" Jul 09 19:41:32.967122 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:32.967106 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7975-140086109580080" Jul 09 19:41:32.973896 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:32.973825 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7975-140086109580080"] Jul 09 19:41:33.035700 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:33.035599 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwh96\" (UniqueName: \"kubernetes.io/projected/5c9c126b-865c-4027-a49a-521c817eb15f-kube-api-access-pwh96\") pod \"test-curl-7975-140086109580080\" (UID: \"5c9c126b-865c-4027-a49a-521c817eb15f\") " pod="openshift-ingress/test-curl-7975-140086109580080" Jul 09 19:41:33.058267 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:33.058237 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fa76514-7dae-4868-a1ba-c5912b48433b" path="/var/lib/kubelet/pods/0fa76514-7dae-4868-a1ba-c5912b48433b/volumes" Jul 09 19:41:33.136964 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:33.136933 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwh96\" (UniqueName: \"kubernetes.io/projected/5c9c126b-865c-4027-a49a-521c817eb15f-kube-api-access-pwh96\") pod \"test-curl-7975-140086109580080\" (UID: \"5c9c126b-865c-4027-a49a-521c817eb15f\") " pod="openshift-ingress/test-curl-7975-140086109580080" Jul 09 19:41:33.145215 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:33.145184 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwh96\" (UniqueName: \"kubernetes.io/projected/5c9c126b-865c-4027-a49a-521c817eb15f-kube-api-access-pwh96\") pod \"test-curl-7975-140086109580080\" (UID: \"5c9c126b-865c-4027-a49a-521c817eb15f\") " pod="openshift-ingress/test-curl-7975-140086109580080" Jul 09 19:41:33.278086 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:33.278040 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7975-140086109580080" Jul 09 19:41:33.416045 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:33.416020 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7975-140086109580080"] Jul 09 19:41:33.417873 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:41:33.417848 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c9c126b_865c_4027_a49a_521c817eb15f.slice/crio-bc293ba332970bb4ab163264e8678b799e65d9890a7281a0aeacdee63521481a WatchSource:0}: Error finding container bc293ba332970bb4ab163264e8678b799e65d9890a7281a0aeacdee63521481a: Status 404 returned error can't find the container with id bc293ba332970bb4ab163264e8678b799e65d9890a7281a0aeacdee63521481a Jul 09 19:41:33.640962 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:33.640869 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7975-140086109579568" Jul 09 19:41:33.640962 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:33.640919 2566 scope.go:117] "RemoveContainer" containerID="1b1f3d64cd6c669e24535c42799247e65e012ab1183879994b05250a7c4bd84f" Jul 09 19:41:33.642875 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:33.642844 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7975-140086109580080" event={"ID":"5c9c126b-865c-4027-a49a-521c817eb15f","Type":"ContainerStarted","Data":"bc293ba332970bb4ab163264e8678b799e65d9890a7281a0aeacdee63521481a"} Jul 09 19:41:34.650490 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:34.650452 2566 generic.go:356] "Generic (PLEG): container finished" podID="5c9c126b-865c-4027-a49a-521c817eb15f" containerID="6f2f1130a1b7d0df5f2f0876a9f92df891ff9c030aca6a3387bf70e8aa1562ef" exitCode=0 Jul 09 19:41:34.650944 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:34.650542 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7975-140086109580080" event={"ID":"5c9c126b-865c-4027-a49a-521c817eb15f","Type":"ContainerDied","Data":"6f2f1130a1b7d0df5f2f0876a9f92df891ff9c030aca6a3387bf70e8aa1562ef"} Jul 09 19:41:35.801655 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:35.801632 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7975-140086109580080" Jul 09 19:41:35.825699 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:35.822258 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7975-140086109580080_5c9c126b-865c-4027-a49a-521c817eb15f/test-curl-7975-140086109580080/0.log" Jul 09 19:41:35.846830 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:35.846795 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7975-140086109580080"] Jul 09 19:41:35.852189 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:35.852154 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7975-140086109580080"] Jul 09 19:41:35.862311 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:35.862270 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwh96\" (UniqueName: \"kubernetes.io/projected/5c9c126b-865c-4027-a49a-521c817eb15f-kube-api-access-pwh96\") pod \"5c9c126b-865c-4027-a49a-521c817eb15f\" (UID: \"5c9c126b-865c-4027-a49a-521c817eb15f\") " Jul 09 19:41:35.864342 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:35.864318 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c9c126b-865c-4027-a49a-521c817eb15f-kube-api-access-pwh96" (OuterVolumeSpecName: "kube-api-access-pwh96") pod "5c9c126b-865c-4027-a49a-521c817eb15f" (UID: "5c9c126b-865c-4027-a49a-521c817eb15f"). InnerVolumeSpecName "kube-api-access-pwh96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:41:35.963906 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:35.963865 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwh96\" (UniqueName: \"kubernetes.io/projected/5c9c126b-865c-4027-a49a-521c817eb15f-kube-api-access-pwh96\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:41:36.004473 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:36.004439 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7975-140086109439872"] Jul 09 19:41:36.004895 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:36.004879 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c9c126b-865c-4027-a49a-521c817eb15f" containerName="test-curl-7975-140086109580080" Jul 09 19:41:36.004895 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:36.004895 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c9c126b-865c-4027-a49a-521c817eb15f" containerName="test-curl-7975-140086109580080" Jul 09 19:41:36.005060 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:36.004972 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c9c126b-865c-4027-a49a-521c817eb15f" containerName="test-curl-7975-140086109580080" Jul 09 19:41:36.007960 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:36.007939 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7975-140086109439872" Jul 09 19:41:36.015999 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:36.015973 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7975-140086109439872"] Jul 09 19:41:36.065313 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:36.065284 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9tbt\" (UniqueName: \"kubernetes.io/projected/4d7f9155-e8cc-402a-8eed-f359a43da784-kube-api-access-g9tbt\") pod \"test-curl-7975-140086109439872\" (UID: \"4d7f9155-e8cc-402a-8eed-f359a43da784\") " pod="openshift-ingress/test-curl-7975-140086109439872" Jul 09 19:41:36.166661 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:36.166598 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9tbt\" (UniqueName: \"kubernetes.io/projected/4d7f9155-e8cc-402a-8eed-f359a43da784-kube-api-access-g9tbt\") pod \"test-curl-7975-140086109439872\" (UID: \"4d7f9155-e8cc-402a-8eed-f359a43da784\") " pod="openshift-ingress/test-curl-7975-140086109439872" Jul 09 19:41:36.175482 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:36.175451 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9tbt\" (UniqueName: \"kubernetes.io/projected/4d7f9155-e8cc-402a-8eed-f359a43da784-kube-api-access-g9tbt\") pod \"test-curl-7975-140086109439872\" (UID: \"4d7f9155-e8cc-402a-8eed-f359a43da784\") " pod="openshift-ingress/test-curl-7975-140086109439872" Jul 09 19:41:36.318079 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:36.317976 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7975-140086109439872" Jul 09 19:41:36.455600 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:36.455575 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7975-140086109439872"] Jul 09 19:41:36.457039 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:41:36.457015 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d7f9155_e8cc_402a_8eed_f359a43da784.slice/crio-70a62b1de32a9c9ce26c558a29162e2928a364198a6271af01153db3bb73e966 WatchSource:0}: Error finding container 70a62b1de32a9c9ce26c558a29162e2928a364198a6271af01153db3bb73e966: Status 404 returned error can't find the container with id 70a62b1de32a9c9ce26c558a29162e2928a364198a6271af01153db3bb73e966 Jul 09 19:41:36.664935 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:36.664850 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc293ba332970bb4ab163264e8678b799e65d9890a7281a0aeacdee63521481a" Jul 09 19:41:36.664935 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:36.664852 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7975-140086109580080" Jul 09 19:41:36.666088 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:36.666054 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7975-140086109439872" event={"ID":"4d7f9155-e8cc-402a-8eed-f359a43da784","Type":"ContainerStarted","Data":"70a62b1de32a9c9ce26c558a29162e2928a364198a6271af01153db3bb73e966"} Jul 09 19:41:36.675997 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:41:36.675968 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7975-140086109580080\" is forbidden: User \"system:node:ip-10-0-140-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-140-153.ec2.internal' and this object" podUID="5c9c126b-865c-4027-a49a-521c817eb15f" pod="openshift-ingress/test-curl-7975-140086109580080" Jul 09 19:41:37.059298 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:37.059263 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c9c126b-865c-4027-a49a-521c817eb15f" path="/var/lib/kubelet/pods/5c9c126b-865c-4027-a49a-521c817eb15f/volumes" Jul 09 19:41:37.673672 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:37.673642 2566 generic.go:356] "Generic (PLEG): container finished" podID="4d7f9155-e8cc-402a-8eed-f359a43da784" containerID="df92f9985ede08f99685beeebffde4743e2da4162b4f61eee0816cf36866d899" exitCode=0 Jul 09 19:41:37.673840 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:37.673776 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7975-140086109439872" event={"ID":"4d7f9155-e8cc-402a-8eed-f359a43da784","Type":"ContainerDied","Data":"df92f9985ede08f99685beeebffde4743e2da4162b4f61eee0816cf36866d899"} Jul 09 19:41:38.834505 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:38.834478 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7975-140086109439872" Jul 09 19:41:38.853009 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:38.852983 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7975-140086109439872_4d7f9155-e8cc-402a-8eed-f359a43da784/test-curl-7975-140086109439872/0.log" Jul 09 19:41:38.874135 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:38.874106 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7975-140086109439872"] Jul 09 19:41:38.877958 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:38.877919 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7975-140086109439872"] Jul 09 19:41:38.889284 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:38.889261 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9tbt\" (UniqueName: \"kubernetes.io/projected/4d7f9155-e8cc-402a-8eed-f359a43da784-kube-api-access-g9tbt\") pod \"4d7f9155-e8cc-402a-8eed-f359a43da784\" (UID: \"4d7f9155-e8cc-402a-8eed-f359a43da784\") " Jul 09 19:41:38.891237 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:38.891213 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d7f9155-e8cc-402a-8eed-f359a43da784-kube-api-access-g9tbt" (OuterVolumeSpecName: "kube-api-access-g9tbt") pod "4d7f9155-e8cc-402a-8eed-f359a43da784" (UID: "4d7f9155-e8cc-402a-8eed-f359a43da784"). InnerVolumeSpecName "kube-api-access-g9tbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:41:38.993654 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:38.990770 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9tbt\" (UniqueName: \"kubernetes.io/projected/4d7f9155-e8cc-402a-8eed-f359a43da784-kube-api-access-g9tbt\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:41:38.993654 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:38.991085 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7975-140086109439872"] Jul 09 19:41:38.993654 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:38.992092 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d7f9155-e8cc-402a-8eed-f359a43da784" containerName="test-curl-7975-140086109439872" Jul 09 19:41:38.993654 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:38.992111 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d7f9155-e8cc-402a-8eed-f359a43da784" containerName="test-curl-7975-140086109439872" Jul 09 19:41:38.993654 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:38.992316 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d7f9155-e8cc-402a-8eed-f359a43da784" containerName="test-curl-7975-140086109439872" Jul 09 19:41:38.999685 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:38.997622 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7975-140086109439872" Jul 09 19:41:39.000643 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:39.000595 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7975-140086109439872"] Jul 09 19:41:39.057435 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:39.057403 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d7f9155-e8cc-402a-8eed-f359a43da784" path="/var/lib/kubelet/pods/4d7f9155-e8cc-402a-8eed-f359a43da784/volumes" Jul 09 19:41:39.091630 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:39.091578 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5lvg\" (UniqueName: \"kubernetes.io/projected/1901c743-644d-4817-a107-ea57199bac07-kube-api-access-n5lvg\") pod \"test-curl-7975-140086109439872\" (UID: \"1901c743-644d-4817-a107-ea57199bac07\") " pod="openshift-ingress/test-curl-7975-140086109439872" Jul 09 19:41:39.192159 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:39.192127 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5lvg\" (UniqueName: \"kubernetes.io/projected/1901c743-644d-4817-a107-ea57199bac07-kube-api-access-n5lvg\") pod \"test-curl-7975-140086109439872\" (UID: \"1901c743-644d-4817-a107-ea57199bac07\") " pod="openshift-ingress/test-curl-7975-140086109439872" Jul 09 19:41:39.200911 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:39.200885 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5lvg\" (UniqueName: \"kubernetes.io/projected/1901c743-644d-4817-a107-ea57199bac07-kube-api-access-n5lvg\") pod \"test-curl-7975-140086109439872\" (UID: \"1901c743-644d-4817-a107-ea57199bac07\") " pod="openshift-ingress/test-curl-7975-140086109439872" Jul 09 19:41:39.311436 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:39.311354 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7975-140086109439872" Jul 09 19:41:39.445896 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:39.445871 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7975-140086109439872"] Jul 09 19:41:39.447688 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:41:39.447666 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1901c743_644d_4817_a107_ea57199bac07.slice/crio-0cd7c461e4f14be7c2b8f5b68b094c8a9b9cebc3add80389932bf37183738200 WatchSource:0}: Error finding container 0cd7c461e4f14be7c2b8f5b68b094c8a9b9cebc3add80389932bf37183738200: Status 404 returned error can't find the container with id 0cd7c461e4f14be7c2b8f5b68b094c8a9b9cebc3add80389932bf37183738200 Jul 09 19:41:39.685529 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:39.685445 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7975-140086109439872" Jul 09 19:41:39.685529 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:39.685479 2566 scope.go:117] "RemoveContainer" containerID="df92f9985ede08f99685beeebffde4743e2da4162b4f61eee0816cf36866d899" Jul 09 19:41:39.686859 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:39.686831 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7975-140086109439872" event={"ID":"1901c743-644d-4817-a107-ea57199bac07","Type":"ContainerStarted","Data":"0cd7c461e4f14be7c2b8f5b68b094c8a9b9cebc3add80389932bf37183738200"} Jul 09 19:41:39.960859 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:39.960832 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:41:40.694959 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:40.694925 2566 generic.go:356] "Generic (PLEG): container finished" podID="1901c743-644d-4817-a107-ea57199bac07" containerID="1bb7721e3fefca5a6c76b913665e662afce6cdb446f0aa3dfbe07058a6ea4d74" exitCode=0 Jul 09 19:41:40.695127 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:40.694997 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7975-140086109439872" event={"ID":"1901c743-644d-4817-a107-ea57199bac07","Type":"ContainerDied","Data":"1bb7721e3fefca5a6c76b913665e662afce6cdb446f0aa3dfbe07058a6ea4d74"} Jul 09 19:41:41.856127 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:41.856103 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7975-140086109439872" Jul 09 19:41:41.875526 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:41.875505 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7975-140086109439872_1901c743-644d-4817-a107-ea57199bac07/test-curl-7975-140086109439872/0.log" Jul 09 19:41:41.898426 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:41.898392 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7975-140086109439872"] Jul 09 19:41:41.902316 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:41.902289 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7975-140086109439872"] Jul 09 19:41:41.918903 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:41.918873 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n5lvg\" (UniqueName: \"kubernetes.io/projected/1901c743-644d-4817-a107-ea57199bac07-kube-api-access-n5lvg\") pod \"1901c743-644d-4817-a107-ea57199bac07\" (UID: \"1901c743-644d-4817-a107-ea57199bac07\") " Jul 09 19:41:41.920933 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:41.920905 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1901c743-644d-4817-a107-ea57199bac07-kube-api-access-n5lvg" (OuterVolumeSpecName: "kube-api-access-n5lvg") pod "1901c743-644d-4817-a107-ea57199bac07" (UID: "1901c743-644d-4817-a107-ea57199bac07"). InnerVolumeSpecName "kube-api-access-n5lvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:41:42.020124 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:42.020036 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5lvg\" (UniqueName: \"kubernetes.io/projected/1901c743-644d-4817-a107-ea57199bac07-kube-api-access-n5lvg\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:41:42.022528 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:42.022503 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7975-140086109440880"] Jul 09 19:41:42.022924 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:42.022911 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1901c743-644d-4817-a107-ea57199bac07" containerName="test-curl-7975-140086109439872" Jul 09 19:41:42.022969 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:42.022927 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1901c743-644d-4817-a107-ea57199bac07" containerName="test-curl-7975-140086109439872" Jul 09 19:41:42.023003 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:42.022980 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1901c743-644d-4817-a107-ea57199bac07" containerName="test-curl-7975-140086109439872" Jul 09 19:41:42.025858 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:42.025840 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7975-140086109440880" Jul 09 19:41:42.030340 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:42.030318 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7975-140086109440880"] Jul 09 19:41:42.121267 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:42.121229 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtgmk\" (UniqueName: \"kubernetes.io/projected/4334a3d2-846a-4e1d-8829-d2894e2e7cf9-kube-api-access-vtgmk\") pod \"test-curl-7975-140086109440880\" (UID: \"4334a3d2-846a-4e1d-8829-d2894e2e7cf9\") " pod="openshift-ingress/test-curl-7975-140086109440880" Jul 09 19:41:42.222526 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:42.222480 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtgmk\" (UniqueName: \"kubernetes.io/projected/4334a3d2-846a-4e1d-8829-d2894e2e7cf9-kube-api-access-vtgmk\") pod \"test-curl-7975-140086109440880\" (UID: \"4334a3d2-846a-4e1d-8829-d2894e2e7cf9\") " pod="openshift-ingress/test-curl-7975-140086109440880" Jul 09 19:41:42.230990 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:42.230928 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtgmk\" (UniqueName: \"kubernetes.io/projected/4334a3d2-846a-4e1d-8829-d2894e2e7cf9-kube-api-access-vtgmk\") pod \"test-curl-7975-140086109440880\" (UID: \"4334a3d2-846a-4e1d-8829-d2894e2e7cf9\") " pod="openshift-ingress/test-curl-7975-140086109440880" Jul 09 19:41:42.336227 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:42.336138 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7975-140086109440880" Jul 09 19:41:42.485310 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:42.484246 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7975-140086109440880"] Jul 09 19:41:42.487241 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:41:42.487213 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4334a3d2_846a_4e1d_8829_d2894e2e7cf9.slice/crio-8bf72f268ed07702333dca29f395754f71e7220c8bf785eeff9614102900ccb4 WatchSource:0}: Error finding container 8bf72f268ed07702333dca29f395754f71e7220c8bf785eeff9614102900ccb4: Status 404 returned error can't find the container with id 8bf72f268ed07702333dca29f395754f71e7220c8bf785eeff9614102900ccb4 Jul 09 19:41:42.707509 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:42.707473 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7975-140086109440880" event={"ID":"4334a3d2-846a-4e1d-8829-d2894e2e7cf9","Type":"ContainerStarted","Data":"8bf72f268ed07702333dca29f395754f71e7220c8bf785eeff9614102900ccb4"} Jul 09 19:41:42.708947 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:42.708925 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7975-140086109439872" Jul 09 19:41:42.709072 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:42.708959 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cd7c461e4f14be7c2b8f5b68b094c8a9b9cebc3add80389932bf37183738200" Jul 09 19:41:42.719587 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:41:42.719554 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7975-140086109439872\" is forbidden: User \"system:node:ip-10-0-140-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-140-153.ec2.internal' and this object" podUID="1901c743-644d-4817-a107-ea57199bac07" pod="openshift-ingress/test-curl-7975-140086109439872" Jul 09 19:41:43.064012 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:43.058133 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1901c743-644d-4817-a107-ea57199bac07" path="/var/lib/kubelet/pods/1901c743-644d-4817-a107-ea57199bac07/volumes" Jul 09 19:41:43.717174 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:43.717138 2566 generic.go:356] "Generic (PLEG): container finished" podID="4334a3d2-846a-4e1d-8829-d2894e2e7cf9" containerID="4de768a9959a5ea874527f3f32d33850ba4ed958a3645139ec7d244bd7d89b51" exitCode=0 Jul 09 19:41:43.717360 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:43.717216 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7975-140086109440880" event={"ID":"4334a3d2-846a-4e1d-8829-d2894e2e7cf9","Type":"ContainerDied","Data":"4de768a9959a5ea874527f3f32d33850ba4ed958a3645139ec7d244bd7d89b51"} Jul 09 19:41:44.873170 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:44.873144 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7975-140086109440880" Jul 09 19:41:44.894772 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:44.894745 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7975-140086109440880_4334a3d2-846a-4e1d-8829-d2894e2e7cf9/test-curl-7975-140086109440880/0.log" Jul 09 19:41:44.919482 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:44.919448 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7975-140086109440880"] Jul 09 19:41:44.923487 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:44.923461 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7975-140086109440880"] Jul 09 19:41:44.947597 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:44.947563 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vtgmk\" (UniqueName: \"kubernetes.io/projected/4334a3d2-846a-4e1d-8829-d2894e2e7cf9-kube-api-access-vtgmk\") pod \"4334a3d2-846a-4e1d-8829-d2894e2e7cf9\" (UID: \"4334a3d2-846a-4e1d-8829-d2894e2e7cf9\") " Jul 09 19:41:44.949728 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:44.949702 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4334a3d2-846a-4e1d-8829-d2894e2e7cf9-kube-api-access-vtgmk" (OuterVolumeSpecName: "kube-api-access-vtgmk") pod "4334a3d2-846a-4e1d-8829-d2894e2e7cf9" (UID: "4334a3d2-846a-4e1d-8829-d2894e2e7cf9"). InnerVolumeSpecName "kube-api-access-vtgmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:41:45.040063 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:45.039984 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7975-140086109440880"] Jul 09 19:41:45.040490 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:45.040472 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4334a3d2-846a-4e1d-8829-d2894e2e7cf9" containerName="test-curl-7975-140086109440880" Jul 09 19:41:45.040490 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:45.040488 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4334a3d2-846a-4e1d-8829-d2894e2e7cf9" containerName="test-curl-7975-140086109440880" Jul 09 19:41:45.040690 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:45.040553 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4334a3d2-846a-4e1d-8829-d2894e2e7cf9" containerName="test-curl-7975-140086109440880" Jul 09 19:41:45.043561 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:45.043546 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7975-140086109440880" Jul 09 19:41:45.048588 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:45.048562 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtgmk\" (UniqueName: \"kubernetes.io/projected/4334a3d2-846a-4e1d-8829-d2894e2e7cf9-kube-api-access-vtgmk\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:41:45.059216 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:45.059184 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4334a3d2-846a-4e1d-8829-d2894e2e7cf9" path="/var/lib/kubelet/pods/4334a3d2-846a-4e1d-8829-d2894e2e7cf9/volumes" Jul 09 19:41:45.059758 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:45.059724 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7975-140086109440880"] Jul 09 19:41:45.150096 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:45.150041 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn9xs\" (UniqueName: \"kubernetes.io/projected/487e41ed-5ed4-4116-b33b-097b425521b3-kube-api-access-zn9xs\") pod \"test-curl-7975-140086109440880\" (UID: \"487e41ed-5ed4-4116-b33b-097b425521b3\") " pod="openshift-ingress/test-curl-7975-140086109440880" Jul 09 19:41:45.251719 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:45.251673 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn9xs\" (UniqueName: \"kubernetes.io/projected/487e41ed-5ed4-4116-b33b-097b425521b3-kube-api-access-zn9xs\") pod \"test-curl-7975-140086109440880\" (UID: \"487e41ed-5ed4-4116-b33b-097b425521b3\") " pod="openshift-ingress/test-curl-7975-140086109440880" Jul 09 19:41:45.261137 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:45.261108 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn9xs\" (UniqueName: \"kubernetes.io/projected/487e41ed-5ed4-4116-b33b-097b425521b3-kube-api-access-zn9xs\") pod \"test-curl-7975-140086109440880\" (UID: \"487e41ed-5ed4-4116-b33b-097b425521b3\") " pod="openshift-ingress/test-curl-7975-140086109440880" Jul 09 19:41:45.354546 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:45.354449 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7975-140086109440880" Jul 09 19:41:45.494698 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:45.494663 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7975-140086109440880"] Jul 09 19:41:45.497099 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:41:45.497067 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod487e41ed_5ed4_4116_b33b_097b425521b3.slice/crio-ff2aec9b8397f0a820d95b91bc335259a753230ce9fac90d49aee69a214bc3e3 WatchSource:0}: Error finding container ff2aec9b8397f0a820d95b91bc335259a753230ce9fac90d49aee69a214bc3e3: Status 404 returned error can't find the container with id ff2aec9b8397f0a820d95b91bc335259a753230ce9fac90d49aee69a214bc3e3 Jul 09 19:41:45.731002 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:45.730971 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7975-140086109440880" event={"ID":"487e41ed-5ed4-4116-b33b-097b425521b3","Type":"ContainerStarted","Data":"ff2aec9b8397f0a820d95b91bc335259a753230ce9fac90d49aee69a214bc3e3"} Jul 09 19:41:45.732221 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:45.732200 2566 scope.go:117] "RemoveContainer" containerID="4de768a9959a5ea874527f3f32d33850ba4ed958a3645139ec7d244bd7d89b51" Jul 09 19:41:45.732345 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:45.732200 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7975-140086109440880" Jul 09 19:41:46.739231 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:46.739200 2566 generic.go:356] "Generic (PLEG): container finished" podID="487e41ed-5ed4-4116-b33b-097b425521b3" containerID="fba4c99066733954a4d4ffcc3014f0d03db944285e84876aa088e92fa33ea1ee" exitCode=0 Jul 09 19:41:46.739689 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:46.739287 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7975-140086109440880" event={"ID":"487e41ed-5ed4-4116-b33b-097b425521b3","Type":"ContainerDied","Data":"fba4c99066733954a4d4ffcc3014f0d03db944285e84876aa088e92fa33ea1ee"} Jul 09 19:41:47.897087 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:47.897055 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7975-140086109440880" Jul 09 19:41:47.916939 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:47.916912 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7975-140086109440880_487e41ed-5ed4-4116-b33b-097b425521b3/test-curl-7975-140086109440880/0.log" Jul 09 19:41:47.937188 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:47.937154 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7975-140086109440880"] Jul 09 19:41:47.942342 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:47.942313 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7975-140086109440880"] Jul 09 19:41:47.976550 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:47.976519 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zn9xs\" (UniqueName: \"kubernetes.io/projected/487e41ed-5ed4-4116-b33b-097b425521b3-kube-api-access-zn9xs\") pod \"487e41ed-5ed4-4116-b33b-097b425521b3\" (UID: \"487e41ed-5ed4-4116-b33b-097b425521b3\") " Jul 09 19:41:47.978628 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:47.978588 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/487e41ed-5ed4-4116-b33b-097b425521b3-kube-api-access-zn9xs" (OuterVolumeSpecName: "kube-api-access-zn9xs") pod "487e41ed-5ed4-4116-b33b-097b425521b3" (UID: "487e41ed-5ed4-4116-b33b-097b425521b3"). InnerVolumeSpecName "kube-api-access-zn9xs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:41:48.078040 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:48.077951 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn9xs\" (UniqueName: \"kubernetes.io/projected/487e41ed-5ed4-4116-b33b-097b425521b3-kube-api-access-zn9xs\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:41:48.752100 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:48.752071 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7975-140086109440880" Jul 09 19:41:48.752100 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:48.752102 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff2aec9b8397f0a820d95b91bc335259a753230ce9fac90d49aee69a214bc3e3" Jul 09 19:41:48.763922 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:41:48.763892 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7975-140086109440880\" is forbidden: User \"system:node:ip-10-0-140-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-140-153.ec2.internal' and this object" podUID="487e41ed-5ed4-4116-b33b-097b425521b3" pod="openshift-ingress/test-curl-7975-140086109440880" Jul 09 19:41:49.057215 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:49.057128 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="487e41ed-5ed4-4116-b33b-097b425521b3" path="/var/lib/kubelet/pods/487e41ed-5ed4-4116-b33b-097b425521b3/volumes" Jul 09 19:41:53.058311 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:53.058279 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:41:55.059168 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:55.059139 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc"] Jul 09 19:41:55.059726 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:41:55.059348 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc" podUID="1afb71de-d888-46a9-ab0f-18e2ab8279b5" containerName="istio-proxy" containerID="cri-o://3a84663c54acce5eb4cc081cf7d73377e265936388e8b81c96d4a28d45d4328f" gracePeriod=30 Jul 09 19:42:00.327382 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:00.327321 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc" Jul 09 19:42:00.495466 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:00.495427 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1afb71de-d888-46a9-ab0f-18e2ab8279b5-istiod-ca-cert\") pod \"1afb71de-d888-46a9-ab0f-18e2ab8279b5\" (UID: \"1afb71de-d888-46a9-ab0f-18e2ab8279b5\") " Jul 09 19:42:00.495676 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:00.495480 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1afb71de-d888-46a9-ab0f-18e2ab8279b5-istio-data\") pod \"1afb71de-d888-46a9-ab0f-18e2ab8279b5\" (UID: \"1afb71de-d888-46a9-ab0f-18e2ab8279b5\") " Jul 09 19:42:00.495676 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:00.495497 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qvtwc\" (UniqueName: \"kubernetes.io/projected/1afb71de-d888-46a9-ab0f-18e2ab8279b5-kube-api-access-qvtwc\") pod \"1afb71de-d888-46a9-ab0f-18e2ab8279b5\" (UID: \"1afb71de-d888-46a9-ab0f-18e2ab8279b5\") " Jul 09 19:42:00.495676 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:00.495541 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1afb71de-d888-46a9-ab0f-18e2ab8279b5-istio-envoy\") pod \"1afb71de-d888-46a9-ab0f-18e2ab8279b5\" (UID: \"1afb71de-d888-46a9-ab0f-18e2ab8279b5\") " Jul 09 19:42:00.495676 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:00.495578 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1afb71de-d888-46a9-ab0f-18e2ab8279b5-istio-token\") pod \"1afb71de-d888-46a9-ab0f-18e2ab8279b5\" (UID: \"1afb71de-d888-46a9-ab0f-18e2ab8279b5\") " Jul 09 19:42:00.495676 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:00.495628 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1afb71de-d888-46a9-ab0f-18e2ab8279b5-workload-certs\") pod \"1afb71de-d888-46a9-ab0f-18e2ab8279b5\" (UID: \"1afb71de-d888-46a9-ab0f-18e2ab8279b5\") " Jul 09 19:42:00.495980 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:00.495766 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1afb71de-d888-46a9-ab0f-18e2ab8279b5-workload-socket\") pod \"1afb71de-d888-46a9-ab0f-18e2ab8279b5\" (UID: \"1afb71de-d888-46a9-ab0f-18e2ab8279b5\") " Jul 09 19:42:00.495980 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:00.495828 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1afb71de-d888-46a9-ab0f-18e2ab8279b5-credential-socket\") pod \"1afb71de-d888-46a9-ab0f-18e2ab8279b5\" (UID: \"1afb71de-d888-46a9-ab0f-18e2ab8279b5\") " Jul 09 19:42:00.495980 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:00.495886 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1afb71de-d888-46a9-ab0f-18e2ab8279b5-istio-podinfo\") pod \"1afb71de-d888-46a9-ab0f-18e2ab8279b5\" (UID: \"1afb71de-d888-46a9-ab0f-18e2ab8279b5\") " Jul 09 19:42:00.495980 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:00.495915 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1afb71de-d888-46a9-ab0f-18e2ab8279b5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1afb71de-d888-46a9-ab0f-18e2ab8279b5" (UID: "1afb71de-d888-46a9-ab0f-18e2ab8279b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 19:42:00.496207 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:00.496119 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1afb71de-d888-46a9-ab0f-18e2ab8279b5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1afb71de-d888-46a9-ab0f-18e2ab8279b5" (UID: "1afb71de-d888-46a9-ab0f-18e2ab8279b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:42:00.496207 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:00.496165 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1afb71de-d888-46a9-ab0f-18e2ab8279b5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1afb71de-d888-46a9-ab0f-18e2ab8279b5" (UID: "1afb71de-d888-46a9-ab0f-18e2ab8279b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:42:00.496311 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:00.496219 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1afb71de-d888-46a9-ab0f-18e2ab8279b5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1afb71de-d888-46a9-ab0f-18e2ab8279b5" (UID: "1afb71de-d888-46a9-ab0f-18e2ab8279b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:42:00.496376 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:00.496307 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1afb71de-d888-46a9-ab0f-18e2ab8279b5-istiod-ca-cert\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:42:00.496376 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:00.496326 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1afb71de-d888-46a9-ab0f-18e2ab8279b5-workload-certs\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:42:00.496376 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:00.496340 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1afb71de-d888-46a9-ab0f-18e2ab8279b5-workload-socket\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:42:00.496376 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:00.496354 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1afb71de-d888-46a9-ab0f-18e2ab8279b5-credential-socket\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:42:00.496376 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:00.496322 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1afb71de-d888-46a9-ab0f-18e2ab8279b5-istio-data" (OuterVolumeSpecName: "istio-data") pod "1afb71de-d888-46a9-ab0f-18e2ab8279b5" (UID: "1afb71de-d888-46a9-ab0f-18e2ab8279b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:42:00.497963 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:00.497928 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1afb71de-d888-46a9-ab0f-18e2ab8279b5-istio-token" (OuterVolumeSpecName: "istio-token") pod "1afb71de-d888-46a9-ab0f-18e2ab8279b5" (UID: "1afb71de-d888-46a9-ab0f-18e2ab8279b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:42:00.498101 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:00.497975 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1afb71de-d888-46a9-ab0f-18e2ab8279b5-kube-api-access-qvtwc" (OuterVolumeSpecName: "kube-api-access-qvtwc") pod "1afb71de-d888-46a9-ab0f-18e2ab8279b5" (UID: "1afb71de-d888-46a9-ab0f-18e2ab8279b5"). InnerVolumeSpecName "kube-api-access-qvtwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:42:00.498101 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:00.498014 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1afb71de-d888-46a9-ab0f-18e2ab8279b5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1afb71de-d888-46a9-ab0f-18e2ab8279b5" (UID: "1afb71de-d888-46a9-ab0f-18e2ab8279b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 19:42:00.498101 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:00.498016 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1afb71de-d888-46a9-ab0f-18e2ab8279b5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1afb71de-d888-46a9-ab0f-18e2ab8279b5" (UID: "1afb71de-d888-46a9-ab0f-18e2ab8279b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:42:00.597917 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:00.597840 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1afb71de-d888-46a9-ab0f-18e2ab8279b5-istio-data\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:42:00.597917 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:00.597865 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvtwc\" (UniqueName: \"kubernetes.io/projected/1afb71de-d888-46a9-ab0f-18e2ab8279b5-kube-api-access-qvtwc\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:42:00.597917 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:00.597876 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1afb71de-d888-46a9-ab0f-18e2ab8279b5-istio-envoy\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:42:00.597917 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:00.597884 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1afb71de-d888-46a9-ab0f-18e2ab8279b5-istio-token\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:42:00.597917 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:00.597892 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1afb71de-d888-46a9-ab0f-18e2ab8279b5-istio-podinfo\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:42:00.820539 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:00.820504 2566 generic.go:356] "Generic (PLEG): container finished" podID="1afb71de-d888-46a9-ab0f-18e2ab8279b5" containerID="3a84663c54acce5eb4cc081cf7d73377e265936388e8b81c96d4a28d45d4328f" exitCode=0 Jul 09 19:42:00.820772 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:00.820563 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc" Jul 09 19:42:00.820772 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:00.820597 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc" event={"ID":"1afb71de-d888-46a9-ab0f-18e2ab8279b5","Type":"ContainerDied","Data":"3a84663c54acce5eb4cc081cf7d73377e265936388e8b81c96d4a28d45d4328f"} Jul 09 19:42:00.820772 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:00.820682 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc" event={"ID":"1afb71de-d888-46a9-ab0f-18e2ab8279b5","Type":"ContainerDied","Data":"bc015b1219409c88ee764a19fd9c64374cf0583380b95194ddd91c78c63856d3"} Jul 09 19:42:00.820772 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:00.820711 2566 scope.go:117] "RemoveContainer" containerID="3a84663c54acce5eb4cc081cf7d73377e265936388e8b81c96d4a28d45d4328f" Jul 09 19:42:00.830873 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:00.830857 2566 scope.go:117] "RemoveContainer" containerID="fee05fd4c465d7497dc12a707a81a4aee8e297c0d4b21909a14e1c59b58eee87" Jul 09 19:42:00.839304 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:00.839286 2566 scope.go:117] "RemoveContainer" containerID="3a84663c54acce5eb4cc081cf7d73377e265936388e8b81c96d4a28d45d4328f" Jul 09 19:42:00.839549 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:42:00.839530 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a84663c54acce5eb4cc081cf7d73377e265936388e8b81c96d4a28d45d4328f\": container with ID starting with 3a84663c54acce5eb4cc081cf7d73377e265936388e8b81c96d4a28d45d4328f not found: ID does not exist" containerID="3a84663c54acce5eb4cc081cf7d73377e265936388e8b81c96d4a28d45d4328f" Jul 09 19:42:00.839597 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:00.839558 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a84663c54acce5eb4cc081cf7d73377e265936388e8b81c96d4a28d45d4328f"} err="failed to get container status \"3a84663c54acce5eb4cc081cf7d73377e265936388e8b81c96d4a28d45d4328f\": rpc error: code = NotFound desc = could not find container \"3a84663c54acce5eb4cc081cf7d73377e265936388e8b81c96d4a28d45d4328f\": container with ID starting with 3a84663c54acce5eb4cc081cf7d73377e265936388e8b81c96d4a28d45d4328f not found: ID does not exist" Jul 09 19:42:00.839597 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:00.839574 2566 scope.go:117] "RemoveContainer" containerID="fee05fd4c465d7497dc12a707a81a4aee8e297c0d4b21909a14e1c59b58eee87" Jul 09 19:42:00.839904 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:42:00.839880 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fee05fd4c465d7497dc12a707a81a4aee8e297c0d4b21909a14e1c59b58eee87\": container with ID starting with fee05fd4c465d7497dc12a707a81a4aee8e297c0d4b21909a14e1c59b58eee87 not found: ID does not exist" containerID="fee05fd4c465d7497dc12a707a81a4aee8e297c0d4b21909a14e1c59b58eee87" Jul 09 19:42:00.839978 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:00.839914 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fee05fd4c465d7497dc12a707a81a4aee8e297c0d4b21909a14e1c59b58eee87"} err="failed to get container status \"fee05fd4c465d7497dc12a707a81a4aee8e297c0d4b21909a14e1c59b58eee87\": rpc error: code = NotFound desc = could not find container \"fee05fd4c465d7497dc12a707a81a4aee8e297c0d4b21909a14e1c59b58eee87\": container with ID starting with fee05fd4c465d7497dc12a707a81a4aee8e297c0d4b21909a14e1c59b58eee87 not found: ID does not exist" Jul 09 19:42:00.843679 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:00.843657 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc"] Jul 09 19:42:00.847062 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:00.847037 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-77ffac-openshift-default-847d55b959-nqrtc"] Jul 09 19:42:01.058159 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:01.058130 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1afb71de-d888-46a9-ab0f-18e2ab8279b5" path="/var/lib/kubelet/pods/1afb71de-d888-46a9-ab0f-18e2ab8279b5/volumes" Jul 09 19:42:02.150964 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:02.150928 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv"] Jul 09 19:42:02.151461 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:02.151148 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv" podUID="513b8c1c-53cc-4a4e-9d16-f945ddde744b" containerName="istio-proxy" containerID="cri-o://1f610ce2eb398ecc7c071911a259642763495ddad43a9e58d23384b5512b95e3" gracePeriod=30 Jul 09 19:42:02.809581 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:02.809550 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-9b647dc98-67ckh_d2c6d1ea-6d72-479d-a5f8-0406b68616f9/authorino/0.log" Jul 09 19:42:06.454028 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:06.453998 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mtw72"] Jul 09 19:42:07.415064 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:07.415040 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv" Jul 09 19:42:07.460146 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:07.460111 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/513b8c1c-53cc-4a4e-9d16-f945ddde744b-istiod-ca-cert\") pod \"513b8c1c-53cc-4a4e-9d16-f945ddde744b\" (UID: \"513b8c1c-53cc-4a4e-9d16-f945ddde744b\") " Jul 09 19:42:07.460593 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:07.460178 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28xb5\" (UniqueName: \"kubernetes.io/projected/513b8c1c-53cc-4a4e-9d16-f945ddde744b-kube-api-access-28xb5\") pod \"513b8c1c-53cc-4a4e-9d16-f945ddde744b\" (UID: \"513b8c1c-53cc-4a4e-9d16-f945ddde744b\") " Jul 09 19:42:07.460593 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:07.460214 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/513b8c1c-53cc-4a4e-9d16-f945ddde744b-istio-data\") pod \"513b8c1c-53cc-4a4e-9d16-f945ddde744b\" (UID: \"513b8c1c-53cc-4a4e-9d16-f945ddde744b\") " Jul 09 19:42:07.460593 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:07.460256 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/513b8c1c-53cc-4a4e-9d16-f945ddde744b-istio-podinfo\") pod \"513b8c1c-53cc-4a4e-9d16-f945ddde744b\" (UID: \"513b8c1c-53cc-4a4e-9d16-f945ddde744b\") " Jul 09 19:42:07.460593 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:07.460286 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/513b8c1c-53cc-4a4e-9d16-f945ddde744b-workload-socket\") pod \"513b8c1c-53cc-4a4e-9d16-f945ddde744b\" (UID: \"513b8c1c-53cc-4a4e-9d16-f945ddde744b\") " Jul 09 19:42:07.460593 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:07.460309 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/513b8c1c-53cc-4a4e-9d16-f945ddde744b-istio-token\") pod \"513b8c1c-53cc-4a4e-9d16-f945ddde744b\" (UID: \"513b8c1c-53cc-4a4e-9d16-f945ddde744b\") " Jul 09 19:42:07.460593 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:07.460379 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/513b8c1c-53cc-4a4e-9d16-f945ddde744b-workload-certs\") pod \"513b8c1c-53cc-4a4e-9d16-f945ddde744b\" (UID: \"513b8c1c-53cc-4a4e-9d16-f945ddde744b\") " Jul 09 19:42:07.460593 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:07.460434 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/513b8c1c-53cc-4a4e-9d16-f945ddde744b-istio-envoy\") pod \"513b8c1c-53cc-4a4e-9d16-f945ddde744b\" (UID: \"513b8c1c-53cc-4a4e-9d16-f945ddde744b\") " Jul 09 19:42:07.460593 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:07.460479 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/513b8c1c-53cc-4a4e-9d16-f945ddde744b-credential-socket\") pod \"513b8c1c-53cc-4a4e-9d16-f945ddde744b\" (UID: \"513b8c1c-53cc-4a4e-9d16-f945ddde744b\") " Jul 09 19:42:07.460593 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:07.460568 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/513b8c1c-53cc-4a4e-9d16-f945ddde744b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "513b8c1c-53cc-4a4e-9d16-f945ddde744b" (UID: "513b8c1c-53cc-4a4e-9d16-f945ddde744b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 19:42:07.461081 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:07.460769 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/513b8c1c-53cc-4a4e-9d16-f945ddde744b-istiod-ca-cert\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:42:07.461081 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:07.460926 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/513b8c1c-53cc-4a4e-9d16-f945ddde744b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "513b8c1c-53cc-4a4e-9d16-f945ddde744b" (UID: "513b8c1c-53cc-4a4e-9d16-f945ddde744b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:42:07.461081 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:07.460940 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/513b8c1c-53cc-4a4e-9d16-f945ddde744b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "513b8c1c-53cc-4a4e-9d16-f945ddde744b" (UID: "513b8c1c-53cc-4a4e-9d16-f945ddde744b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:42:07.461081 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:07.460976 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/513b8c1c-53cc-4a4e-9d16-f945ddde744b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "513b8c1c-53cc-4a4e-9d16-f945ddde744b" (UID: "513b8c1c-53cc-4a4e-9d16-f945ddde744b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:42:07.461315 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:07.461291 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/513b8c1c-53cc-4a4e-9d16-f945ddde744b-istio-data" (OuterVolumeSpecName: "istio-data") pod "513b8c1c-53cc-4a4e-9d16-f945ddde744b" (UID: "513b8c1c-53cc-4a4e-9d16-f945ddde744b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:42:07.462724 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:07.462689 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/513b8c1c-53cc-4a4e-9d16-f945ddde744b-istio-token" (OuterVolumeSpecName: "istio-token") pod "513b8c1c-53cc-4a4e-9d16-f945ddde744b" (UID: "513b8c1c-53cc-4a4e-9d16-f945ddde744b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:42:07.462900 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:07.462872 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/513b8c1c-53cc-4a4e-9d16-f945ddde744b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "513b8c1c-53cc-4a4e-9d16-f945ddde744b" (UID: "513b8c1c-53cc-4a4e-9d16-f945ddde744b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 19:42:07.463009 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:07.462898 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/513b8c1c-53cc-4a4e-9d16-f945ddde744b-kube-api-access-28xb5" (OuterVolumeSpecName: "kube-api-access-28xb5") pod "513b8c1c-53cc-4a4e-9d16-f945ddde744b" (UID: "513b8c1c-53cc-4a4e-9d16-f945ddde744b"). InnerVolumeSpecName "kube-api-access-28xb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 19:42:07.463110 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:07.463088 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/513b8c1c-53cc-4a4e-9d16-f945ddde744b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "513b8c1c-53cc-4a4e-9d16-f945ddde744b" (UID: "513b8c1c-53cc-4a4e-9d16-f945ddde744b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 19:42:07.481958 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:07.481936 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-6sn6c_adc02e16-794f-4f41-ad98-1ca745d50b62/manager/0.log" Jul 09 19:42:07.561599 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:07.561571 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/513b8c1c-53cc-4a4e-9d16-f945ddde744b-istio-envoy\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:42:07.561599 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:07.561596 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/513b8c1c-53cc-4a4e-9d16-f945ddde744b-credential-socket\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:42:07.561599 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:07.561619 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28xb5\" (UniqueName: \"kubernetes.io/projected/513b8c1c-53cc-4a4e-9d16-f945ddde744b-kube-api-access-28xb5\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:42:07.561599 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:07.561631 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/513b8c1c-53cc-4a4e-9d16-f945ddde744b-istio-data\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:42:07.561900 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:07.561643 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/513b8c1c-53cc-4a4e-9d16-f945ddde744b-istio-podinfo\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:42:07.561900 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:07.561656 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/513b8c1c-53cc-4a4e-9d16-f945ddde744b-workload-socket\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:42:07.561900 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:07.561667 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/513b8c1c-53cc-4a4e-9d16-f945ddde744b-istio-token\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:42:07.561900 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:07.561675 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/513b8c1c-53cc-4a4e-9d16-f945ddde744b-workload-certs\") on node \"ip-10-0-140-153.ec2.internal\" DevicePath \"\"" Jul 09 19:42:07.600424 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:07.600376 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-cjd52_1c649895-8c6a-4f67-ae72-872f80af8f79/manager/0.log" Jul 09 19:42:07.863086 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:07.862997 2566 generic.go:356] "Generic (PLEG): container finished" podID="513b8c1c-53cc-4a4e-9d16-f945ddde744b" containerID="1f610ce2eb398ecc7c071911a259642763495ddad43a9e58d23384b5512b95e3" exitCode=0 Jul 09 19:42:07.863086 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:07.863060 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv" Jul 09 19:42:07.863280 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:07.863095 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv" event={"ID":"513b8c1c-53cc-4a4e-9d16-f945ddde744b","Type":"ContainerDied","Data":"1f610ce2eb398ecc7c071911a259642763495ddad43a9e58d23384b5512b95e3"} Jul 09 19:42:07.863280 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:07.863157 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv" event={"ID":"513b8c1c-53cc-4a4e-9d16-f945ddde744b","Type":"ContainerDied","Data":"1f3b2bda504b2f8085b96b455bb65b8f2b11df8471a0952b1fba921f6f667112"} Jul 09 19:42:07.863280 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:07.863185 2566 scope.go:117] "RemoveContainer" containerID="1f610ce2eb398ecc7c071911a259642763495ddad43a9e58d23384b5512b95e3" Jul 09 19:42:07.873639 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:07.873598 2566 scope.go:117] "RemoveContainer" containerID="834b795a74015e0fe2caf876b3830d884da79b742b8d2c4469ca76116193d9ce" Jul 09 19:42:07.881868 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:07.881848 2566 scope.go:117] "RemoveContainer" containerID="1f610ce2eb398ecc7c071911a259642763495ddad43a9e58d23384b5512b95e3" Jul 09 19:42:07.882110 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:42:07.882092 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f610ce2eb398ecc7c071911a259642763495ddad43a9e58d23384b5512b95e3\": container with ID starting with 1f610ce2eb398ecc7c071911a259642763495ddad43a9e58d23384b5512b95e3 not found: ID does not exist" containerID="1f610ce2eb398ecc7c071911a259642763495ddad43a9e58d23384b5512b95e3" Jul 09 19:42:07.882157 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:07.882119 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f610ce2eb398ecc7c071911a259642763495ddad43a9e58d23384b5512b95e3"} err="failed to get container status \"1f610ce2eb398ecc7c071911a259642763495ddad43a9e58d23384b5512b95e3\": rpc error: code = NotFound desc = could not find container \"1f610ce2eb398ecc7c071911a259642763495ddad43a9e58d23384b5512b95e3\": container with ID starting with 1f610ce2eb398ecc7c071911a259642763495ddad43a9e58d23384b5512b95e3 not found: ID does not exist" Jul 09 19:42:07.882157 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:07.882138 2566 scope.go:117] "RemoveContainer" containerID="834b795a74015e0fe2caf876b3830d884da79b742b8d2c4469ca76116193d9ce" Jul 09 19:42:07.882395 ip-10-0-140-153 kubenswrapper[2566]: E0709 19:42:07.882377 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"834b795a74015e0fe2caf876b3830d884da79b742b8d2c4469ca76116193d9ce\": container with ID starting with 834b795a74015e0fe2caf876b3830d884da79b742b8d2c4469ca76116193d9ce not found: ID does not exist" containerID="834b795a74015e0fe2caf876b3830d884da79b742b8d2c4469ca76116193d9ce" Jul 09 19:42:07.882448 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:07.882401 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"834b795a74015e0fe2caf876b3830d884da79b742b8d2c4469ca76116193d9ce"} err="failed to get container status \"834b795a74015e0fe2caf876b3830d884da79b742b8d2c4469ca76116193d9ce\": rpc error: code = NotFound desc = could not find container \"834b795a74015e0fe2caf876b3830d884da79b742b8d2c4469ca76116193d9ce\": container with ID starting with 834b795a74015e0fe2caf876b3830d884da79b742b8d2c4469ca76116193d9ce not found: ID does not exist" Jul 09 19:42:07.886830 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:07.886807 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv"] Jul 09 19:42:07.890319 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:07.890299 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-11fb58-openshift-default-d5dc769f-vjszv"] Jul 09 19:42:07.952788 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:07.952761 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-wpdz8_f9b48c49-05c5-4319-80d3-5b827d22c1a0/manager/0.log" Jul 09 19:42:08.292884 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:08.292855 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-799d55cdcd-82dxc_8483e7be-9ac3-4f1a-834d-cc410f402b93/copy-manifests/0.log" Jul 09 19:42:08.299346 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:08.299326 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-799d55cdcd-82dxc_8483e7be-9ac3-4f1a-834d-cc410f402b93/manager/0.log" Jul 09 19:42:09.057213 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:09.057180 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="513b8c1c-53cc-4a4e-9d16-f945ddde744b" path="/var/lib/kubelet/pods/513b8c1c-53cc-4a4e-9d16-f945ddde744b/volumes" Jul 09 19:42:09.477564 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:09.477533 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfmnp_1ccc3e7f-f900-431b-8310-8945c6fb00ca/util/0.log" Jul 09 19:42:09.485633 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:09.485562 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfmnp_1ccc3e7f-f900-431b-8310-8945c6fb00ca/pull/0.log" Jul 09 19:42:09.491037 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:09.491022 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfmnp_1ccc3e7f-f900-431b-8310-8945c6fb00ca/extract/0.log" Jul 09 19:42:09.603521 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:09.603495 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04qjbb_53a4270c-599b-4314-95f4-e0ce293d6568/pull/0.log" Jul 09 19:42:09.627213 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:09.627186 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04qjbb_53a4270c-599b-4314-95f4-e0ce293d6568/extract/0.log" Jul 09 19:42:09.637892 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:09.637860 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04qjbb_53a4270c-599b-4314-95f4-e0ce293d6568/util/0.log" Jul 09 19:42:09.742377 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:09.742286 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xvp5x_bbfbf153-e930-4709-bb2a-f5d20eb37cc5/extract/0.log" Jul 09 19:42:09.748193 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:09.748173 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xvp5x_bbfbf153-e930-4709-bb2a-f5d20eb37cc5/util/0.log" Jul 09 19:42:09.754198 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:09.754179 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xvp5x_bbfbf153-e930-4709-bb2a-f5d20eb37cc5/pull/0.log" Jul 09 19:42:09.857140 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:09.857092 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dt6dn_ab362fb3-ec00-4d48-b5ae-494b00c79e84/extract/0.log" Jul 09 19:42:09.862701 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:09.862678 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dt6dn_ab362fb3-ec00-4d48-b5ae-494b00c79e84/util/0.log" Jul 09 19:42:09.868355 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:09.868339 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dt6dn_ab362fb3-ec00-4d48-b5ae-494b00c79e84/pull/0.log" Jul 09 19:42:09.980741 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:09.980711 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-9b647dc98-67ckh_d2c6d1ea-6d72-479d-a5f8-0406b68616f9/authorino/0.log" Jul 09 19:42:10.092298 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:10.092203 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-2jcmm_e91380bf-c93b-499e-9b14-78f4e6d1df18/manager/0.log" Jul 09 19:42:10.434646 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:10.434542 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-xxpcc_25daeb84-f792-4cb4-b20b-ae07589df438/registry-server/0.log" Jul 09 19:42:10.561385 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:10.561353 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-6ckvt_e7d5db49-6211-44f9-98a3-9ca7946d714d/manager/0.log" Jul 09 19:42:10.699041 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:10.699013 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-mtw72_a755661f-8b26-45ab-845a-c746557b80bc/limitador/0.log" Jul 09 19:42:10.813879 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:10.813849 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-k4ws6_d8cbf6a2-9f8a-45ef-b075-cd0830fd285e/manager/0.log" Jul 09 19:42:11.151794 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:11.151718 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4z4jll_a05d1fbf-cb86-4936-99c7-fc3811c2192a/istio-proxy/0.log" Jul 09 19:42:11.485090 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:11.485061 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-db49944d8-vqbn4_9a42cd05-1cb0-4b3b-9140-76554fdc5202/kube-auth-proxy/0.log" Jul 09 19:42:11.608459 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:11.608407 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-qwmxj_e5df48a3-3a56-4ff6-967e-984b325a7a2e/istio-proxy/1.log" Jul 09 19:42:16.597784 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:16.597696 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-797qx/must-gather-db4vp"] Jul 09 19:42:16.598319 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:16.598301 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1afb71de-d888-46a9-ab0f-18e2ab8279b5" containerName="istio-proxy" Jul 09 19:42:16.598402 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:16.598323 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1afb71de-d888-46a9-ab0f-18e2ab8279b5" containerName="istio-proxy" Jul 09 19:42:16.598402 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:16.598339 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="513b8c1c-53cc-4a4e-9d16-f945ddde744b" containerName="istio-proxy" Jul 09 19:42:16.598402 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:16.598347 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="513b8c1c-53cc-4a4e-9d16-f945ddde744b" containerName="istio-proxy" Jul 09 19:42:16.598402 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:16.598364 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1afb71de-d888-46a9-ab0f-18e2ab8279b5" containerName="istio-proxy" Jul 09 19:42:16.598402 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:16.598372 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1afb71de-d888-46a9-ab0f-18e2ab8279b5" containerName="istio-proxy" Jul 09 19:42:16.598402 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:16.598381 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="487e41ed-5ed4-4116-b33b-097b425521b3" containerName="test-curl-7975-140086109440880" Jul 09 19:42:16.598402 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:16.598390 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="487e41ed-5ed4-4116-b33b-097b425521b3" containerName="test-curl-7975-140086109440880" Jul 09 19:42:16.598402 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:16.598404 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="513b8c1c-53cc-4a4e-9d16-f945ddde744b" containerName="istio-proxy" Jul 09 19:42:16.598832 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:16.598412 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="513b8c1c-53cc-4a4e-9d16-f945ddde744b" containerName="istio-proxy" Jul 09 19:42:16.598832 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:16.598511 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="513b8c1c-53cc-4a4e-9d16-f945ddde744b" containerName="istio-proxy" Jul 09 19:42:16.598832 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:16.598523 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1afb71de-d888-46a9-ab0f-18e2ab8279b5" containerName="istio-proxy" Jul 09 19:42:16.598832 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:16.598533 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="487e41ed-5ed4-4116-b33b-097b425521b3" containerName="test-curl-7975-140086109440880" Jul 09 19:42:16.598832 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:16.598734 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="513b8c1c-53cc-4a4e-9d16-f945ddde744b" containerName="istio-proxy" Jul 09 19:42:16.598832 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:16.598748 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1afb71de-d888-46a9-ab0f-18e2ab8279b5" containerName="istio-proxy" Jul 09 19:42:16.601900 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:16.601881 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-797qx/must-gather-db4vp" Jul 09 19:42:16.604794 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:16.604769 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-797qx\"/\"openshift-service-ca.crt\"" Jul 09 19:42:16.606248 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:16.606229 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-797qx\"/\"kube-root-ca.crt\"" Jul 09 19:42:16.606345 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:16.606269 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-797qx\"/\"default-dockercfg-7ljkd\"" Jul 09 19:42:16.610936 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:16.610910 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-797qx/must-gather-db4vp"] Jul 09 19:42:16.636815 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:16.636784 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f71d68f3-6f6e-4607-9f1c-905f1b48917b-must-gather-output\") pod \"must-gather-db4vp\" (UID: \"f71d68f3-6f6e-4607-9f1c-905f1b48917b\") " pod="openshift-must-gather-797qx/must-gather-db4vp" Jul 09 19:42:16.636999 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:16.636928 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw44x\" (UniqueName: \"kubernetes.io/projected/f71d68f3-6f6e-4607-9f1c-905f1b48917b-kube-api-access-gw44x\") pod \"must-gather-db4vp\" (UID: \"f71d68f3-6f6e-4607-9f1c-905f1b48917b\") " pod="openshift-must-gather-797qx/must-gather-db4vp" Jul 09 19:42:16.738184 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:16.738146 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f71d68f3-6f6e-4607-9f1c-905f1b48917b-must-gather-output\") pod \"must-gather-db4vp\" (UID: \"f71d68f3-6f6e-4607-9f1c-905f1b48917b\") " pod="openshift-must-gather-797qx/must-gather-db4vp" Jul 09 19:42:16.738365 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:16.738222 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw44x\" (UniqueName: \"kubernetes.io/projected/f71d68f3-6f6e-4607-9f1c-905f1b48917b-kube-api-access-gw44x\") pod \"must-gather-db4vp\" (UID: \"f71d68f3-6f6e-4607-9f1c-905f1b48917b\") " pod="openshift-must-gather-797qx/must-gather-db4vp" Jul 09 19:42:16.738513 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:16.738491 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f71d68f3-6f6e-4607-9f1c-905f1b48917b-must-gather-output\") pod \"must-gather-db4vp\" (UID: \"f71d68f3-6f6e-4607-9f1c-905f1b48917b\") " pod="openshift-must-gather-797qx/must-gather-db4vp" Jul 09 19:42:16.749909 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:16.749878 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw44x\" (UniqueName: \"kubernetes.io/projected/f71d68f3-6f6e-4607-9f1c-905f1b48917b-kube-api-access-gw44x\") pod \"must-gather-db4vp\" (UID: \"f71d68f3-6f6e-4607-9f1c-905f1b48917b\") " pod="openshift-must-gather-797qx/must-gather-db4vp" Jul 09 19:42:16.913100 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:16.913020 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-797qx/must-gather-db4vp" Jul 09 19:42:17.055893 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:42:17.055861 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf71d68f3_6f6e_4607_9f1c_905f1b48917b.slice/crio-14a4af6bac5877c3224f184c121f21950f57c613d57bdc6127faa15bff133381 WatchSource:0}: Error finding container 14a4af6bac5877c3224f184c121f21950f57c613d57bdc6127faa15bff133381: Status 404 returned error can't find the container with id 14a4af6bac5877c3224f184c121f21950f57c613d57bdc6127faa15bff133381 Jul 09 19:42:17.058013 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:17.057995 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-797qx/must-gather-db4vp"] Jul 09 19:42:17.921148 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:17.921117 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-797qx/must-gather-db4vp" event={"ID":"f71d68f3-6f6e-4607-9f1c-905f1b48917b","Type":"ContainerStarted","Data":"71ea45ee999e2944a97db9cc38c5dacc943df16164557fd5cb5334b2a4d589b3"} Jul 09 19:42:17.921490 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:17.921158 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-797qx/must-gather-db4vp" event={"ID":"f71d68f3-6f6e-4607-9f1c-905f1b48917b","Type":"ContainerStarted","Data":"14a4af6bac5877c3224f184c121f21950f57c613d57bdc6127faa15bff133381"} Jul 09 19:42:18.930416 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:18.930374 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-797qx/must-gather-db4vp" event={"ID":"f71d68f3-6f6e-4607-9f1c-905f1b48917b","Type":"ContainerStarted","Data":"161d80d7a03bf7ed9d861a4996fb5ba1331a0063a2740df8c6187cd24fbb6fdf"} Jul 09 19:42:18.952811 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:18.952752 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-797qx/must-gather-db4vp" podStartSLOduration=2.303379582 podStartE2EDuration="2.952729085s" totalImagesPullingTime="649.349503ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:42:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:42:18.947878305 +0000 UTC m=+2608.455658075" watchObservedRunningTime="2026-07-09 19:42:18.952729085 +0000 UTC m=+2608.460508860" Jul 09 19:42:19.297535 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:19.297506 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-tqtv5_7435737c-5954-4ef0-8b41-1889b8231561/global-pull-secret-syncer/0.log" Jul 09 19:42:19.379759 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:19.379732 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-c2bn7_51b7fa90-97cc-4b69-80d4-d63ea5f30d6f/konnectivity-agent/0.log" Jul 09 19:42:19.476682 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:19.476650 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-153.ec2.internal_2bd4f2bfb833729e4a5ed7d4607ff8c1/haproxy/0.log" Jul 09 19:42:23.198206 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:23.198167 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfmnp_1ccc3e7f-f900-431b-8310-8945c6fb00ca/extract/0.log" Jul 09 19:42:23.220272 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:23.220242 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfmnp_1ccc3e7f-f900-431b-8310-8945c6fb00ca/util/0.log" Jul 09 19:42:23.246930 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:23.246895 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qfmnp_1ccc3e7f-f900-431b-8310-8945c6fb00ca/pull/0.log" Jul 09 19:42:23.269585 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:23.269553 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04qjbb_53a4270c-599b-4314-95f4-e0ce293d6568/extract/0.log" Jul 09 19:42:23.294782 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:23.294756 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04qjbb_53a4270c-599b-4314-95f4-e0ce293d6568/util/0.log" Jul 09 19:42:23.325586 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:23.325563 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04qjbb_53a4270c-599b-4314-95f4-e0ce293d6568/pull/0.log" Jul 09 19:42:23.352060 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:23.352013 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xvp5x_bbfbf153-e930-4709-bb2a-f5d20eb37cc5/extract/0.log" Jul 09 19:42:23.373033 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:23.373007 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xvp5x_bbfbf153-e930-4709-bb2a-f5d20eb37cc5/util/0.log" Jul 09 19:42:23.393840 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:23.393799 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xvp5x_bbfbf153-e930-4709-bb2a-f5d20eb37cc5/pull/0.log" Jul 09 19:42:23.460758 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:23.460671 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dt6dn_ab362fb3-ec00-4d48-b5ae-494b00c79e84/extract/0.log" Jul 09 19:42:23.484581 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:23.484548 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dt6dn_ab362fb3-ec00-4d48-b5ae-494b00c79e84/util/0.log" Jul 09 19:42:23.507099 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:23.507057 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dt6dn_ab362fb3-ec00-4d48-b5ae-494b00c79e84/pull/0.log" Jul 09 19:42:23.934478 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:23.934265 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-9b647dc98-67ckh_d2c6d1ea-6d72-479d-a5f8-0406b68616f9/authorino/0.log" Jul 09 19:42:23.966032 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:23.966003 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-2jcmm_e91380bf-c93b-499e-9b14-78f4e6d1df18/manager/0.log" Jul 09 19:42:24.070955 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:24.070924 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-xxpcc_25daeb84-f792-4cb4-b20b-ae07589df438/registry-server/0.log" Jul 09 19:42:24.253592 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:24.253000 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-6ckvt_e7d5db49-6211-44f9-98a3-9ca7946d714d/manager/0.log" Jul 09 19:42:24.277198 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:24.277156 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-mtw72_a755661f-8b26-45ab-845a-c746557b80bc/limitador/0.log" Jul 09 19:42:24.356576 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:24.356387 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-k4ws6_d8cbf6a2-9f8a-45ef-b075-cd0830fd285e/manager/0.log" Jul 09 19:42:26.190125 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:26.190090 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-cz67s_9cb3de9f-2c0d-4e83-a0a1-fb96c9fb5281/kube-state-metrics/0.log" Jul 09 19:42:26.208052 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:26.208022 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-cz67s_9cb3de9f-2c0d-4e83-a0a1-fb96c9fb5281/kube-rbac-proxy-main/0.log" Jul 09 19:42:26.229912 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:26.229855 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-cz67s_9cb3de9f-2c0d-4e83-a0a1-fb96c9fb5281/kube-rbac-proxy-self/0.log" Jul 09 19:42:26.289714 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:26.289690 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-xtr7d_13d71f7d-dc42-44c6-98be-d8ed4d2173c9/monitoring-plugin/0.log" Jul 09 19:42:26.407095 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:26.407066 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kvwlg_40fbb88a-0919-45cb-88e9-53a6f4177875/node-exporter/0.log" Jul 09 19:42:26.431184 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:26.431161 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kvwlg_40fbb88a-0919-45cb-88e9-53a6f4177875/kube-rbac-proxy/0.log" Jul 09 19:42:26.455460 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:26.455360 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kvwlg_40fbb88a-0919-45cb-88e9-53a6f4177875/init-textfile/0.log" Jul 09 19:42:26.554684 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:26.554654 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-wnllf_6402e20a-0bd8-4e91-bf65-e5de9a1a6f50/kube-rbac-proxy-main/0.log" Jul 09 19:42:26.578552 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:26.578513 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-wnllf_6402e20a-0bd8-4e91-bf65-e5de9a1a6f50/kube-rbac-proxy-self/0.log" Jul 09 19:42:26.619072 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:26.619039 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-wnllf_6402e20a-0bd8-4e91-bf65-e5de9a1a6f50/openshift-state-metrics/0.log" Jul 09 19:42:29.164490 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:29.164459 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-vws4c_7cf4af17-5218-4b92-bceb-addbefcc03cb/console-operator/1.log" Jul 09 19:42:29.170813 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:29.170786 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-vws4c_7cf4af17-5218-4b92-bceb-addbefcc03cb/console-operator/2.log" Jul 09 19:42:29.760476 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:29.760446 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-74dbl_acc4f579-a12a-488f-b17a-8503ac4be7b4/download-server/0.log" Jul 09 19:42:29.784464 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:29.784430 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-797qx/perf-node-gather-daemonset-4rqgp"] Jul 09 19:42:29.792198 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:29.792171 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-797qx/perf-node-gather-daemonset-4rqgp" Jul 09 19:42:29.797910 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:29.797876 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-797qx/perf-node-gather-daemonset-4rqgp"] Jul 09 19:42:29.887631 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:29.887489 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/80289613-5bf3-4f4d-8840-d3de525b97cb-proc\") pod \"perf-node-gather-daemonset-4rqgp\" (UID: \"80289613-5bf3-4f4d-8840-d3de525b97cb\") " pod="openshift-must-gather-797qx/perf-node-gather-daemonset-4rqgp" Jul 09 19:42:29.887631 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:29.887586 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/80289613-5bf3-4f4d-8840-d3de525b97cb-sys\") pod \"perf-node-gather-daemonset-4rqgp\" (UID: \"80289613-5bf3-4f4d-8840-d3de525b97cb\") " pod="openshift-must-gather-797qx/perf-node-gather-daemonset-4rqgp" Jul 09 19:42:29.887631 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:29.887637 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/80289613-5bf3-4f4d-8840-d3de525b97cb-podres\") pod \"perf-node-gather-daemonset-4rqgp\" (UID: \"80289613-5bf3-4f4d-8840-d3de525b97cb\") " pod="openshift-must-gather-797qx/perf-node-gather-daemonset-4rqgp" Jul 09 19:42:29.887955 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:29.887677 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvbvm\" (UniqueName: \"kubernetes.io/projected/80289613-5bf3-4f4d-8840-d3de525b97cb-kube-api-access-bvbvm\") pod \"perf-node-gather-daemonset-4rqgp\" (UID: \"80289613-5bf3-4f4d-8840-d3de525b97cb\") " pod="openshift-must-gather-797qx/perf-node-gather-daemonset-4rqgp" Jul 09 19:42:29.887955 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:29.887732 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/80289613-5bf3-4f4d-8840-d3de525b97cb-lib-modules\") pod \"perf-node-gather-daemonset-4rqgp\" (UID: \"80289613-5bf3-4f4d-8840-d3de525b97cb\") " pod="openshift-must-gather-797qx/perf-node-gather-daemonset-4rqgp" Jul 09 19:42:29.988494 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:29.988454 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/80289613-5bf3-4f4d-8840-d3de525b97cb-podres\") pod \"perf-node-gather-daemonset-4rqgp\" (UID: \"80289613-5bf3-4f4d-8840-d3de525b97cb\") " pod="openshift-must-gather-797qx/perf-node-gather-daemonset-4rqgp" Jul 09 19:42:29.988494 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:29.988495 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvbvm\" (UniqueName: \"kubernetes.io/projected/80289613-5bf3-4f4d-8840-d3de525b97cb-kube-api-access-bvbvm\") pod \"perf-node-gather-daemonset-4rqgp\" (UID: \"80289613-5bf3-4f4d-8840-d3de525b97cb\") " pod="openshift-must-gather-797qx/perf-node-gather-daemonset-4rqgp" Jul 09 19:42:29.988764 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:29.988577 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/80289613-5bf3-4f4d-8840-d3de525b97cb-lib-modules\") pod \"perf-node-gather-daemonset-4rqgp\" (UID: \"80289613-5bf3-4f4d-8840-d3de525b97cb\") " pod="openshift-must-gather-797qx/perf-node-gather-daemonset-4rqgp" Jul 09 19:42:29.988764 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:29.988645 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/80289613-5bf3-4f4d-8840-d3de525b97cb-podres\") pod \"perf-node-gather-daemonset-4rqgp\" (UID: \"80289613-5bf3-4f4d-8840-d3de525b97cb\") " pod="openshift-must-gather-797qx/perf-node-gather-daemonset-4rqgp" Jul 09 19:42:29.988764 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:29.988676 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/80289613-5bf3-4f4d-8840-d3de525b97cb-proc\") pod \"perf-node-gather-daemonset-4rqgp\" (UID: \"80289613-5bf3-4f4d-8840-d3de525b97cb\") " pod="openshift-must-gather-797qx/perf-node-gather-daemonset-4rqgp" Jul 09 19:42:29.988764 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:29.988736 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/80289613-5bf3-4f4d-8840-d3de525b97cb-lib-modules\") pod \"perf-node-gather-daemonset-4rqgp\" (UID: \"80289613-5bf3-4f4d-8840-d3de525b97cb\") " pod="openshift-must-gather-797qx/perf-node-gather-daemonset-4rqgp" Jul 09 19:42:29.988764 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:29.988750 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/80289613-5bf3-4f4d-8840-d3de525b97cb-sys\") pod \"perf-node-gather-daemonset-4rqgp\" (UID: \"80289613-5bf3-4f4d-8840-d3de525b97cb\") " pod="openshift-must-gather-797qx/perf-node-gather-daemonset-4rqgp" Jul 09 19:42:29.989038 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:29.988769 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/80289613-5bf3-4f4d-8840-d3de525b97cb-proc\") pod \"perf-node-gather-daemonset-4rqgp\" (UID: \"80289613-5bf3-4f4d-8840-d3de525b97cb\") " pod="openshift-must-gather-797qx/perf-node-gather-daemonset-4rqgp" Jul 09 19:42:29.989038 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:29.988885 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/80289613-5bf3-4f4d-8840-d3de525b97cb-sys\") pod \"perf-node-gather-daemonset-4rqgp\" (UID: \"80289613-5bf3-4f4d-8840-d3de525b97cb\") " pod="openshift-must-gather-797qx/perf-node-gather-daemonset-4rqgp" Jul 09 19:42:29.998890 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:29.998864 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvbvm\" (UniqueName: \"kubernetes.io/projected/80289613-5bf3-4f4d-8840-d3de525b97cb-kube-api-access-bvbvm\") pod \"perf-node-gather-daemonset-4rqgp\" (UID: \"80289613-5bf3-4f4d-8840-d3de525b97cb\") " pod="openshift-must-gather-797qx/perf-node-gather-daemonset-4rqgp" Jul 09 19:42:30.117911 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:30.117816 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-797qx/perf-node-gather-daemonset-4rqgp" Jul 09 19:42:30.321654 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:30.320690 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-797qx/perf-node-gather-daemonset-4rqgp"] Jul 09 19:42:30.323371 ip-10-0-140-153 kubenswrapper[2566]: W0709 19:42:30.323329 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod80289613_5bf3_4f4d_8840_d3de525b97cb.slice/crio-eee68d76678e16c9382ff3995308dac33d34ad8538377c2cbf62532e233b241e WatchSource:0}: Error finding container eee68d76678e16c9382ff3995308dac33d34ad8538377c2cbf62532e233b241e: Status 404 returned error can't find the container with id eee68d76678e16c9382ff3995308dac33d34ad8538377c2cbf62532e233b241e Jul 09 19:42:30.490980 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:30.490946 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78495597f6-l7m79_01d01444-63e0-4637-9edb-9fbe2273de23/volume-data-source-validator/0.log" Jul 09 19:42:31.042558 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:31.042456 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-797qx/perf-node-gather-daemonset-4rqgp" event={"ID":"80289613-5bf3-4f4d-8840-d3de525b97cb","Type":"ContainerStarted","Data":"ebcc521d9233401cbfffec4f07661a581810e0aa377440252df662f7a0b044c5"} Jul 09 19:42:31.042558 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:31.042524 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-797qx/perf-node-gather-daemonset-4rqgp" event={"ID":"80289613-5bf3-4f4d-8840-d3de525b97cb","Type":"ContainerStarted","Data":"eee68d76678e16c9382ff3995308dac33d34ad8538377c2cbf62532e233b241e"} Jul 09 19:42:31.043657 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:31.043633 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-797qx/perf-node-gather-daemonset-4rqgp" Jul 09 19:42:31.062265 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:31.062215 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-797qx/perf-node-gather-daemonset-4rqgp" podStartSLOduration=2.062199655 podStartE2EDuration="2.062199655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 19:42:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:42:31.061018216 +0000 UTC m=+2620.568797986" watchObservedRunningTime="2026-07-09 19:42:31.062199655 +0000 UTC m=+2620.569979428" Jul 09 19:42:31.469340 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:31.469310 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9xn8w_aa9f1324-a4e6-4f69-93f5-101c3aed7a0c/dns/0.log" Jul 09 19:42:31.499325 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:31.499290 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9xn8w_aa9f1324-a4e6-4f69-93f5-101c3aed7a0c/kube-rbac-proxy/0.log" Jul 09 19:42:31.760740 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:31.760664 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-x8k65_0da71879-5de7-47a0-819a-0b1951b1e817/dns-node-resolver/0.log" Jul 09 19:42:32.298542 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:32.298515 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-c5759b49-2l8kr_4cb1ea83-2644-457c-b6dd-e730c35effe1/registry/0.log" Jul 09 19:42:32.370651 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:32.370621 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jnt6g_5c298798-c68a-4450-b9dc-358fa978305a/node-ca/0.log" Jul 09 19:42:33.294888 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:33.294843 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4z4jll_a05d1fbf-cb86-4936-99c7-fc3811c2192a/istio-proxy/0.log" Jul 09 19:42:33.447538 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:33.447507 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-db49944d8-vqbn4_9a42cd05-1cb0-4b3b-9140-76554fdc5202/kube-auth-proxy/0.log" Jul 09 19:42:33.482478 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:33.482448 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-qwmxj_e5df48a3-3a56-4ff6-967e-984b325a7a2e/istio-proxy/0.log" Jul 09 19:42:33.528386 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:33.528349 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-qwmxj_e5df48a3-3a56-4ff6-967e-984b325a7a2e/istio-proxy/1.log" Jul 09 19:42:34.299861 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:34.299807 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7dgq9_a209a84f-cc92-4f17-a299-4f4161322946/serve-healthcheck-canary/0.log" Jul 09 19:42:34.803325 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:34.803277 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-bwggg_7dc9a1c9-93e9-4e98-b49d-90f25331cea6/insights-operator/0.log" Jul 09 19:42:34.803817 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:34.803781 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-bwggg_7dc9a1c9-93e9-4e98-b49d-90f25331cea6/insights-operator/1.log" Jul 09 19:42:34.890509 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:34.890477 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8zh4c_e057c3f1-8832-4c08-b9c2-7e1353a8a82c/kube-rbac-proxy/0.log" Jul 09 19:42:34.911511 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:34.911476 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8zh4c_e057c3f1-8832-4c08-b9c2-7e1353a8a82c/exporter/0.log" Jul 09 19:42:34.931473 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:34.931446 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8zh4c_e057c3f1-8832-4c08-b9c2-7e1353a8a82c/extractor/0.log" Jul 09 19:42:37.052070 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:37.052040 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-6sn6c_adc02e16-794f-4f41-ad98-1ca745d50b62/manager/0.log" Jul 09 19:42:37.218963 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:37.218929 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-cjd52_1c649895-8c6a-4f67-ae72-872f80af8f79/manager/0.log" Jul 09 19:42:37.409110 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:37.409025 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-wpdz8_f9b48c49-05c5-4319-80d3-5b827d22c1a0/manager/0.log" Jul 09 19:42:37.586824 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:37.586792 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-799d55cdcd-82dxc_8483e7be-9ac3-4f1a-834d-cc410f402b93/manager/0.log" Jul 09 19:42:37.607447 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:37.607420 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-799d55cdcd-82dxc_8483e7be-9ac3-4f1a-834d-cc410f402b93/copy-manifests/0.log" Jul 09 19:42:38.071962 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:38.071920 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-797qx/perf-node-gather-daemonset-4rqgp" Jul 09 19:42:40.038725 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:40.038657 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4z4jll_a05d1fbf-cb86-4936-99c7-fc3811c2192a/istio-proxy/0.log" Jul 09 19:42:40.179054 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:40.178977 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-db49944d8-vqbn4_9a42cd05-1cb0-4b3b-9140-76554fdc5202/kube-auth-proxy/0.log" Jul 09 19:42:40.203056 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:40.202880 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-qwmxj_e5df48a3-3a56-4ff6-967e-984b325a7a2e/istio-proxy/0.log" Jul 09 19:42:40.249562 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:40.249519 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-qwmxj_e5df48a3-3a56-4ff6-967e-984b325a7a2e/istio-proxy/1.log" Jul 09 19:42:41.590959 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:41.590925 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4z4jll_a05d1fbf-cb86-4936-99c7-fc3811c2192a/istio-proxy/0.log" Jul 09 19:42:41.728363 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:41.728330 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-db49944d8-vqbn4_9a42cd05-1cb0-4b3b-9140-76554fdc5202/kube-auth-proxy/0.log" Jul 09 19:42:41.754252 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:41.754217 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-qwmxj_e5df48a3-3a56-4ff6-967e-984b325a7a2e/istio-proxy/0.log" Jul 09 19:42:41.795580 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:41.795546 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-qwmxj_e5df48a3-3a56-4ff6-967e-984b325a7a2e/istio-proxy/1.log" Jul 09 19:42:44.101235 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:44.101199 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-z4t8w_fbfea8e9-4071-48a1-894c-6c6e0549e5cd/migrator/0.log" Jul 09 19:42:44.122830 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:44.122800 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-z4t8w_fbfea8e9-4071-48a1-894c-6c6e0549e5cd/graceful-termination/0.log" Jul 09 19:42:45.450581 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:45.450554 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-5mwmg_e5c65950-27b0-41de-8ee1-e2ced769d6b4/kube-multus/0.log" Jul 09 19:42:45.853360 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:45.853287 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-swdmr_db52b7ec-a201-4d65-81b3-56e37f0b6d36/kube-multus-additional-cni-plugins/0.log" Jul 09 19:42:45.877208 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:45.877182 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-swdmr_db52b7ec-a201-4d65-81b3-56e37f0b6d36/egress-router-binary-copy/0.log" Jul 09 19:42:45.900251 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:45.900228 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-swdmr_db52b7ec-a201-4d65-81b3-56e37f0b6d36/cni-plugins/0.log" Jul 09 19:42:45.925736 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:45.925710 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-swdmr_db52b7ec-a201-4d65-81b3-56e37f0b6d36/bond-cni-plugin/0.log" Jul 09 19:42:45.950794 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:45.950763 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-swdmr_db52b7ec-a201-4d65-81b3-56e37f0b6d36/routeoverride-cni/0.log" Jul 09 19:42:45.972505 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:45.972481 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-swdmr_db52b7ec-a201-4d65-81b3-56e37f0b6d36/whereabouts-cni-bincopy/0.log" Jul 09 19:42:45.996414 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:45.996390 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-swdmr_db52b7ec-a201-4d65-81b3-56e37f0b6d36/whereabouts-cni/0.log" Jul 09 19:42:46.137170 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:46.137051 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vtc8p_30cf5feb-cf9a-40c0-8a6a-de8fc6c60506/network-metrics-daemon/0.log" Jul 09 19:42:46.156816 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:46.156790 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vtc8p_30cf5feb-cf9a-40c0-8a6a-de8fc6c60506/kube-rbac-proxy/0.log" Jul 09 19:42:46.923619 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:46.923582 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6dh7g_db233460-134d-4d9d-8da9-31d89366a4e0/ovn-controller/0.log" Jul 09 19:42:46.938992 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:46.938965 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6dh7g_db233460-134d-4d9d-8da9-31d89366a4e0/ovn-acl-logging/1.log" Jul 09 19:42:46.939137 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:46.939037 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6dh7g_db233460-134d-4d9d-8da9-31d89366a4e0/ovn-acl-logging/0.log" Jul 09 19:42:46.958263 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:46.958236 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6dh7g_db233460-134d-4d9d-8da9-31d89366a4e0/kube-rbac-proxy-node/0.log" Jul 09 19:42:46.977479 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:46.977457 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6dh7g_db233460-134d-4d9d-8da9-31d89366a4e0/kube-rbac-proxy-ovn-metrics/0.log" Jul 09 19:42:46.995054 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:46.995017 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6dh7g_db233460-134d-4d9d-8da9-31d89366a4e0/northd/0.log" Jul 09 19:42:47.015255 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:47.015229 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6dh7g_db233460-134d-4d9d-8da9-31d89366a4e0/nbdb/0.log" Jul 09 19:42:47.037094 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:47.037065 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6dh7g_db233460-134d-4d9d-8da9-31d89366a4e0/sbdb/0.log" Jul 09 19:42:47.102969 ip-10-0-140-153 kubenswrapper[2566]: I0709 19:42:47.102928 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6dh7g_db233460-134d-4d9d-8da9-31d89366a4e0/ovnkube-controller/0.log"