Jul 23 03:55:19.997714 ip-10-0-140-6 systemd[1]: Starting Kubernetes Kubelet... Jul 23 03:55:20.320447 ip-10-0-140-6 kubenswrapper[2567]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 03:55:20.320447 ip-10-0-140-6 kubenswrapper[2567]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 23 03:55:20.320447 ip-10-0-140-6 kubenswrapper[2567]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 03:55:20.320447 ip-10-0-140-6 kubenswrapper[2567]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 23 03:55:20.320447 ip-10-0-140-6 kubenswrapper[2567]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 03:55:20.321122 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.321038 2567 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 23 03:55:20.325219 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325203 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 03:55:20.325219 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325219 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 03:55:20.325282 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325223 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 03:55:20.325282 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325226 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 03:55:20.325282 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325231 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 03:55:20.325282 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325234 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 03:55:20.325282 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325237 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 03:55:20.325282 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325240 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 03:55:20.325282 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325242 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 03:55:20.325282 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325245 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 03:55:20.325282 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325248 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 03:55:20.325282 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325250 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 03:55:20.325282 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325253 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 03:55:20.325282 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325256 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 03:55:20.325282 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325258 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 03:55:20.325282 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325261 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 03:55:20.325282 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325264 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 03:55:20.325282 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325266 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 03:55:20.325282 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325269 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 03:55:20.325282 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325272 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 03:55:20.325282 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325274 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 03:55:20.325282 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325277 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 03:55:20.325775 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325279 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 03:55:20.325775 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325282 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 03:55:20.325775 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325284 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 03:55:20.325775 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325287 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 03:55:20.325775 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325290 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 03:55:20.325775 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325293 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 03:55:20.325775 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325296 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 03:55:20.325775 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325298 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 03:55:20.325775 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325301 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 03:55:20.325775 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325303 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 03:55:20.325775 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325306 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 03:55:20.325775 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325309 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 03:55:20.325775 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325311 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 03:55:20.325775 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325314 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 03:55:20.325775 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325316 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 03:55:20.325775 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325319 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 03:55:20.325775 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325323 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 03:55:20.325775 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325327 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 03:55:20.325775 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325330 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 03:55:20.326261 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325333 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 03:55:20.326261 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325335 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 03:55:20.326261 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325338 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 03:55:20.326261 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325341 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 03:55:20.326261 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325344 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 03:55:20.326261 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325347 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 03:55:20.326261 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325349 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 03:55:20.326261 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325352 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 03:55:20.326261 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325354 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 03:55:20.326261 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325357 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 03:55:20.326261 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325360 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 03:55:20.326261 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325362 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 03:55:20.326261 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325365 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 03:55:20.326261 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325368 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 03:55:20.326261 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325371 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 03:55:20.326261 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325373 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 03:55:20.326261 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325375 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 23 03:55:20.326261 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325378 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 03:55:20.326261 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325381 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 03:55:20.326741 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325384 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 03:55:20.326741 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325386 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 03:55:20.326741 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325389 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 03:55:20.326741 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325393 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 03:55:20.326741 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325397 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 03:55:20.326741 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325399 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 03:55:20.326741 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325402 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 03:55:20.326741 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325404 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 03:55:20.326741 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325407 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 03:55:20.326741 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325410 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 03:55:20.326741 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325412 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 03:55:20.326741 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325415 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 03:55:20.326741 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325418 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 03:55:20.326741 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325421 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 03:55:20.326741 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325442 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 03:55:20.326741 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325447 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 03:55:20.326741 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325451 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 03:55:20.326741 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325458 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 03:55:20.326741 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325462 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 03:55:20.327190 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325465 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 03:55:20.327190 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325468 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 03:55:20.327190 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325471 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 03:55:20.327190 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325474 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 03:55:20.327190 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325477 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 03:55:20.327190 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325479 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 03:55:20.327190 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325483 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 03:55:20.327190 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325486 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 03:55:20.327190 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325490 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 03:55:20.327190 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325493 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 03:55:20.327190 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325495 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 03:55:20.327190 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325498 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 03:55:20.327190 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325501 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 03:55:20.327190 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325504 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 03:55:20.327190 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325506 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 03:55:20.327190 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325510 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 03:55:20.327190 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325513 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 03:55:20.327190 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325515 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 03:55:20.327190 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325518 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 03:55:20.327190 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.325521 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 03:55:20.327665 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326212 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 03:55:20.327665 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326218 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 03:55:20.327665 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326222 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 03:55:20.327665 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326224 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 03:55:20.327665 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326227 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 03:55:20.327665 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326230 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 03:55:20.327665 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326233 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 03:55:20.327665 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326236 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 23 03:55:20.327665 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326238 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 03:55:20.327665 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326241 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 03:55:20.327665 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326245 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 03:55:20.327665 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326248 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 03:55:20.327665 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326252 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 03:55:20.327665 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326254 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 03:55:20.327665 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326257 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 03:55:20.327665 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326259 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 03:55:20.327665 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326262 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 03:55:20.327665 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326264 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 03:55:20.327665 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326268 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 03:55:20.328132 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326270 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 03:55:20.328132 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326273 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 03:55:20.328132 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326276 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 03:55:20.328132 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326279 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 03:55:20.328132 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326282 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 03:55:20.328132 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326284 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 03:55:20.328132 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326287 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 03:55:20.328132 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326290 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 03:55:20.328132 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326293 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 03:55:20.328132 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326295 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 03:55:20.328132 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326298 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 03:55:20.328132 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326301 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 03:55:20.328132 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326304 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 03:55:20.328132 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326306 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 03:55:20.328132 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326310 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 03:55:20.328132 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326313 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 03:55:20.328132 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326316 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 03:55:20.328132 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326318 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 03:55:20.328132 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326321 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 03:55:20.328132 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326324 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 03:55:20.328630 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326326 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 03:55:20.328630 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326329 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 03:55:20.328630 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326331 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 03:55:20.328630 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326334 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 03:55:20.328630 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326336 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 03:55:20.328630 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326339 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 03:55:20.328630 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326341 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 03:55:20.328630 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326343 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 03:55:20.328630 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326346 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 03:55:20.328630 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326348 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 03:55:20.328630 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326351 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 03:55:20.328630 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326353 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 03:55:20.328630 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326356 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 03:55:20.328630 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326359 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 03:55:20.328630 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326362 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 03:55:20.328630 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326364 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 03:55:20.328630 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326367 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 03:55:20.328630 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326370 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 03:55:20.328630 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326373 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 03:55:20.328630 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326375 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 03:55:20.329108 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326378 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 03:55:20.329108 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326381 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 03:55:20.329108 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326384 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 03:55:20.329108 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326386 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 03:55:20.329108 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326388 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 03:55:20.329108 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326391 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 03:55:20.329108 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326393 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 03:55:20.329108 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326395 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 03:55:20.329108 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326398 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 03:55:20.329108 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326400 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 03:55:20.329108 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326403 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 03:55:20.329108 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326405 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 03:55:20.329108 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326408 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 03:55:20.329108 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326412 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 03:55:20.329108 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326415 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 03:55:20.329108 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326418 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 03:55:20.329108 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326439 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 03:55:20.329108 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326442 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 03:55:20.329108 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326444 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 03:55:20.329592 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326447 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 03:55:20.329592 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326449 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 03:55:20.329592 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326452 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 03:55:20.329592 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326454 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 03:55:20.329592 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326457 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 03:55:20.329592 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326460 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 03:55:20.329592 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326463 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 03:55:20.329592 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326465 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 03:55:20.329592 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326468 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 03:55:20.329592 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326471 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 03:55:20.329592 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326473 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 03:55:20.329592 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326476 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 03:55:20.329592 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326479 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 03:55:20.329592 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326482 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 03:55:20.329592 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326484 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 03:55:20.329592 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326487 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 03:55:20.329592 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326489 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 03:55:20.329592 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326492 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 03:55:20.329592 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326494 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 03:55:20.329592 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326497 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 03:55:20.330134 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326499 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 03:55:20.330134 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326847 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 03:55:20.330134 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326853 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 03:55:20.330134 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326856 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 03:55:20.330134 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326859 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 23 03:55:20.330134 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326867 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 03:55:20.330134 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326870 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 03:55:20.330134 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326873 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 03:55:20.330134 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326876 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 03:55:20.330134 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326878 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 03:55:20.330134 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326881 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 03:55:20.330134 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326883 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 03:55:20.330134 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326886 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 03:55:20.330134 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326888 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 03:55:20.330134 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326891 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 03:55:20.330134 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326893 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 03:55:20.330134 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326896 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 03:55:20.330134 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326899 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 03:55:20.330134 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326901 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 03:55:20.330682 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326904 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 03:55:20.330682 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326907 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 03:55:20.330682 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326909 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 03:55:20.330682 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326911 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 03:55:20.330682 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326914 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 03:55:20.330682 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326916 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 03:55:20.330682 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326919 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 03:55:20.330682 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326921 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 03:55:20.330682 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326925 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 03:55:20.330682 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326927 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 03:55:20.330682 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326930 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 03:55:20.330682 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326932 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 03:55:20.330682 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326935 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 03:55:20.330682 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326937 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 03:55:20.330682 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326939 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 03:55:20.330682 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326942 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 03:55:20.330682 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326944 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 03:55:20.330682 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326946 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 03:55:20.330682 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326949 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 03:55:20.330682 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326952 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 03:55:20.331238 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326954 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 03:55:20.331238 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326956 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 03:55:20.331238 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326959 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 03:55:20.331238 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326961 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 03:55:20.331238 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326964 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 03:55:20.331238 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326966 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 03:55:20.331238 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326969 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 03:55:20.331238 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326971 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 03:55:20.331238 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326974 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 03:55:20.331238 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326976 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 03:55:20.331238 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326979 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 03:55:20.331238 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326982 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 03:55:20.331238 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326986 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 03:55:20.331238 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326989 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 03:55:20.331238 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326991 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 03:55:20.331238 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326994 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 03:55:20.331238 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326996 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 03:55:20.331238 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.326999 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 03:55:20.331238 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327001 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 03:55:20.331238 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327004 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 03:55:20.331830 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327006 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 03:55:20.331830 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327009 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 03:55:20.331830 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327011 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 03:55:20.331830 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327014 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 03:55:20.331830 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327016 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 03:55:20.331830 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327018 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 03:55:20.331830 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327020 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 03:55:20.331830 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327023 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 03:55:20.331830 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327027 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 03:55:20.331830 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327029 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 03:55:20.331830 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327031 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 03:55:20.331830 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327034 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 03:55:20.331830 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327036 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 03:55:20.331830 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327039 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 03:55:20.331830 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327042 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 03:55:20.331830 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327044 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 03:55:20.331830 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327046 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 03:55:20.331830 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327049 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 03:55:20.331830 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327051 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 03:55:20.331830 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327053 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 03:55:20.332388 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327057 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 03:55:20.332388 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327062 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 03:55:20.332388 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327065 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 03:55:20.332388 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327068 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 03:55:20.332388 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327071 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 03:55:20.332388 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327074 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 03:55:20.332388 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327077 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 03:55:20.332388 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327079 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 03:55:20.332388 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327082 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 03:55:20.332388 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327084 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 03:55:20.332388 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327087 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 03:55:20.332388 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327089 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 03:55:20.332388 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327092 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 03:55:20.332388 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327095 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 03:55:20.332388 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327098 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 03:55:20.332388 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327100 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 03:55:20.332388 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327103 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 03:55:20.332388 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327106 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 03:55:20.332388 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327108 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 03:55:20.332926 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327110 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 03:55:20.332926 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327113 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 03:55:20.332926 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327183 2567 flags.go:64] FLAG: --address="0.0.0.0" Jul 23 03:55:20.332926 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327190 2567 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 23 03:55:20.332926 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327197 2567 flags.go:64] FLAG: --anonymous-auth="true" Jul 23 03:55:20.332926 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327202 2567 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 23 03:55:20.332926 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327206 2567 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 23 03:55:20.332926 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327210 2567 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 23 03:55:20.332926 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327214 2567 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 23 03:55:20.332926 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327218 2567 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 23 03:55:20.332926 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327221 2567 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 23 03:55:20.332926 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327224 2567 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 23 03:55:20.332926 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327228 2567 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 23 03:55:20.332926 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327232 2567 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 23 03:55:20.332926 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327235 2567 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 23 03:55:20.332926 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327238 2567 flags.go:64] FLAG: --cgroup-root="" Jul 23 03:55:20.332926 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327241 2567 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 23 03:55:20.332926 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327244 2567 flags.go:64] FLAG: --client-ca-file="" Jul 23 03:55:20.332926 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327247 2567 flags.go:64] FLAG: --cloud-provider="external" Jul 23 03:55:20.332926 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327250 2567 flags.go:64] FLAG: --cluster-dns="[]" Jul 23 03:55:20.332926 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327254 2567 flags.go:64] FLAG: --cluster-domain="" Jul 23 03:55:20.332926 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327257 2567 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 23 03:55:20.332926 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327259 2567 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 23 03:55:20.333623 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327262 2567 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 23 03:55:20.333623 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327266 2567 flags.go:64] FLAG: --container-log-max-files="5" Jul 23 03:55:20.333623 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327269 2567 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 23 03:55:20.333623 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327272 2567 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 23 03:55:20.333623 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327275 2567 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 23 03:55:20.333623 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327278 2567 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 23 03:55:20.333623 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327281 2567 flags.go:64] FLAG: --contention-profiling="false" Jul 23 03:55:20.333623 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327284 2567 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 23 03:55:20.333623 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327286 2567 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 23 03:55:20.333623 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327289 2567 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 23 03:55:20.333623 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327292 2567 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 23 03:55:20.333623 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327296 2567 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 23 03:55:20.333623 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327299 2567 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 23 03:55:20.333623 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327302 2567 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 23 03:55:20.333623 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327304 2567 flags.go:64] FLAG: --enable-load-reader="false" Jul 23 03:55:20.333623 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327307 2567 flags.go:64] FLAG: --enable-server="true" Jul 23 03:55:20.333623 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327313 2567 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 23 03:55:20.333623 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327317 2567 flags.go:64] FLAG: --event-burst="100" Jul 23 03:55:20.333623 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327320 2567 flags.go:64] FLAG: --event-qps="50" Jul 23 03:55:20.333623 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327323 2567 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 23 03:55:20.333623 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327326 2567 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 23 03:55:20.333623 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327329 2567 flags.go:64] FLAG: --eviction-hard="" Jul 23 03:55:20.333623 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327334 2567 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 23 03:55:20.333623 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327337 2567 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 23 03:55:20.334264 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327340 2567 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 23 03:55:20.334264 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327343 2567 flags.go:64] FLAG: --eviction-soft="" Jul 23 03:55:20.334264 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327346 2567 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 23 03:55:20.334264 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327349 2567 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 23 03:55:20.334264 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327351 2567 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 23 03:55:20.334264 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327354 2567 flags.go:64] FLAG: --experimental-mounter-path="" Jul 23 03:55:20.334264 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327357 2567 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 23 03:55:20.334264 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327360 2567 flags.go:64] FLAG: --fail-swap-on="true" Jul 23 03:55:20.334264 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327362 2567 flags.go:64] FLAG: --feature-gates="" Jul 23 03:55:20.334264 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327366 2567 flags.go:64] FLAG: --file-check-frequency="20s" Jul 23 03:55:20.334264 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327369 2567 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 23 03:55:20.334264 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327372 2567 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 23 03:55:20.334264 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327375 2567 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 23 03:55:20.334264 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327378 2567 flags.go:64] FLAG: --healthz-port="10248" Jul 23 03:55:20.334264 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327381 2567 flags.go:64] FLAG: --help="false" Jul 23 03:55:20.334264 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327384 2567 flags.go:64] FLAG: --hostname-override="ip-10-0-140-6.ec2.internal" Jul 23 03:55:20.334264 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327387 2567 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 23 03:55:20.334264 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327390 2567 flags.go:64] FLAG: --http-check-frequency="20s" Jul 23 03:55:20.334264 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327393 2567 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 23 03:55:20.334264 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327396 2567 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 23 03:55:20.334264 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327399 2567 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 23 03:55:20.334264 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327402 2567 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 23 03:55:20.334264 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327405 2567 flags.go:64] FLAG: --image-service-endpoint="" Jul 23 03:55:20.334876 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327407 2567 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 23 03:55:20.334876 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327411 2567 flags.go:64] FLAG: --kube-api-burst="100" Jul 23 03:55:20.334876 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327414 2567 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 23 03:55:20.334876 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327417 2567 flags.go:64] FLAG: --kube-api-qps="50" Jul 23 03:55:20.334876 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327420 2567 flags.go:64] FLAG: --kube-reserved="" Jul 23 03:55:20.334876 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327438 2567 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 23 03:55:20.334876 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327441 2567 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 23 03:55:20.334876 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327446 2567 flags.go:64] FLAG: --kubelet-cgroups="" Jul 23 03:55:20.334876 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327448 2567 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 23 03:55:20.334876 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327451 2567 flags.go:64] FLAG: --lock-file="" Jul 23 03:55:20.334876 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327454 2567 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 23 03:55:20.334876 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327457 2567 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 23 03:55:20.334876 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327460 2567 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 23 03:55:20.334876 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327469 2567 flags.go:64] FLAG: --log-json-split-stream="false" Jul 23 03:55:20.334876 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327473 2567 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 23 03:55:20.334876 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327476 2567 flags.go:64] FLAG: --log-text-split-stream="false" Jul 23 03:55:20.334876 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327479 2567 flags.go:64] FLAG: --logging-format="text" Jul 23 03:55:20.334876 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327481 2567 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 23 03:55:20.334876 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327485 2567 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 23 03:55:20.334876 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327488 2567 flags.go:64] FLAG: --manifest-url="" Jul 23 03:55:20.334876 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327490 2567 flags.go:64] FLAG: --manifest-url-header="" Jul 23 03:55:20.334876 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327495 2567 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 23 03:55:20.334876 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327497 2567 flags.go:64] FLAG: --max-open-files="1000000" Jul 23 03:55:20.334876 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327501 2567 flags.go:64] FLAG: --max-pods="110" Jul 23 03:55:20.334876 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327504 2567 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 23 03:55:20.335493 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327507 2567 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 23 03:55:20.335493 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327510 2567 flags.go:64] FLAG: --memory-manager-policy="None" Jul 23 03:55:20.335493 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327513 2567 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 23 03:55:20.335493 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327516 2567 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 23 03:55:20.335493 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327519 2567 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 23 03:55:20.335493 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327522 2567 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 23 03:55:20.335493 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327529 2567 flags.go:64] FLAG: --node-status-max-images="50" Jul 23 03:55:20.335493 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327532 2567 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 23 03:55:20.335493 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327536 2567 flags.go:64] FLAG: --oom-score-adj="-999" Jul 23 03:55:20.335493 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327539 2567 flags.go:64] FLAG: --pod-cidr="" Jul 23 03:55:20.335493 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327542 2567 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 23 03:55:20.335493 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327547 2567 flags.go:64] FLAG: --pod-manifest-path="" Jul 23 03:55:20.335493 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327550 2567 flags.go:64] FLAG: --pod-max-pids="-1" Jul 23 03:55:20.335493 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327553 2567 flags.go:64] FLAG: --pods-per-core="0" Jul 23 03:55:20.335493 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327556 2567 flags.go:64] FLAG: --port="10250" Jul 23 03:55:20.335493 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327560 2567 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 23 03:55:20.335493 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327563 2567 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d73a4e63399916c4" Jul 23 03:55:20.335493 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327566 2567 flags.go:64] FLAG: --qos-reserved="" Jul 23 03:55:20.335493 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327569 2567 flags.go:64] FLAG: --read-only-port="10255" Jul 23 03:55:20.335493 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327572 2567 flags.go:64] FLAG: --register-node="true" Jul 23 03:55:20.335493 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327575 2567 flags.go:64] FLAG: --register-with-taints="" Jul 23 03:55:20.335493 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327578 2567 flags.go:64] FLAG: --registry-burst="10" Jul 23 03:55:20.335493 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327581 2567 flags.go:64] FLAG: --registry-qps="5" Jul 23 03:55:20.335493 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327583 2567 flags.go:64] FLAG: --reserved-cpus="" Jul 23 03:55:20.336125 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327586 2567 flags.go:64] FLAG: --reserved-memory="" Jul 23 03:55:20.336125 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327589 2567 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 23 03:55:20.336125 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327592 2567 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 23 03:55:20.336125 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327595 2567 flags.go:64] FLAG: --rotate-certificates="false" Jul 23 03:55:20.336125 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327598 2567 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 23 03:55:20.336125 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327601 2567 flags.go:64] FLAG: --runonce="false" Jul 23 03:55:20.336125 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327603 2567 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 23 03:55:20.336125 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327606 2567 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 23 03:55:20.336125 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327609 2567 flags.go:64] FLAG: --seccomp-default="false" Jul 23 03:55:20.336125 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327612 2567 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 23 03:55:20.336125 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327615 2567 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 23 03:55:20.336125 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327617 2567 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 23 03:55:20.336125 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327620 2567 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 23 03:55:20.336125 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327623 2567 flags.go:64] FLAG: --storage-driver-password="root" Jul 23 03:55:20.336125 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327626 2567 flags.go:64] FLAG: --storage-driver-secure="false" Jul 23 03:55:20.336125 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327629 2567 flags.go:64] FLAG: --storage-driver-table="stats" Jul 23 03:55:20.336125 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327633 2567 flags.go:64] FLAG: --storage-driver-user="root" Jul 23 03:55:20.336125 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327635 2567 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 23 03:55:20.336125 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327638 2567 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 23 03:55:20.336125 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327641 2567 flags.go:64] FLAG: --system-cgroups="" Jul 23 03:55:20.336125 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327644 2567 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 23 03:55:20.336125 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327649 2567 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 23 03:55:20.336125 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327652 2567 flags.go:64] FLAG: --tls-cert-file="" Jul 23 03:55:20.336125 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327656 2567 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 23 03:55:20.336125 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327659 2567 flags.go:64] FLAG: --tls-min-version="" Jul 23 03:55:20.336763 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327662 2567 flags.go:64] FLAG: --tls-private-key-file="" Jul 23 03:55:20.336763 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327664 2567 flags.go:64] FLAG: --topology-manager-policy="none" Jul 23 03:55:20.336763 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327667 2567 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 23 03:55:20.336763 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327670 2567 flags.go:64] FLAG: --topology-manager-scope="container" Jul 23 03:55:20.336763 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327673 2567 flags.go:64] FLAG: --v="2" Jul 23 03:55:20.336763 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327677 2567 flags.go:64] FLAG: --version="false" Jul 23 03:55:20.336763 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327681 2567 flags.go:64] FLAG: --vmodule="" Jul 23 03:55:20.336763 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327684 2567 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 23 03:55:20.336763 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.327687 2567 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 23 03:55:20.336763 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327800 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 03:55:20.336763 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327804 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 03:55:20.336763 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327807 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 03:55:20.336763 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327810 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 03:55:20.336763 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327813 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 03:55:20.336763 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327815 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 03:55:20.336763 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327818 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 03:55:20.336763 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327820 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 03:55:20.336763 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327823 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 03:55:20.336763 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327826 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 03:55:20.336763 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327828 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 03:55:20.336763 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327831 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 03:55:20.336763 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327833 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 03:55:20.337362 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327836 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 03:55:20.337362 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327840 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 03:55:20.337362 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327843 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 03:55:20.337362 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327846 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 03:55:20.337362 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327848 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 03:55:20.337362 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327851 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 03:55:20.337362 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327854 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 03:55:20.337362 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327856 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 03:55:20.337362 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327860 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 03:55:20.337362 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327862 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 03:55:20.337362 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327865 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 03:55:20.337362 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327868 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 03:55:20.337362 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327870 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 03:55:20.337362 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327873 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 03:55:20.337362 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327875 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 03:55:20.337362 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327878 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 03:55:20.337362 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327880 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 03:55:20.337362 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327882 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 03:55:20.337362 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327885 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 03:55:20.337362 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327887 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 03:55:20.338149 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327890 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 03:55:20.338149 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327892 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 03:55:20.338149 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327896 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 03:55:20.338149 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327900 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 03:55:20.338149 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327903 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 03:55:20.338149 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327905 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 03:55:20.338149 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327908 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 03:55:20.338149 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327911 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 03:55:20.338149 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327913 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 03:55:20.338149 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327916 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 03:55:20.338149 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327918 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 03:55:20.338149 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327921 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 03:55:20.338149 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327923 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 03:55:20.338149 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327928 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 03:55:20.338149 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327931 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 03:55:20.338149 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327933 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 03:55:20.338149 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327935 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 03:55:20.338149 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327938 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 03:55:20.338149 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327941 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 03:55:20.338149 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327943 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 03:55:20.338756 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327947 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 03:55:20.338756 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327949 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 03:55:20.338756 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327952 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 03:55:20.338756 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327954 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 03:55:20.338756 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327957 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 03:55:20.338756 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327959 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 03:55:20.338756 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327962 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 03:55:20.338756 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327965 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 03:55:20.338756 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327967 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 03:55:20.338756 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327970 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 03:55:20.338756 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327972 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 03:55:20.338756 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327974 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 23 03:55:20.338756 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327977 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 03:55:20.338756 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327979 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 03:55:20.338756 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327982 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 03:55:20.338756 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327984 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 03:55:20.338756 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327986 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 03:55:20.338756 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327989 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 03:55:20.338756 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327992 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 03:55:20.338756 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327994 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 03:55:20.339243 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327996 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 03:55:20.339243 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.327999 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 03:55:20.339243 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.328001 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 03:55:20.339243 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.328003 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 03:55:20.339243 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.328006 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 03:55:20.339243 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.328010 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 03:55:20.339243 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.328012 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 03:55:20.339243 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.328015 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 03:55:20.339243 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.328017 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 03:55:20.339243 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.328020 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 03:55:20.339243 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.328022 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 03:55:20.339243 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.328025 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 03:55:20.339243 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.328029 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 03:55:20.339243 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.328031 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 03:55:20.339243 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.328035 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 03:55:20.339243 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.328039 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 03:55:20.339243 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.328043 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 03:55:20.339243 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.328047 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 03:55:20.339243 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.328049 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 03:55:20.339818 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.328052 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 03:55:20.339818 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.328055 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 03:55:20.339818 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.328057 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 03:55:20.339818 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.328060 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 03:55:20.339818 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.328062 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 03:55:20.339818 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.328066 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 03:55:20.339818 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.328068 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 03:55:20.339818 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.328972 2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 03:55:20.339818 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.335311 2567 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 23 03:55:20.339818 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.335324 2567 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 23 03:55:20.339818 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.335337 2567 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 23 03:55:20.339818 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.335343 2567 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 23 03:55:20.339818 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335397 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 03:55:20.339818 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335402 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 03:55:20.340167 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335405 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 03:55:20.340167 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335408 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 03:55:20.340167 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335411 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 03:55:20.340167 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335414 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 03:55:20.340167 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335417 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 03:55:20.340167 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335420 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 03:55:20.340167 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335440 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 03:55:20.340167 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335443 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 03:55:20.340167 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335446 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 03:55:20.340167 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335448 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 03:55:20.340167 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335451 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 03:55:20.340167 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335453 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 03:55:20.340167 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335456 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 03:55:20.340167 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335458 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 03:55:20.340167 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335461 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 03:55:20.340167 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335464 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 03:55:20.340167 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335466 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 03:55:20.340167 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335469 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 03:55:20.340167 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335471 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 03:55:20.340167 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335473 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 03:55:20.340790 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335476 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 03:55:20.340790 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335478 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 03:55:20.340790 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335481 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 03:55:20.340790 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335483 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 03:55:20.340790 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335486 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 03:55:20.340790 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335489 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 03:55:20.340790 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335491 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 03:55:20.340790 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335496 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 03:55:20.340790 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335500 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 03:55:20.340790 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335502 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 03:55:20.340790 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335505 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 03:55:20.340790 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335508 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 03:55:20.340790 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335510 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 03:55:20.340790 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335513 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 03:55:20.340790 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335516 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 03:55:20.340790 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335518 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 03:55:20.340790 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335521 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 03:55:20.340790 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335523 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 03:55:20.340790 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335526 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 03:55:20.340790 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335528 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 03:55:20.341543 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335530 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 03:55:20.341543 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335533 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 03:55:20.341543 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335535 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 03:55:20.341543 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335538 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 03:55:20.341543 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335540 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 03:55:20.341543 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335543 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 03:55:20.341543 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335545 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 03:55:20.341543 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335548 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 03:55:20.341543 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335550 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 03:55:20.341543 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335553 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 03:55:20.341543 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335555 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 03:55:20.341543 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335557 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 03:55:20.341543 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335560 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 03:55:20.341543 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335562 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 03:55:20.341543 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335565 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 03:55:20.341543 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335568 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 03:55:20.341543 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335570 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 03:55:20.341543 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335573 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 03:55:20.341543 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335575 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 03:55:20.341543 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335578 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 03:55:20.342315 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335581 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 03:55:20.342315 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335584 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 03:55:20.342315 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335587 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 03:55:20.342315 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335589 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 03:55:20.342315 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335591 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 03:55:20.342315 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335594 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 03:55:20.342315 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335598 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 03:55:20.342315 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335601 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 03:55:20.342315 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335604 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 03:55:20.342315 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335606 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 03:55:20.342315 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335609 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 03:55:20.342315 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335611 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 03:55:20.342315 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335614 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 03:55:20.342315 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335616 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 03:55:20.342315 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335619 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 03:55:20.342315 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335621 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 03:55:20.342315 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335623 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 03:55:20.342315 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335626 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 03:55:20.342315 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335628 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 03:55:20.342315 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335630 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 03:55:20.342820 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335633 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 03:55:20.342820 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335637 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 03:55:20.342820 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335640 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 03:55:20.342820 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335643 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 03:55:20.342820 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335646 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 03:55:20.342820 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335648 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 03:55:20.342820 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335651 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 03:55:20.342820 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335655 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 03:55:20.342820 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335658 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 03:55:20.342820 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335661 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 23 03:55:20.342820 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335663 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 03:55:20.342820 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335666 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 03:55:20.342820 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335669 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 03:55:20.342820 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335672 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 03:55:20.342820 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335675 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 03:55:20.342820 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335677 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 03:55:20.342820 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335679 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 03:55:20.343415 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.335684 2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 03:55:20.343415 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335790 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 03:55:20.343415 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335794 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 03:55:20.343415 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335797 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 03:55:20.343415 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335800 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 03:55:20.343415 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335803 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 03:55:20.343415 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335805 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 03:55:20.343415 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335808 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 03:55:20.343415 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335811 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 03:55:20.343415 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335813 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 03:55:20.343415 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335815 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 03:55:20.343415 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335818 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 03:55:20.343415 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335820 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 03:55:20.343415 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335822 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 03:55:20.343415 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335825 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 03:55:20.343885 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335827 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 03:55:20.343885 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335829 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 03:55:20.343885 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335832 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 03:55:20.343885 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335834 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 03:55:20.343885 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335837 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 03:55:20.343885 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335839 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 03:55:20.343885 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335841 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 03:55:20.343885 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335844 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 03:55:20.343885 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335847 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 03:55:20.343885 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335849 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 03:55:20.343885 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335852 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 03:55:20.343885 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335854 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 03:55:20.343885 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335857 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 03:55:20.343885 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335860 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 03:55:20.343885 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335863 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 03:55:20.343885 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335865 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 03:55:20.343885 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335869 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 03:55:20.343885 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335872 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 03:55:20.343885 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335875 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 03:55:20.344365 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335878 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 03:55:20.344365 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335881 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 03:55:20.344365 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335883 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 03:55:20.344365 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335886 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 03:55:20.344365 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335889 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 03:55:20.344365 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335892 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 03:55:20.344365 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335896 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 03:55:20.344365 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335898 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 03:55:20.344365 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335901 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 03:55:20.344365 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335904 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 03:55:20.344365 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335906 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 03:55:20.344365 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335909 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 03:55:20.344365 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335911 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 03:55:20.344365 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335913 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 03:55:20.344365 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335916 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 03:55:20.344365 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335918 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 03:55:20.344365 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335920 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 03:55:20.344365 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335923 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 03:55:20.344365 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335925 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 03:55:20.344365 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335927 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 03:55:20.345030 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335930 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 03:55:20.345030 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335932 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 03:55:20.345030 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335934 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 03:55:20.345030 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335937 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 03:55:20.345030 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335939 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 03:55:20.345030 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335942 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 03:55:20.345030 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335944 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 03:55:20.345030 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335946 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 03:55:20.345030 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335949 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 23 03:55:20.345030 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335952 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 03:55:20.345030 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335954 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 03:55:20.345030 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335956 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 03:55:20.345030 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335959 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 03:55:20.345030 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335961 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 03:55:20.345030 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335963 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 03:55:20.345030 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335966 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 03:55:20.345030 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335968 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 03:55:20.345030 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335970 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 03:55:20.345030 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335973 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 03:55:20.345030 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335975 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 03:55:20.345631 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335978 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 03:55:20.345631 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335980 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 03:55:20.345631 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335983 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 03:55:20.345631 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335985 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 03:55:20.345631 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335988 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 03:55:20.345631 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335992 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 03:55:20.345631 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335995 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 03:55:20.345631 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.335997 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 03:55:20.345631 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.336000 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 03:55:20.345631 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.336002 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 03:55:20.345631 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.336005 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 03:55:20.345631 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.336008 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 03:55:20.345631 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.336010 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 03:55:20.345631 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.336013 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 03:55:20.345631 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.336015 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 03:55:20.345631 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.336017 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 03:55:20.345631 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.336020 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 03:55:20.345631 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.336023 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 03:55:20.345631 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.336025 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 03:55:20.346113 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.336028 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 03:55:20.346113 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.336030 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 03:55:20.346113 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.336033 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 03:55:20.346113 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.336036 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 03:55:20.346113 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.336038 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 03:55:20.346113 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.336040 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 03:55:20.346113 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:20.336043 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 03:55:20.346113 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.336047 2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 03:55:20.346113 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.336136 2567 server.go:962] "Client rotation is on, will bootstrap in background" Jul 23 03:55:20.346113 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.339094 2567 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 23 03:55:20.346113 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.340209 2567 server.go:1019] "Starting client certificate rotation" Jul 23 03:55:20.346113 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.340323 2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 23 03:55:20.346113 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.340375 2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 23 03:55:20.346113 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.342393 2567 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 23 03:55:20.346506 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.343449 2567 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 23 03:55:20.352271 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.352257 2567 log.go:25] "Validated CRI v1 runtime API" Jul 23 03:55:20.356130 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.356115 2567 log.go:25] "Validated CRI v1 image API" Jul 23 03:55:20.357007 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.356988 2567 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 23 03:55:20.359956 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.359938 2567 fs.go:135] Filesystem UUIDs: map[79e80cc5-cc81-4d30-971b-6cd8ce63707d:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d828fbe5-d0af-4b78-bb54-cf5d946c0a35:/dev/nvme0n1p3] Jul 23 03:55:20.360012 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.359956 2567 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 23 03:55:20.364568 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.364413 2567 manager.go:216] Machine: {Timestamp:2026-07-23 03:55:20.363401221 +0000 UTC m=+0.268705514 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3106148 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e0cb10ab1f525e2a7fa2a98f6b8b5 SystemUUID:ec2e0cb1-0ab1-f525-e2a7-fa2a98f6b8b5 BootID:3fb7440d-4178-43ee-9eca-ca9ca0a6bf19 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:d9:3b:e3:96:5b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:d9:3b:e3:96:5b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:26:7b:91:58:47:aa Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 23 03:55:20.364568 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.364563 2567 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 23 03:55:20.364687 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.364637 2567 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 23 03:55:20.365570 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.365546 2567 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 23 03:55:20.365692 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.365573 2567 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-6.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 23 03:55:20.366170 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.366161 2567 topology_manager.go:138] "Creating topology manager with none policy" Jul 23 03:55:20.366201 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.366172 2567 container_manager_linux.go:309] "Creating device plugin manager" Jul 23 03:55:20.366201 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.366179 2567 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 23 03:55:20.366698 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.366687 2567 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 23 03:55:20.366828 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.366814 2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 23 03:55:20.367901 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.367887 2567 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 23 03:55:20.371536 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.371524 2567 state_mem.go:36] "Initialized new in-memory state store" Jul 23 03:55:20.371671 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.371661 2567 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 23 03:55:20.371704 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.371691 2567 kubelet.go:486] "Attempting to sync node with API server" Jul 23 03:55:20.371704 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.371700 2567 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 23 03:55:20.371756 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.371714 2567 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 23 03:55:20.371756 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.371722 2567 kubelet.go:398] "Adding apiserver pod source" Jul 23 03:55:20.371756 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.371732 2567 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 23 03:55:20.372121 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.372099 2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 23 03:55:20.372160 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.372132 2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 23 03:55:20.379984 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.379958 2567 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 23 03:55:20.381770 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.381754 2567 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 23 03:55:20.381844 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.381784 2567 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 23 03:55:20.382823 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.382811 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 23 03:55:20.382859 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.382827 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 23 03:55:20.382859 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.382832 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 23 03:55:20.382859 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.382838 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 23 03:55:20.382859 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.382846 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 23 03:55:20.382859 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.382852 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 23 03:55:20.382859 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.382860 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 23 03:55:20.383014 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.382865 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 23 03:55:20.383014 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.382871 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 23 03:55:20.383014 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.382876 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 23 03:55:20.383014 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.382885 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 23 03:55:20.383014 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.382889 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 23 03:55:20.383596 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.383584 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 23 03:55:20.383596 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:20.383584 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 23 03:55:20.383708 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:20.383587 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-6.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 23 03:55:20.383708 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.383597 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 23 03:55:20.386656 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.386644 2567 server.go:1268] "Started kubelet" Jul 23 03:55:20.386735 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.386709 2567 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 23 03:55:20.386782 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.386706 2567 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 23 03:55:20.386838 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.386780 2567 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 23 03:55:20.387409 ip-10-0-140-6 systemd[1]: Started Kubernetes Kubelet. Jul 23 03:55:20.388721 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.388708 2567 server.go:310] "Adding debug handlers to kubelet server" Jul 23 03:55:20.391032 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.391016 2567 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 23 03:55:20.391705 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:20.391689 2567 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 23 03:55:20.393626 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.393610 2567 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-6.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 23 03:55:20.394544 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:20.393628 2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-6.ec2.internal.18c4cef8d03fe20c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-6.ec2.internal,UID:ip-10-0-140-6.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-6.ec2.internal,},FirstTimestamp:2026-07-23 03:55:20.386617868 +0000 UTC m=+0.291922211,LastTimestamp:2026-07-23 03:55:20.386617868 +0000 UTC m=+0.291922211,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-6.ec2.internal,}" Jul 23 03:55:20.396601 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.396587 2567 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 23 03:55:20.396601 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.396596 2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 23 03:55:20.396704 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:20.396694 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-6.ec2.internal\" not found" Jul 23 03:55:20.397222 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.397194 2567 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 23 03:55:20.397322 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.397196 2567 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 23 03:55:20.397322 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.397243 2567 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 23 03:55:20.397322 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.397302 2567 reconstruct.go:97] "Volume reconstruction finished" Jul 23 03:55:20.397322 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.397313 2567 reconciler.go:29] "Reconciler: start to sync state" Jul 23 03:55:20.397518 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.397377 2567 factory.go:55] Registering systemd factory Jul 23 03:55:20.397518 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.397391 2567 factory.go:223] Registration of the systemd container factory successfully Jul 23 03:55:20.397624 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.397608 2567 factory.go:153] Registering CRI-O factory Jul 23 03:55:20.397671 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.397629 2567 factory.go:223] Registration of the crio container factory successfully Jul 23 03:55:20.397721 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.397676 2567 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 23 03:55:20.397721 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.397704 2567 factory.go:103] Registering Raw factory Jul 23 03:55:20.397814 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.397722 2567 manager.go:1196] Started watching for new ooms in manager Jul 23 03:55:20.398058 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.398044 2567 manager.go:349] Starting recovery of all containers Jul 23 03:55:20.399191 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:20.398647 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 23 03:55:20.399191 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:20.398668 2567 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-6.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 23 03:55:20.406982 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.406854 2567 manager.go:354] Recovery completed Jul 23 03:55:20.413074 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.413056 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 03:55:20.414711 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.414694 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-6.ec2.internal" event="NodeHasSufficientMemory" Jul 23 03:55:20.414788 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.414723 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-6.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 03:55:20.414788 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.414733 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-6.ec2.internal" event="NodeHasSufficientPID" Jul 23 03:55:20.415237 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.415220 2567 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 23 03:55:20.415237 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.415236 2567 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 23 03:55:20.415379 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.415247 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 03:55:20.415379 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.415250 2567 state_mem.go:36] "Initialized new in-memory state store" Jul 23 03:55:20.415927 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.415909 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-6.ec2.internal" event="NodeHasSufficientMemory" Jul 23 03:55:20.415992 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.415938 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-6.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 03:55:20.415992 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.415948 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-6.ec2.internal" event="NodeHasSufficientPID" Jul 23 03:55:20.417490 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.417446 2567 policy_none.go:49] "None policy: Start" Jul 23 03:55:20.417490 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.417465 2567 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 23 03:55:20.417490 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.417484 2567 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 23 03:55:20.419456 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.419443 2567 policy_none.go:47] "Start" Jul 23 03:55:20.422637 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:20.422566 2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-6.ec2.internal.18c4cef8d1ec87eb default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-6.ec2.internal,UID:ip-10-0-140-6.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-6.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-6.ec2.internal,},FirstTimestamp:2026-07-23 03:55:20.414709739 +0000 UTC m=+0.320014022,LastTimestamp:2026-07-23 03:55:20.414709739 +0000 UTC m=+0.320014022,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-6.ec2.internal,}" Jul 23 03:55:20.426728 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:20.426658 2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-6.ec2.internal.18c4cef8d1ecce9c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-6.ec2.internal,UID:ip-10-0-140-6.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-6.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-6.ec2.internal,},FirstTimestamp:2026-07-23 03:55:20.414727836 +0000 UTC m=+0.320032119,LastTimestamp:2026-07-23 03:55:20.414727836 +0000 UTC m=+0.320032119,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-6.ec2.internal,}" Jul 23 03:55:20.432393 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:20.432336 2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-6.ec2.internal.18c4cef8d1ecf274 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-6.ec2.internal,UID:ip-10-0-140-6.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-6.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-6.ec2.internal,},FirstTimestamp:2026-07-23 03:55:20.414737012 +0000 UTC m=+0.320041295,LastTimestamp:2026-07-23 03:55:20.414737012 +0000 UTC m=+0.320041295,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-6.ec2.internal,}" Jul 23 03:55:20.436253 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.436230 2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h7rqn" Jul 23 03:55:20.436813 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:20.436752 2567 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-6.ec2.internal.18c4cef8d1ec87eb\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-6.ec2.internal.18c4cef8d1ec87eb default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-6.ec2.internal,UID:ip-10-0-140-6.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-6.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-6.ec2.internal,},FirstTimestamp:2026-07-23 03:55:20.414709739 +0000 UTC m=+0.320014022,LastTimestamp:2026-07-23 03:55:20.415926507 +0000 UTC m=+0.321230790,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-6.ec2.internal,}" Jul 23 03:55:20.440787 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:20.440732 2567 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-6.ec2.internal.18c4cef8d1ecce9c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-6.ec2.internal.18c4cef8d1ecce9c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-6.ec2.internal,UID:ip-10-0-140-6.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-6.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-6.ec2.internal,},FirstTimestamp:2026-07-23 03:55:20.414727836 +0000 UTC m=+0.320032119,LastTimestamp:2026-07-23 03:55:20.415942917 +0000 UTC m=+0.321247199,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-6.ec2.internal,}" Jul 23 03:55:20.443474 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.443443 2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h7rqn" Jul 23 03:55:20.447049 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:20.446998 2567 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-6.ec2.internal.18c4cef8d1ecf274\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-6.ec2.internal.18c4cef8d1ecf274 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-6.ec2.internal,UID:ip-10-0-140-6.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-6.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-6.ec2.internal,},FirstTimestamp:2026-07-23 03:55:20.414737012 +0000 UTC m=+0.320041295,LastTimestamp:2026-07-23 03:55:20.415952039 +0000 UTC m=+0.321256322,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-6.ec2.internal,}" Jul 23 03:55:20.452567 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.452555 2567 manager.go:337] "Starting Device Plugin manager" Jul 23 03:55:20.452617 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:20.452582 2567 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 23 03:55:20.452617 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.452590 2567 server.go:87] "Starting device plugin registration server" Jul 23 03:55:20.452812 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.452800 2567 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 23 03:55:20.452853 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.452815 2567 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 23 03:55:20.452988 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.452968 2567 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 23 03:55:20.453081 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.453046 2567 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 23 03:55:20.453081 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.453053 2567 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 23 03:55:20.453471 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:20.453454 2567 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 23 03:55:20.453572 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:20.453494 2567 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-6.ec2.internal\" not found" Jul 23 03:55:20.484903 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.484882 2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 23 03:55:20.493381 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.486160 2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 23 03:55:20.493381 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.486196 2567 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 23 03:55:20.493381 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.486217 2567 kubelet.go:2448] "Starting kubelet main sync loop" Jul 23 03:55:20.493381 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:20.486256 2567 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 23 03:55:20.493381 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.488100 2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 03:55:20.553494 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.553462 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 03:55:20.555324 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.555310 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-6.ec2.internal" event="NodeHasSufficientMemory" Jul 23 03:55:20.555388 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.555338 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-6.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 03:55:20.555388 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.555348 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-6.ec2.internal" event="NodeHasSufficientPID" Jul 23 03:55:20.555388 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.555370 2567 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-6.ec2.internal" Jul 23 03:55:20.563023 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.563009 2567 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-6.ec2.internal" Jul 23 03:55:20.563076 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:20.563032 2567 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-6.ec2.internal\": node \"ip-10-0-140-6.ec2.internal\" not found" Jul 23 03:55:20.576458 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:20.576384 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-6.ec2.internal\" not found" Jul 23 03:55:20.586462 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.586440 2567 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-6.ec2.internal"] Jul 23 03:55:20.586541 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.586531 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 03:55:20.587919 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.587898 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-6.ec2.internal" event="NodeHasSufficientMemory" Jul 23 03:55:20.587979 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.587928 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-6.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 03:55:20.587979 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.587938 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-6.ec2.internal" event="NodeHasSufficientPID" Jul 23 03:55:20.590307 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.590293 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 03:55:20.590397 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.590383 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal" Jul 23 03:55:20.590468 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.590414 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 03:55:20.590933 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.590915 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-6.ec2.internal" event="NodeHasSufficientMemory" Jul 23 03:55:20.591012 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.590944 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-6.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 03:55:20.591012 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.590953 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-6.ec2.internal" event="NodeHasSufficientPID" Jul 23 03:55:20.591012 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.590915 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-6.ec2.internal" event="NodeHasSufficientMemory" Jul 23 03:55:20.591012 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.591009 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-6.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 03:55:20.591133 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.591023 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-6.ec2.internal" event="NodeHasSufficientPID" Jul 23 03:55:20.593255 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.593240 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-6.ec2.internal" Jul 23 03:55:20.593297 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.593272 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 03:55:20.594023 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.594007 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-6.ec2.internal" event="NodeHasSufficientMemory" Jul 23 03:55:20.594121 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.594037 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-6.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 03:55:20.594121 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.594052 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-6.ec2.internal" event="NodeHasSufficientPID" Jul 23 03:55:20.599085 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.599066 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2d6aefa033c82b0f8916925a50156cfa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal\" (UID: \"2d6aefa033c82b0f8916925a50156cfa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal" Jul 23 03:55:20.599170 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.599092 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d6aefa033c82b0f8916925a50156cfa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal\" (UID: \"2d6aefa033c82b0f8916925a50156cfa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal" Jul 23 03:55:20.599170 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.599108 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b2183c9004d8f7a985e4b7edb883b58a-config\") pod \"kube-apiserver-proxy-ip-10-0-140-6.ec2.internal\" (UID: \"b2183c9004d8f7a985e4b7edb883b58a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-6.ec2.internal" Jul 23 03:55:20.616086 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:20.616069 2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-6.ec2.internal\" not found" node="ip-10-0-140-6.ec2.internal" Jul 23 03:55:20.620730 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:20.620715 2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-6.ec2.internal\" not found" node="ip-10-0-140-6.ec2.internal" Jul 23 03:55:20.676478 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:20.676462 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-6.ec2.internal\" not found" Jul 23 03:55:20.699875 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.699858 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2d6aefa033c82b0f8916925a50156cfa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal\" (UID: \"2d6aefa033c82b0f8916925a50156cfa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal" Jul 23 03:55:20.699932 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.699880 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d6aefa033c82b0f8916925a50156cfa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal\" (UID: \"2d6aefa033c82b0f8916925a50156cfa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal" Jul 23 03:55:20.699932 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.699902 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b2183c9004d8f7a985e4b7edb883b58a-config\") pod \"kube-apiserver-proxy-ip-10-0-140-6.ec2.internal\" (UID: \"b2183c9004d8f7a985e4b7edb883b58a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-6.ec2.internal" Jul 23 03:55:20.699932 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.699925 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b2183c9004d8f7a985e4b7edb883b58a-config\") pod \"kube-apiserver-proxy-ip-10-0-140-6.ec2.internal\" (UID: \"b2183c9004d8f7a985e4b7edb883b58a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-6.ec2.internal" Jul 23 03:55:20.700053 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.699956 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2d6aefa033c82b0f8916925a50156cfa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal\" (UID: \"2d6aefa033c82b0f8916925a50156cfa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal" Jul 23 03:55:20.700053 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.699975 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d6aefa033c82b0f8916925a50156cfa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal\" (UID: \"2d6aefa033c82b0f8916925a50156cfa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal" Jul 23 03:55:20.776830 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:20.776793 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-6.ec2.internal\" not found" Jul 23 03:55:20.877639 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:20.877577 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-6.ec2.internal\" not found" Jul 23 03:55:20.919041 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.919021 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal" Jul 23 03:55:20.924200 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:20.923835 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-6.ec2.internal" Jul 23 03:55:20.978374 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:20.978338 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-6.ec2.internal\" not found" Jul 23 03:55:21.078839 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:21.078800 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-6.ec2.internal\" not found" Jul 23 03:55:21.179360 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:21.179329 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-6.ec2.internal\" not found" Jul 23 03:55:21.279867 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:21.279844 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-6.ec2.internal\" not found" Jul 23 03:55:21.340389 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:21.340359 2567 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 23 03:55:21.340921 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:21.340566 2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 23 03:55:21.380797 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:21.380770 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-6.ec2.internal\" not found" Jul 23 03:55:21.434845 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:21.434796 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2183c9004d8f7a985e4b7edb883b58a.slice/crio-9e1c852a2621ab7628ea59cef2b237f6ad7351105f6f9129a5af6906dc315cab WatchSource:0}: Error finding container 9e1c852a2621ab7628ea59cef2b237f6ad7351105f6f9129a5af6906dc315cab: Status 404 returned error can't find the container with id 9e1c852a2621ab7628ea59cef2b237f6ad7351105f6f9129a5af6906dc315cab Jul 23 03:55:21.435026 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:21.435008 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d6aefa033c82b0f8916925a50156cfa.slice/crio-9b93ed8f033babb85ba953f4ecb5e99a646f10d1b4f6ca7cedf2b6b8d8fa88cc WatchSource:0}: Error finding container 9b93ed8f033babb85ba953f4ecb5e99a646f10d1b4f6ca7cedf2b6b8d8fa88cc: Status 404 returned error can't find the container with id 9b93ed8f033babb85ba953f4ecb5e99a646f10d1b4f6ca7cedf2b6b8d8fa88cc Jul 23 03:55:21.440139 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:21.440120 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 03:55:21.448417 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:21.448389 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-22 03:50:20 +0000 UTC" deadline="2028-04-09 06:38:01.29935041 +0000 UTC" Jul 23 03:55:21.448501 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:21.448415 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15026h42m39.850938288s" Jul 23 03:55:21.468020 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:21.468003 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 03:55:21.468912 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:21.468898 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-6.ec2.internal" event="NodeHasSufficientMemory" Jul 23 03:55:21.468968 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:21.468924 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-6.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 03:55:21.468968 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:21.468940 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-6.ec2.internal" event="NodeHasSufficientPID" Jul 23 03:55:21.473116 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:21.473104 2567 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 03:55:21.481496 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:21.481480 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-6.ec2.internal\" not found" Jul 23 03:55:21.489261 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:21.489223 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal" event={"ID":"2d6aefa033c82b0f8916925a50156cfa","Type":"ContainerStarted","Data":"9b93ed8f033babb85ba953f4ecb5e99a646f10d1b4f6ca7cedf2b6b8d8fa88cc"} Jul 23 03:55:21.490238 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:21.490218 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-6.ec2.internal" event={"ID":"b2183c9004d8f7a985e4b7edb883b58a","Type":"ContainerStarted","Data":"9e1c852a2621ab7628ea59cef2b237f6ad7351105f6f9129a5af6906dc315cab"} Jul 23 03:55:21.581666 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:21.581643 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-6.ec2.internal\" not found" Jul 23 03:55:21.592368 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:21.592351 2567 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 03:55:21.597532 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:21.597497 2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal" Jul 23 03:55:21.608079 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:21.608063 2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 23 03:55:21.609383 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:21.609368 2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-6.ec2.internal" Jul 23 03:55:21.617584 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:21.617568 2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 23 03:55:21.729657 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:21.729600 2567 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 03:55:22.373542 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.373513 2567 apiserver.go:52] "Watching apiserver" Jul 23 03:55:22.381105 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.381080 2567 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 23 03:55:22.382307 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.382280 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-w99tk","openshift-network-operator/iptables-alerter-d8c2n","kube-system/global-pull-secret-syncer-bjprg","kube-system/kube-apiserver-proxy-ip-10-0-140-6.ec2.internal","openshift-dns/node-resolver-gpl6h","openshift-image-registry/node-ca-kw46l","openshift-multus/multus-hk2bm","openshift-ovn-kubernetes/ovnkube-node-wd7gs","kube-system/konnectivity-agent-kt7n9","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f84jx","openshift-cluster-node-tuning-operator/tuned-rc7rw","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal","openshift-multus/multus-additional-cni-plugins-j4hnz","openshift-multus/network-metrics-daemon-pq2xh"] Jul 23 03:55:22.385445 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.384815 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f84jx" Jul 23 03:55:22.387459 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.387438 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-j4hnz" Jul 23 03:55:22.387835 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.387812 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 23 03:55:22.387934 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.387841 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 23 03:55:22.388174 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.388157 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 23 03:55:22.388514 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.388493 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gz7wj\"" Jul 23 03:55:22.389517 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.389499 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjprg" Jul 23 03:55:22.389610 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:22.389551 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjprg" podUID="ec74b46f-3165-4772-9090-b8adbf536767" Jul 23 03:55:22.391786 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.391764 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 23 03:55:22.391846 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.391804 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 23 03:55:22.391846 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.391822 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 23 03:55:22.391846 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.391827 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 23 03:55:22.391955 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.391939 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nvwm8\"" Jul 23 03:55:22.392056 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.392039 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 23 03:55:22.392257 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.392239 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pq2xh" Jul 23 03:55:22.392348 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:22.392329 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pq2xh" podUID="73cf013c-7d0b-4139-bf40-8d5d528474a3" Jul 23 03:55:22.395880 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.395861 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gpl6h" Jul 23 03:55:22.398565 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.397083 2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 23 03:55:22.398565 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.398202 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 23 03:55:22.398565 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.398407 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hwrkm\"" Jul 23 03:55:22.398565 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.398504 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 23 03:55:22.398565 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.398537 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kw46l" Jul 23 03:55:22.400180 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.400164 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kt7n9" Jul 23 03:55:22.401151 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.401130 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 23 03:55:22.401524 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.401496 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 23 03:55:22.401613 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.401533 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 23 03:55:22.401613 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.401567 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-q7pb9\"" Jul 23 03:55:22.402294 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.402278 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 23 03:55:22.402500 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.402485 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 23 03:55:22.402575 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.402517 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5vkwf\"" Jul 23 03:55:22.403590 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.403574 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hk2bm" Jul 23 03:55:22.405969 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.405945 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-zgj4r\"" Jul 23 03:55:22.406101 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.405953 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 23 03:55:22.407690 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.407522 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-d8c2n" Jul 23 03:55:22.408525 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.408411 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec74b46f-3165-4772-9090-b8adbf536767-original-pull-secret\") pod \"global-pull-secret-syncer-bjprg\" (UID: \"ec74b46f-3165-4772-9090-b8adbf536767\") " pod="kube-system/global-pull-secret-syncer-bjprg" Jul 23 03:55:22.408525 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.408469 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/46a75c21-dc90-40a4-b933-cc1e92dac8c7-agent-certs\") pod \"konnectivity-agent-kt7n9\" (UID: \"46a75c21-dc90-40a4-b933-cc1e92dac8c7\") " pod="kube-system/konnectivity-agent-kt7n9" Jul 23 03:55:22.408525 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.408519 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/aa5c99e2-68a8-4f38-9cfe-8c917fb7c61a-sys-fs\") pod \"aws-ebs-csi-driver-node-f84jx\" (UID: \"aa5c99e2-68a8-4f38-9cfe-8c917fb7c61a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f84jx" Jul 23 03:55:22.408678 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.408544 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b6f66fe-a3d8-4ef6-95ee-9016ff91606f-tmp-dir\") pod \"node-resolver-gpl6h\" (UID: \"6b6f66fe-a3d8-4ef6-95ee-9016ff91606f\") " pod="openshift-dns/node-resolver-gpl6h" Jul 23 03:55:22.408757 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.408573 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ea356156-f0f7-42dd-8c96-b30bdede6205-system-cni-dir\") pod \"multus-additional-cni-plugins-j4hnz\" (UID: \"ea356156-f0f7-42dd-8c96-b30bdede6205\") " pod="openshift-multus/multus-additional-cni-plugins-j4hnz" Jul 23 03:55:22.408846 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.408770 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ea356156-f0f7-42dd-8c96-b30bdede6205-cnibin\") pod \"multus-additional-cni-plugins-j4hnz\" (UID: \"ea356156-f0f7-42dd-8c96-b30bdede6205\") " pod="openshift-multus/multus-additional-cni-plugins-j4hnz" Jul 23 03:55:22.408846 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.408796 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ec74b46f-3165-4772-9090-b8adbf536767-dbus\") pod \"global-pull-secret-syncer-bjprg\" (UID: \"ec74b46f-3165-4772-9090-b8adbf536767\") " pod="kube-system/global-pull-secret-syncer-bjprg" Jul 23 03:55:22.408999 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.408956 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/aa5c99e2-68a8-4f38-9cfe-8c917fb7c61a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f84jx\" (UID: \"aa5c99e2-68a8-4f38-9cfe-8c917fb7c61a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f84jx" Jul 23 03:55:22.408999 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.408982 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73cf013c-7d0b-4139-bf40-8d5d528474a3-metrics-certs\") pod \"network-metrics-daemon-pq2xh\" (UID: \"73cf013c-7d0b-4139-bf40-8d5d528474a3\") " pod="openshift-multus/network-metrics-daemon-pq2xh" Jul 23 03:55:22.409499 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.409475 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x29cq\" (UniqueName: \"kubernetes.io/projected/787ce18c-a72b-426f-a9f0-51f2c6f7c9bb-kube-api-access-x29cq\") pod \"node-ca-kw46l\" (UID: \"787ce18c-a72b-426f-a9f0-51f2c6f7c9bb\") " pod="openshift-image-registry/node-ca-kw46l" Jul 23 03:55:22.409583 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.409504 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w99tk" Jul 23 03:55:22.409583 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.409524 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkq68\" (UniqueName: \"kubernetes.io/projected/73cf013c-7d0b-4139-bf40-8d5d528474a3-kube-api-access-zkq68\") pod \"network-metrics-daemon-pq2xh\" (UID: \"73cf013c-7d0b-4139-bf40-8d5d528474a3\") " pod="openshift-multus/network-metrics-daemon-pq2xh" Jul 23 03:55:22.409583 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.409553 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/787ce18c-a72b-426f-a9f0-51f2c6f7c9bb-host\") pod \"node-ca-kw46l\" (UID: \"787ce18c-a72b-426f-a9f0-51f2c6f7c9bb\") " pod="openshift-image-registry/node-ca-kw46l" Jul 23 03:55:22.409583 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.409580 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ea356156-f0f7-42dd-8c96-b30bdede6205-cni-binary-copy\") pod \"multus-additional-cni-plugins-j4hnz\" (UID: \"ea356156-f0f7-42dd-8c96-b30bdede6205\") " pod="openshift-multus/multus-additional-cni-plugins-j4hnz" Jul 23 03:55:22.409765 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:22.409705 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w99tk" podUID="1e14ea08-0647-4056-9f4a-e639fb16c928" Jul 23 03:55:22.409765 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.409713 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/aa5c99e2-68a8-4f38-9cfe-8c917fb7c61a-registration-dir\") pod \"aws-ebs-csi-driver-node-f84jx\" (UID: \"aa5c99e2-68a8-4f38-9cfe-8c917fb7c61a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f84jx" Jul 23 03:55:22.409847 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.409768 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgkds\" (UniqueName: \"kubernetes.io/projected/aa5c99e2-68a8-4f38-9cfe-8c917fb7c61a-kube-api-access-qgkds\") pod \"aws-ebs-csi-driver-node-f84jx\" (UID: \"aa5c99e2-68a8-4f38-9cfe-8c917fb7c61a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f84jx" Jul 23 03:55:22.409847 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.409802 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnzls\" (UniqueName: \"kubernetes.io/projected/ea356156-f0f7-42dd-8c96-b30bdede6205-kube-api-access-cnzls\") pod \"multus-additional-cni-plugins-j4hnz\" (UID: \"ea356156-f0f7-42dd-8c96-b30bdede6205\") " pod="openshift-multus/multus-additional-cni-plugins-j4hnz" Jul 23 03:55:22.409847 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.409834 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ea356156-f0f7-42dd-8c96-b30bdede6205-os-release\") pod \"multus-additional-cni-plugins-j4hnz\" (UID: \"ea356156-f0f7-42dd-8c96-b30bdede6205\") " pod="openshift-multus/multus-additional-cni-plugins-j4hnz" Jul 23 03:55:22.410013 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.409990 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ea356156-f0f7-42dd-8c96-b30bdede6205-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j4hnz\" (UID: \"ea356156-f0f7-42dd-8c96-b30bdede6205\") " pod="openshift-multus/multus-additional-cni-plugins-j4hnz" Jul 23 03:55:22.410145 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.410037 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ec74b46f-3165-4772-9090-b8adbf536767-kubelet-config\") pod \"global-pull-secret-syncer-bjprg\" (UID: \"ec74b46f-3165-4772-9090-b8adbf536767\") " pod="kube-system/global-pull-secret-syncer-bjprg" Jul 23 03:55:22.410145 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.410037 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 23 03:55:22.410145 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.410072 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/aa5c99e2-68a8-4f38-9cfe-8c917fb7c61a-etc-selinux\") pod \"aws-ebs-csi-driver-node-f84jx\" (UID: \"aa5c99e2-68a8-4f38-9cfe-8c917fb7c61a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f84jx" Jul 23 03:55:22.410145 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.410110 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnhj8\" (UniqueName: \"kubernetes.io/projected/6b6f66fe-a3d8-4ef6-95ee-9016ff91606f-kube-api-access-bnhj8\") pod \"node-resolver-gpl6h\" (UID: \"6b6f66fe-a3d8-4ef6-95ee-9016ff91606f\") " pod="openshift-dns/node-resolver-gpl6h" Jul 23 03:55:22.410145 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.410138 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ea356156-f0f7-42dd-8c96-b30bdede6205-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j4hnz\" (UID: \"ea356156-f0f7-42dd-8c96-b30bdede6205\") " pod="openshift-multus/multus-additional-cni-plugins-j4hnz" Jul 23 03:55:22.410355 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.410165 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ea356156-f0f7-42dd-8c96-b30bdede6205-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j4hnz\" (UID: \"ea356156-f0f7-42dd-8c96-b30bdede6205\") " pod="openshift-multus/multus-additional-cni-plugins-j4hnz" Jul 23 03:55:22.410355 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.410247 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/aa5c99e2-68a8-4f38-9cfe-8c917fb7c61a-socket-dir\") pod \"aws-ebs-csi-driver-node-f84jx\" (UID: \"aa5c99e2-68a8-4f38-9cfe-8c917fb7c61a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f84jx" Jul 23 03:55:22.410355 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.410303 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6b6f66fe-a3d8-4ef6-95ee-9016ff91606f-hosts-file\") pod \"node-resolver-gpl6h\" (UID: \"6b6f66fe-a3d8-4ef6-95ee-9016ff91606f\") " pod="openshift-dns/node-resolver-gpl6h" Jul 23 03:55:22.410355 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.410346 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/46a75c21-dc90-40a4-b933-cc1e92dac8c7-konnectivity-ca\") pod \"konnectivity-agent-kt7n9\" (UID: \"46a75c21-dc90-40a4-b933-cc1e92dac8c7\") " pod="kube-system/konnectivity-agent-kt7n9" Jul 23 03:55:22.410565 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.410380 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/aa5c99e2-68a8-4f38-9cfe-8c917fb7c61a-device-dir\") pod \"aws-ebs-csi-driver-node-f84jx\" (UID: \"aa5c99e2-68a8-4f38-9cfe-8c917fb7c61a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f84jx" Jul 23 03:55:22.410565 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.410421 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/787ce18c-a72b-426f-a9f0-51f2c6f7c9bb-serviceca\") pod \"node-ca-kw46l\" (UID: \"787ce18c-a72b-426f-a9f0-51f2c6f7c9bb\") " pod="openshift-image-registry/node-ca-kw46l" Jul 23 03:55:22.411060 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.411042 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-t4w9j\"" Jul 23 03:55:22.411225 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.411206 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 23 03:55:22.411291 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.411272 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 23 03:55:22.412705 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.412685 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rc7rw" Jul 23 03:55:22.415377 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.414960 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 23 03:55:22.415377 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.415043 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 23 03:55:22.415377 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.415073 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ds9tf\"" Jul 23 03:55:22.415377 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.415147 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.418677 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.418657 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 23 03:55:22.418677 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.418669 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 23 03:55:22.418856 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.418840 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 23 03:55:22.418928 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.418915 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-j4vp2\"" Jul 23 03:55:22.418968 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.418952 2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 23 03:55:22.419069 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.419032 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 23 03:55:22.419160 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.419124 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 23 03:55:22.419216 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.419208 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 23 03:55:22.438488 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.438469 2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9qptn" Jul 23 03:55:22.443889 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.443638 2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9qptn" Jul 23 03:55:22.498316 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.498296 2567 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 23 03:55:22.511288 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.511263 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25n67\" (UniqueName: \"kubernetes.io/projected/ec45f896-36a7-4e95-bb7b-98597c796f20-kube-api-access-25n67\") pod \"multus-hk2bm\" (UID: \"ec45f896-36a7-4e95-bb7b-98597c796f20\") " pod="openshift-multus/multus-hk2bm" Jul 23 03:55:22.511407 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.511303 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c4a09228-913b-4aa4-8d53-2252a51162f9-etc-tuned\") pod \"tuned-rc7rw\" (UID: \"c4a09228-913b-4aa4-8d53-2252a51162f9\") " pod="openshift-cluster-node-tuning-operator/tuned-rc7rw" Jul 23 03:55:22.511407 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.511332 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ec74b46f-3165-4772-9090-b8adbf536767-dbus\") pod \"global-pull-secret-syncer-bjprg\" (UID: \"ec74b46f-3165-4772-9090-b8adbf536767\") " pod="kube-system/global-pull-secret-syncer-bjprg" Jul 23 03:55:22.511407 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.511355 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/aa5c99e2-68a8-4f38-9cfe-8c917fb7c61a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f84jx\" (UID: \"aa5c99e2-68a8-4f38-9cfe-8c917fb7c61a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f84jx" Jul 23 03:55:22.511407 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.511379 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73cf013c-7d0b-4139-bf40-8d5d528474a3-metrics-certs\") pod \"network-metrics-daemon-pq2xh\" (UID: \"73cf013c-7d0b-4139-bf40-8d5d528474a3\") " pod="openshift-multus/network-metrics-daemon-pq2xh" Jul 23 03:55:22.511407 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.511403 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x29cq\" (UniqueName: \"kubernetes.io/projected/787ce18c-a72b-426f-a9f0-51f2c6f7c9bb-kube-api-access-x29cq\") pod \"node-ca-kw46l\" (UID: \"787ce18c-a72b-426f-a9f0-51f2c6f7c9bb\") " pod="openshift-image-registry/node-ca-kw46l" Jul 23 03:55:22.511641 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.511442 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ec45f896-36a7-4e95-bb7b-98597c796f20-os-release\") pod \"multus-hk2bm\" (UID: \"ec45f896-36a7-4e95-bb7b-98597c796f20\") " pod="openshift-multus/multus-hk2bm" Jul 23 03:55:22.511641 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.511447 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/aa5c99e2-68a8-4f38-9cfe-8c917fb7c61a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f84jx\" (UID: \"aa5c99e2-68a8-4f38-9cfe-8c917fb7c61a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f84jx" Jul 23 03:55:22.511641 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.511467 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/00f9a9d0-8ca6-412a-85c2-3a6d01bc6264-host-slash\") pod \"iptables-alerter-d8c2n\" (UID: \"00f9a9d0-8ca6-412a-85c2-3a6d01bc6264\") " pod="openshift-network-operator/iptables-alerter-d8c2n" Jul 23 03:55:22.511641 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.511497 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ec74b46f-3165-4772-9090-b8adbf536767-dbus\") pod \"global-pull-secret-syncer-bjprg\" (UID: \"ec74b46f-3165-4772-9090-b8adbf536767\") " pod="kube-system/global-pull-secret-syncer-bjprg" Jul 23 03:55:22.511641 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.511493 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwlgg\" (UniqueName: \"kubernetes.io/projected/c4a09228-913b-4aa4-8d53-2252a51162f9-kube-api-access-pwlgg\") pod \"tuned-rc7rw\" (UID: \"c4a09228-913b-4aa4-8d53-2252a51162f9\") " pod="openshift-cluster-node-tuning-operator/tuned-rc7rw" Jul 23 03:55:22.511641 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:22.511543 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 03:55:22.511641 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:22.511599 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73cf013c-7d0b-4139-bf40-8d5d528474a3-metrics-certs podName:73cf013c-7d0b-4139-bf40-8d5d528474a3 nodeName:}" failed. No retries permitted until 2026-07-23 03:55:23.011583508 +0000 UTC m=+2.916887778 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/73cf013c-7d0b-4139-bf40-8d5d528474a3-metrics-certs") pod "network-metrics-daemon-pq2xh" (UID: "73cf013c-7d0b-4139-bf40-8d5d528474a3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 03:55:22.511641 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.511544 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-host-kubelet\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.511641 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.511636 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/787ce18c-a72b-426f-a9f0-51f2c6f7c9bb-host\") pod \"node-ca-kw46l\" (UID: \"787ce18c-a72b-426f-a9f0-51f2c6f7c9bb\") " pod="openshift-image-registry/node-ca-kw46l" Jul 23 03:55:22.512018 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.511658 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ec45f896-36a7-4e95-bb7b-98597c796f20-cnibin\") pod \"multus-hk2bm\" (UID: \"ec45f896-36a7-4e95-bb7b-98597c796f20\") " pod="openshift-multus/multus-hk2bm" Jul 23 03:55:22.512018 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.511672 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4a09228-913b-4aa4-8d53-2252a51162f9-var-lib-kubelet\") pod \"tuned-rc7rw\" (UID: \"c4a09228-913b-4aa4-8d53-2252a51162f9\") " pod="openshift-cluster-node-tuning-operator/tuned-rc7rw" Jul 23 03:55:22.512018 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.511691 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-ovn-node-metrics-cert\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.512018 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.511722 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/aa5c99e2-68a8-4f38-9cfe-8c917fb7c61a-registration-dir\") pod \"aws-ebs-csi-driver-node-f84jx\" (UID: \"aa5c99e2-68a8-4f38-9cfe-8c917fb7c61a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f84jx" Jul 23 03:55:22.512018 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.511755 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/787ce18c-a72b-426f-a9f0-51f2c6f7c9bb-host\") pod \"node-ca-kw46l\" (UID: \"787ce18c-a72b-426f-a9f0-51f2c6f7c9bb\") " pod="openshift-image-registry/node-ca-kw46l" Jul 23 03:55:22.512018 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.511783 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgkds\" (UniqueName: \"kubernetes.io/projected/aa5c99e2-68a8-4f38-9cfe-8c917fb7c61a-kube-api-access-qgkds\") pod \"aws-ebs-csi-driver-node-f84jx\" (UID: \"aa5c99e2-68a8-4f38-9cfe-8c917fb7c61a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f84jx" Jul 23 03:55:22.512018 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.511811 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/aa5c99e2-68a8-4f38-9cfe-8c917fb7c61a-registration-dir\") pod \"aws-ebs-csi-driver-node-f84jx\" (UID: \"aa5c99e2-68a8-4f38-9cfe-8c917fb7c61a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f84jx" Jul 23 03:55:22.512018 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.511813 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ec45f896-36a7-4e95-bb7b-98597c796f20-host-run-netns\") pod \"multus-hk2bm\" (UID: \"ec45f896-36a7-4e95-bb7b-98597c796f20\") " pod="openshift-multus/multus-hk2bm" Jul 23 03:55:22.512018 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.511922 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ec45f896-36a7-4e95-bb7b-98597c796f20-hostroot\") pod \"multus-hk2bm\" (UID: \"ec45f896-36a7-4e95-bb7b-98597c796f20\") " pod="openshift-multus/multus-hk2bm" Jul 23 03:55:22.512018 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.511937 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-node-log\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.512018 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.511986 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5lbh\" (UniqueName: \"kubernetes.io/projected/00f9a9d0-8ca6-412a-85c2-3a6d01bc6264-kube-api-access-b5lbh\") pod \"iptables-alerter-d8c2n\" (UID: \"00f9a9d0-8ca6-412a-85c2-3a6d01bc6264\") " pod="openshift-network-operator/iptables-alerter-d8c2n" Jul 23 03:55:22.512018 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.512017 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4a09228-913b-4aa4-8d53-2252a51162f9-tmp\") pod \"tuned-rc7rw\" (UID: \"c4a09228-913b-4aa4-8d53-2252a51162f9\") " pod="openshift-cluster-node-tuning-operator/tuned-rc7rw" Jul 23 03:55:22.512561 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.512044 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-systemd-units\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.512561 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.512072 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.512561 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.512112 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/aa5c99e2-68a8-4f38-9cfe-8c917fb7c61a-etc-selinux\") pod \"aws-ebs-csi-driver-node-f84jx\" (UID: \"aa5c99e2-68a8-4f38-9cfe-8c917fb7c61a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f84jx" Jul 23 03:55:22.512561 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.512136 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnhj8\" (UniqueName: \"kubernetes.io/projected/6b6f66fe-a3d8-4ef6-95ee-9016ff91606f-kube-api-access-bnhj8\") pod \"node-resolver-gpl6h\" (UID: \"6b6f66fe-a3d8-4ef6-95ee-9016ff91606f\") " pod="openshift-dns/node-resolver-gpl6h" Jul 23 03:55:22.512561 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.512159 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ec45f896-36a7-4e95-bb7b-98597c796f20-etc-kubernetes\") pod \"multus-hk2bm\" (UID: \"ec45f896-36a7-4e95-bb7b-98597c796f20\") " pod="openshift-multus/multus-hk2bm" Jul 23 03:55:22.512561 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.512181 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c4a09228-913b-4aa4-8d53-2252a51162f9-run\") pod \"tuned-rc7rw\" (UID: \"c4a09228-913b-4aa4-8d53-2252a51162f9\") " pod="openshift-cluster-node-tuning-operator/tuned-rc7rw" Jul 23 03:55:22.512561 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.512223 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c4a09228-913b-4aa4-8d53-2252a51162f9-sys\") pod \"tuned-rc7rw\" (UID: \"c4a09228-913b-4aa4-8d53-2252a51162f9\") " pod="openshift-cluster-node-tuning-operator/tuned-rc7rw" Jul 23 03:55:22.512561 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.512258 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ea356156-f0f7-42dd-8c96-b30bdede6205-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j4hnz\" (UID: \"ea356156-f0f7-42dd-8c96-b30bdede6205\") " pod="openshift-multus/multus-additional-cni-plugins-j4hnz" Jul 23 03:55:22.512561 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.512284 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/aa5c99e2-68a8-4f38-9cfe-8c917fb7c61a-socket-dir\") pod \"aws-ebs-csi-driver-node-f84jx\" (UID: \"aa5c99e2-68a8-4f38-9cfe-8c917fb7c61a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f84jx" Jul 23 03:55:22.512561 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.512306 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6b6f66fe-a3d8-4ef6-95ee-9016ff91606f-hosts-file\") pod \"node-resolver-gpl6h\" (UID: \"6b6f66fe-a3d8-4ef6-95ee-9016ff91606f\") " pod="openshift-dns/node-resolver-gpl6h" Jul 23 03:55:22.512561 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.512331 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ec45f896-36a7-4e95-bb7b-98597c796f20-host-var-lib-cni-bin\") pod \"multus-hk2bm\" (UID: \"ec45f896-36a7-4e95-bb7b-98597c796f20\") " pod="openshift-multus/multus-hk2bm" Jul 23 03:55:22.512561 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.512353 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/00f9a9d0-8ca6-412a-85c2-3a6d01bc6264-iptables-alerter-script\") pod \"iptables-alerter-d8c2n\" (UID: \"00f9a9d0-8ca6-412a-85c2-3a6d01bc6264\") " pod="openshift-network-operator/iptables-alerter-d8c2n" Jul 23 03:55:22.512561 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.512377 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-run-openvswitch\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.512561 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.512401 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-env-overrides\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.512561 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.512420 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ea356156-f0f7-42dd-8c96-b30bdede6205-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j4hnz\" (UID: \"ea356156-f0f7-42dd-8c96-b30bdede6205\") " pod="openshift-multus/multus-additional-cni-plugins-j4hnz" Jul 23 03:55:22.512561 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.512447 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6b6f66fe-a3d8-4ef6-95ee-9016ff91606f-hosts-file\") pod \"node-resolver-gpl6h\" (UID: \"6b6f66fe-a3d8-4ef6-95ee-9016ff91606f\") " pod="openshift-dns/node-resolver-gpl6h" Jul 23 03:55:22.512561 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.512460 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ec45f896-36a7-4e95-bb7b-98597c796f20-host-run-k8s-cni-cncf-io\") pod \"multus-hk2bm\" (UID: \"ec45f896-36a7-4e95-bb7b-98597c796f20\") " pod="openshift-multus/multus-hk2bm" Jul 23 03:55:22.513232 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.512494 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-var-lib-openvswitch\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.513232 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.512526 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec74b46f-3165-4772-9090-b8adbf536767-original-pull-secret\") pod \"global-pull-secret-syncer-bjprg\" (UID: \"ec74b46f-3165-4772-9090-b8adbf536767\") " pod="kube-system/global-pull-secret-syncer-bjprg" Jul 23 03:55:22.513232 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.512550 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b6f66fe-a3d8-4ef6-95ee-9016ff91606f-tmp-dir\") pod \"node-resolver-gpl6h\" (UID: \"6b6f66fe-a3d8-4ef6-95ee-9016ff91606f\") " pod="openshift-dns/node-resolver-gpl6h" Jul 23 03:55:22.513232 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.512566 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/aa5c99e2-68a8-4f38-9cfe-8c917fb7c61a-socket-dir\") pod \"aws-ebs-csi-driver-node-f84jx\" (UID: \"aa5c99e2-68a8-4f38-9cfe-8c917fb7c61a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f84jx" Jul 23 03:55:22.513232 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.512575 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ec45f896-36a7-4e95-bb7b-98597c796f20-multus-cni-dir\") pod \"multus-hk2bm\" (UID: \"ec45f896-36a7-4e95-bb7b-98597c796f20\") " pod="openshift-multus/multus-hk2bm" Jul 23 03:55:22.513232 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.512620 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zrms\" (UniqueName: \"kubernetes.io/projected/1e14ea08-0647-4056-9f4a-e639fb16c928-kube-api-access-4zrms\") pod \"network-check-target-w99tk\" (UID: \"1e14ea08-0647-4056-9f4a-e639fb16c928\") " pod="openshift-network-diagnostics/network-check-target-w99tk" Jul 23 03:55:22.513232 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:22.512656 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 03:55:22.513232 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.512245 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/aa5c99e2-68a8-4f38-9cfe-8c917fb7c61a-etc-selinux\") pod \"aws-ebs-csi-driver-node-f84jx\" (UID: \"aa5c99e2-68a8-4f38-9cfe-8c917fb7c61a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f84jx" Jul 23 03:55:22.513232 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:22.512696 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec74b46f-3165-4772-9090-b8adbf536767-original-pull-secret podName:ec74b46f-3165-4772-9090-b8adbf536767 nodeName:}" failed. No retries permitted until 2026-07-23 03:55:23.012682362 +0000 UTC m=+2.917986633 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ec74b46f-3165-4772-9090-b8adbf536767-original-pull-secret") pod "global-pull-secret-syncer-bjprg" (UID: "ec74b46f-3165-4772-9090-b8adbf536767") : object "kube-system"/"original-pull-secret" not registered Jul 23 03:55:22.513232 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.512722 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4pfd\" (UniqueName: \"kubernetes.io/projected/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-kube-api-access-m4pfd\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.513232 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.512752 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ea356156-f0f7-42dd-8c96-b30bdede6205-system-cni-dir\") pod \"multus-additional-cni-plugins-j4hnz\" (UID: \"ea356156-f0f7-42dd-8c96-b30bdede6205\") " pod="openshift-multus/multus-additional-cni-plugins-j4hnz" Jul 23 03:55:22.513232 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.512774 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ea356156-f0f7-42dd-8c96-b30bdede6205-cnibin\") pod \"multus-additional-cni-plugins-j4hnz\" (UID: \"ea356156-f0f7-42dd-8c96-b30bdede6205\") " pod="openshift-multus/multus-additional-cni-plugins-j4hnz" Jul 23 03:55:22.513232 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.512789 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ec45f896-36a7-4e95-bb7b-98597c796f20-host-var-lib-kubelet\") pod \"multus-hk2bm\" (UID: \"ec45f896-36a7-4e95-bb7b-98597c796f20\") " pod="openshift-multus/multus-hk2bm" Jul 23 03:55:22.513232 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.512810 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c4a09228-913b-4aa4-8d53-2252a51162f9-lib-modules\") pod \"tuned-rc7rw\" (UID: \"c4a09228-913b-4aa4-8d53-2252a51162f9\") " pod="openshift-cluster-node-tuning-operator/tuned-rc7rw" Jul 23 03:55:22.513232 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.512824 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c4a09228-913b-4aa4-8d53-2252a51162f9-host\") pod \"tuned-rc7rw\" (UID: \"c4a09228-913b-4aa4-8d53-2252a51162f9\") " pod="openshift-cluster-node-tuning-operator/tuned-rc7rw" Jul 23 03:55:22.513232 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.512834 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ea356156-f0f7-42dd-8c96-b30bdede6205-system-cni-dir\") pod \"multus-additional-cni-plugins-j4hnz\" (UID: \"ea356156-f0f7-42dd-8c96-b30bdede6205\") " pod="openshift-multus/multus-additional-cni-plugins-j4hnz" Jul 23 03:55:22.513900 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.512848 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-run-systemd\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.513900 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.512864 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ea356156-f0f7-42dd-8c96-b30bdede6205-cnibin\") pod \"multus-additional-cni-plugins-j4hnz\" (UID: \"ea356156-f0f7-42dd-8c96-b30bdede6205\") " pod="openshift-multus/multus-additional-cni-plugins-j4hnz" Jul 23 03:55:22.513900 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.512878 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-ovnkube-config\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.513900 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.512907 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b6f66fe-a3d8-4ef6-95ee-9016ff91606f-tmp-dir\") pod \"node-resolver-gpl6h\" (UID: \"6b6f66fe-a3d8-4ef6-95ee-9016ff91606f\") " pod="openshift-dns/node-resolver-gpl6h" Jul 23 03:55:22.513900 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.512913 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkq68\" (UniqueName: \"kubernetes.io/projected/73cf013c-7d0b-4139-bf40-8d5d528474a3-kube-api-access-zkq68\") pod \"network-metrics-daemon-pq2xh\" (UID: \"73cf013c-7d0b-4139-bf40-8d5d528474a3\") " pod="openshift-multus/network-metrics-daemon-pq2xh" Jul 23 03:55:22.513900 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.512978 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c4a09228-913b-4aa4-8d53-2252a51162f9-etc-sysconfig\") pod \"tuned-rc7rw\" (UID: \"c4a09228-913b-4aa4-8d53-2252a51162f9\") " pod="openshift-cluster-node-tuning-operator/tuned-rc7rw" Jul 23 03:55:22.513900 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.513003 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ea356156-f0f7-42dd-8c96-b30bdede6205-cni-binary-copy\") pod \"multus-additional-cni-plugins-j4hnz\" (UID: \"ea356156-f0f7-42dd-8c96-b30bdede6205\") " pod="openshift-multus/multus-additional-cni-plugins-j4hnz" Jul 23 03:55:22.513900 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.513027 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ec45f896-36a7-4e95-bb7b-98597c796f20-host-var-lib-cni-multus\") pod \"multus-hk2bm\" (UID: \"ec45f896-36a7-4e95-bb7b-98597c796f20\") " pod="openshift-multus/multus-hk2bm" Jul 23 03:55:22.513900 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.513048 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-host-cni-bin\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.513900 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.513072 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnzls\" (UniqueName: \"kubernetes.io/projected/ea356156-f0f7-42dd-8c96-b30bdede6205-kube-api-access-cnzls\") pod \"multus-additional-cni-plugins-j4hnz\" (UID: \"ea356156-f0f7-42dd-8c96-b30bdede6205\") " pod="openshift-multus/multus-additional-cni-plugins-j4hnz" Jul 23 03:55:22.513900 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.513094 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ec45f896-36a7-4e95-bb7b-98597c796f20-multus-daemon-config\") pod \"multus-hk2bm\" (UID: \"ec45f896-36a7-4e95-bb7b-98597c796f20\") " pod="openshift-multus/multus-hk2bm" Jul 23 03:55:22.513900 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.513116 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ec45f896-36a7-4e95-bb7b-98597c796f20-host-run-multus-certs\") pod \"multus-hk2bm\" (UID: \"ec45f896-36a7-4e95-bb7b-98597c796f20\") " pod="openshift-multus/multus-hk2bm" Jul 23 03:55:22.513900 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.513138 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c4a09228-913b-4aa4-8d53-2252a51162f9-etc-modprobe-d\") pod \"tuned-rc7rw\" (UID: \"c4a09228-913b-4aa4-8d53-2252a51162f9\") " pod="openshift-cluster-node-tuning-operator/tuned-rc7rw" Jul 23 03:55:22.513900 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.513159 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4a09228-913b-4aa4-8d53-2252a51162f9-etc-kubernetes\") pod \"tuned-rc7rw\" (UID: \"c4a09228-913b-4aa4-8d53-2252a51162f9\") " pod="openshift-cluster-node-tuning-operator/tuned-rc7rw" Jul 23 03:55:22.513900 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.513180 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c4a09228-913b-4aa4-8d53-2252a51162f9-etc-sysctl-d\") pod \"tuned-rc7rw\" (UID: \"c4a09228-913b-4aa4-8d53-2252a51162f9\") " pod="openshift-cluster-node-tuning-operator/tuned-rc7rw" Jul 23 03:55:22.513900 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.513204 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-host-run-netns\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.513900 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.513227 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-etc-openvswitch\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.514695 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.513261 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ea356156-f0f7-42dd-8c96-b30bdede6205-os-release\") pod \"multus-additional-cni-plugins-j4hnz\" (UID: \"ea356156-f0f7-42dd-8c96-b30bdede6205\") " pod="openshift-multus/multus-additional-cni-plugins-j4hnz" Jul 23 03:55:22.514695 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.513291 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ea356156-f0f7-42dd-8c96-b30bdede6205-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j4hnz\" (UID: \"ea356156-f0f7-42dd-8c96-b30bdede6205\") " pod="openshift-multus/multus-additional-cni-plugins-j4hnz" Jul 23 03:55:22.514695 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.513328 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ec74b46f-3165-4772-9090-b8adbf536767-kubelet-config\") pod \"global-pull-secret-syncer-bjprg\" (UID: \"ec74b46f-3165-4772-9090-b8adbf536767\") " pod="kube-system/global-pull-secret-syncer-bjprg" Jul 23 03:55:22.514695 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.513335 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ea356156-f0f7-42dd-8c96-b30bdede6205-os-release\") pod \"multus-additional-cni-plugins-j4hnz\" (UID: \"ea356156-f0f7-42dd-8c96-b30bdede6205\") " pod="openshift-multus/multus-additional-cni-plugins-j4hnz" Jul 23 03:55:22.514695 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.513354 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c4a09228-913b-4aa4-8d53-2252a51162f9-etc-systemd\") pod \"tuned-rc7rw\" (UID: \"c4a09228-913b-4aa4-8d53-2252a51162f9\") " pod="openshift-cluster-node-tuning-operator/tuned-rc7rw" Jul 23 03:55:22.514695 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.513377 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-log-socket\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.514695 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.513411 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-host-run-ovn-kubernetes\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.514695 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.513454 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-host-cni-netd\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.514695 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.513477 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ea356156-f0f7-42dd-8c96-b30bdede6205-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j4hnz\" (UID: \"ea356156-f0f7-42dd-8c96-b30bdede6205\") " pod="openshift-multus/multus-additional-cni-plugins-j4hnz" Jul 23 03:55:22.514695 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.513502 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ec45f896-36a7-4e95-bb7b-98597c796f20-cni-binary-copy\") pod \"multus-hk2bm\" (UID: \"ec45f896-36a7-4e95-bb7b-98597c796f20\") " pod="openshift-multus/multus-hk2bm" Jul 23 03:55:22.514695 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.513524 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ec45f896-36a7-4e95-bb7b-98597c796f20-multus-socket-dir-parent\") pod \"multus-hk2bm\" (UID: \"ec45f896-36a7-4e95-bb7b-98597c796f20\") " pod="openshift-multus/multus-hk2bm" Jul 23 03:55:22.514695 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.513547 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c4a09228-913b-4aa4-8d53-2252a51162f9-etc-sysctl-conf\") pod \"tuned-rc7rw\" (UID: \"c4a09228-913b-4aa4-8d53-2252a51162f9\") " pod="openshift-cluster-node-tuning-operator/tuned-rc7rw" Jul 23 03:55:22.514695 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.513569 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-host-slash\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.514695 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.513596 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/46a75c21-dc90-40a4-b933-cc1e92dac8c7-konnectivity-ca\") pod \"konnectivity-agent-kt7n9\" (UID: \"46a75c21-dc90-40a4-b933-cc1e92dac8c7\") " pod="kube-system/konnectivity-agent-kt7n9" Jul 23 03:55:22.514695 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.513620 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/aa5c99e2-68a8-4f38-9cfe-8c917fb7c61a-device-dir\") pod \"aws-ebs-csi-driver-node-f84jx\" (UID: \"aa5c99e2-68a8-4f38-9cfe-8c917fb7c61a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f84jx" Jul 23 03:55:22.514695 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.513628 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ea356156-f0f7-42dd-8c96-b30bdede6205-cni-binary-copy\") pod \"multus-additional-cni-plugins-j4hnz\" (UID: \"ea356156-f0f7-42dd-8c96-b30bdede6205\") " pod="openshift-multus/multus-additional-cni-plugins-j4hnz" Jul 23 03:55:22.515307 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.513646 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/787ce18c-a72b-426f-a9f0-51f2c6f7c9bb-serviceca\") pod \"node-ca-kw46l\" (UID: \"787ce18c-a72b-426f-a9f0-51f2c6f7c9bb\") " pod="openshift-image-registry/node-ca-kw46l" Jul 23 03:55:22.515307 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.513708 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-run-ovn\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.515307 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.513732 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-ovnkube-script-lib\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.515307 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.513750 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/46a75c21-dc90-40a4-b933-cc1e92dac8c7-agent-certs\") pod \"konnectivity-agent-kt7n9\" (UID: \"46a75c21-dc90-40a4-b933-cc1e92dac8c7\") " pod="kube-system/konnectivity-agent-kt7n9" Jul 23 03:55:22.515307 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.513765 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/aa5c99e2-68a8-4f38-9cfe-8c917fb7c61a-sys-fs\") pod \"aws-ebs-csi-driver-node-f84jx\" (UID: \"aa5c99e2-68a8-4f38-9cfe-8c917fb7c61a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f84jx" Jul 23 03:55:22.515307 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.513779 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ec45f896-36a7-4e95-bb7b-98597c796f20-system-cni-dir\") pod \"multus-hk2bm\" (UID: \"ec45f896-36a7-4e95-bb7b-98597c796f20\") " pod="openshift-multus/multus-hk2bm" Jul 23 03:55:22.515307 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.513812 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ea356156-f0f7-42dd-8c96-b30bdede6205-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j4hnz\" (UID: \"ea356156-f0f7-42dd-8c96-b30bdede6205\") " pod="openshift-multus/multus-additional-cni-plugins-j4hnz" Jul 23 03:55:22.515307 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.513842 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ec45f896-36a7-4e95-bb7b-98597c796f20-multus-conf-dir\") pod \"multus-hk2bm\" (UID: \"ec45f896-36a7-4e95-bb7b-98597c796f20\") " pod="openshift-multus/multus-hk2bm" Jul 23 03:55:22.515307 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.514139 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ea356156-f0f7-42dd-8c96-b30bdede6205-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j4hnz\" (UID: \"ea356156-f0f7-42dd-8c96-b30bdede6205\") " pod="openshift-multus/multus-additional-cni-plugins-j4hnz" Jul 23 03:55:22.515307 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.514222 2567 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 23 03:55:22.515307 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.514230 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/787ce18c-a72b-426f-a9f0-51f2c6f7c9bb-serviceca\") pod \"node-ca-kw46l\" (UID: \"787ce18c-a72b-426f-a9f0-51f2c6f7c9bb\") " pod="openshift-image-registry/node-ca-kw46l" Jul 23 03:55:22.515307 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.514295 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ec74b46f-3165-4772-9090-b8adbf536767-kubelet-config\") pod \"global-pull-secret-syncer-bjprg\" (UID: \"ec74b46f-3165-4772-9090-b8adbf536767\") " pod="kube-system/global-pull-secret-syncer-bjprg" Jul 23 03:55:22.515307 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.514618 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/46a75c21-dc90-40a4-b933-cc1e92dac8c7-konnectivity-ca\") pod \"konnectivity-agent-kt7n9\" (UID: \"46a75c21-dc90-40a4-b933-cc1e92dac8c7\") " pod="kube-system/konnectivity-agent-kt7n9" Jul 23 03:55:22.515307 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.514639 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/aa5c99e2-68a8-4f38-9cfe-8c917fb7c61a-device-dir\") pod \"aws-ebs-csi-driver-node-f84jx\" (UID: \"aa5c99e2-68a8-4f38-9cfe-8c917fb7c61a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f84jx" Jul 23 03:55:22.515307 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.514683 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/aa5c99e2-68a8-4f38-9cfe-8c917fb7c61a-sys-fs\") pod \"aws-ebs-csi-driver-node-f84jx\" (UID: \"aa5c99e2-68a8-4f38-9cfe-8c917fb7c61a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f84jx" Jul 23 03:55:22.517519 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.517501 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/46a75c21-dc90-40a4-b933-cc1e92dac8c7-agent-certs\") pod \"konnectivity-agent-kt7n9\" (UID: \"46a75c21-dc90-40a4-b933-cc1e92dac8c7\") " pod="kube-system/konnectivity-agent-kt7n9" Jul 23 03:55:22.525090 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.525065 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnhj8\" (UniqueName: \"kubernetes.io/projected/6b6f66fe-a3d8-4ef6-95ee-9016ff91606f-kube-api-access-bnhj8\") pod \"node-resolver-gpl6h\" (UID: \"6b6f66fe-a3d8-4ef6-95ee-9016ff91606f\") " pod="openshift-dns/node-resolver-gpl6h" Jul 23 03:55:22.525521 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.525470 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkq68\" (UniqueName: \"kubernetes.io/projected/73cf013c-7d0b-4139-bf40-8d5d528474a3-kube-api-access-zkq68\") pod \"network-metrics-daemon-pq2xh\" (UID: \"73cf013c-7d0b-4139-bf40-8d5d528474a3\") " pod="openshift-multus/network-metrics-daemon-pq2xh" Jul 23 03:55:22.525753 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.525732 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x29cq\" (UniqueName: \"kubernetes.io/projected/787ce18c-a72b-426f-a9f0-51f2c6f7c9bb-kube-api-access-x29cq\") pod \"node-ca-kw46l\" (UID: \"787ce18c-a72b-426f-a9f0-51f2c6f7c9bb\") " pod="openshift-image-registry/node-ca-kw46l" Jul 23 03:55:22.526028 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.526010 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgkds\" (UniqueName: \"kubernetes.io/projected/aa5c99e2-68a8-4f38-9cfe-8c917fb7c61a-kube-api-access-qgkds\") pod \"aws-ebs-csi-driver-node-f84jx\" (UID: \"aa5c99e2-68a8-4f38-9cfe-8c917fb7c61a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f84jx" Jul 23 03:55:22.526111 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.526079 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnzls\" (UniqueName: \"kubernetes.io/projected/ea356156-f0f7-42dd-8c96-b30bdede6205-kube-api-access-cnzls\") pod \"multus-additional-cni-plugins-j4hnz\" (UID: \"ea356156-f0f7-42dd-8c96-b30bdede6205\") " pod="openshift-multus/multus-additional-cni-plugins-j4hnz" Jul 23 03:55:22.614920 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.614884 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c4a09228-913b-4aa4-8d53-2252a51162f9-etc-sysctl-conf\") pod \"tuned-rc7rw\" (UID: \"c4a09228-913b-4aa4-8d53-2252a51162f9\") " pod="openshift-cluster-node-tuning-operator/tuned-rc7rw" Jul 23 03:55:22.615068 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.614930 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-host-slash\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.615068 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.614956 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-run-ovn\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.615068 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.614980 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-ovnkube-script-lib\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.615068 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.615008 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ec45f896-36a7-4e95-bb7b-98597c796f20-system-cni-dir\") pod \"multus-hk2bm\" (UID: \"ec45f896-36a7-4e95-bb7b-98597c796f20\") " pod="openshift-multus/multus-hk2bm" Jul 23 03:55:22.615068 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.615031 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ec45f896-36a7-4e95-bb7b-98597c796f20-multus-conf-dir\") pod \"multus-hk2bm\" (UID: \"ec45f896-36a7-4e95-bb7b-98597c796f20\") " pod="openshift-multus/multus-hk2bm" Jul 23 03:55:22.615068 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.615052 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25n67\" (UniqueName: \"kubernetes.io/projected/ec45f896-36a7-4e95-bb7b-98597c796f20-kube-api-access-25n67\") pod \"multus-hk2bm\" (UID: \"ec45f896-36a7-4e95-bb7b-98597c796f20\") " pod="openshift-multus/multus-hk2bm" Jul 23 03:55:22.615291 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.615074 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c4a09228-913b-4aa4-8d53-2252a51162f9-etc-tuned\") pod \"tuned-rc7rw\" (UID: \"c4a09228-913b-4aa4-8d53-2252a51162f9\") " pod="openshift-cluster-node-tuning-operator/tuned-rc7rw" Jul 23 03:55:22.615291 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.615085 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c4a09228-913b-4aa4-8d53-2252a51162f9-etc-sysctl-conf\") pod \"tuned-rc7rw\" (UID: \"c4a09228-913b-4aa4-8d53-2252a51162f9\") " pod="openshift-cluster-node-tuning-operator/tuned-rc7rw" Jul 23 03:55:22.615291 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.615137 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ec45f896-36a7-4e95-bb7b-98597c796f20-os-release\") pod \"multus-hk2bm\" (UID: \"ec45f896-36a7-4e95-bb7b-98597c796f20\") " pod="openshift-multus/multus-hk2bm" Jul 23 03:55:22.615291 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.615166 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/00f9a9d0-8ca6-412a-85c2-3a6d01bc6264-host-slash\") pod \"iptables-alerter-d8c2n\" (UID: \"00f9a9d0-8ca6-412a-85c2-3a6d01bc6264\") " pod="openshift-network-operator/iptables-alerter-d8c2n" Jul 23 03:55:22.615291 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.615193 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwlgg\" (UniqueName: \"kubernetes.io/projected/c4a09228-913b-4aa4-8d53-2252a51162f9-kube-api-access-pwlgg\") pod \"tuned-rc7rw\" (UID: \"c4a09228-913b-4aa4-8d53-2252a51162f9\") " pod="openshift-cluster-node-tuning-operator/tuned-rc7rw" Jul 23 03:55:22.615291 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.615220 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-host-kubelet\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.615291 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.615245 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ec45f896-36a7-4e95-bb7b-98597c796f20-cnibin\") pod \"multus-hk2bm\" (UID: \"ec45f896-36a7-4e95-bb7b-98597c796f20\") " pod="openshift-multus/multus-hk2bm" Jul 23 03:55:22.615291 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.615269 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4a09228-913b-4aa4-8d53-2252a51162f9-var-lib-kubelet\") pod \"tuned-rc7rw\" (UID: \"c4a09228-913b-4aa4-8d53-2252a51162f9\") " pod="openshift-cluster-node-tuning-operator/tuned-rc7rw" Jul 23 03:55:22.615606 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.615293 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-ovn-node-metrics-cert\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.615606 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.615318 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ec45f896-36a7-4e95-bb7b-98597c796f20-host-run-netns\") pod \"multus-hk2bm\" (UID: \"ec45f896-36a7-4e95-bb7b-98597c796f20\") " pod="openshift-multus/multus-hk2bm" Jul 23 03:55:22.615606 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.615341 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ec45f896-36a7-4e95-bb7b-98597c796f20-hostroot\") pod \"multus-hk2bm\" (UID: \"ec45f896-36a7-4e95-bb7b-98597c796f20\") " pod="openshift-multus/multus-hk2bm" Jul 23 03:55:22.615606 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.615363 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-node-log\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.615606 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.615387 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5lbh\" (UniqueName: \"kubernetes.io/projected/00f9a9d0-8ca6-412a-85c2-3a6d01bc6264-kube-api-access-b5lbh\") pod \"iptables-alerter-d8c2n\" (UID: \"00f9a9d0-8ca6-412a-85c2-3a6d01bc6264\") " pod="openshift-network-operator/iptables-alerter-d8c2n" Jul 23 03:55:22.615606 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.615411 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4a09228-913b-4aa4-8d53-2252a51162f9-tmp\") pod \"tuned-rc7rw\" (UID: \"c4a09228-913b-4aa4-8d53-2252a51162f9\") " pod="openshift-cluster-node-tuning-operator/tuned-rc7rw" Jul 23 03:55:22.615606 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.615450 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-systemd-units\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.615606 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.615475 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.615606 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.615505 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ec45f896-36a7-4e95-bb7b-98597c796f20-etc-kubernetes\") pod \"multus-hk2bm\" (UID: \"ec45f896-36a7-4e95-bb7b-98597c796f20\") " pod="openshift-multus/multus-hk2bm" Jul 23 03:55:22.615606 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.615527 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c4a09228-913b-4aa4-8d53-2252a51162f9-run\") pod \"tuned-rc7rw\" (UID: \"c4a09228-913b-4aa4-8d53-2252a51162f9\") " pod="openshift-cluster-node-tuning-operator/tuned-rc7rw" Jul 23 03:55:22.615606 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.615548 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c4a09228-913b-4aa4-8d53-2252a51162f9-sys\") pod \"tuned-rc7rw\" (UID: \"c4a09228-913b-4aa4-8d53-2252a51162f9\") " pod="openshift-cluster-node-tuning-operator/tuned-rc7rw" Jul 23 03:55:22.615606 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.615574 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ec45f896-36a7-4e95-bb7b-98597c796f20-host-var-lib-cni-bin\") pod \"multus-hk2bm\" (UID: \"ec45f896-36a7-4e95-bb7b-98597c796f20\") " pod="openshift-multus/multus-hk2bm" Jul 23 03:55:22.615606 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.615600 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/00f9a9d0-8ca6-412a-85c2-3a6d01bc6264-iptables-alerter-script\") pod \"iptables-alerter-d8c2n\" (UID: \"00f9a9d0-8ca6-412a-85c2-3a6d01bc6264\") " pod="openshift-network-operator/iptables-alerter-d8c2n" Jul 23 03:55:22.616028 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.615626 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-run-openvswitch\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.616028 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.615649 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-env-overrides\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.616028 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.615679 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ec45f896-36a7-4e95-bb7b-98597c796f20-host-run-k8s-cni-cncf-io\") pod \"multus-hk2bm\" (UID: \"ec45f896-36a7-4e95-bb7b-98597c796f20\") " pod="openshift-multus/multus-hk2bm" Jul 23 03:55:22.616028 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.615703 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-var-lib-openvswitch\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.616028 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.615743 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ec45f896-36a7-4e95-bb7b-98597c796f20-multus-cni-dir\") pod \"multus-hk2bm\" (UID: \"ec45f896-36a7-4e95-bb7b-98597c796f20\") " pod="openshift-multus/multus-hk2bm" Jul 23 03:55:22.616028 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.615765 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zrms\" (UniqueName: \"kubernetes.io/projected/1e14ea08-0647-4056-9f4a-e639fb16c928-kube-api-access-4zrms\") pod \"network-check-target-w99tk\" (UID: \"1e14ea08-0647-4056-9f4a-e639fb16c928\") " pod="openshift-network-diagnostics/network-check-target-w99tk" Jul 23 03:55:22.616028 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.615793 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4pfd\" (UniqueName: \"kubernetes.io/projected/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-kube-api-access-m4pfd\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.616028 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.615821 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ec45f896-36a7-4e95-bb7b-98597c796f20-host-var-lib-kubelet\") pod \"multus-hk2bm\" (UID: \"ec45f896-36a7-4e95-bb7b-98597c796f20\") " pod="openshift-multus/multus-hk2bm" Jul 23 03:55:22.616028 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.615846 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c4a09228-913b-4aa4-8d53-2252a51162f9-lib-modules\") pod \"tuned-rc7rw\" (UID: \"c4a09228-913b-4aa4-8d53-2252a51162f9\") " pod="openshift-cluster-node-tuning-operator/tuned-rc7rw" Jul 23 03:55:22.616028 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.615867 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c4a09228-913b-4aa4-8d53-2252a51162f9-host\") pod \"tuned-rc7rw\" (UID: \"c4a09228-913b-4aa4-8d53-2252a51162f9\") " pod="openshift-cluster-node-tuning-operator/tuned-rc7rw" Jul 23 03:55:22.616028 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.615891 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-run-systemd\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.616028 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.615918 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-ovnkube-config\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.616028 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.615944 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c4a09228-913b-4aa4-8d53-2252a51162f9-etc-sysconfig\") pod \"tuned-rc7rw\" (UID: \"c4a09228-913b-4aa4-8d53-2252a51162f9\") " pod="openshift-cluster-node-tuning-operator/tuned-rc7rw" Jul 23 03:55:22.616028 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.615970 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ec45f896-36a7-4e95-bb7b-98597c796f20-host-var-lib-cni-multus\") pod \"multus-hk2bm\" (UID: \"ec45f896-36a7-4e95-bb7b-98597c796f20\") " pod="openshift-multus/multus-hk2bm" Jul 23 03:55:22.616028 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.615995 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-host-cni-bin\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.616028 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.616020 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ec45f896-36a7-4e95-bb7b-98597c796f20-multus-daemon-config\") pod \"multus-hk2bm\" (UID: \"ec45f896-36a7-4e95-bb7b-98597c796f20\") " pod="openshift-multus/multus-hk2bm" Jul 23 03:55:22.616641 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.616053 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ec45f896-36a7-4e95-bb7b-98597c796f20-host-run-multus-certs\") pod \"multus-hk2bm\" (UID: \"ec45f896-36a7-4e95-bb7b-98597c796f20\") " pod="openshift-multus/multus-hk2bm" Jul 23 03:55:22.616641 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.616077 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c4a09228-913b-4aa4-8d53-2252a51162f9-etc-modprobe-d\") pod \"tuned-rc7rw\" (UID: \"c4a09228-913b-4aa4-8d53-2252a51162f9\") " pod="openshift-cluster-node-tuning-operator/tuned-rc7rw" Jul 23 03:55:22.616641 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.616100 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4a09228-913b-4aa4-8d53-2252a51162f9-etc-kubernetes\") pod \"tuned-rc7rw\" (UID: \"c4a09228-913b-4aa4-8d53-2252a51162f9\") " pod="openshift-cluster-node-tuning-operator/tuned-rc7rw" Jul 23 03:55:22.616641 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.616124 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c4a09228-913b-4aa4-8d53-2252a51162f9-etc-sysctl-d\") pod \"tuned-rc7rw\" (UID: \"c4a09228-913b-4aa4-8d53-2252a51162f9\") " pod="openshift-cluster-node-tuning-operator/tuned-rc7rw" Jul 23 03:55:22.616641 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.616147 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-host-run-netns\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.616641 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.616170 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-etc-openvswitch\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.616641 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.616198 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c4a09228-913b-4aa4-8d53-2252a51162f9-etc-systemd\") pod \"tuned-rc7rw\" (UID: \"c4a09228-913b-4aa4-8d53-2252a51162f9\") " pod="openshift-cluster-node-tuning-operator/tuned-rc7rw" Jul 23 03:55:22.616641 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.616222 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-log-socket\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.616641 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.616250 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-host-run-ovn-kubernetes\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.616641 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.616275 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-host-cni-netd\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.616641 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.616301 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ec45f896-36a7-4e95-bb7b-98597c796f20-cni-binary-copy\") pod \"multus-hk2bm\" (UID: \"ec45f896-36a7-4e95-bb7b-98597c796f20\") " pod="openshift-multus/multus-hk2bm" Jul 23 03:55:22.616641 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.616329 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ec45f896-36a7-4e95-bb7b-98597c796f20-multus-socket-dir-parent\") pod \"multus-hk2bm\" (UID: \"ec45f896-36a7-4e95-bb7b-98597c796f20\") " pod="openshift-multus/multus-hk2bm" Jul 23 03:55:22.616641 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.616395 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-ovnkube-script-lib\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.616641 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.616446 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ec45f896-36a7-4e95-bb7b-98597c796f20-multus-socket-dir-parent\") pod \"multus-hk2bm\" (UID: \"ec45f896-36a7-4e95-bb7b-98597c796f20\") " pod="openshift-multus/multus-hk2bm" Jul 23 03:55:22.616641 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.616466 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-host-slash\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.616641 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.616495 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-run-ovn\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.616641 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.616506 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ec45f896-36a7-4e95-bb7b-98597c796f20-os-release\") pod \"multus-hk2bm\" (UID: \"ec45f896-36a7-4e95-bb7b-98597c796f20\") " pod="openshift-multus/multus-hk2bm" Jul 23 03:55:22.616641 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.616524 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ec45f896-36a7-4e95-bb7b-98597c796f20-multus-conf-dir\") pod \"multus-hk2bm\" (UID: \"ec45f896-36a7-4e95-bb7b-98597c796f20\") " pod="openshift-multus/multus-hk2bm" Jul 23 03:55:22.617195 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.616546 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/00f9a9d0-8ca6-412a-85c2-3a6d01bc6264-host-slash\") pod \"iptables-alerter-d8c2n\" (UID: \"00f9a9d0-8ca6-412a-85c2-3a6d01bc6264\") " pod="openshift-network-operator/iptables-alerter-d8c2n" Jul 23 03:55:22.617195 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.616576 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ec45f896-36a7-4e95-bb7b-98597c796f20-system-cni-dir\") pod \"multus-hk2bm\" (UID: \"ec45f896-36a7-4e95-bb7b-98597c796f20\") " pod="openshift-multus/multus-hk2bm" Jul 23 03:55:22.617195 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.616795 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-host-kubelet\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.617195 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.616838 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ec45f896-36a7-4e95-bb7b-98597c796f20-cnibin\") pod \"multus-hk2bm\" (UID: \"ec45f896-36a7-4e95-bb7b-98597c796f20\") " pod="openshift-multus/multus-hk2bm" Jul 23 03:55:22.617195 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.616870 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4a09228-913b-4aa4-8d53-2252a51162f9-var-lib-kubelet\") pod \"tuned-rc7rw\" (UID: \"c4a09228-913b-4aa4-8d53-2252a51162f9\") " pod="openshift-cluster-node-tuning-operator/tuned-rc7rw" Jul 23 03:55:22.617195 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.616909 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ec45f896-36a7-4e95-bb7b-98597c796f20-host-var-lib-kubelet\") pod \"multus-hk2bm\" (UID: \"ec45f896-36a7-4e95-bb7b-98597c796f20\") " pod="openshift-multus/multus-hk2bm" Jul 23 03:55:22.617195 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.617034 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c4a09228-913b-4aa4-8d53-2252a51162f9-lib-modules\") pod \"tuned-rc7rw\" (UID: \"c4a09228-913b-4aa4-8d53-2252a51162f9\") " pod="openshift-cluster-node-tuning-operator/tuned-rc7rw" Jul 23 03:55:22.617195 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.617095 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c4a09228-913b-4aa4-8d53-2252a51162f9-host\") pod \"tuned-rc7rw\" (UID: \"c4a09228-913b-4aa4-8d53-2252a51162f9\") " pod="openshift-cluster-node-tuning-operator/tuned-rc7rw" Jul 23 03:55:22.617195 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.617142 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-run-systemd\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.618564 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.617562 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c4a09228-913b-4aa4-8d53-2252a51162f9-run\") pod \"tuned-rc7rw\" (UID: \"c4a09228-913b-4aa4-8d53-2252a51162f9\") " pod="openshift-cluster-node-tuning-operator/tuned-rc7rw" Jul 23 03:55:22.618564 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.617624 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ec45f896-36a7-4e95-bb7b-98597c796f20-host-run-netns\") pod \"multus-hk2bm\" (UID: \"ec45f896-36a7-4e95-bb7b-98597c796f20\") " pod="openshift-multus/multus-hk2bm" Jul 23 03:55:22.618564 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.617677 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c4a09228-913b-4aa4-8d53-2252a51162f9-etc-sysconfig\") pod \"tuned-rc7rw\" (UID: \"c4a09228-913b-4aa4-8d53-2252a51162f9\") " pod="openshift-cluster-node-tuning-operator/tuned-rc7rw" Jul 23 03:55:22.618564 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.617677 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-run-openvswitch\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.618564 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.617711 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-node-log\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.618564 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.617728 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c4a09228-913b-4aa4-8d53-2252a51162f9-etc-tuned\") pod \"tuned-rc7rw\" (UID: \"c4a09228-913b-4aa4-8d53-2252a51162f9\") " pod="openshift-cluster-node-tuning-operator/tuned-rc7rw" Jul 23 03:55:22.618564 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.617738 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ec45f896-36a7-4e95-bb7b-98597c796f20-host-var-lib-cni-multus\") pod \"multus-hk2bm\" (UID: \"ec45f896-36a7-4e95-bb7b-98597c796f20\") " pod="openshift-multus/multus-hk2bm" Jul 23 03:55:22.618564 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.617762 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c4a09228-913b-4aa4-8d53-2252a51162f9-sys\") pod \"tuned-rc7rw\" (UID: \"c4a09228-913b-4aa4-8d53-2252a51162f9\") " pod="openshift-cluster-node-tuning-operator/tuned-rc7rw" Jul 23 03:55:22.618564 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.617785 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-host-cni-bin\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.618564 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.617798 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-host-run-netns\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.618564 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.617826 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-etc-openvswitch\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.618564 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.617836 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ec45f896-36a7-4e95-bb7b-98597c796f20-host-var-lib-cni-bin\") pod \"multus-hk2bm\" (UID: \"ec45f896-36a7-4e95-bb7b-98597c796f20\") " pod="openshift-multus/multus-hk2bm" Jul 23 03:55:22.618564 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.617876 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c4a09228-913b-4aa4-8d53-2252a51162f9-etc-systemd\") pod \"tuned-rc7rw\" (UID: \"c4a09228-913b-4aa4-8d53-2252a51162f9\") " pod="openshift-cluster-node-tuning-operator/tuned-rc7rw" Jul 23 03:55:22.618564 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.617917 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-log-socket\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.618564 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.617955 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-host-run-ovn-kubernetes\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.618564 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.617992 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-host-cni-netd\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.618564 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.618119 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ec45f896-36a7-4e95-bb7b-98597c796f20-hostroot\") pod \"multus-hk2bm\" (UID: \"ec45f896-36a7-4e95-bb7b-98597c796f20\") " pod="openshift-multus/multus-hk2bm" Jul 23 03:55:22.618564 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.618230 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c4a09228-913b-4aa4-8d53-2252a51162f9-etc-modprobe-d\") pod \"tuned-rc7rw\" (UID: \"c4a09228-913b-4aa4-8d53-2252a51162f9\") " pod="openshift-cluster-node-tuning-operator/tuned-rc7rw" Jul 23 03:55:22.619364 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.618283 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ec45f896-36a7-4e95-bb7b-98597c796f20-host-run-multus-certs\") pod \"multus-hk2bm\" (UID: \"ec45f896-36a7-4e95-bb7b-98597c796f20\") " pod="openshift-multus/multus-hk2bm" Jul 23 03:55:22.619364 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.618297 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ec45f896-36a7-4e95-bb7b-98597c796f20-multus-daemon-config\") pod \"multus-hk2bm\" (UID: \"ec45f896-36a7-4e95-bb7b-98597c796f20\") " pod="openshift-multus/multus-hk2bm" Jul 23 03:55:22.619364 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.618335 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4a09228-913b-4aa4-8d53-2252a51162f9-etc-kubernetes\") pod \"tuned-rc7rw\" (UID: \"c4a09228-913b-4aa4-8d53-2252a51162f9\") " pod="openshift-cluster-node-tuning-operator/tuned-rc7rw" Jul 23 03:55:22.619364 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.618357 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/00f9a9d0-8ca6-412a-85c2-3a6d01bc6264-iptables-alerter-script\") pod \"iptables-alerter-d8c2n\" (UID: \"00f9a9d0-8ca6-412a-85c2-3a6d01bc6264\") " pod="openshift-network-operator/iptables-alerter-d8c2n" Jul 23 03:55:22.619364 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.618380 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ec45f896-36a7-4e95-bb7b-98597c796f20-host-run-k8s-cni-cncf-io\") pod \"multus-hk2bm\" (UID: \"ec45f896-36a7-4e95-bb7b-98597c796f20\") " pod="openshift-multus/multus-hk2bm" Jul 23 03:55:22.619364 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.618418 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c4a09228-913b-4aa4-8d53-2252a51162f9-etc-sysctl-d\") pod \"tuned-rc7rw\" (UID: \"c4a09228-913b-4aa4-8d53-2252a51162f9\") " pod="openshift-cluster-node-tuning-operator/tuned-rc7rw" Jul 23 03:55:22.619364 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.618453 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-systemd-units\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.619364 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.618481 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ec45f896-36a7-4e95-bb7b-98597c796f20-etc-kubernetes\") pod \"multus-hk2bm\" (UID: \"ec45f896-36a7-4e95-bb7b-98597c796f20\") " pod="openshift-multus/multus-hk2bm" Jul 23 03:55:22.619364 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.618498 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ec45f896-36a7-4e95-bb7b-98597c796f20-cni-binary-copy\") pod \"multus-hk2bm\" (UID: \"ec45f896-36a7-4e95-bb7b-98597c796f20\") " pod="openshift-multus/multus-hk2bm" Jul 23 03:55:22.619364 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.618501 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-var-lib-openvswitch\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.619364 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.618530 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ec45f896-36a7-4e95-bb7b-98597c796f20-multus-cni-dir\") pod \"multus-hk2bm\" (UID: \"ec45f896-36a7-4e95-bb7b-98597c796f20\") " pod="openshift-multus/multus-hk2bm" Jul 23 03:55:22.619364 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.618561 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.619364 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.618728 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-ovnkube-config\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.619364 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.618752 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-env-overrides\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.620069 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.620050 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-ovn-node-metrics-cert\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.620297 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.620281 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4a09228-913b-4aa4-8d53-2252a51162f9-tmp\") pod \"tuned-rc7rw\" (UID: \"c4a09228-913b-4aa4-8d53-2252a51162f9\") " pod="openshift-cluster-node-tuning-operator/tuned-rc7rw" Jul 23 03:55:22.629563 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:22.629509 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 03:55:22.629563 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:22.629535 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 03:55:22.629563 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:22.629549 2567 projected.go:196] Error preparing data for projected volume kube-api-access-4zrms for pod openshift-network-diagnostics/network-check-target-w99tk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 03:55:22.629801 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:22.629618 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e14ea08-0647-4056-9f4a-e639fb16c928-kube-api-access-4zrms podName:1e14ea08-0647-4056-9f4a-e639fb16c928 nodeName:}" failed. No retries permitted until 2026-07-23 03:55:23.129601781 +0000 UTC m=+3.034906068 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4zrms" (UniqueName: "kubernetes.io/projected/1e14ea08-0647-4056-9f4a-e639fb16c928-kube-api-access-4zrms") pod "network-check-target-w99tk" (UID: "1e14ea08-0647-4056-9f4a-e639fb16c928") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 03:55:22.631251 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.631227 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5lbh\" (UniqueName: \"kubernetes.io/projected/00f9a9d0-8ca6-412a-85c2-3a6d01bc6264-kube-api-access-b5lbh\") pod \"iptables-alerter-d8c2n\" (UID: \"00f9a9d0-8ca6-412a-85c2-3a6d01bc6264\") " pod="openshift-network-operator/iptables-alerter-d8c2n" Jul 23 03:55:22.631884 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.631863 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25n67\" (UniqueName: \"kubernetes.io/projected/ec45f896-36a7-4e95-bb7b-98597c796f20-kube-api-access-25n67\") pod \"multus-hk2bm\" (UID: \"ec45f896-36a7-4e95-bb7b-98597c796f20\") " pod="openshift-multus/multus-hk2bm" Jul 23 03:55:22.632166 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.632147 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwlgg\" (UniqueName: \"kubernetes.io/projected/c4a09228-913b-4aa4-8d53-2252a51162f9-kube-api-access-pwlgg\") pod \"tuned-rc7rw\" (UID: \"c4a09228-913b-4aa4-8d53-2252a51162f9\") " pod="openshift-cluster-node-tuning-operator/tuned-rc7rw" Jul 23 03:55:22.632255 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.632237 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4pfd\" (UniqueName: \"kubernetes.io/projected/4c32148f-cbbb-4823-95ae-cd3da6b92cd9-kube-api-access-m4pfd\") pod \"ovnkube-node-wd7gs\" (UID: \"4c32148f-cbbb-4823-95ae-cd3da6b92cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.696476 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.696448 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f84jx" Jul 23 03:55:22.704547 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.704526 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-j4hnz" Jul 23 03:55:22.714303 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.714282 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gpl6h" Jul 23 03:55:22.720988 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.720970 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kw46l" Jul 23 03:55:22.726569 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.726554 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kt7n9" Jul 23 03:55:22.734150 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.734132 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hk2bm" Jul 23 03:55:22.742667 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.742649 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-d8c2n" Jul 23 03:55:22.750326 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.750310 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rc7rw" Jul 23 03:55:22.756980 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.756963 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:22.774096 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:22.774078 2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 03:55:23.018288 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:23.018242 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73cf013c-7d0b-4139-bf40-8d5d528474a3-metrics-certs\") pod \"network-metrics-daemon-pq2xh\" (UID: \"73cf013c-7d0b-4139-bf40-8d5d528474a3\") " pod="openshift-multus/network-metrics-daemon-pq2xh" Jul 23 03:55:23.018472 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:23.018300 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec74b46f-3165-4772-9090-b8adbf536767-original-pull-secret\") pod \"global-pull-secret-syncer-bjprg\" (UID: \"ec74b46f-3165-4772-9090-b8adbf536767\") " pod="kube-system/global-pull-secret-syncer-bjprg" Jul 23 03:55:23.018472 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:23.018378 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 03:55:23.018472 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:23.018388 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 03:55:23.018472 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:23.018449 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec74b46f-3165-4772-9090-b8adbf536767-original-pull-secret podName:ec74b46f-3165-4772-9090-b8adbf536767 nodeName:}" failed. No retries permitted until 2026-07-23 03:55:24.018408184 +0000 UTC m=+3.923712454 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ec74b46f-3165-4772-9090-b8adbf536767-original-pull-secret") pod "global-pull-secret-syncer-bjprg" (UID: "ec74b46f-3165-4772-9090-b8adbf536767") : object "kube-system"/"original-pull-secret" not registered Jul 23 03:55:23.018658 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:23.018536 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73cf013c-7d0b-4139-bf40-8d5d528474a3-metrics-certs podName:73cf013c-7d0b-4139-bf40-8d5d528474a3 nodeName:}" failed. No retries permitted until 2026-07-23 03:55:24.018517022 +0000 UTC m=+3.923821299 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/73cf013c-7d0b-4139-bf40-8d5d528474a3-metrics-certs") pod "network-metrics-daemon-pq2xh" (UID: "73cf013c-7d0b-4139-bf40-8d5d528474a3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 03:55:23.219667 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:23.219638 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zrms\" (UniqueName: \"kubernetes.io/projected/1e14ea08-0647-4056-9f4a-e639fb16c928-kube-api-access-4zrms\") pod \"network-check-target-w99tk\" (UID: \"1e14ea08-0647-4056-9f4a-e639fb16c928\") " pod="openshift-network-diagnostics/network-check-target-w99tk" Jul 23 03:55:23.219841 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:23.219750 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 03:55:23.219841 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:23.219764 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 03:55:23.219841 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:23.219773 2567 projected.go:196] Error preparing data for projected volume kube-api-access-4zrms for pod openshift-network-diagnostics/network-check-target-w99tk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 03:55:23.219841 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:23.219819 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e14ea08-0647-4056-9f4a-e639fb16c928-kube-api-access-4zrms podName:1e14ea08-0647-4056-9f4a-e639fb16c928 nodeName:}" failed. No retries permitted until 2026-07-23 03:55:24.219807403 +0000 UTC m=+4.125111672 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4zrms" (UniqueName: "kubernetes.io/projected/1e14ea08-0647-4056-9f4a-e639fb16c928-kube-api-access-4zrms") pod "network-check-target-w99tk" (UID: "1e14ea08-0647-4056-9f4a-e639fb16c928") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 03:55:23.253611 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:23.253582 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c32148f_cbbb_4823_95ae_cd3da6b92cd9.slice/crio-d6b30b782f57bb63af113a1d10ceeb60da698c76e4b6983bce0b58132dabceee WatchSource:0}: Error finding container d6b30b782f57bb63af113a1d10ceeb60da698c76e4b6983bce0b58132dabceee: Status 404 returned error can't find the container with id d6b30b782f57bb63af113a1d10ceeb60da698c76e4b6983bce0b58132dabceee Jul 23 03:55:23.279654 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:23.279625 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec45f896_36a7_4e95_bb7b_98597c796f20.slice/crio-d2c039d964f9786d1b684817146f9ad3709f2c7dbd6790ed7c7146ebc4d12d3e WatchSource:0}: Error finding container d2c039d964f9786d1b684817146f9ad3709f2c7dbd6790ed7c7146ebc4d12d3e: Status 404 returned error can't find the container with id d2c039d964f9786d1b684817146f9ad3709f2c7dbd6790ed7c7146ebc4d12d3e Jul 23 03:55:23.280151 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:23.280100 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea356156_f0f7_42dd_8c96_b30bdede6205.slice/crio-96acbf33de84841012bc36736c0faa96e32515578a3e8a38fcc6e7d4bdc42251 WatchSource:0}: Error finding container 96acbf33de84841012bc36736c0faa96e32515578a3e8a38fcc6e7d4bdc42251: Status 404 returned error can't find the container with id 96acbf33de84841012bc36736c0faa96e32515578a3e8a38fcc6e7d4bdc42251 Jul 23 03:55:23.281288 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:23.281265 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa5c99e2_68a8_4f38_9cfe_8c917fb7c61a.slice/crio-fe976b11a72072cc8b12102488a792f76285cfa27ef69c15d1988d90683f3d6d WatchSource:0}: Error finding container fe976b11a72072cc8b12102488a792f76285cfa27ef69c15d1988d90683f3d6d: Status 404 returned error can't find the container with id fe976b11a72072cc8b12102488a792f76285cfa27ef69c15d1988d90683f3d6d Jul 23 03:55:23.282391 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:23.282372 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b6f66fe_a3d8_4ef6_95ee_9016ff91606f.slice/crio-97828782e9ed142ba4e5f389d2aedf6c1f145c31fa21ed469b6ce10a02404ca1 WatchSource:0}: Error finding container 97828782e9ed142ba4e5f389d2aedf6c1f145c31fa21ed469b6ce10a02404ca1: Status 404 returned error can't find the container with id 97828782e9ed142ba4e5f389d2aedf6c1f145c31fa21ed469b6ce10a02404ca1 Jul 23 03:55:23.285555 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:23.285507 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00f9a9d0_8ca6_412a_85c2_3a6d01bc6264.slice/crio-9388e046d4aa4ebcaa861a7bebb92431d2fe3701e262e213e8ef1f1d323524c1 WatchSource:0}: Error finding container 9388e046d4aa4ebcaa861a7bebb92431d2fe3701e262e213e8ef1f1d323524c1: Status 404 returned error can't find the container with id 9388e046d4aa4ebcaa861a7bebb92431d2fe3701e262e213e8ef1f1d323524c1 Jul 23 03:55:23.447088 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:23.446949 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 03:50:22 +0000 UTC" deadline="2028-02-21 05:49:43.592621952 +0000 UTC" Jul 23 03:55:23.447088 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:23.447086 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13873h54m20.145539006s" Jul 23 03:55:23.486741 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:23.486720 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pq2xh" Jul 23 03:55:23.486833 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:23.486819 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pq2xh" podUID="73cf013c-7d0b-4139-bf40-8d5d528474a3" Jul 23 03:55:23.497214 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:23.497179 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rc7rw" event={"ID":"c4a09228-913b-4aa4-8d53-2252a51162f9","Type":"ContainerStarted","Data":"b5660abd2f3efc9534ae1174de6d4862f7293069382f3cac2ddb463b0db48164"} Jul 23 03:55:23.499361 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:23.499335 2567 generic.go:356] "Generic (PLEG): container finished" podID="2d6aefa033c82b0f8916925a50156cfa" containerID="659b811eb28dc7d6dd9c6e0420029645da4632d36a5c6e0ee81a59d99601950d" exitCode=0 Jul 23 03:55:23.499478 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:23.499421 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal" event={"ID":"2d6aefa033c82b0f8916925a50156cfa","Type":"ContainerDied","Data":"659b811eb28dc7d6dd9c6e0420029645da4632d36a5c6e0ee81a59d99601950d"} Jul 23 03:55:23.500994 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:23.500977 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hk2bm" event={"ID":"ec45f896-36a7-4e95-bb7b-98597c796f20","Type":"ContainerStarted","Data":"d2c039d964f9786d1b684817146f9ad3709f2c7dbd6790ed7c7146ebc4d12d3e"} Jul 23 03:55:23.502031 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:23.502006 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kw46l" event={"ID":"787ce18c-a72b-426f-a9f0-51f2c6f7c9bb","Type":"ContainerStarted","Data":"6f441f6c0be38044ad9f5ee08134dba49cb92f2c2d7e3c89b90dd296379e645b"} Jul 23 03:55:23.502938 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:23.502921 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kt7n9" event={"ID":"46a75c21-dc90-40a4-b933-cc1e92dac8c7","Type":"ContainerStarted","Data":"3111ef43e698ea0e5fe68fd88d484b15e5334640d506d9ed602d122eb09dbd18"} Jul 23 03:55:23.503983 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:23.503962 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" event={"ID":"4c32148f-cbbb-4823-95ae-cd3da6b92cd9","Type":"ContainerStarted","Data":"d6b30b782f57bb63af113a1d10ceeb60da698c76e4b6983bce0b58132dabceee"} Jul 23 03:55:23.505294 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:23.505278 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-6.ec2.internal" event={"ID":"b2183c9004d8f7a985e4b7edb883b58a","Type":"ContainerStarted","Data":"624d1ccef39ca38c3265307fd254e00d9fc4360b7f2b0601bc0725e2037afde5"} Jul 23 03:55:23.506295 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:23.506277 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-d8c2n" event={"ID":"00f9a9d0-8ca6-412a-85c2-3a6d01bc6264","Type":"ContainerStarted","Data":"9388e046d4aa4ebcaa861a7bebb92431d2fe3701e262e213e8ef1f1d323524c1"} Jul 23 03:55:23.507310 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:23.507294 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gpl6h" event={"ID":"6b6f66fe-a3d8-4ef6-95ee-9016ff91606f","Type":"ContainerStarted","Data":"97828782e9ed142ba4e5f389d2aedf6c1f145c31fa21ed469b6ce10a02404ca1"} Jul 23 03:55:23.508345 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:23.508327 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f84jx" event={"ID":"aa5c99e2-68a8-4f38-9cfe-8c917fb7c61a","Type":"ContainerStarted","Data":"fe976b11a72072cc8b12102488a792f76285cfa27ef69c15d1988d90683f3d6d"} Jul 23 03:55:23.509473 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:23.509414 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j4hnz" event={"ID":"ea356156-f0f7-42dd-8c96-b30bdede6205","Type":"ContainerStarted","Data":"96acbf33de84841012bc36736c0faa96e32515578a3e8a38fcc6e7d4bdc42251"} Jul 23 03:55:23.531709 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:23.531607 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-6.ec2.internal" podStartSLOduration=2.531589737 podStartE2EDuration="2.531589737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 03:55:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 03:55:23.531336454 +0000 UTC m=+3.436640746" watchObservedRunningTime="2026-07-23 03:55:23.531589737 +0000 UTC m=+3.436894031" Jul 23 03:55:24.025196 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:24.025155 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec74b46f-3165-4772-9090-b8adbf536767-original-pull-secret\") pod \"global-pull-secret-syncer-bjprg\" (UID: \"ec74b46f-3165-4772-9090-b8adbf536767\") " pod="kube-system/global-pull-secret-syncer-bjprg" Jul 23 03:55:24.025402 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:24.025246 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73cf013c-7d0b-4139-bf40-8d5d528474a3-metrics-certs\") pod \"network-metrics-daemon-pq2xh\" (UID: \"73cf013c-7d0b-4139-bf40-8d5d528474a3\") " pod="openshift-multus/network-metrics-daemon-pq2xh" Jul 23 03:55:24.025402 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:24.025386 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 03:55:24.025563 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:24.025472 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73cf013c-7d0b-4139-bf40-8d5d528474a3-metrics-certs podName:73cf013c-7d0b-4139-bf40-8d5d528474a3 nodeName:}" failed. No retries permitted until 2026-07-23 03:55:26.025450642 +0000 UTC m=+5.930754950 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/73cf013c-7d0b-4139-bf40-8d5d528474a3-metrics-certs") pod "network-metrics-daemon-pq2xh" (UID: "73cf013c-7d0b-4139-bf40-8d5d528474a3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 03:55:24.025923 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:24.025900 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 03:55:24.026049 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:24.025951 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec74b46f-3165-4772-9090-b8adbf536767-original-pull-secret podName:ec74b46f-3165-4772-9090-b8adbf536767 nodeName:}" failed. No retries permitted until 2026-07-23 03:55:26.025935608 +0000 UTC m=+5.931239881 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ec74b46f-3165-4772-9090-b8adbf536767-original-pull-secret") pod "global-pull-secret-syncer-bjprg" (UID: "ec74b46f-3165-4772-9090-b8adbf536767") : object "kube-system"/"original-pull-secret" not registered Jul 23 03:55:24.226722 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:24.226679 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zrms\" (UniqueName: \"kubernetes.io/projected/1e14ea08-0647-4056-9f4a-e639fb16c928-kube-api-access-4zrms\") pod \"network-check-target-w99tk\" (UID: \"1e14ea08-0647-4056-9f4a-e639fb16c928\") " pod="openshift-network-diagnostics/network-check-target-w99tk" Jul 23 03:55:24.226920 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:24.226900 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 03:55:24.226989 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:24.226929 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 03:55:24.226989 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:24.226944 2567 projected.go:196] Error preparing data for projected volume kube-api-access-4zrms for pod openshift-network-diagnostics/network-check-target-w99tk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 03:55:24.227107 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:24.227002 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e14ea08-0647-4056-9f4a-e639fb16c928-kube-api-access-4zrms podName:1e14ea08-0647-4056-9f4a-e639fb16c928 nodeName:}" failed. No retries permitted until 2026-07-23 03:55:26.226983528 +0000 UTC m=+6.132287800 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4zrms" (UniqueName: "kubernetes.io/projected/1e14ea08-0647-4056-9f4a-e639fb16c928-kube-api-access-4zrms") pod "network-check-target-w99tk" (UID: "1e14ea08-0647-4056-9f4a-e639fb16c928") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 03:55:24.447461 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:24.447380 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 03:50:22 +0000 UTC" deadline="2028-02-29 08:44:04.985221536 +0000 UTC" Jul 23 03:55:24.447461 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:24.447459 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14068h48m40.537767182s" Jul 23 03:55:24.489372 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:24.489341 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjprg" Jul 23 03:55:24.489561 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:24.489481 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjprg" podUID="ec74b46f-3165-4772-9090-b8adbf536767" Jul 23 03:55:24.489561 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:24.489556 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w99tk" Jul 23 03:55:24.489678 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:24.489633 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w99tk" podUID="1e14ea08-0647-4056-9f4a-e639fb16c928" Jul 23 03:55:24.544822 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:24.544786 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal" event={"ID":"2d6aefa033c82b0f8916925a50156cfa","Type":"ContainerStarted","Data":"f3242cae2ea5dd6b8a1cbdfdea678ae46d687c88182c2f682a70b018e4aed216"} Jul 23 03:55:24.568163 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:24.568109 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal" podStartSLOduration=3.5680883100000003 podStartE2EDuration="3.56808831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 03:55:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 03:55:24.56784304 +0000 UTC m=+4.473147334" watchObservedRunningTime="2026-07-23 03:55:24.56808831 +0000 UTC m=+4.473392604" Jul 23 03:55:25.487292 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:25.487262 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pq2xh" Jul 23 03:55:25.487789 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:25.487408 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pq2xh" podUID="73cf013c-7d0b-4139-bf40-8d5d528474a3" Jul 23 03:55:26.043490 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:26.042658 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec74b46f-3165-4772-9090-b8adbf536767-original-pull-secret\") pod \"global-pull-secret-syncer-bjprg\" (UID: \"ec74b46f-3165-4772-9090-b8adbf536767\") " pod="kube-system/global-pull-secret-syncer-bjprg" Jul 23 03:55:26.043490 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:26.042753 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73cf013c-7d0b-4139-bf40-8d5d528474a3-metrics-certs\") pod \"network-metrics-daemon-pq2xh\" (UID: \"73cf013c-7d0b-4139-bf40-8d5d528474a3\") " pod="openshift-multus/network-metrics-daemon-pq2xh" Jul 23 03:55:26.043490 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:26.042880 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 03:55:26.043490 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:26.042947 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73cf013c-7d0b-4139-bf40-8d5d528474a3-metrics-certs podName:73cf013c-7d0b-4139-bf40-8d5d528474a3 nodeName:}" failed. No retries permitted until 2026-07-23 03:55:30.042927231 +0000 UTC m=+9.948231518 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/73cf013c-7d0b-4139-bf40-8d5d528474a3-metrics-certs") pod "network-metrics-daemon-pq2xh" (UID: "73cf013c-7d0b-4139-bf40-8d5d528474a3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 03:55:26.043490 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:26.043342 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 03:55:26.043490 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:26.043401 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec74b46f-3165-4772-9090-b8adbf536767-original-pull-secret podName:ec74b46f-3165-4772-9090-b8adbf536767 nodeName:}" failed. No retries permitted until 2026-07-23 03:55:30.043385181 +0000 UTC m=+9.948689457 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ec74b46f-3165-4772-9090-b8adbf536767-original-pull-secret") pod "global-pull-secret-syncer-bjprg" (UID: "ec74b46f-3165-4772-9090-b8adbf536767") : object "kube-system"/"original-pull-secret" not registered Jul 23 03:55:26.244758 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:26.244717 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zrms\" (UniqueName: \"kubernetes.io/projected/1e14ea08-0647-4056-9f4a-e639fb16c928-kube-api-access-4zrms\") pod \"network-check-target-w99tk\" (UID: \"1e14ea08-0647-4056-9f4a-e639fb16c928\") " pod="openshift-network-diagnostics/network-check-target-w99tk" Jul 23 03:55:26.244969 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:26.244899 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 03:55:26.244969 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:26.244923 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 03:55:26.244969 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:26.244935 2567 projected.go:196] Error preparing data for projected volume kube-api-access-4zrms for pod openshift-network-diagnostics/network-check-target-w99tk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 03:55:26.245104 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:26.244992 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e14ea08-0647-4056-9f4a-e639fb16c928-kube-api-access-4zrms podName:1e14ea08-0647-4056-9f4a-e639fb16c928 nodeName:}" failed. No retries permitted until 2026-07-23 03:55:30.244973558 +0000 UTC m=+10.150277845 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4zrms" (UniqueName: "kubernetes.io/projected/1e14ea08-0647-4056-9f4a-e639fb16c928-kube-api-access-4zrms") pod "network-check-target-w99tk" (UID: "1e14ea08-0647-4056-9f4a-e639fb16c928") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 03:55:26.487959 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:26.487327 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjprg" Jul 23 03:55:26.487959 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:26.487488 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjprg" podUID="ec74b46f-3165-4772-9090-b8adbf536767" Jul 23 03:55:26.487959 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:26.487833 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w99tk" Jul 23 03:55:26.487959 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:26.487922 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w99tk" podUID="1e14ea08-0647-4056-9f4a-e639fb16c928" Jul 23 03:55:27.487393 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:27.487333 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pq2xh" Jul 23 03:55:27.487593 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:27.487509 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pq2xh" podUID="73cf013c-7d0b-4139-bf40-8d5d528474a3" Jul 23 03:55:28.488956 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:28.488921 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w99tk" Jul 23 03:55:28.489348 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:28.489041 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w99tk" podUID="1e14ea08-0647-4056-9f4a-e639fb16c928" Jul 23 03:55:28.489481 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:28.489384 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjprg" Jul 23 03:55:28.489540 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:28.489483 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjprg" podUID="ec74b46f-3165-4772-9090-b8adbf536767" Jul 23 03:55:29.487246 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:29.487207 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pq2xh" Jul 23 03:55:29.487460 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:29.487407 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pq2xh" podUID="73cf013c-7d0b-4139-bf40-8d5d528474a3" Jul 23 03:55:30.079203 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:30.078614 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73cf013c-7d0b-4139-bf40-8d5d528474a3-metrics-certs\") pod \"network-metrics-daemon-pq2xh\" (UID: \"73cf013c-7d0b-4139-bf40-8d5d528474a3\") " pod="openshift-multus/network-metrics-daemon-pq2xh" Jul 23 03:55:30.079203 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:30.078677 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec74b46f-3165-4772-9090-b8adbf536767-original-pull-secret\") pod \"global-pull-secret-syncer-bjprg\" (UID: \"ec74b46f-3165-4772-9090-b8adbf536767\") " pod="kube-system/global-pull-secret-syncer-bjprg" Jul 23 03:55:30.079203 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:30.078821 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 03:55:30.079203 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:30.078865 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 03:55:30.079203 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:30.078884 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec74b46f-3165-4772-9090-b8adbf536767-original-pull-secret podName:ec74b46f-3165-4772-9090-b8adbf536767 nodeName:}" failed. No retries permitted until 2026-07-23 03:55:38.078866313 +0000 UTC m=+17.984170596 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ec74b46f-3165-4772-9090-b8adbf536767-original-pull-secret") pod "global-pull-secret-syncer-bjprg" (UID: "ec74b46f-3165-4772-9090-b8adbf536767") : object "kube-system"/"original-pull-secret" not registered Jul 23 03:55:30.079203 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:30.078911 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73cf013c-7d0b-4139-bf40-8d5d528474a3-metrics-certs podName:73cf013c-7d0b-4139-bf40-8d5d528474a3 nodeName:}" failed. No retries permitted until 2026-07-23 03:55:38.078891329 +0000 UTC m=+17.984195601 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/73cf013c-7d0b-4139-bf40-8d5d528474a3-metrics-certs") pod "network-metrics-daemon-pq2xh" (UID: "73cf013c-7d0b-4139-bf40-8d5d528474a3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 03:55:30.281229 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:30.280697 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zrms\" (UniqueName: \"kubernetes.io/projected/1e14ea08-0647-4056-9f4a-e639fb16c928-kube-api-access-4zrms\") pod \"network-check-target-w99tk\" (UID: \"1e14ea08-0647-4056-9f4a-e639fb16c928\") " pod="openshift-network-diagnostics/network-check-target-w99tk" Jul 23 03:55:30.281229 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:30.280898 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 03:55:30.281229 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:30.280921 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 03:55:30.281229 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:30.280931 2567 projected.go:196] Error preparing data for projected volume kube-api-access-4zrms for pod openshift-network-diagnostics/network-check-target-w99tk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 03:55:30.281229 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:30.280975 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e14ea08-0647-4056-9f4a-e639fb16c928-kube-api-access-4zrms podName:1e14ea08-0647-4056-9f4a-e639fb16c928 nodeName:}" failed. No retries permitted until 2026-07-23 03:55:38.28096298 +0000 UTC m=+18.186267251 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4zrms" (UniqueName: "kubernetes.io/projected/1e14ea08-0647-4056-9f4a-e639fb16c928-kube-api-access-4zrms") pod "network-check-target-w99tk" (UID: "1e14ea08-0647-4056-9f4a-e639fb16c928") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 03:55:30.488304 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:30.488254 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w99tk" Jul 23 03:55:30.488501 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:30.488380 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w99tk" podUID="1e14ea08-0647-4056-9f4a-e639fb16c928" Jul 23 03:55:30.488772 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:30.488754 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjprg" Jul 23 03:55:30.488873 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:30.488849 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjprg" podUID="ec74b46f-3165-4772-9090-b8adbf536767" Jul 23 03:55:31.487339 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:31.487310 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pq2xh" Jul 23 03:55:31.487817 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:31.487440 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pq2xh" podUID="73cf013c-7d0b-4139-bf40-8d5d528474a3" Jul 23 03:55:32.488953 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:32.488461 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjprg" Jul 23 03:55:32.488953 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:32.488560 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjprg" podUID="ec74b46f-3165-4772-9090-b8adbf536767" Jul 23 03:55:32.488953 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:32.488833 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w99tk" Jul 23 03:55:32.488953 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:32.488915 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w99tk" podUID="1e14ea08-0647-4056-9f4a-e639fb16c928" Jul 23 03:55:33.487132 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:33.487100 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pq2xh" Jul 23 03:55:33.487290 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:33.487223 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pq2xh" podUID="73cf013c-7d0b-4139-bf40-8d5d528474a3" Jul 23 03:55:34.487037 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:34.486996 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjprg" Jul 23 03:55:34.487482 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:34.487140 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjprg" podUID="ec74b46f-3165-4772-9090-b8adbf536767" Jul 23 03:55:34.487482 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:34.487189 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w99tk" Jul 23 03:55:34.487482 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:34.487279 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w99tk" podUID="1e14ea08-0647-4056-9f4a-e639fb16c928" Jul 23 03:55:35.486476 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:35.486445 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pq2xh" Jul 23 03:55:35.486661 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:35.486594 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pq2xh" podUID="73cf013c-7d0b-4139-bf40-8d5d528474a3" Jul 23 03:55:36.486766 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:36.486737 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjprg" Jul 23 03:55:36.487199 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:36.486853 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjprg" podUID="ec74b46f-3165-4772-9090-b8adbf536767" Jul 23 03:55:36.487199 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:36.486926 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w99tk" Jul 23 03:55:36.487199 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:36.487042 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w99tk" podUID="1e14ea08-0647-4056-9f4a-e639fb16c928" Jul 23 03:55:37.487130 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:37.487101 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pq2xh" Jul 23 03:55:37.487609 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:37.487225 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pq2xh" podUID="73cf013c-7d0b-4139-bf40-8d5d528474a3" Jul 23 03:55:38.143671 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:38.143639 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73cf013c-7d0b-4139-bf40-8d5d528474a3-metrics-certs\") pod \"network-metrics-daemon-pq2xh\" (UID: \"73cf013c-7d0b-4139-bf40-8d5d528474a3\") " pod="openshift-multus/network-metrics-daemon-pq2xh" Jul 23 03:55:38.143811 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:38.143681 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec74b46f-3165-4772-9090-b8adbf536767-original-pull-secret\") pod \"global-pull-secret-syncer-bjprg\" (UID: \"ec74b46f-3165-4772-9090-b8adbf536767\") " pod="kube-system/global-pull-secret-syncer-bjprg" Jul 23 03:55:38.143811 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:38.143775 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 03:55:38.143811 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:38.143780 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 03:55:38.143951 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:38.143826 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec74b46f-3165-4772-9090-b8adbf536767-original-pull-secret podName:ec74b46f-3165-4772-9090-b8adbf536767 nodeName:}" failed. No retries permitted until 2026-07-23 03:55:54.143811843 +0000 UTC m=+34.049116115 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ec74b46f-3165-4772-9090-b8adbf536767-original-pull-secret") pod "global-pull-secret-syncer-bjprg" (UID: "ec74b46f-3165-4772-9090-b8adbf536767") : object "kube-system"/"original-pull-secret" not registered Jul 23 03:55:38.143951 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:38.143842 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73cf013c-7d0b-4139-bf40-8d5d528474a3-metrics-certs podName:73cf013c-7d0b-4139-bf40-8d5d528474a3 nodeName:}" failed. No retries permitted until 2026-07-23 03:55:54.143834359 +0000 UTC m=+34.049138629 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/73cf013c-7d0b-4139-bf40-8d5d528474a3-metrics-certs") pod "network-metrics-daemon-pq2xh" (UID: "73cf013c-7d0b-4139-bf40-8d5d528474a3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 03:55:38.344832 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:38.344804 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zrms\" (UniqueName: \"kubernetes.io/projected/1e14ea08-0647-4056-9f4a-e639fb16c928-kube-api-access-4zrms\") pod \"network-check-target-w99tk\" (UID: \"1e14ea08-0647-4056-9f4a-e639fb16c928\") " pod="openshift-network-diagnostics/network-check-target-w99tk" Jul 23 03:55:38.344983 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:38.344964 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 03:55:38.345021 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:38.344989 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 03:55:38.345021 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:38.344999 2567 projected.go:196] Error preparing data for projected volume kube-api-access-4zrms for pod openshift-network-diagnostics/network-check-target-w99tk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 03:55:38.345083 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:38.345044 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e14ea08-0647-4056-9f4a-e639fb16c928-kube-api-access-4zrms podName:1e14ea08-0647-4056-9f4a-e639fb16c928 nodeName:}" failed. No retries permitted until 2026-07-23 03:55:54.345030712 +0000 UTC m=+34.250334982 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4zrms" (UniqueName: "kubernetes.io/projected/1e14ea08-0647-4056-9f4a-e639fb16c928-kube-api-access-4zrms") pod "network-check-target-w99tk" (UID: "1e14ea08-0647-4056-9f4a-e639fb16c928") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 03:55:38.486522 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:38.486489 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w99tk" Jul 23 03:55:38.486624 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:38.486491 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjprg" Jul 23 03:55:38.486624 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:38.486588 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w99tk" podUID="1e14ea08-0647-4056-9f4a-e639fb16c928" Jul 23 03:55:38.486713 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:38.486673 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjprg" podUID="ec74b46f-3165-4772-9090-b8adbf536767" Jul 23 03:55:39.486711 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:39.486669 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pq2xh" Jul 23 03:55:39.487155 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:39.486830 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pq2xh" podUID="73cf013c-7d0b-4139-bf40-8d5d528474a3" Jul 23 03:55:40.487822 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:40.487789 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjprg" Jul 23 03:55:40.488201 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:40.487830 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w99tk" Jul 23 03:55:40.488201 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:40.487883 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjprg" podUID="ec74b46f-3165-4772-9090-b8adbf536767" Jul 23 03:55:40.488201 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:40.487906 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w99tk" podUID="1e14ea08-0647-4056-9f4a-e639fb16c928" Jul 23 03:55:41.487004 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:41.486976 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pq2xh" Jul 23 03:55:41.487159 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:41.487135 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pq2xh" podUID="73cf013c-7d0b-4139-bf40-8d5d528474a3" Jul 23 03:55:42.487089 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:42.486770 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjprg" Jul 23 03:55:42.487089 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:42.487054 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjprg" podUID="ec74b46f-3165-4772-9090-b8adbf536767" Jul 23 03:55:42.487659 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:42.487529 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w99tk" Jul 23 03:55:42.487659 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:42.487640 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w99tk" podUID="1e14ea08-0647-4056-9f4a-e639fb16c928" Jul 23 03:55:42.589670 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:42.589642 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hk2bm" event={"ID":"ec45f896-36a7-4e95-bb7b-98597c796f20","Type":"ContainerStarted","Data":"2ea1f9fa41e614c8fcff969d2f079d4d568236296295d4b08a26e8f3597c02e8"} Jul 23 03:55:42.591137 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:42.591105 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kw46l" event={"ID":"787ce18c-a72b-426f-a9f0-51f2c6f7c9bb","Type":"ContainerStarted","Data":"4a1f62b27f111fb47526ed2b06aaf30af1e9579c2b87beac62bbc4d66faef786"} Jul 23 03:55:42.592897 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:42.592860 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kt7n9" event={"ID":"46a75c21-dc90-40a4-b933-cc1e92dac8c7","Type":"ContainerStarted","Data":"0afe17a6158f81ed2db015822bfbc365a45255f04879274705e7515787dec968"} Jul 23 03:55:42.595676 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:42.595651 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" event={"ID":"4c32148f-cbbb-4823-95ae-cd3da6b92cd9","Type":"ContainerStarted","Data":"dc3272d45e0d25f2b975172e7e22ded2a65027cb3892c9aeb388a9eee9f8ba8a"} Jul 23 03:55:42.595771 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:42.595735 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" event={"ID":"4c32148f-cbbb-4823-95ae-cd3da6b92cd9","Type":"ContainerStarted","Data":"47de14297c6562313fbda111f14a351a83a0dfff0ecf6484059e1bc6d187cbce"} Jul 23 03:55:42.595818 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:42.595786 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" event={"ID":"4c32148f-cbbb-4823-95ae-cd3da6b92cd9","Type":"ContainerStarted","Data":"9fdb9b6af307a240dfef574cf82ac1b049df507a97a526d3f58195c420afc989"} Jul 23 03:55:42.597177 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:42.597156 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gpl6h" event={"ID":"6b6f66fe-a3d8-4ef6-95ee-9016ff91606f","Type":"ContainerStarted","Data":"535ceb0cedeb95f9df53a410bd2630997b79afc73d095cf43871758ae97edd7f"} Jul 23 03:55:42.598577 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:42.598558 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f84jx" event={"ID":"aa5c99e2-68a8-4f38-9cfe-8c917fb7c61a","Type":"ContainerStarted","Data":"6be44a19fbe3388cf4d2634982fe8eed551291ecb8563a0026be6d07c0cae77d"} Jul 23 03:55:42.600135 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:42.600114 2567 generic.go:356] "Generic (PLEG): container finished" podID="ea356156-f0f7-42dd-8c96-b30bdede6205" containerID="6835301873f734cc75760945db3236efa2e5204be05fbe9bd64ceede4d33e9a7" exitCode=0 Jul 23 03:55:42.600236 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:42.600176 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j4hnz" event={"ID":"ea356156-f0f7-42dd-8c96-b30bdede6205","Type":"ContainerDied","Data":"6835301873f734cc75760945db3236efa2e5204be05fbe9bd64ceede4d33e9a7"} Jul 23 03:55:42.601794 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:42.601774 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rc7rw" event={"ID":"c4a09228-913b-4aa4-8d53-2252a51162f9","Type":"ContainerStarted","Data":"914039f3efe449a39346c414574b19efaf68585f30524bdcfab7dc1f52bc47de"} Jul 23 03:55:42.610294 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:42.610259 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-hk2bm" podStartSLOduration=3.576595955 podStartE2EDuration="22.61024619s" totalImagesPullingTime="19.033650235s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 03:55:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 03:55:42.609673553 +0000 UTC m=+22.514977860" watchObservedRunningTime="2026-07-23 03:55:42.61024619 +0000 UTC m=+22.515550483" Jul 23 03:55:42.645605 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:42.645568 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-kt7n9" podStartSLOduration=7.401361695 podStartE2EDuration="22.645554084s" totalImagesPullingTime="15.244192389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 03:55:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 03:55:42.645311262 +0000 UTC m=+22.550615553" watchObservedRunningTime="2026-07-23 03:55:42.645554084 +0000 UTC m=+22.550858375" Jul 23 03:55:42.659443 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:42.659386 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-gpl6h" podStartSLOduration=3.728649109 podStartE2EDuration="22.659373983s" totalImagesPullingTime="18.930724874s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 03:55:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 03:55:42.659167516 +0000 UTC m=+22.564471828" watchObservedRunningTime="2026-07-23 03:55:42.659373983 +0000 UTC m=+22.564678269" Jul 23 03:55:42.673842 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:42.673811 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-kw46l" podStartSLOduration=3.714925624 podStartE2EDuration="22.673796951s" totalImagesPullingTime="18.958871327s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 03:55:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 03:55:42.673706726 +0000 UTC m=+22.579011018" watchObservedRunningTime="2026-07-23 03:55:42.673796951 +0000 UTC m=+22.579101244" Jul 23 03:55:42.682932 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:42.682915 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-kt7n9" Jul 23 03:55:42.683326 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:42.683306 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-kt7n9" Jul 23 03:55:42.693731 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:42.693698 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rc7rw" podStartSLOduration=3.742064738 podStartE2EDuration="22.693686369s" totalImagesPullingTime="18.951621631s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 03:55:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 03:55:42.693410808 +0000 UTC m=+22.598715099" watchObservedRunningTime="2026-07-23 03:55:42.693686369 +0000 UTC m=+22.598990660" Jul 23 03:55:43.486660 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:43.486631 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pq2xh" Jul 23 03:55:43.486873 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:43.486737 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pq2xh" podUID="73cf013c-7d0b-4139-bf40-8d5d528474a3" Jul 23 03:55:43.607685 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:43.607106 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" event={"ID":"4c32148f-cbbb-4823-95ae-cd3da6b92cd9","Type":"ContainerStarted","Data":"2238dc4ce66113a132b5aaf8f6b3b86155b1343e2c794a7af7dc21eee02e460b"} Jul 23 03:55:43.607685 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:43.607188 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" event={"ID":"4c32148f-cbbb-4823-95ae-cd3da6b92cd9","Type":"ContainerStarted","Data":"7228db1ff85d8c7ff6523b0c1bfedfc62e227c341ee2bb602c106530099cafe3"} Jul 23 03:55:43.607685 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:43.607219 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" event={"ID":"4c32148f-cbbb-4823-95ae-cd3da6b92cd9","Type":"ContainerStarted","Data":"2f6973f9e930ea4e9478013a2489bb745f634bf2a7663857992219c88f3af6cc"} Jul 23 03:55:43.607685 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:43.607577 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-kt7n9" Jul 23 03:55:43.608274 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:43.608044 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kt7n9" Jul 23 03:55:43.851146 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:43.851116 2567 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 23 03:55:44.469606 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:44.469495 2567 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-23T03:55:43.851131926Z","UUID":"a3d6d509-a566-4505-b6a0-b384876420bb","Handler":null,"Name":"","Endpoint":""} Jul 23 03:55:44.471065 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:44.471042 2567 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 23 03:55:44.471201 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:44.471089 2567 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 23 03:55:44.487449 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:44.487407 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjprg" Jul 23 03:55:44.487696 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:44.487481 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w99tk" Jul 23 03:55:44.487807 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:44.487713 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjprg" podUID="ec74b46f-3165-4772-9090-b8adbf536767" Jul 23 03:55:44.487807 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:44.487779 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w99tk" podUID="1e14ea08-0647-4056-9f4a-e639fb16c928" Jul 23 03:55:44.611458 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:44.610900 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-d8c2n" event={"ID":"00f9a9d0-8ca6-412a-85c2-3a6d01bc6264","Type":"ContainerStarted","Data":"7444da3f0fa3e07312d736e7f06729c49b344dab755739a012cbc99bcbb89051"} Jul 23 03:55:44.614488 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:44.614464 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f84jx" event={"ID":"aa5c99e2-68a8-4f38-9cfe-8c917fb7c61a","Type":"ContainerStarted","Data":"a01ac6ff630563290b877ecf0872300f7dfb336a1aebd4f7cef903fb69dadbb0"} Jul 23 03:55:44.627922 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:44.627872 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-d8c2n" podStartSLOduration=5.689730504 podStartE2EDuration="24.627853521s" totalImagesPullingTime="18.938123017s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 03:55:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 03:55:44.627490919 +0000 UTC m=+24.532795240" watchObservedRunningTime="2026-07-23 03:55:44.627853521 +0000 UTC m=+24.533157817" Jul 23 03:55:45.486985 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:45.486950 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pq2xh" Jul 23 03:55:45.487179 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:45.487067 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pq2xh" podUID="73cf013c-7d0b-4139-bf40-8d5d528474a3" Jul 23 03:55:45.620408 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:45.620373 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" event={"ID":"4c32148f-cbbb-4823-95ae-cd3da6b92cd9","Type":"ContainerStarted","Data":"0c83c3049d5aaa72ab0359ec7264390e2352d872fed97cb01040fc1a8140e779"} Jul 23 03:55:45.622797 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:45.622766 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f84jx" event={"ID":"aa5c99e2-68a8-4f38-9cfe-8c917fb7c61a","Type":"ContainerStarted","Data":"e52412e29e74d1216798586cf2ed0c8a3a6d19c27bc1ccfa58dc510beca68fae"} Jul 23 03:55:45.642602 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:45.642563 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f84jx" podStartSLOduration=4.494552142 podStartE2EDuration="25.642549717s" totalImagesPullingTime="21.147997575s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 03:55:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 03:55:45.642467576 +0000 UTC m=+25.547771882" watchObservedRunningTime="2026-07-23 03:55:45.642549717 +0000 UTC m=+25.547854008" Jul 23 03:55:46.486732 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:46.486698 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjprg" Jul 23 03:55:46.486913 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:46.486702 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w99tk" Jul 23 03:55:46.486913 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:46.486835 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjprg" podUID="ec74b46f-3165-4772-9090-b8adbf536767" Jul 23 03:55:46.486913 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:46.486885 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w99tk" podUID="1e14ea08-0647-4056-9f4a-e639fb16c928" Jul 23 03:55:47.487567 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:47.487392 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pq2xh" Jul 23 03:55:47.488153 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:47.487656 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pq2xh" podUID="73cf013c-7d0b-4139-bf40-8d5d528474a3" Jul 23 03:55:47.630564 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:47.630521 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" event={"ID":"4c32148f-cbbb-4823-95ae-cd3da6b92cd9","Type":"ContainerStarted","Data":"6cf5f89c538658fff7097841fdaa1f526b47e9c1bedd90e1454a6f0a960e8567"} Jul 23 03:55:47.630861 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:47.630838 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:47.632323 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:47.632298 2567 generic.go:356] "Generic (PLEG): container finished" podID="ea356156-f0f7-42dd-8c96-b30bdede6205" containerID="8d051a5429e67859269f1554adb657b1cd8dcb150f75535a074a9f6e8733290c" exitCode=0 Jul 23 03:55:47.632502 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:47.632331 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j4hnz" event={"ID":"ea356156-f0f7-42dd-8c96-b30bdede6205","Type":"ContainerDied","Data":"8d051a5429e67859269f1554adb657b1cd8dcb150f75535a074a9f6e8733290c"} Jul 23 03:55:47.647494 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:47.647475 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:47.659947 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:47.659907 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" podStartSLOduration=8.634752373 podStartE2EDuration="27.659894572s" totalImagesPullingTime="19.025142199s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 03:55:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 03:55:47.659522009 +0000 UTC m=+27.564826317" watchObservedRunningTime="2026-07-23 03:55:47.659894572 +0000 UTC m=+27.565198892" Jul 23 03:55:48.487259 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:48.487088 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjprg" Jul 23 03:55:48.487516 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:48.487098 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w99tk" Jul 23 03:55:48.487516 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:48.487330 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjprg" podUID="ec74b46f-3165-4772-9090-b8adbf536767" Jul 23 03:55:48.487516 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:48.487454 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w99tk" podUID="1e14ea08-0647-4056-9f4a-e639fb16c928" Jul 23 03:55:48.634662 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:48.634635 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:48.634818 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:48.634720 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:48.649658 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:48.649634 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:55:49.138478 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:49.138390 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bjprg"] Jul 23 03:55:49.138616 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:49.138523 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjprg" Jul 23 03:55:49.138675 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:49.138630 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjprg" podUID="ec74b46f-3165-4772-9090-b8adbf536767" Jul 23 03:55:49.141951 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:49.141924 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pq2xh"] Jul 23 03:55:49.142062 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:49.142040 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pq2xh" Jul 23 03:55:49.142172 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:49.142151 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pq2xh" podUID="73cf013c-7d0b-4139-bf40-8d5d528474a3" Jul 23 03:55:49.142564 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:49.142536 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-w99tk"] Jul 23 03:55:49.142671 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:49.142616 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w99tk" Jul 23 03:55:49.142725 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:49.142684 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w99tk" podUID="1e14ea08-0647-4056-9f4a-e639fb16c928" Jul 23 03:55:49.638475 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:49.638445 2567 generic.go:356] "Generic (PLEG): container finished" podID="ea356156-f0f7-42dd-8c96-b30bdede6205" containerID="e67999763f553decbacb6c72062471bb48421185591233c1dafbf6f160eda47e" exitCode=0 Jul 23 03:55:49.638983 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:49.638531 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j4hnz" event={"ID":"ea356156-f0f7-42dd-8c96-b30bdede6205","Type":"ContainerDied","Data":"e67999763f553decbacb6c72062471bb48421185591233c1dafbf6f160eda47e"} Jul 23 03:55:50.488364 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:50.488332 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjprg" Jul 23 03:55:50.488564 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:50.488468 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjprg" podUID="ec74b46f-3165-4772-9090-b8adbf536767" Jul 23 03:55:51.486781 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:51.486572 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w99tk" Jul 23 03:55:51.487115 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:51.486617 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pq2xh" Jul 23 03:55:51.487115 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:51.486820 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w99tk" podUID="1e14ea08-0647-4056-9f4a-e639fb16c928" Jul 23 03:55:51.487115 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:51.486890 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pq2xh" podUID="73cf013c-7d0b-4139-bf40-8d5d528474a3" Jul 23 03:55:51.645145 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:51.645113 2567 generic.go:356] "Generic (PLEG): container finished" podID="ea356156-f0f7-42dd-8c96-b30bdede6205" containerID="2b6e4686687b38b4218fcb3b162ec9734b21aedabc7857fe4f9578eb90687617" exitCode=0 Jul 23 03:55:51.645284 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:51.645175 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j4hnz" event={"ID":"ea356156-f0f7-42dd-8c96-b30bdede6205","Type":"ContainerDied","Data":"2b6e4686687b38b4218fcb3b162ec9734b21aedabc7857fe4f9578eb90687617"} Jul 23 03:55:52.486816 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:52.486786 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjprg" Jul 23 03:55:52.487402 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:52.486934 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjprg" podUID="ec74b46f-3165-4772-9090-b8adbf536767" Jul 23 03:55:53.487182 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:53.487151 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w99tk" Jul 23 03:55:53.487616 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:53.487151 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pq2xh" Jul 23 03:55:53.487616 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:53.487283 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w99tk" podUID="1e14ea08-0647-4056-9f4a-e639fb16c928" Jul 23 03:55:53.487616 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:53.487345 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pq2xh" podUID="73cf013c-7d0b-4139-bf40-8d5d528474a3" Jul 23 03:55:54.171029 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:54.170993 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec74b46f-3165-4772-9090-b8adbf536767-original-pull-secret\") pod \"global-pull-secret-syncer-bjprg\" (UID: \"ec74b46f-3165-4772-9090-b8adbf536767\") " pod="kube-system/global-pull-secret-syncer-bjprg" Jul 23 03:55:54.171192 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:54.171084 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73cf013c-7d0b-4139-bf40-8d5d528474a3-metrics-certs\") pod \"network-metrics-daemon-pq2xh\" (UID: \"73cf013c-7d0b-4139-bf40-8d5d528474a3\") " pod="openshift-multus/network-metrics-daemon-pq2xh" Jul 23 03:55:54.171192 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:54.171156 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 03:55:54.171192 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:54.171175 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 03:55:54.171310 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:54.171231 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec74b46f-3165-4772-9090-b8adbf536767-original-pull-secret podName:ec74b46f-3165-4772-9090-b8adbf536767 nodeName:}" failed. No retries permitted until 2026-07-23 03:56:26.171209639 +0000 UTC m=+66.076513913 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ec74b46f-3165-4772-9090-b8adbf536767-original-pull-secret") pod "global-pull-secret-syncer-bjprg" (UID: "ec74b46f-3165-4772-9090-b8adbf536767") : object "kube-system"/"original-pull-secret" not registered Jul 23 03:55:54.171310 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:54.171252 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73cf013c-7d0b-4139-bf40-8d5d528474a3-metrics-certs podName:73cf013c-7d0b-4139-bf40-8d5d528474a3 nodeName:}" failed. No retries permitted until 2026-07-23 03:56:26.171241395 +0000 UTC m=+66.076545670 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/73cf013c-7d0b-4139-bf40-8d5d528474a3-metrics-certs") pod "network-metrics-daemon-pq2xh" (UID: "73cf013c-7d0b-4139-bf40-8d5d528474a3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 03:55:54.372610 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:54.372574 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zrms\" (UniqueName: \"kubernetes.io/projected/1e14ea08-0647-4056-9f4a-e639fb16c928-kube-api-access-4zrms\") pod \"network-check-target-w99tk\" (UID: \"1e14ea08-0647-4056-9f4a-e639fb16c928\") " pod="openshift-network-diagnostics/network-check-target-w99tk" Jul 23 03:55:54.372790 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:54.372722 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 03:55:54.372790 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:54.372746 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 03:55:54.372790 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:54.372756 2567 projected.go:196] Error preparing data for projected volume kube-api-access-4zrms for pod openshift-network-diagnostics/network-check-target-w99tk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 03:55:54.372937 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:54.372804 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e14ea08-0647-4056-9f4a-e639fb16c928-kube-api-access-4zrms podName:1e14ea08-0647-4056-9f4a-e639fb16c928 nodeName:}" failed. No retries permitted until 2026-07-23 03:56:26.372789795 +0000 UTC m=+66.278094065 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-4zrms" (UniqueName: "kubernetes.io/projected/1e14ea08-0647-4056-9f4a-e639fb16c928-kube-api-access-4zrms") pod "network-check-target-w99tk" (UID: "1e14ea08-0647-4056-9f4a-e639fb16c928") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 03:55:54.487087 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:54.487011 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjprg" Jul 23 03:55:54.487259 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:54.487127 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjprg" podUID="ec74b46f-3165-4772-9090-b8adbf536767" Jul 23 03:55:55.487252 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:55.487218 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pq2xh" Jul 23 03:55:55.487252 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:55.487243 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w99tk" Jul 23 03:55:55.487800 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:55.487336 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w99tk" podUID="1e14ea08-0647-4056-9f4a-e639fb16c928" Jul 23 03:55:55.487800 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:55.487454 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pq2xh" podUID="73cf013c-7d0b-4139-bf40-8d5d528474a3" Jul 23 03:55:55.958340 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:55.958316 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-6.ec2.internal" event="NodeReady" Jul 23 03:55:55.958512 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:55.958498 2567 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 23 03:55:56.014234 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:56.014182 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-w64s6"] Jul 23 03:55:56.017659 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:56.017638 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-nt27s"] Jul 23 03:55:56.017770 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:56.017754 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w64s6" Jul 23 03:55:56.019604 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:56.019585 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nt27s" Jul 23 03:55:56.020263 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:56.020245 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 23 03:55:56.020340 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:56.020265 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jzncz\"" Jul 23 03:55:56.024601 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:56.024587 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 23 03:55:56.024852 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:56.024839 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 23 03:55:56.024944 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:56.024926 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-c4p5f\"" Jul 23 03:55:56.024990 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:56.024936 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 23 03:55:56.025180 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:56.025167 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 23 03:55:56.031750 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:56.031605 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w64s6"] Jul 23 03:55:56.032477 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:56.032376 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nt27s"] Jul 23 03:55:56.184693 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:56.184667 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d-metrics-tls\") pod \"dns-default-w64s6\" (UID: \"2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d\") " pod="openshift-dns/dns-default-w64s6" Jul 23 03:55:56.184780 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:56.184728 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m98xs\" (UniqueName: \"kubernetes.io/projected/2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d-kube-api-access-m98xs\") pod \"dns-default-w64s6\" (UID: \"2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d\") " pod="openshift-dns/dns-default-w64s6" Jul 23 03:55:56.184780 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:56.184753 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d-config-volume\") pod \"dns-default-w64s6\" (UID: \"2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d\") " pod="openshift-dns/dns-default-w64s6" Jul 23 03:55:56.184853 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:56.184789 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d-tmp-dir\") pod \"dns-default-w64s6\" (UID: \"2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d\") " pod="openshift-dns/dns-default-w64s6" Jul 23 03:55:56.184893 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:56.184866 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc9ae7fe-6710-4847-bacc-01096f89cd3c-cert\") pod \"ingress-canary-nt27s\" (UID: \"cc9ae7fe-6710-4847-bacc-01096f89cd3c\") " pod="openshift-ingress-canary/ingress-canary-nt27s" Jul 23 03:55:56.184929 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:56.184892 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlfvf\" (UniqueName: \"kubernetes.io/projected/cc9ae7fe-6710-4847-bacc-01096f89cd3c-kube-api-access-qlfvf\") pod \"ingress-canary-nt27s\" (UID: \"cc9ae7fe-6710-4847-bacc-01096f89cd3c\") " pod="openshift-ingress-canary/ingress-canary-nt27s" Jul 23 03:55:56.285983 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:56.285918 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m98xs\" (UniqueName: \"kubernetes.io/projected/2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d-kube-api-access-m98xs\") pod \"dns-default-w64s6\" (UID: \"2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d\") " pod="openshift-dns/dns-default-w64s6" Jul 23 03:55:56.285983 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:56.285954 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d-config-volume\") pod \"dns-default-w64s6\" (UID: \"2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d\") " pod="openshift-dns/dns-default-w64s6" Jul 23 03:55:56.286113 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:56.286061 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d-tmp-dir\") pod \"dns-default-w64s6\" (UID: \"2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d\") " pod="openshift-dns/dns-default-w64s6" Jul 23 03:55:56.286113 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:56.286100 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc9ae7fe-6710-4847-bacc-01096f89cd3c-cert\") pod \"ingress-canary-nt27s\" (UID: \"cc9ae7fe-6710-4847-bacc-01096f89cd3c\") " pod="openshift-ingress-canary/ingress-canary-nt27s" Jul 23 03:55:56.286192 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:56.286128 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlfvf\" (UniqueName: \"kubernetes.io/projected/cc9ae7fe-6710-4847-bacc-01096f89cd3c-kube-api-access-qlfvf\") pod \"ingress-canary-nt27s\" (UID: \"cc9ae7fe-6710-4847-bacc-01096f89cd3c\") " pod="openshift-ingress-canary/ingress-canary-nt27s" Jul 23 03:55:56.286192 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:56.286164 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d-metrics-tls\") pod \"dns-default-w64s6\" (UID: \"2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d\") " pod="openshift-dns/dns-default-w64s6" Jul 23 03:55:56.286290 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:56.286274 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 03:55:56.286350 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:56.286340 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d-metrics-tls podName:2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d nodeName:}" failed. No retries permitted until 2026-07-23 03:55:56.786320699 +0000 UTC m=+36.691624972 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d-metrics-tls") pod "dns-default-w64s6" (UID: "2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d") : secret "dns-default-metrics-tls" not found Jul 23 03:55:56.286476 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:56.286457 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d-tmp-dir\") pod \"dns-default-w64s6\" (UID: \"2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d\") " pod="openshift-dns/dns-default-w64s6" Jul 23 03:55:56.286636 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:56.286615 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d-config-volume\") pod \"dns-default-w64s6\" (UID: \"2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d\") " pod="openshift-dns/dns-default-w64s6" Jul 23 03:55:56.290031 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:56.290010 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc9ae7fe-6710-4847-bacc-01096f89cd3c-cert\") pod \"ingress-canary-nt27s\" (UID: \"cc9ae7fe-6710-4847-bacc-01096f89cd3c\") " pod="openshift-ingress-canary/ingress-canary-nt27s" Jul 23 03:55:56.296660 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:56.296479 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlfvf\" (UniqueName: \"kubernetes.io/projected/cc9ae7fe-6710-4847-bacc-01096f89cd3c-kube-api-access-qlfvf\") pod \"ingress-canary-nt27s\" (UID: \"cc9ae7fe-6710-4847-bacc-01096f89cd3c\") " pod="openshift-ingress-canary/ingress-canary-nt27s" Jul 23 03:55:56.300845 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:56.300822 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m98xs\" (UniqueName: \"kubernetes.io/projected/2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d-kube-api-access-m98xs\") pod \"dns-default-w64s6\" (UID: \"2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d\") " pod="openshift-dns/dns-default-w64s6" Jul 23 03:55:56.357386 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:56.357353 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nt27s" Jul 23 03:55:56.487298 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:56.487146 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjprg" Jul 23 03:55:56.490720 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:56.490699 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nt27s"] Jul 23 03:55:56.490842 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:56.490765 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 23 03:55:56.492513 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:55:56.492494 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc9ae7fe_6710_4847_bacc_01096f89cd3c.slice/crio-19f207f2f12fb14244a190d544fa9eacd2a2e73d9496a523ad14b66b440119ff WatchSource:0}: Error finding container 19f207f2f12fb14244a190d544fa9eacd2a2e73d9496a523ad14b66b440119ff: Status 404 returned error can't find the container with id 19f207f2f12fb14244a190d544fa9eacd2a2e73d9496a523ad14b66b440119ff Jul 23 03:55:56.657980 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:56.657879 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nt27s" event={"ID":"cc9ae7fe-6710-4847-bacc-01096f89cd3c","Type":"ContainerStarted","Data":"19f207f2f12fb14244a190d544fa9eacd2a2e73d9496a523ad14b66b440119ff"} Jul 23 03:55:56.660301 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:56.660274 2567 generic.go:356] "Generic (PLEG): container finished" podID="ea356156-f0f7-42dd-8c96-b30bdede6205" containerID="56452cbbd1f9084f6b0be14da899bc2229d8b366bf91f539a608b52c1225d976" exitCode=0 Jul 23 03:55:56.660403 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:56.660308 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j4hnz" event={"ID":"ea356156-f0f7-42dd-8c96-b30bdede6205","Type":"ContainerDied","Data":"56452cbbd1f9084f6b0be14da899bc2229d8b366bf91f539a608b52c1225d976"} Jul 23 03:55:56.789880 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:56.789844 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d-metrics-tls\") pod \"dns-default-w64s6\" (UID: \"2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d\") " pod="openshift-dns/dns-default-w64s6" Jul 23 03:55:56.789998 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:56.789979 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 03:55:56.790063 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:56.790052 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d-metrics-tls podName:2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d nodeName:}" failed. No retries permitted until 2026-07-23 03:55:57.790031652 +0000 UTC m=+37.695335922 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d-metrics-tls") pod "dns-default-w64s6" (UID: "2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d") : secret "dns-default-metrics-tls" not found Jul 23 03:55:57.487411 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:57.487378 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pq2xh" Jul 23 03:55:57.487893 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:57.487418 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w99tk" Jul 23 03:55:57.491647 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:57.491460 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 23 03:55:57.491647 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:57.491481 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 23 03:55:57.491647 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:57.491511 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zlbwn\"" Jul 23 03:55:57.491647 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:57.491467 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l57h5\"" Jul 23 03:55:57.491647 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:57.491471 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 23 03:55:57.666291 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:57.666256 2567 generic.go:356] "Generic (PLEG): container finished" podID="ea356156-f0f7-42dd-8c96-b30bdede6205" containerID="9b1936988c966435c391cc5cb6204200d1c205817fbf92860f5ee53c6f3f7f14" exitCode=0 Jul 23 03:55:57.666479 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:57.666302 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j4hnz" event={"ID":"ea356156-f0f7-42dd-8c96-b30bdede6205","Type":"ContainerDied","Data":"9b1936988c966435c391cc5cb6204200d1c205817fbf92860f5ee53c6f3f7f14"} Jul 23 03:55:57.797330 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:57.797304 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d-metrics-tls\") pod \"dns-default-w64s6\" (UID: \"2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d\") " pod="openshift-dns/dns-default-w64s6" Jul 23 03:55:57.797527 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:57.797505 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 03:55:57.797601 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:57.797579 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d-metrics-tls podName:2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d nodeName:}" failed. No retries permitted until 2026-07-23 03:55:59.797555238 +0000 UTC m=+39.702859528 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d-metrics-tls") pod "dns-default-w64s6" (UID: "2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d") : secret "dns-default-metrics-tls" not found Jul 23 03:55:58.671065 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:58.670914 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j4hnz" event={"ID":"ea356156-f0f7-42dd-8c96-b30bdede6205","Type":"ContainerStarted","Data":"bf4ec32fa0cb47fa56843cb8e37ed5c2c19a013246e2d1a14a8c2c5986070752"} Jul 23 03:55:58.695919 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:58.695867 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-j4hnz" podStartSLOduration=8.820573801 podStartE2EDuration="38.695848489s" totalImagesPullingTime="29.875274688s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 03:55:20 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 03:55:58.694445911 +0000 UTC m=+38.599750215" watchObservedRunningTime="2026-07-23 03:55:58.695848489 +0000 UTC m=+38.601152786" Jul 23 03:55:59.675044 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:59.675012 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nt27s" event={"ID":"cc9ae7fe-6710-4847-bacc-01096f89cd3c","Type":"ContainerStarted","Data":"f44b1b8a0511eaa00f0f53374f7fa5d66042a6f8ad0e08c539ea070d0a71ddd4"} Jul 23 03:55:59.694953 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:59.694906 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-nt27s" podStartSLOduration=2.531004275 podStartE2EDuration="4.694890437s" totalImagesPullingTime="2.163886162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 03:55:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 03:55:59.694698655 +0000 UTC m=+39.600002949" watchObservedRunningTime="2026-07-23 03:55:59.694890437 +0000 UTC m=+39.600194728" Jul 23 03:55:59.810350 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:55:59.810314 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d-metrics-tls\") pod \"dns-default-w64s6\" (UID: \"2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d\") " pod="openshift-dns/dns-default-w64s6" Jul 23 03:55:59.810547 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:59.810492 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 03:55:59.810605 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:55:59.810565 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d-metrics-tls podName:2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d nodeName:}" failed. No retries permitted until 2026-07-23 03:56:03.810545789 +0000 UTC m=+43.715850076 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d-metrics-tls") pod "dns-default-w64s6" (UID: "2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d") : secret "dns-default-metrics-tls" not found Jul 23 03:56:00.275877 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.275832 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-qwlg8"] Jul 23 03:56:00.278745 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.278728 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-qwlg8" Jul 23 03:56:00.281227 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.281201 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 23 03:56:00.281485 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.281471 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-8d78w\"" Jul 23 03:56:00.281563 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.281540 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 23 03:56:00.287191 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.287168 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-qwlg8"] Jul 23 03:56:00.377049 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.377023 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2nm67"] Jul 23 03:56:00.379698 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.379683 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2nm67" Jul 23 03:56:00.382164 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.382135 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 23 03:56:00.382276 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.382144 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 23 03:56:00.382359 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.382279 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 23 03:56:00.382507 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.382491 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-q4zkg\"" Jul 23 03:56:00.382593 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.382580 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 23 03:56:00.384251 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.384229 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-76fccd85c4-bhbr2"] Jul 23 03:56:00.387064 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.387046 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7467ff7dc-dlggz"] Jul 23 03:56:00.387196 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.387174 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-76fccd85c4-bhbr2" Jul 23 03:56:00.390154 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.390135 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7467ff7dc-dlggz" Jul 23 03:56:00.391094 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.391078 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 23 03:56:00.391216 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.391198 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 23 03:56:00.391292 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.391266 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 23 03:56:00.391342 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.391327 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 23 03:56:00.391405 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.391383 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 23 03:56:00.391405 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.391388 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-887bn\"" Jul 23 03:56:00.394146 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.394126 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 23 03:56:00.394374 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.394321 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 23 03:56:00.394374 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.394345 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 23 03:56:00.394374 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.394357 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dj82d\"" Jul 23 03:56:00.399562 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.399544 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2nm67"] Jul 23 03:56:00.402982 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.402966 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 23 03:56:00.405195 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.405180 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 23 03:56:00.409335 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.407987 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-76fccd85c4-bhbr2"] Jul 23 03:56:00.414680 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.414661 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5k2w\" (UniqueName: \"kubernetes.io/projected/b8b0f82e-c3c2-4bee-80c2-8ed759b47910-kube-api-access-r5k2w\") pod \"volume-data-source-validator-78cf4c6f6-qwlg8\" (UID: \"b8b0f82e-c3c2-4bee-80c2-8ed759b47910\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-qwlg8" Jul 23 03:56:00.428721 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.428701 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7467ff7dc-dlggz"] Jul 23 03:56:00.481900 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.481881 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-wzgn9"] Jul 23 03:56:00.484717 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.484700 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-wzgn9" Jul 23 03:56:00.487422 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.487398 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 23 03:56:00.487529 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.487488 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 23 03:56:00.487529 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.487509 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 23 03:56:00.487529 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.487519 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-smlgg\"" Jul 23 03:56:00.487529 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.487526 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 23 03:56:00.492652 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.492633 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 23 03:56:00.495295 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.495276 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-wzgn9"] Jul 23 03:56:00.514995 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.514974 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-registry-tls\") pod \"image-registry-7467ff7dc-dlggz\" (UID: \"1db0e3eb-7604-4de1-ac10-2e11ebe20a2a\") " pod="openshift-image-registry/image-registry-7467ff7dc-dlggz" Jul 23 03:56:00.515095 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.515000 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-bound-sa-token\") pod \"image-registry-7467ff7dc-dlggz\" (UID: \"1db0e3eb-7604-4de1-ac10-2e11ebe20a2a\") " pod="openshift-image-registry/image-registry-7467ff7dc-dlggz" Jul 23 03:56:00.515095 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.515021 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/83d126c6-3769-45ac-8b1e-8e04b8f775b3-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-2nm67\" (UID: \"83d126c6-3769-45ac-8b1e-8e04b8f775b3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2nm67" Jul 23 03:56:00.515095 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.515039 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0a513261-862c-43e7-bf4a-2011100221f5-stats-auth\") pod \"router-default-76fccd85c4-bhbr2\" (UID: \"0a513261-862c-43e7-bf4a-2011100221f5\") " pod="openshift-ingress/router-default-76fccd85c4-bhbr2" Jul 23 03:56:00.515095 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.515065 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0a513261-862c-43e7-bf4a-2011100221f5-default-certificate\") pod \"router-default-76fccd85c4-bhbr2\" (UID: \"0a513261-862c-43e7-bf4a-2011100221f5\") " pod="openshift-ingress/router-default-76fccd85c4-bhbr2" Jul 23 03:56:00.515269 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.515140 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-trusted-ca\") pod \"image-registry-7467ff7dc-dlggz\" (UID: \"1db0e3eb-7604-4de1-ac10-2e11ebe20a2a\") " pod="openshift-image-registry/image-registry-7467ff7dc-dlggz" Jul 23 03:56:00.515269 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.515195 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a513261-862c-43e7-bf4a-2011100221f5-service-ca-bundle\") pod \"router-default-76fccd85c4-bhbr2\" (UID: \"0a513261-862c-43e7-bf4a-2011100221f5\") " pod="openshift-ingress/router-default-76fccd85c4-bhbr2" Jul 23 03:56:00.515269 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.515248 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5k2w\" (UniqueName: \"kubernetes.io/projected/b8b0f82e-c3c2-4bee-80c2-8ed759b47910-kube-api-access-r5k2w\") pod \"volume-data-source-validator-78cf4c6f6-qwlg8\" (UID: \"b8b0f82e-c3c2-4bee-80c2-8ed759b47910\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-qwlg8" Jul 23 03:56:00.515405 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.515269 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-ca-trust-extracted\") pod \"image-registry-7467ff7dc-dlggz\" (UID: \"1db0e3eb-7604-4de1-ac10-2e11ebe20a2a\") " pod="openshift-image-registry/image-registry-7467ff7dc-dlggz" Jul 23 03:56:00.515405 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.515284 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pgqc\" (UniqueName: \"kubernetes.io/projected/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-kube-api-access-8pgqc\") pod \"image-registry-7467ff7dc-dlggz\" (UID: \"1db0e3eb-7604-4de1-ac10-2e11ebe20a2a\") " pod="openshift-image-registry/image-registry-7467ff7dc-dlggz" Jul 23 03:56:00.515405 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.515299 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/83d126c6-3769-45ac-8b1e-8e04b8f775b3-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-2nm67\" (UID: \"83d126c6-3769-45ac-8b1e-8e04b8f775b3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2nm67" Jul 23 03:56:00.515405 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.515319 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnn8c\" (UniqueName: \"kubernetes.io/projected/83d126c6-3769-45ac-8b1e-8e04b8f775b3-kube-api-access-jnn8c\") pod \"kube-storage-version-migrator-operator-7658d9b749-2nm67\" (UID: \"83d126c6-3769-45ac-8b1e-8e04b8f775b3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2nm67" Jul 23 03:56:00.515405 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.515349 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whwhd\" (UniqueName: \"kubernetes.io/projected/0a513261-862c-43e7-bf4a-2011100221f5-kube-api-access-whwhd\") pod \"router-default-76fccd85c4-bhbr2\" (UID: \"0a513261-862c-43e7-bf4a-2011100221f5\") " pod="openshift-ingress/router-default-76fccd85c4-bhbr2" Jul 23 03:56:00.515405 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.515407 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-installation-pull-secrets\") pod \"image-registry-7467ff7dc-dlggz\" (UID: \"1db0e3eb-7604-4de1-ac10-2e11ebe20a2a\") " pod="openshift-image-registry/image-registry-7467ff7dc-dlggz" Jul 23 03:56:00.515645 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.515462 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-registry-certificates\") pod \"image-registry-7467ff7dc-dlggz\" (UID: \"1db0e3eb-7604-4de1-ac10-2e11ebe20a2a\") " pod="openshift-image-registry/image-registry-7467ff7dc-dlggz" Jul 23 03:56:00.515645 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.515478 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a513261-862c-43e7-bf4a-2011100221f5-metrics-certs\") pod \"router-default-76fccd85c4-bhbr2\" (UID: \"0a513261-862c-43e7-bf4a-2011100221f5\") " pod="openshift-ingress/router-default-76fccd85c4-bhbr2" Jul 23 03:56:00.515645 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.515495 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-image-registry-private-configuration\") pod \"image-registry-7467ff7dc-dlggz\" (UID: \"1db0e3eb-7604-4de1-ac10-2e11ebe20a2a\") " pod="openshift-image-registry/image-registry-7467ff7dc-dlggz" Jul 23 03:56:00.525627 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.525612 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5k2w\" (UniqueName: \"kubernetes.io/projected/b8b0f82e-c3c2-4bee-80c2-8ed759b47910-kube-api-access-r5k2w\") pod \"volume-data-source-validator-78cf4c6f6-qwlg8\" (UID: \"b8b0f82e-c3c2-4bee-80c2-8ed759b47910\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-qwlg8" Jul 23 03:56:00.587675 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.587617 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-qwlg8" Jul 23 03:56:00.616597 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.616572 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a513261-862c-43e7-bf4a-2011100221f5-metrics-certs\") pod \"router-default-76fccd85c4-bhbr2\" (UID: \"0a513261-862c-43e7-bf4a-2011100221f5\") " pod="openshift-ingress/router-default-76fccd85c4-bhbr2" Jul 23 03:56:00.616706 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.616614 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dbd53593-e5e7-473a-b71c-8fbf9272913f-serving-cert\") pod \"console-operator-565b7b569d-wzgn9\" (UID: \"dbd53593-e5e7-473a-b71c-8fbf9272913f\") " pod="openshift-console-operator/console-operator-565b7b569d-wzgn9" Jul 23 03:56:00.616706 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.616641 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-registry-certificates\") pod \"image-registry-7467ff7dc-dlggz\" (UID: \"1db0e3eb-7604-4de1-ac10-2e11ebe20a2a\") " pod="openshift-image-registry/image-registry-7467ff7dc-dlggz" Jul 23 03:56:00.616706 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.616682 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-image-registry-private-configuration\") pod \"image-registry-7467ff7dc-dlggz\" (UID: \"1db0e3eb-7604-4de1-ac10-2e11ebe20a2a\") " pod="openshift-image-registry/image-registry-7467ff7dc-dlggz" Jul 23 03:56:00.616877 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.616709 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ntqb\" (UniqueName: \"kubernetes.io/projected/dbd53593-e5e7-473a-b71c-8fbf9272913f-kube-api-access-5ntqb\") pod \"console-operator-565b7b569d-wzgn9\" (UID: \"dbd53593-e5e7-473a-b71c-8fbf9272913f\") " pod="openshift-console-operator/console-operator-565b7b569d-wzgn9" Jul 23 03:56:00.616877 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:00.616726 2567 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 03:56:00.616877 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.616755 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-registry-tls\") pod \"image-registry-7467ff7dc-dlggz\" (UID: \"1db0e3eb-7604-4de1-ac10-2e11ebe20a2a\") " pod="openshift-image-registry/image-registry-7467ff7dc-dlggz" Jul 23 03:56:00.616877 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:00.616789 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a513261-862c-43e7-bf4a-2011100221f5-metrics-certs podName:0a513261-862c-43e7-bf4a-2011100221f5 nodeName:}" failed. No retries permitted until 2026-07-23 03:56:01.116771287 +0000 UTC m=+41.022075557 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a513261-862c-43e7-bf4a-2011100221f5-metrics-certs") pod "router-default-76fccd85c4-bhbr2" (UID: "0a513261-862c-43e7-bf4a-2011100221f5") : secret "router-metrics-certs-default" not found Jul 23 03:56:00.616877 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.616812 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/83d126c6-3769-45ac-8b1e-8e04b8f775b3-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-2nm67\" (UID: \"83d126c6-3769-45ac-8b1e-8e04b8f775b3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2nm67" Jul 23 03:56:00.616877 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:00.616840 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 03:56:00.616877 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:00.616855 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7467ff7dc-dlggz: secret "image-registry-tls" not found Jul 23 03:56:00.617227 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:00.616901 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-registry-tls podName:1db0e3eb-7604-4de1-ac10-2e11ebe20a2a nodeName:}" failed. No retries permitted until 2026-07-23 03:56:01.116883707 +0000 UTC m=+41.022187979 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-registry-tls") pod "image-registry-7467ff7dc-dlggz" (UID: "1db0e3eb-7604-4de1-ac10-2e11ebe20a2a") : secret "image-registry-tls" not found Jul 23 03:56:00.617227 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.616839 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0a513261-862c-43e7-bf4a-2011100221f5-default-certificate\") pod \"router-default-76fccd85c4-bhbr2\" (UID: \"0a513261-862c-43e7-bf4a-2011100221f5\") " pod="openshift-ingress/router-default-76fccd85c4-bhbr2" Jul 23 03:56:00.617227 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.616939 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-trusted-ca\") pod \"image-registry-7467ff7dc-dlggz\" (UID: \"1db0e3eb-7604-4de1-ac10-2e11ebe20a2a\") " pod="openshift-image-registry/image-registry-7467ff7dc-dlggz" Jul 23 03:56:00.617227 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.616981 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a513261-862c-43e7-bf4a-2011100221f5-service-ca-bundle\") pod \"router-default-76fccd85c4-bhbr2\" (UID: \"0a513261-862c-43e7-bf4a-2011100221f5\") " pod="openshift-ingress/router-default-76fccd85c4-bhbr2" Jul 23 03:56:00.617227 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.617007 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dbd53593-e5e7-473a-b71c-8fbf9272913f-config\") pod \"console-operator-565b7b569d-wzgn9\" (UID: \"dbd53593-e5e7-473a-b71c-8fbf9272913f\") " pod="openshift-console-operator/console-operator-565b7b569d-wzgn9" Jul 23 03:56:00.617227 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.617030 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pgqc\" (UniqueName: \"kubernetes.io/projected/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-kube-api-access-8pgqc\") pod \"image-registry-7467ff7dc-dlggz\" (UID: \"1db0e3eb-7604-4de1-ac10-2e11ebe20a2a\") " pod="openshift-image-registry/image-registry-7467ff7dc-dlggz" Jul 23 03:56:00.617227 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.617046 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnn8c\" (UniqueName: \"kubernetes.io/projected/83d126c6-3769-45ac-8b1e-8e04b8f775b3-kube-api-access-jnn8c\") pod \"kube-storage-version-migrator-operator-7658d9b749-2nm67\" (UID: \"83d126c6-3769-45ac-8b1e-8e04b8f775b3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2nm67" Jul 23 03:56:00.617227 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.617073 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dbd53593-e5e7-473a-b71c-8fbf9272913f-trusted-ca\") pod \"console-operator-565b7b569d-wzgn9\" (UID: \"dbd53593-e5e7-473a-b71c-8fbf9272913f\") " pod="openshift-console-operator/console-operator-565b7b569d-wzgn9" Jul 23 03:56:00.617227 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.617093 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-bound-sa-token\") pod \"image-registry-7467ff7dc-dlggz\" (UID: \"1db0e3eb-7604-4de1-ac10-2e11ebe20a2a\") " pod="openshift-image-registry/image-registry-7467ff7dc-dlggz" Jul 23 03:56:00.617227 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.617132 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0a513261-862c-43e7-bf4a-2011100221f5-stats-auth\") pod \"router-default-76fccd85c4-bhbr2\" (UID: \"0a513261-862c-43e7-bf4a-2011100221f5\") " pod="openshift-ingress/router-default-76fccd85c4-bhbr2" Jul 23 03:56:00.617227 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.617182 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-ca-trust-extracted\") pod \"image-registry-7467ff7dc-dlggz\" (UID: \"1db0e3eb-7604-4de1-ac10-2e11ebe20a2a\") " pod="openshift-image-registry/image-registry-7467ff7dc-dlggz" Jul 23 03:56:00.617227 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.617208 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/83d126c6-3769-45ac-8b1e-8e04b8f775b3-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-2nm67\" (UID: \"83d126c6-3769-45ac-8b1e-8e04b8f775b3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2nm67" Jul 23 03:56:00.617227 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.617230 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whwhd\" (UniqueName: \"kubernetes.io/projected/0a513261-862c-43e7-bf4a-2011100221f5-kube-api-access-whwhd\") pod \"router-default-76fccd85c4-bhbr2\" (UID: \"0a513261-862c-43e7-bf4a-2011100221f5\") " pod="openshift-ingress/router-default-76fccd85c4-bhbr2" Jul 23 03:56:00.617890 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.617257 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-installation-pull-secrets\") pod \"image-registry-7467ff7dc-dlggz\" (UID: \"1db0e3eb-7604-4de1-ac10-2e11ebe20a2a\") " pod="openshift-image-registry/image-registry-7467ff7dc-dlggz" Jul 23 03:56:00.617890 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:00.617710 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0a513261-862c-43e7-bf4a-2011100221f5-service-ca-bundle podName:0a513261-862c-43e7-bf4a-2011100221f5 nodeName:}" failed. No retries permitted until 2026-07-23 03:56:01.117695721 +0000 UTC m=+41.023000027 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0a513261-862c-43e7-bf4a-2011100221f5-service-ca-bundle") pod "router-default-76fccd85c4-bhbr2" (UID: "0a513261-862c-43e7-bf4a-2011100221f5") : configmap references non-existent config key: service-ca.crt Jul 23 03:56:00.617890 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.617839 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-registry-certificates\") pod \"image-registry-7467ff7dc-dlggz\" (UID: \"1db0e3eb-7604-4de1-ac10-2e11ebe20a2a\") " pod="openshift-image-registry/image-registry-7467ff7dc-dlggz" Jul 23 03:56:00.618172 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.618144 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-trusted-ca\") pod \"image-registry-7467ff7dc-dlggz\" (UID: \"1db0e3eb-7604-4de1-ac10-2e11ebe20a2a\") " pod="openshift-image-registry/image-registry-7467ff7dc-dlggz" Jul 23 03:56:00.618244 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.618171 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/83d126c6-3769-45ac-8b1e-8e04b8f775b3-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-2nm67\" (UID: \"83d126c6-3769-45ac-8b1e-8e04b8f775b3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2nm67" Jul 23 03:56:00.618601 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.618497 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-ca-trust-extracted\") pod \"image-registry-7467ff7dc-dlggz\" (UID: \"1db0e3eb-7604-4de1-ac10-2e11ebe20a2a\") " pod="openshift-image-registry/image-registry-7467ff7dc-dlggz" Jul 23 03:56:00.620331 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.619806 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0a513261-862c-43e7-bf4a-2011100221f5-default-certificate\") pod \"router-default-76fccd85c4-bhbr2\" (UID: \"0a513261-862c-43e7-bf4a-2011100221f5\") " pod="openshift-ingress/router-default-76fccd85c4-bhbr2" Jul 23 03:56:00.620331 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.619892 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-image-registry-private-configuration\") pod \"image-registry-7467ff7dc-dlggz\" (UID: \"1db0e3eb-7604-4de1-ac10-2e11ebe20a2a\") " pod="openshift-image-registry/image-registry-7467ff7dc-dlggz" Jul 23 03:56:00.621048 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.620347 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-installation-pull-secrets\") pod \"image-registry-7467ff7dc-dlggz\" (UID: \"1db0e3eb-7604-4de1-ac10-2e11ebe20a2a\") " pod="openshift-image-registry/image-registry-7467ff7dc-dlggz" Jul 23 03:56:00.621048 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.620671 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/83d126c6-3769-45ac-8b1e-8e04b8f775b3-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-2nm67\" (UID: \"83d126c6-3769-45ac-8b1e-8e04b8f775b3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2nm67" Jul 23 03:56:00.621048 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.620992 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0a513261-862c-43e7-bf4a-2011100221f5-stats-auth\") pod \"router-default-76fccd85c4-bhbr2\" (UID: \"0a513261-862c-43e7-bf4a-2011100221f5\") " pod="openshift-ingress/router-default-76fccd85c4-bhbr2" Jul 23 03:56:00.627670 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.627642 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnn8c\" (UniqueName: \"kubernetes.io/projected/83d126c6-3769-45ac-8b1e-8e04b8f775b3-kube-api-access-jnn8c\") pod \"kube-storage-version-migrator-operator-7658d9b749-2nm67\" (UID: \"83d126c6-3769-45ac-8b1e-8e04b8f775b3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2nm67" Jul 23 03:56:00.628184 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.628158 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whwhd\" (UniqueName: \"kubernetes.io/projected/0a513261-862c-43e7-bf4a-2011100221f5-kube-api-access-whwhd\") pod \"router-default-76fccd85c4-bhbr2\" (UID: \"0a513261-862c-43e7-bf4a-2011100221f5\") " pod="openshift-ingress/router-default-76fccd85c4-bhbr2" Jul 23 03:56:00.628273 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.628221 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pgqc\" (UniqueName: \"kubernetes.io/projected/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-kube-api-access-8pgqc\") pod \"image-registry-7467ff7dc-dlggz\" (UID: \"1db0e3eb-7604-4de1-ac10-2e11ebe20a2a\") " pod="openshift-image-registry/image-registry-7467ff7dc-dlggz" Jul 23 03:56:00.628929 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.628898 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-bound-sa-token\") pod \"image-registry-7467ff7dc-dlggz\" (UID: \"1db0e3eb-7604-4de1-ac10-2e11ebe20a2a\") " pod="openshift-image-registry/image-registry-7467ff7dc-dlggz" Jul 23 03:56:00.688372 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.688347 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2nm67" Jul 23 03:56:00.717750 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.717712 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-qwlg8"] Jul 23 03:56:00.717941 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.717818 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dbd53593-e5e7-473a-b71c-8fbf9272913f-serving-cert\") pod \"console-operator-565b7b569d-wzgn9\" (UID: \"dbd53593-e5e7-473a-b71c-8fbf9272913f\") " pod="openshift-console-operator/console-operator-565b7b569d-wzgn9" Jul 23 03:56:00.717941 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.717848 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ntqb\" (UniqueName: \"kubernetes.io/projected/dbd53593-e5e7-473a-b71c-8fbf9272913f-kube-api-access-5ntqb\") pod \"console-operator-565b7b569d-wzgn9\" (UID: \"dbd53593-e5e7-473a-b71c-8fbf9272913f\") " pod="openshift-console-operator/console-operator-565b7b569d-wzgn9" Jul 23 03:56:00.717941 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.717912 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dbd53593-e5e7-473a-b71c-8fbf9272913f-config\") pod \"console-operator-565b7b569d-wzgn9\" (UID: \"dbd53593-e5e7-473a-b71c-8fbf9272913f\") " pod="openshift-console-operator/console-operator-565b7b569d-wzgn9" Jul 23 03:56:00.717941 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.717940 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dbd53593-e5e7-473a-b71c-8fbf9272913f-trusted-ca\") pod \"console-operator-565b7b569d-wzgn9\" (UID: \"dbd53593-e5e7-473a-b71c-8fbf9272913f\") " pod="openshift-console-operator/console-operator-565b7b569d-wzgn9" Jul 23 03:56:00.718646 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.718560 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dbd53593-e5e7-473a-b71c-8fbf9272913f-config\") pod \"console-operator-565b7b569d-wzgn9\" (UID: \"dbd53593-e5e7-473a-b71c-8fbf9272913f\") " pod="openshift-console-operator/console-operator-565b7b569d-wzgn9" Jul 23 03:56:00.719077 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.718994 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dbd53593-e5e7-473a-b71c-8fbf9272913f-trusted-ca\") pod \"console-operator-565b7b569d-wzgn9\" (UID: \"dbd53593-e5e7-473a-b71c-8fbf9272913f\") " pod="openshift-console-operator/console-operator-565b7b569d-wzgn9" Jul 23 03:56:00.720315 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.720298 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dbd53593-e5e7-473a-b71c-8fbf9272913f-serving-cert\") pod \"console-operator-565b7b569d-wzgn9\" (UID: \"dbd53593-e5e7-473a-b71c-8fbf9272913f\") " pod="openshift-console-operator/console-operator-565b7b569d-wzgn9" Jul 23 03:56:00.721165 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:56:00.721142 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8b0f82e_c3c2_4bee_80c2_8ed759b47910.slice/crio-dd0b7091ecbc052015d6047720f5652aec832ba944c1c20e45b90aeef00d23e6 WatchSource:0}: Error finding container dd0b7091ecbc052015d6047720f5652aec832ba944c1c20e45b90aeef00d23e6: Status 404 returned error can't find the container with id dd0b7091ecbc052015d6047720f5652aec832ba944c1c20e45b90aeef00d23e6 Jul 23 03:56:00.726177 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.726157 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ntqb\" (UniqueName: \"kubernetes.io/projected/dbd53593-e5e7-473a-b71c-8fbf9272913f-kube-api-access-5ntqb\") pod \"console-operator-565b7b569d-wzgn9\" (UID: \"dbd53593-e5e7-473a-b71c-8fbf9272913f\") " pod="openshift-console-operator/console-operator-565b7b569d-wzgn9" Jul 23 03:56:00.794146 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.794116 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-wzgn9" Jul 23 03:56:00.810511 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.810481 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2nm67"] Jul 23 03:56:00.814686 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:56:00.814659 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83d126c6_3769_45ac_8b1e_8e04b8f775b3.slice/crio-c033be097528d4995462261ef035cc9e7c41fae23c273eba33b5e75e21199cbd WatchSource:0}: Error finding container c033be097528d4995462261ef035cc9e7c41fae23c273eba33b5e75e21199cbd: Status 404 returned error can't find the container with id c033be097528d4995462261ef035cc9e7c41fae23c273eba33b5e75e21199cbd Jul 23 03:56:00.933711 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:00.933679 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-wzgn9"] Jul 23 03:56:00.936213 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:56:00.936188 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbd53593_e5e7_473a_b71c_8fbf9272913f.slice/crio-43e3c4d0d62472d9ca2069cb853fc70278d7c1c6e716dbb7bfc002a6525ab382 WatchSource:0}: Error finding container 43e3c4d0d62472d9ca2069cb853fc70278d7c1c6e716dbb7bfc002a6525ab382: Status 404 returned error can't find the container with id 43e3c4d0d62472d9ca2069cb853fc70278d7c1c6e716dbb7bfc002a6525ab382 Jul 23 03:56:01.120637 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:01.120543 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a513261-862c-43e7-bf4a-2011100221f5-metrics-certs\") pod \"router-default-76fccd85c4-bhbr2\" (UID: \"0a513261-862c-43e7-bf4a-2011100221f5\") " pod="openshift-ingress/router-default-76fccd85c4-bhbr2" Jul 23 03:56:01.120637 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:01.120611 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-registry-tls\") pod \"image-registry-7467ff7dc-dlggz\" (UID: \"1db0e3eb-7604-4de1-ac10-2e11ebe20a2a\") " pod="openshift-image-registry/image-registry-7467ff7dc-dlggz" Jul 23 03:56:01.120848 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:01.120658 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a513261-862c-43e7-bf4a-2011100221f5-service-ca-bundle\") pod \"router-default-76fccd85c4-bhbr2\" (UID: \"0a513261-862c-43e7-bf4a-2011100221f5\") " pod="openshift-ingress/router-default-76fccd85c4-bhbr2" Jul 23 03:56:01.120848 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:01.120683 2567 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 03:56:01.120848 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:01.120735 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a513261-862c-43e7-bf4a-2011100221f5-metrics-certs podName:0a513261-862c-43e7-bf4a-2011100221f5 nodeName:}" failed. No retries permitted until 2026-07-23 03:56:02.120722106 +0000 UTC m=+42.026026376 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a513261-862c-43e7-bf4a-2011100221f5-metrics-certs") pod "router-default-76fccd85c4-bhbr2" (UID: "0a513261-862c-43e7-bf4a-2011100221f5") : secret "router-metrics-certs-default" not found Jul 23 03:56:01.120848 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:01.120751 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 03:56:01.120848 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:01.120768 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7467ff7dc-dlggz: secret "image-registry-tls" not found Jul 23 03:56:01.120848 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:01.120818 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0a513261-862c-43e7-bf4a-2011100221f5-service-ca-bundle podName:0a513261-862c-43e7-bf4a-2011100221f5 nodeName:}" failed. No retries permitted until 2026-07-23 03:56:02.120802203 +0000 UTC m=+42.026106474 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0a513261-862c-43e7-bf4a-2011100221f5-service-ca-bundle") pod "router-default-76fccd85c4-bhbr2" (UID: "0a513261-862c-43e7-bf4a-2011100221f5") : configmap references non-existent config key: service-ca.crt Jul 23 03:56:01.120848 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:01.120840 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-registry-tls podName:1db0e3eb-7604-4de1-ac10-2e11ebe20a2a nodeName:}" failed. No retries permitted until 2026-07-23 03:56:02.12083015 +0000 UTC m=+42.026134425 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-registry-tls") pod "image-registry-7467ff7dc-dlggz" (UID: "1db0e3eb-7604-4de1-ac10-2e11ebe20a2a") : secret "image-registry-tls" not found Jul 23 03:56:01.683581 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:01.683538 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-wzgn9" event={"ID":"dbd53593-e5e7-473a-b71c-8fbf9272913f","Type":"ContainerStarted","Data":"43e3c4d0d62472d9ca2069cb853fc70278d7c1c6e716dbb7bfc002a6525ab382"} Jul 23 03:56:01.684984 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:01.684954 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-qwlg8" event={"ID":"b8b0f82e-c3c2-4bee-80c2-8ed759b47910","Type":"ContainerStarted","Data":"dd0b7091ecbc052015d6047720f5652aec832ba944c1c20e45b90aeef00d23e6"} Jul 23 03:56:01.686152 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:01.686127 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2nm67" event={"ID":"83d126c6-3769-45ac-8b1e-8e04b8f775b3","Type":"ContainerStarted","Data":"c033be097528d4995462261ef035cc9e7c41fae23c273eba33b5e75e21199cbd"} Jul 23 03:56:02.130738 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:02.130655 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a513261-862c-43e7-bf4a-2011100221f5-service-ca-bundle\") pod \"router-default-76fccd85c4-bhbr2\" (UID: \"0a513261-862c-43e7-bf4a-2011100221f5\") " pod="openshift-ingress/router-default-76fccd85c4-bhbr2" Jul 23 03:56:02.131182 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:02.130769 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a513261-862c-43e7-bf4a-2011100221f5-metrics-certs\") pod \"router-default-76fccd85c4-bhbr2\" (UID: \"0a513261-862c-43e7-bf4a-2011100221f5\") " pod="openshift-ingress/router-default-76fccd85c4-bhbr2" Jul 23 03:56:02.131182 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:02.130824 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-registry-tls\") pod \"image-registry-7467ff7dc-dlggz\" (UID: \"1db0e3eb-7604-4de1-ac10-2e11ebe20a2a\") " pod="openshift-image-registry/image-registry-7467ff7dc-dlggz" Jul 23 03:56:02.131182 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:02.130862 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0a513261-862c-43e7-bf4a-2011100221f5-service-ca-bundle podName:0a513261-862c-43e7-bf4a-2011100221f5 nodeName:}" failed. No retries permitted until 2026-07-23 03:56:04.130838743 +0000 UTC m=+44.036143026 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0a513261-862c-43e7-bf4a-2011100221f5-service-ca-bundle") pod "router-default-76fccd85c4-bhbr2" (UID: "0a513261-862c-43e7-bf4a-2011100221f5") : configmap references non-existent config key: service-ca.crt Jul 23 03:56:02.131182 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:02.130933 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 03:56:02.131182 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:02.130946 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7467ff7dc-dlggz: secret "image-registry-tls" not found Jul 23 03:56:02.131182 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:02.131055 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-registry-tls podName:1db0e3eb-7604-4de1-ac10-2e11ebe20a2a nodeName:}" failed. No retries permitted until 2026-07-23 03:56:04.131037206 +0000 UTC m=+44.036341490 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-registry-tls") pod "image-registry-7467ff7dc-dlggz" (UID: "1db0e3eb-7604-4de1-ac10-2e11ebe20a2a") : secret "image-registry-tls" not found Jul 23 03:56:02.131182 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:02.131132 2567 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 03:56:02.131182 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:02.131167 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a513261-862c-43e7-bf4a-2011100221f5-metrics-certs podName:0a513261-862c-43e7-bf4a-2011100221f5 nodeName:}" failed. No retries permitted until 2026-07-23 03:56:04.131156424 +0000 UTC m=+44.036460695 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a513261-862c-43e7-bf4a-2011100221f5-metrics-certs") pod "router-default-76fccd85c4-bhbr2" (UID: "0a513261-862c-43e7-bf4a-2011100221f5") : secret "router-metrics-certs-default" not found Jul 23 03:56:03.844757 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:03.844722 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d-metrics-tls\") pod \"dns-default-w64s6\" (UID: \"2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d\") " pod="openshift-dns/dns-default-w64s6" Jul 23 03:56:03.845134 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:03.844836 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 03:56:03.845134 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:03.844887 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d-metrics-tls podName:2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d nodeName:}" failed. No retries permitted until 2026-07-23 03:56:11.844874331 +0000 UTC m=+51.750178601 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d-metrics-tls") pod "dns-default-w64s6" (UID: "2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d") : secret "dns-default-metrics-tls" not found Jul 23 03:56:04.147313 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:04.147284 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a513261-862c-43e7-bf4a-2011100221f5-metrics-certs\") pod \"router-default-76fccd85c4-bhbr2\" (UID: \"0a513261-862c-43e7-bf4a-2011100221f5\") " pod="openshift-ingress/router-default-76fccd85c4-bhbr2" Jul 23 03:56:04.147440 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:04.147330 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-registry-tls\") pod \"image-registry-7467ff7dc-dlggz\" (UID: \"1db0e3eb-7604-4de1-ac10-2e11ebe20a2a\") " pod="openshift-image-registry/image-registry-7467ff7dc-dlggz" Jul 23 03:56:04.147440 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:04.147371 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a513261-862c-43e7-bf4a-2011100221f5-service-ca-bundle\") pod \"router-default-76fccd85c4-bhbr2\" (UID: \"0a513261-862c-43e7-bf4a-2011100221f5\") " pod="openshift-ingress/router-default-76fccd85c4-bhbr2" Jul 23 03:56:04.147555 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:04.147451 2567 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 03:56:04.147555 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:04.147507 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0a513261-862c-43e7-bf4a-2011100221f5-service-ca-bundle podName:0a513261-862c-43e7-bf4a-2011100221f5 nodeName:}" failed. No retries permitted until 2026-07-23 03:56:08.14749452 +0000 UTC m=+48.052798790 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0a513261-862c-43e7-bf4a-2011100221f5-service-ca-bundle") pod "router-default-76fccd85c4-bhbr2" (UID: "0a513261-862c-43e7-bf4a-2011100221f5") : configmap references non-existent config key: service-ca.crt Jul 23 03:56:04.147555 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:04.147523 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a513261-862c-43e7-bf4a-2011100221f5-metrics-certs podName:0a513261-862c-43e7-bf4a-2011100221f5 nodeName:}" failed. No retries permitted until 2026-07-23 03:56:08.147516581 +0000 UTC m=+48.052820850 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a513261-862c-43e7-bf4a-2011100221f5-metrics-certs") pod "router-default-76fccd85c4-bhbr2" (UID: "0a513261-862c-43e7-bf4a-2011100221f5") : secret "router-metrics-certs-default" not found Jul 23 03:56:04.147555 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:04.147526 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 03:56:04.147555 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:04.147544 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7467ff7dc-dlggz: secret "image-registry-tls" not found Jul 23 03:56:04.147735 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:04.147594 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-registry-tls podName:1db0e3eb-7604-4de1-ac10-2e11ebe20a2a nodeName:}" failed. No retries permitted until 2026-07-23 03:56:08.147576676 +0000 UTC m=+48.052880947 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-registry-tls") pod "image-registry-7467ff7dc-dlggz" (UID: "1db0e3eb-7604-4de1-ac10-2e11ebe20a2a") : secret "image-registry-tls" not found Jul 23 03:56:04.696307 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:04.696271 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2nm67" event={"ID":"83d126c6-3769-45ac-8b1e-8e04b8f775b3","Type":"ContainerStarted","Data":"1f6f3fc43d84d3161be47546d70ebb60601aae871a28688d130cf948dd062b02"} Jul 23 03:56:04.697902 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:04.697879 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wzgn9_dbd53593-e5e7-473a-b71c-8fbf9272913f/console-operator/0.log" Jul 23 03:56:04.698030 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:04.697915 2567 generic.go:356] "Generic (PLEG): container finished" podID="dbd53593-e5e7-473a-b71c-8fbf9272913f" containerID="764e49dee955d23f2e12482c4fbff90dfcba96b334251f504b94af22ccc47c82" exitCode=255 Jul 23 03:56:04.698030 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:04.697989 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-wzgn9" event={"ID":"dbd53593-e5e7-473a-b71c-8fbf9272913f","Type":"ContainerDied","Data":"764e49dee955d23f2e12482c4fbff90dfcba96b334251f504b94af22ccc47c82"} Jul 23 03:56:04.698204 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:04.698181 2567 scope.go:117] "RemoveContainer" containerID="764e49dee955d23f2e12482c4fbff90dfcba96b334251f504b94af22ccc47c82" Jul 23 03:56:04.699658 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:04.699638 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-qwlg8" event={"ID":"b8b0f82e-c3c2-4bee-80c2-8ed759b47910","Type":"ContainerStarted","Data":"98208f111a79afc277a2fce016fc45a9081b411b8aef6750a58c8361cb45351e"} Jul 23 03:56:04.713267 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:04.713222 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2nm67" podStartSLOduration=1.449073581 podStartE2EDuration="4.713206423s" totalImagesPullingTime="3.264132842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 03:56:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 03:56:04.712571399 +0000 UTC m=+44.617875691" watchObservedRunningTime="2026-07-23 03:56:04.713206423 +0000 UTC m=+44.618510718" Jul 23 03:56:04.747801 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:04.747760 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-qwlg8" podStartSLOduration=1.420343336 podStartE2EDuration="4.747741915s" totalImagesPullingTime="3.327398579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 03:56:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 03:56:04.72996809 +0000 UTC m=+44.635272383" watchObservedRunningTime="2026-07-23 03:56:04.747741915 +0000 UTC m=+44.653046185" Jul 23 03:56:05.180884 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:05.180847 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rhkfn"] Jul 23 03:56:05.199323 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:05.199298 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rhkfn"] Jul 23 03:56:05.199481 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:05.199403 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rhkfn" Jul 23 03:56:05.201834 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:05.201808 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 23 03:56:05.202169 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:05.201835 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kd7d8\"" Jul 23 03:56:05.202169 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:05.201952 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 23 03:56:05.203094 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:05.203073 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 23 03:56:05.203204 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:05.203073 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 23 03:56:05.356046 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:05.355975 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8462a1d3-1d08-4b40-9790-9a60d9ff231e-data-volume\") pod \"insights-runtime-extractor-rhkfn\" (UID: \"8462a1d3-1d08-4b40-9790-9a60d9ff231e\") " pod="openshift-insights/insights-runtime-extractor-rhkfn" Jul 23 03:56:05.356046 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:05.356037 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8462a1d3-1d08-4b40-9790-9a60d9ff231e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rhkfn\" (UID: \"8462a1d3-1d08-4b40-9790-9a60d9ff231e\") " pod="openshift-insights/insights-runtime-extractor-rhkfn" Jul 23 03:56:05.356224 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:05.356063 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8462a1d3-1d08-4b40-9790-9a60d9ff231e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rhkfn\" (UID: \"8462a1d3-1d08-4b40-9790-9a60d9ff231e\") " pod="openshift-insights/insights-runtime-extractor-rhkfn" Jul 23 03:56:05.356224 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:05.356135 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6mnr\" (UniqueName: \"kubernetes.io/projected/8462a1d3-1d08-4b40-9790-9a60d9ff231e-kube-api-access-p6mnr\") pod \"insights-runtime-extractor-rhkfn\" (UID: \"8462a1d3-1d08-4b40-9790-9a60d9ff231e\") " pod="openshift-insights/insights-runtime-extractor-rhkfn" Jul 23 03:56:05.356224 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:05.356180 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8462a1d3-1d08-4b40-9790-9a60d9ff231e-crio-socket\") pod \"insights-runtime-extractor-rhkfn\" (UID: \"8462a1d3-1d08-4b40-9790-9a60d9ff231e\") " pod="openshift-insights/insights-runtime-extractor-rhkfn" Jul 23 03:56:05.457488 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:05.457455 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8462a1d3-1d08-4b40-9790-9a60d9ff231e-crio-socket\") pod \"insights-runtime-extractor-rhkfn\" (UID: \"8462a1d3-1d08-4b40-9790-9a60d9ff231e\") " pod="openshift-insights/insights-runtime-extractor-rhkfn" Jul 23 03:56:05.457634 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:05.457541 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8462a1d3-1d08-4b40-9790-9a60d9ff231e-data-volume\") pod \"insights-runtime-extractor-rhkfn\" (UID: \"8462a1d3-1d08-4b40-9790-9a60d9ff231e\") " pod="openshift-insights/insights-runtime-extractor-rhkfn" Jul 23 03:56:05.457634 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:05.457599 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8462a1d3-1d08-4b40-9790-9a60d9ff231e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rhkfn\" (UID: \"8462a1d3-1d08-4b40-9790-9a60d9ff231e\") " pod="openshift-insights/insights-runtime-extractor-rhkfn" Jul 23 03:56:05.457634 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:05.457624 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8462a1d3-1d08-4b40-9790-9a60d9ff231e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rhkfn\" (UID: \"8462a1d3-1d08-4b40-9790-9a60d9ff231e\") " pod="openshift-insights/insights-runtime-extractor-rhkfn" Jul 23 03:56:05.457782 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:05.457672 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8462a1d3-1d08-4b40-9790-9a60d9ff231e-crio-socket\") pod \"insights-runtime-extractor-rhkfn\" (UID: \"8462a1d3-1d08-4b40-9790-9a60d9ff231e\") " pod="openshift-insights/insights-runtime-extractor-rhkfn" Jul 23 03:56:05.457782 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:05.457697 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 03:56:05.457782 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:05.457780 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8462a1d3-1d08-4b40-9790-9a60d9ff231e-kube-rbac-proxy-cm podName:8462a1d3-1d08-4b40-9790-9a60d9ff231e nodeName:}" failed. No retries permitted until 2026-07-23 03:56:05.957759047 +0000 UTC m=+45.863063334 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8462a1d3-1d08-4b40-9790-9a60d9ff231e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rhkfn" (UID: "8462a1d3-1d08-4b40-9790-9a60d9ff231e") : configmap "kube-rbac-proxy" not found Jul 23 03:56:05.457927 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:05.457780 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 03:56:05.457927 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:05.457834 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8462a1d3-1d08-4b40-9790-9a60d9ff231e-insights-runtime-extractor-tls podName:8462a1d3-1d08-4b40-9790-9a60d9ff231e nodeName:}" failed. No retries permitted until 2026-07-23 03:56:05.957822621 +0000 UTC m=+45.863126890 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8462a1d3-1d08-4b40-9790-9a60d9ff231e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rhkfn" (UID: "8462a1d3-1d08-4b40-9790-9a60d9ff231e") : secret "insights-runtime-extractor-tls" not found Jul 23 03:56:05.457927 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:05.457697 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6mnr\" (UniqueName: \"kubernetes.io/projected/8462a1d3-1d08-4b40-9790-9a60d9ff231e-kube-api-access-p6mnr\") pod \"insights-runtime-extractor-rhkfn\" (UID: \"8462a1d3-1d08-4b40-9790-9a60d9ff231e\") " pod="openshift-insights/insights-runtime-extractor-rhkfn" Jul 23 03:56:05.457927 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:05.457889 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8462a1d3-1d08-4b40-9790-9a60d9ff231e-data-volume\") pod \"insights-runtime-extractor-rhkfn\" (UID: \"8462a1d3-1d08-4b40-9790-9a60d9ff231e\") " pod="openshift-insights/insights-runtime-extractor-rhkfn" Jul 23 03:56:05.466246 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:05.466222 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6mnr\" (UniqueName: \"kubernetes.io/projected/8462a1d3-1d08-4b40-9790-9a60d9ff231e-kube-api-access-p6mnr\") pod \"insights-runtime-extractor-rhkfn\" (UID: \"8462a1d3-1d08-4b40-9790-9a60d9ff231e\") " pod="openshift-insights/insights-runtime-extractor-rhkfn" Jul 23 03:56:05.704619 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:05.704594 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wzgn9_dbd53593-e5e7-473a-b71c-8fbf9272913f/console-operator/1.log" Jul 23 03:56:05.705072 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:05.705051 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wzgn9_dbd53593-e5e7-473a-b71c-8fbf9272913f/console-operator/0.log" Jul 23 03:56:05.705176 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:05.705098 2567 generic.go:356] "Generic (PLEG): container finished" podID="dbd53593-e5e7-473a-b71c-8fbf9272913f" containerID="412156f822c919824d802fcc3e7cc3847a04a46aedf2a1c4ac6e82828a36b614" exitCode=255 Jul 23 03:56:05.705289 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:05.705185 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-wzgn9" event={"ID":"dbd53593-e5e7-473a-b71c-8fbf9272913f","Type":"ContainerDied","Data":"412156f822c919824d802fcc3e7cc3847a04a46aedf2a1c4ac6e82828a36b614"} Jul 23 03:56:05.705289 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:05.705240 2567 scope.go:117] "RemoveContainer" containerID="764e49dee955d23f2e12482c4fbff90dfcba96b334251f504b94af22ccc47c82" Jul 23 03:56:05.705491 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:05.705470 2567 scope.go:117] "RemoveContainer" containerID="412156f822c919824d802fcc3e7cc3847a04a46aedf2a1c4ac6e82828a36b614" Jul 23 03:56:05.705704 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:05.705668 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-wzgn9_openshift-console-operator(dbd53593-e5e7-473a-b71c-8fbf9272913f)\"" pod="openshift-console-operator/console-operator-565b7b569d-wzgn9" podUID="dbd53593-e5e7-473a-b71c-8fbf9272913f" Jul 23 03:56:05.962276 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:05.962202 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8462a1d3-1d08-4b40-9790-9a60d9ff231e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rhkfn\" (UID: \"8462a1d3-1d08-4b40-9790-9a60d9ff231e\") " pod="openshift-insights/insights-runtime-extractor-rhkfn" Jul 23 03:56:05.962276 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:05.962236 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8462a1d3-1d08-4b40-9790-9a60d9ff231e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rhkfn\" (UID: \"8462a1d3-1d08-4b40-9790-9a60d9ff231e\") " pod="openshift-insights/insights-runtime-extractor-rhkfn" Jul 23 03:56:05.962544 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:05.962323 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 03:56:05.962544 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:05.962340 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 03:56:05.962544 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:05.962395 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8462a1d3-1d08-4b40-9790-9a60d9ff231e-insights-runtime-extractor-tls podName:8462a1d3-1d08-4b40-9790-9a60d9ff231e nodeName:}" failed. No retries permitted until 2026-07-23 03:56:06.962381938 +0000 UTC m=+46.867686209 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8462a1d3-1d08-4b40-9790-9a60d9ff231e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rhkfn" (UID: "8462a1d3-1d08-4b40-9790-9a60d9ff231e") : secret "insights-runtime-extractor-tls" not found Jul 23 03:56:05.962544 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:05.962408 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8462a1d3-1d08-4b40-9790-9a60d9ff231e-kube-rbac-proxy-cm podName:8462a1d3-1d08-4b40-9790-9a60d9ff231e nodeName:}" failed. No retries permitted until 2026-07-23 03:56:06.962401872 +0000 UTC m=+46.867706141 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8462a1d3-1d08-4b40-9790-9a60d9ff231e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rhkfn" (UID: "8462a1d3-1d08-4b40-9790-9a60d9ff231e") : configmap "kube-rbac-proxy" not found Jul 23 03:56:06.708687 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:06.708661 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wzgn9_dbd53593-e5e7-473a-b71c-8fbf9272913f/console-operator/1.log" Jul 23 03:56:06.709063 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:06.708954 2567 scope.go:117] "RemoveContainer" containerID="412156f822c919824d802fcc3e7cc3847a04a46aedf2a1c4ac6e82828a36b614" Jul 23 03:56:06.709115 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:06.709097 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-wzgn9_openshift-console-operator(dbd53593-e5e7-473a-b71c-8fbf9272913f)\"" pod="openshift-console-operator/console-operator-565b7b569d-wzgn9" podUID="dbd53593-e5e7-473a-b71c-8fbf9272913f" Jul 23 03:56:06.778070 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:06.778045 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gpl6h_6b6f66fe-a3d8-4ef6-95ee-9016ff91606f/dns-node-resolver/0.log" Jul 23 03:56:06.972311 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:06.972234 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8462a1d3-1d08-4b40-9790-9a60d9ff231e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rhkfn\" (UID: \"8462a1d3-1d08-4b40-9790-9a60d9ff231e\") " pod="openshift-insights/insights-runtime-extractor-rhkfn" Jul 23 03:56:06.972311 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:06.972269 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8462a1d3-1d08-4b40-9790-9a60d9ff231e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rhkfn\" (UID: \"8462a1d3-1d08-4b40-9790-9a60d9ff231e\") " pod="openshift-insights/insights-runtime-extractor-rhkfn" Jul 23 03:56:06.972512 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:06.972366 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 03:56:06.972512 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:06.972375 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 03:56:06.972512 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:06.972453 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8462a1d3-1d08-4b40-9790-9a60d9ff231e-kube-rbac-proxy-cm podName:8462a1d3-1d08-4b40-9790-9a60d9ff231e nodeName:}" failed. No retries permitted until 2026-07-23 03:56:08.972418534 +0000 UTC m=+48.877722803 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8462a1d3-1d08-4b40-9790-9a60d9ff231e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rhkfn" (UID: "8462a1d3-1d08-4b40-9790-9a60d9ff231e") : configmap "kube-rbac-proxy" not found Jul 23 03:56:06.972512 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:06.972510 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8462a1d3-1d08-4b40-9790-9a60d9ff231e-insights-runtime-extractor-tls podName:8462a1d3-1d08-4b40-9790-9a60d9ff231e nodeName:}" failed. No retries permitted until 2026-07-23 03:56:08.972496207 +0000 UTC m=+48.877800478 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8462a1d3-1d08-4b40-9790-9a60d9ff231e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rhkfn" (UID: "8462a1d3-1d08-4b40-9790-9a60d9ff231e") : secret "insights-runtime-extractor-tls" not found Jul 23 03:56:07.216610 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:07.216573 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-h6ksc"] Jul 23 03:56:07.250189 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:07.250114 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-h6ksc"] Jul 23 03:56:07.250336 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:07.250224 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-h6ksc" Jul 23 03:56:07.253103 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:07.253082 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 23 03:56:07.253205 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:07.253084 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-hwgqb\"" Jul 23 03:56:07.253386 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:07.253369 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 23 03:56:07.274870 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:07.274851 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b6d9f954-7eb4-4d0d-9961-ad00ed214cd3-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-h6ksc\" (UID: \"b6d9f954-7eb4-4d0d-9961-ad00ed214cd3\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-h6ksc" Jul 23 03:56:07.274870 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:07.274876 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b6d9f954-7eb4-4d0d-9961-ad00ed214cd3-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-h6ksc\" (UID: \"b6d9f954-7eb4-4d0d-9961-ad00ed214cd3\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-h6ksc" Jul 23 03:56:07.376154 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:07.376124 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b6d9f954-7eb4-4d0d-9961-ad00ed214cd3-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-h6ksc\" (UID: \"b6d9f954-7eb4-4d0d-9961-ad00ed214cd3\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-h6ksc" Jul 23 03:56:07.376292 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:07.376161 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b6d9f954-7eb4-4d0d-9961-ad00ed214cd3-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-h6ksc\" (UID: \"b6d9f954-7eb4-4d0d-9961-ad00ed214cd3\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-h6ksc" Jul 23 03:56:07.376352 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:07.376337 2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 23 03:56:07.376407 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:07.376397 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6d9f954-7eb4-4d0d-9961-ad00ed214cd3-networking-console-plugin-cert podName:b6d9f954-7eb4-4d0d-9961-ad00ed214cd3 nodeName:}" failed. No retries permitted until 2026-07-23 03:56:07.876382856 +0000 UTC m=+47.781687126 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b6d9f954-7eb4-4d0d-9961-ad00ed214cd3-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-h6ksc" (UID: "b6d9f954-7eb4-4d0d-9961-ad00ed214cd3") : secret "networking-console-plugin-cert" not found Jul 23 03:56:07.376748 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:07.376731 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b6d9f954-7eb4-4d0d-9961-ad00ed214cd3-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-h6ksc\" (UID: \"b6d9f954-7eb4-4d0d-9961-ad00ed214cd3\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-h6ksc" Jul 23 03:56:07.579280 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:07.579220 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kw46l_787ce18c-a72b-426f-a9f0-51f2c6f7c9bb/node-ca/0.log" Jul 23 03:56:07.880454 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:07.880362 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b6d9f954-7eb4-4d0d-9961-ad00ed214cd3-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-h6ksc\" (UID: \"b6d9f954-7eb4-4d0d-9961-ad00ed214cd3\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-h6ksc" Jul 23 03:56:07.880794 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:07.880542 2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 23 03:56:07.880794 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:07.880611 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6d9f954-7eb4-4d0d-9961-ad00ed214cd3-networking-console-plugin-cert podName:b6d9f954-7eb4-4d0d-9961-ad00ed214cd3 nodeName:}" failed. No retries permitted until 2026-07-23 03:56:08.88059333 +0000 UTC m=+48.785897601 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b6d9f954-7eb4-4d0d-9961-ad00ed214cd3-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-h6ksc" (UID: "b6d9f954-7eb4-4d0d-9961-ad00ed214cd3") : secret "networking-console-plugin-cert" not found Jul 23 03:56:08.182253 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:08.182223 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a513261-862c-43e7-bf4a-2011100221f5-metrics-certs\") pod \"router-default-76fccd85c4-bhbr2\" (UID: \"0a513261-862c-43e7-bf4a-2011100221f5\") " pod="openshift-ingress/router-default-76fccd85c4-bhbr2" Jul 23 03:56:08.182472 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:08.182277 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-registry-tls\") pod \"image-registry-7467ff7dc-dlggz\" (UID: \"1db0e3eb-7604-4de1-ac10-2e11ebe20a2a\") " pod="openshift-image-registry/image-registry-7467ff7dc-dlggz" Jul 23 03:56:08.182472 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:08.182341 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a513261-862c-43e7-bf4a-2011100221f5-service-ca-bundle\") pod \"router-default-76fccd85c4-bhbr2\" (UID: \"0a513261-862c-43e7-bf4a-2011100221f5\") " pod="openshift-ingress/router-default-76fccd85c4-bhbr2" Jul 23 03:56:08.182472 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:08.182379 2567 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 03:56:08.182472 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:08.182462 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 03:56:08.182626 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:08.182484 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7467ff7dc-dlggz: secret "image-registry-tls" not found Jul 23 03:56:08.182626 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:08.182469 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a513261-862c-43e7-bf4a-2011100221f5-metrics-certs podName:0a513261-862c-43e7-bf4a-2011100221f5 nodeName:}" failed. No retries permitted until 2026-07-23 03:56:16.182450666 +0000 UTC m=+56.087754961 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a513261-862c-43e7-bf4a-2011100221f5-metrics-certs") pod "router-default-76fccd85c4-bhbr2" (UID: "0a513261-862c-43e7-bf4a-2011100221f5") : secret "router-metrics-certs-default" not found Jul 23 03:56:08.182626 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:08.182544 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-registry-tls podName:1db0e3eb-7604-4de1-ac10-2e11ebe20a2a nodeName:}" failed. No retries permitted until 2026-07-23 03:56:16.182530724 +0000 UTC m=+56.087834995 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-registry-tls") pod "image-registry-7467ff7dc-dlggz" (UID: "1db0e3eb-7604-4de1-ac10-2e11ebe20a2a") : secret "image-registry-tls" not found Jul 23 03:56:08.182626 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:08.182557 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0a513261-862c-43e7-bf4a-2011100221f5-service-ca-bundle podName:0a513261-862c-43e7-bf4a-2011100221f5 nodeName:}" failed. No retries permitted until 2026-07-23 03:56:16.182550485 +0000 UTC m=+56.087854754 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0a513261-862c-43e7-bf4a-2011100221f5-service-ca-bundle") pod "router-default-76fccd85c4-bhbr2" (UID: "0a513261-862c-43e7-bf4a-2011100221f5") : configmap references non-existent config key: service-ca.crt Jul 23 03:56:08.377513 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:08.377489 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nt27s_cc9ae7fe-6710-4847-bacc-01096f89cd3c/serve-healthcheck-canary/0.log" Jul 23 03:56:08.889608 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:08.889561 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b6d9f954-7eb4-4d0d-9961-ad00ed214cd3-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-h6ksc\" (UID: \"b6d9f954-7eb4-4d0d-9961-ad00ed214cd3\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-h6ksc" Jul 23 03:56:08.890004 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:08.889720 2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 23 03:56:08.890004 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:08.889801 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6d9f954-7eb4-4d0d-9961-ad00ed214cd3-networking-console-plugin-cert podName:b6d9f954-7eb4-4d0d-9961-ad00ed214cd3 nodeName:}" failed. No retries permitted until 2026-07-23 03:56:10.889784342 +0000 UTC m=+50.795088611 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b6d9f954-7eb4-4d0d-9961-ad00ed214cd3-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-h6ksc" (UID: "b6d9f954-7eb4-4d0d-9961-ad00ed214cd3") : secret "networking-console-plugin-cert" not found Jul 23 03:56:08.990881 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:08.990843 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8462a1d3-1d08-4b40-9790-9a60d9ff231e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rhkfn\" (UID: \"8462a1d3-1d08-4b40-9790-9a60d9ff231e\") " pod="openshift-insights/insights-runtime-extractor-rhkfn" Jul 23 03:56:08.991008 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:08.990885 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8462a1d3-1d08-4b40-9790-9a60d9ff231e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rhkfn\" (UID: \"8462a1d3-1d08-4b40-9790-9a60d9ff231e\") " pod="openshift-insights/insights-runtime-extractor-rhkfn" Jul 23 03:56:08.991008 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:08.990969 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 03:56:08.991008 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:08.991000 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 03:56:08.991118 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:08.991045 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8462a1d3-1d08-4b40-9790-9a60d9ff231e-kube-rbac-proxy-cm podName:8462a1d3-1d08-4b40-9790-9a60d9ff231e nodeName:}" failed. No retries permitted until 2026-07-23 03:56:12.99103044 +0000 UTC m=+52.896334710 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8462a1d3-1d08-4b40-9790-9a60d9ff231e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rhkfn" (UID: "8462a1d3-1d08-4b40-9790-9a60d9ff231e") : configmap "kube-rbac-proxy" not found Jul 23 03:56:08.991118 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:08.991077 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8462a1d3-1d08-4b40-9790-9a60d9ff231e-insights-runtime-extractor-tls podName:8462a1d3-1d08-4b40-9790-9a60d9ff231e nodeName:}" failed. No retries permitted until 2026-07-23 03:56:12.991065787 +0000 UTC m=+52.896370057 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8462a1d3-1d08-4b40-9790-9a60d9ff231e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rhkfn" (UID: "8462a1d3-1d08-4b40-9790-9a60d9ff231e") : secret "insights-runtime-extractor-tls" not found Jul 23 03:56:10.794983 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:10.794950 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-wzgn9" Jul 23 03:56:10.794983 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:10.794994 2567 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-wzgn9" Jul 23 03:56:10.795389 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:10.795319 2567 scope.go:117] "RemoveContainer" containerID="412156f822c919824d802fcc3e7cc3847a04a46aedf2a1c4ac6e82828a36b614" Jul 23 03:56:10.795507 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:10.795488 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-wzgn9_openshift-console-operator(dbd53593-e5e7-473a-b71c-8fbf9272913f)\"" pod="openshift-console-operator/console-operator-565b7b569d-wzgn9" podUID="dbd53593-e5e7-473a-b71c-8fbf9272913f" Jul 23 03:56:10.907262 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:10.907229 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b6d9f954-7eb4-4d0d-9961-ad00ed214cd3-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-h6ksc\" (UID: \"b6d9f954-7eb4-4d0d-9961-ad00ed214cd3\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-h6ksc" Jul 23 03:56:10.907402 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:10.907322 2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 23 03:56:10.907402 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:10.907386 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6d9f954-7eb4-4d0d-9961-ad00ed214cd3-networking-console-plugin-cert podName:b6d9f954-7eb4-4d0d-9961-ad00ed214cd3 nodeName:}" failed. No retries permitted until 2026-07-23 03:56:14.907373174 +0000 UTC m=+54.812677445 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b6d9f954-7eb4-4d0d-9961-ad00ed214cd3-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-h6ksc" (UID: "b6d9f954-7eb4-4d0d-9961-ad00ed214cd3") : secret "networking-console-plugin-cert" not found Jul 23 03:56:11.916686 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:11.916653 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d-metrics-tls\") pod \"dns-default-w64s6\" (UID: \"2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d\") " pod="openshift-dns/dns-default-w64s6" Jul 23 03:56:11.917077 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:11.916780 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 03:56:11.917077 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:11.916830 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d-metrics-tls podName:2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d nodeName:}" failed. No retries permitted until 2026-07-23 03:56:27.916818415 +0000 UTC m=+67.822122685 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d-metrics-tls") pod "dns-default-w64s6" (UID: "2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d") : secret "dns-default-metrics-tls" not found Jul 23 03:56:13.025125 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:13.025090 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8462a1d3-1d08-4b40-9790-9a60d9ff231e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rhkfn\" (UID: \"8462a1d3-1d08-4b40-9790-9a60d9ff231e\") " pod="openshift-insights/insights-runtime-extractor-rhkfn" Jul 23 03:56:13.025125 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:13.025124 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8462a1d3-1d08-4b40-9790-9a60d9ff231e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rhkfn\" (UID: \"8462a1d3-1d08-4b40-9790-9a60d9ff231e\") " pod="openshift-insights/insights-runtime-extractor-rhkfn" Jul 23 03:56:13.025636 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:13.025275 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 03:56:13.025636 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:13.025355 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8462a1d3-1d08-4b40-9790-9a60d9ff231e-insights-runtime-extractor-tls podName:8462a1d3-1d08-4b40-9790-9a60d9ff231e nodeName:}" failed. No retries permitted until 2026-07-23 03:56:21.02534116 +0000 UTC m=+60.930645430 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8462a1d3-1d08-4b40-9790-9a60d9ff231e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rhkfn" (UID: "8462a1d3-1d08-4b40-9790-9a60d9ff231e") : secret "insights-runtime-extractor-tls" not found Jul 23 03:56:13.025713 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:13.025684 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8462a1d3-1d08-4b40-9790-9a60d9ff231e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rhkfn\" (UID: \"8462a1d3-1d08-4b40-9790-9a60d9ff231e\") " pod="openshift-insights/insights-runtime-extractor-rhkfn" Jul 23 03:56:14.941866 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:14.941824 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b6d9f954-7eb4-4d0d-9961-ad00ed214cd3-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-h6ksc\" (UID: \"b6d9f954-7eb4-4d0d-9961-ad00ed214cd3\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-h6ksc" Jul 23 03:56:14.942248 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:14.941986 2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 23 03:56:14.942248 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:14.942047 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6d9f954-7eb4-4d0d-9961-ad00ed214cd3-networking-console-plugin-cert podName:b6d9f954-7eb4-4d0d-9961-ad00ed214cd3 nodeName:}" failed. No retries permitted until 2026-07-23 03:56:22.942030572 +0000 UTC m=+62.847334842 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b6d9f954-7eb4-4d0d-9961-ad00ed214cd3-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-h6ksc" (UID: "b6d9f954-7eb4-4d0d-9961-ad00ed214cd3") : secret "networking-console-plugin-cert" not found Jul 23 03:56:16.252387 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:16.252351 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a513261-862c-43e7-bf4a-2011100221f5-metrics-certs\") pod \"router-default-76fccd85c4-bhbr2\" (UID: \"0a513261-862c-43e7-bf4a-2011100221f5\") " pod="openshift-ingress/router-default-76fccd85c4-bhbr2" Jul 23 03:56:16.252773 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:16.252400 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-registry-tls\") pod \"image-registry-7467ff7dc-dlggz\" (UID: \"1db0e3eb-7604-4de1-ac10-2e11ebe20a2a\") " pod="openshift-image-registry/image-registry-7467ff7dc-dlggz" Jul 23 03:56:16.252773 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:16.252462 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a513261-862c-43e7-bf4a-2011100221f5-service-ca-bundle\") pod \"router-default-76fccd85c4-bhbr2\" (UID: \"0a513261-862c-43e7-bf4a-2011100221f5\") " pod="openshift-ingress/router-default-76fccd85c4-bhbr2" Jul 23 03:56:16.252988 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:16.252969 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a513261-862c-43e7-bf4a-2011100221f5-service-ca-bundle\") pod \"router-default-76fccd85c4-bhbr2\" (UID: \"0a513261-862c-43e7-bf4a-2011100221f5\") " pod="openshift-ingress/router-default-76fccd85c4-bhbr2" Jul 23 03:56:16.254891 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:16.254870 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a513261-862c-43e7-bf4a-2011100221f5-metrics-certs\") pod \"router-default-76fccd85c4-bhbr2\" (UID: \"0a513261-862c-43e7-bf4a-2011100221f5\") " pod="openshift-ingress/router-default-76fccd85c4-bhbr2" Jul 23 03:56:16.254973 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:16.254920 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-registry-tls\") pod \"image-registry-7467ff7dc-dlggz\" (UID: \"1db0e3eb-7604-4de1-ac10-2e11ebe20a2a\") " pod="openshift-image-registry/image-registry-7467ff7dc-dlggz" Jul 23 03:56:16.297003 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:16.296973 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-76fccd85c4-bhbr2" Jul 23 03:56:16.303205 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:16.303182 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7467ff7dc-dlggz" Jul 23 03:56:16.463116 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:16.463096 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-76fccd85c4-bhbr2"] Jul 23 03:56:16.465349 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:56:16.465312 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a513261_862c_43e7_bf4a_2011100221f5.slice/crio-eb21cb781953bdca386de692a2a6fbd5f23582f225facf3ffba6f96d9c88081a WatchSource:0}: Error finding container eb21cb781953bdca386de692a2a6fbd5f23582f225facf3ffba6f96d9c88081a: Status 404 returned error can't find the container with id eb21cb781953bdca386de692a2a6fbd5f23582f225facf3ffba6f96d9c88081a Jul 23 03:56:16.479914 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:16.479890 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7467ff7dc-dlggz"] Jul 23 03:56:16.482653 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:56:16.482630 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1db0e3eb_7604_4de1_ac10_2e11ebe20a2a.slice/crio-c507a6b4851d9d3f2bc3342fc616d38e038d944b9c0bc5a121a9df8c9298a4d2 WatchSource:0}: Error finding container c507a6b4851d9d3f2bc3342fc616d38e038d944b9c0bc5a121a9df8c9298a4d2: Status 404 returned error can't find the container with id c507a6b4851d9d3f2bc3342fc616d38e038d944b9c0bc5a121a9df8c9298a4d2 Jul 23 03:56:16.736496 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:16.736460 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7467ff7dc-dlggz" event={"ID":"1db0e3eb-7604-4de1-ac10-2e11ebe20a2a","Type":"ContainerStarted","Data":"338bca6cd89b02b1560c6c1c5b5c881b6d97a3ac877fbd7b251d3c8b5180b820"} Jul 23 03:56:16.736668 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:16.736518 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7467ff7dc-dlggz" event={"ID":"1db0e3eb-7604-4de1-ac10-2e11ebe20a2a","Type":"ContainerStarted","Data":"c507a6b4851d9d3f2bc3342fc616d38e038d944b9c0bc5a121a9df8c9298a4d2"} Jul 23 03:56:16.736668 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:16.736620 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7467ff7dc-dlggz" Jul 23 03:56:16.737997 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:16.737968 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-76fccd85c4-bhbr2" event={"ID":"0a513261-862c-43e7-bf4a-2011100221f5","Type":"ContainerStarted","Data":"2f2fd2414e568db4480eaff892ab8c055b7517a52dda4cfc3e1085dd4714f736"} Jul 23 03:56:16.738110 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:16.738013 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-76fccd85c4-bhbr2" event={"ID":"0a513261-862c-43e7-bf4a-2011100221f5","Type":"ContainerStarted","Data":"eb21cb781953bdca386de692a2a6fbd5f23582f225facf3ffba6f96d9c88081a"} Jul 23 03:56:16.756684 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:16.756645 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7467ff7dc-dlggz" podStartSLOduration=16.756630553 podStartE2EDuration="16.756630553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 03:56:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 03:56:16.755634663 +0000 UTC m=+56.660938980" watchObservedRunningTime="2026-07-23 03:56:16.756630553 +0000 UTC m=+56.661934845" Jul 23 03:56:16.775596 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:16.775525 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-76fccd85c4-bhbr2" podStartSLOduration=16.775510145 podStartE2EDuration="16.775510145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 03:56:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 03:56:16.774608369 +0000 UTC m=+56.679912706" watchObservedRunningTime="2026-07-23 03:56:16.775510145 +0000 UTC m=+56.680814439" Jul 23 03:56:17.297319 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:17.297289 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-76fccd85c4-bhbr2" Jul 23 03:56:17.300698 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:17.300677 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-76fccd85c4-bhbr2" Jul 23 03:56:17.741488 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:17.741457 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-76fccd85c4-bhbr2" Jul 23 03:56:17.742660 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:17.742639 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-76fccd85c4-bhbr2" Jul 23 03:56:20.651945 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:20.651914 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wd7gs" Jul 23 03:56:21.089617 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:21.089580 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8462a1d3-1d08-4b40-9790-9a60d9ff231e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rhkfn\" (UID: \"8462a1d3-1d08-4b40-9790-9a60d9ff231e\") " pod="openshift-insights/insights-runtime-extractor-rhkfn" Jul 23 03:56:21.092091 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:21.092055 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8462a1d3-1d08-4b40-9790-9a60d9ff231e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rhkfn\" (UID: \"8462a1d3-1d08-4b40-9790-9a60d9ff231e\") " pod="openshift-insights/insights-runtime-extractor-rhkfn" Jul 23 03:56:21.110408 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:21.110382 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kd7d8\"" Jul 23 03:56:21.118674 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:21.118659 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rhkfn" Jul 23 03:56:21.245972 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:21.245942 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rhkfn"] Jul 23 03:56:21.249093 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:56:21.249070 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8462a1d3_1d08_4b40_9790_9a60d9ff231e.slice/crio-d7770a7d4f0eae69d57760768d9e2b046465e70e94e80df1202b32727620c39a WatchSource:0}: Error finding container d7770a7d4f0eae69d57760768d9e2b046465e70e94e80df1202b32727620c39a: Status 404 returned error can't find the container with id d7770a7d4f0eae69d57760768d9e2b046465e70e94e80df1202b32727620c39a Jul 23 03:56:21.753784 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:21.753746 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rhkfn" event={"ID":"8462a1d3-1d08-4b40-9790-9a60d9ff231e","Type":"ContainerStarted","Data":"d7770a7d4f0eae69d57760768d9e2b046465e70e94e80df1202b32727620c39a"} Jul 23 03:56:23.003868 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:23.003832 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b6d9f954-7eb4-4d0d-9961-ad00ed214cd3-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-h6ksc\" (UID: \"b6d9f954-7eb4-4d0d-9961-ad00ed214cd3\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-h6ksc" Jul 23 03:56:23.006692 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:23.006659 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b6d9f954-7eb4-4d0d-9961-ad00ed214cd3-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-h6ksc\" (UID: \"b6d9f954-7eb4-4d0d-9961-ad00ed214cd3\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-h6ksc" Jul 23 03:56:23.161663 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:23.161627 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-hwgqb\"" Jul 23 03:56:23.168968 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:23.168942 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-h6ksc" Jul 23 03:56:23.312958 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:23.312789 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-h6ksc"] Jul 23 03:56:23.315610 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:56:23.315582 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6d9f954_7eb4_4d0d_9961_ad00ed214cd3.slice/crio-51a011905a67cc6931b9f3d03e5c481c4f7f3f14cc88db828adf7a51cd03ea44 WatchSource:0}: Error finding container 51a011905a67cc6931b9f3d03e5c481c4f7f3f14cc88db828adf7a51cd03ea44: Status 404 returned error can't find the container with id 51a011905a67cc6931b9f3d03e5c481c4f7f3f14cc88db828adf7a51cd03ea44 Jul 23 03:56:23.487848 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:23.487820 2567 scope.go:117] "RemoveContainer" containerID="412156f822c919824d802fcc3e7cc3847a04a46aedf2a1c4ac6e82828a36b614" Jul 23 03:56:23.762575 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:23.762550 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wzgn9_dbd53593-e5e7-473a-b71c-8fbf9272913f/console-operator/1.log" Jul 23 03:56:23.762750 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:23.762663 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-wzgn9" event={"ID":"dbd53593-e5e7-473a-b71c-8fbf9272913f","Type":"ContainerStarted","Data":"e1be40eec240c773e4fb6eee0ecfbeda570f52bb97fc11388b6e83634bdd4130"} Jul 23 03:56:23.763234 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:23.763215 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-wzgn9" Jul 23 03:56:23.764215 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:23.764177 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-h6ksc" event={"ID":"b6d9f954-7eb4-4d0d-9961-ad00ed214cd3","Type":"ContainerStarted","Data":"51a011905a67cc6931b9f3d03e5c481c4f7f3f14cc88db828adf7a51cd03ea44"} Jul 23 03:56:23.764656 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:23.764620 2567 patch_prober.go:28] interesting pod/console-operator-565b7b569d-wzgn9 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.133.0.12:8443/readyz\": dial tcp 10.133.0.12:8443: connect: connection refused" start-of-body= Jul 23 03:56:23.764751 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:23.764672 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-565b7b569d-wzgn9" podUID="dbd53593-e5e7-473a-b71c-8fbf9272913f" containerName="console-operator" probeResult="failure" output="Get \"https://10.133.0.12:8443/readyz\": dial tcp 10.133.0.12:8443: connect: connection refused" Jul 23 03:56:23.779838 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:23.779615 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-565b7b569d-wzgn9" podStartSLOduration=20.64304374 podStartE2EDuration="23.779598642s" totalImagesPullingTime="3.136554902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 03:56:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 03:56:23.779189522 +0000 UTC m=+63.684493839" watchObservedRunningTime="2026-07-23 03:56:23.779598642 +0000 UTC m=+63.684902936" Jul 23 03:56:24.767760 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:24.767718 2567 patch_prober.go:28] interesting pod/console-operator-565b7b569d-wzgn9 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.133.0.12:8443/readyz\": dial tcp 10.133.0.12:8443: connect: connection refused" start-of-body= Jul 23 03:56:24.768195 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:24.767782 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-565b7b569d-wzgn9" podUID="dbd53593-e5e7-473a-b71c-8fbf9272913f" containerName="console-operator" probeResult="failure" output="Get \"https://10.133.0.12:8443/readyz\": dial tcp 10.133.0.12:8443: connect: connection refused" Jul 23 03:56:25.772776 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:25.772632 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wzgn9_dbd53593-e5e7-473a-b71c-8fbf9272913f/console-operator/2.log" Jul 23 03:56:25.773468 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:25.773420 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wzgn9_dbd53593-e5e7-473a-b71c-8fbf9272913f/console-operator/1.log" Jul 23 03:56:25.773581 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:25.773481 2567 generic.go:356] "Generic (PLEG): container finished" podID="dbd53593-e5e7-473a-b71c-8fbf9272913f" containerID="e1be40eec240c773e4fb6eee0ecfbeda570f52bb97fc11388b6e83634bdd4130" exitCode=255 Jul 23 03:56:25.773581 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:25.773564 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-wzgn9" event={"ID":"dbd53593-e5e7-473a-b71c-8fbf9272913f","Type":"ContainerDied","Data":"e1be40eec240c773e4fb6eee0ecfbeda570f52bb97fc11388b6e83634bdd4130"} Jul 23 03:56:25.773703 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:25.773616 2567 scope.go:117] "RemoveContainer" containerID="412156f822c919824d802fcc3e7cc3847a04a46aedf2a1c4ac6e82828a36b614" Jul 23 03:56:25.774062 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:25.774046 2567 scope.go:117] "RemoveContainer" containerID="e1be40eec240c773e4fb6eee0ecfbeda570f52bb97fc11388b6e83634bdd4130" Jul 23 03:56:25.774249 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:25.774221 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-565b7b569d-wzgn9_openshift-console-operator(dbd53593-e5e7-473a-b71c-8fbf9272913f)\"" pod="openshift-console-operator/console-operator-565b7b569d-wzgn9" podUID="dbd53593-e5e7-473a-b71c-8fbf9272913f" Jul 23 03:56:26.234882 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:26.234856 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec74b46f-3165-4772-9090-b8adbf536767-original-pull-secret\") pod \"global-pull-secret-syncer-bjprg\" (UID: \"ec74b46f-3165-4772-9090-b8adbf536767\") " pod="kube-system/global-pull-secret-syncer-bjprg" Jul 23 03:56:26.235081 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:26.234902 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73cf013c-7d0b-4139-bf40-8d5d528474a3-metrics-certs\") pod \"network-metrics-daemon-pq2xh\" (UID: \"73cf013c-7d0b-4139-bf40-8d5d528474a3\") " pod="openshift-multus/network-metrics-daemon-pq2xh" Jul 23 03:56:26.237615 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:26.237594 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 23 03:56:26.238578 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:26.238564 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 23 03:56:26.247840 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:26.247817 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec74b46f-3165-4772-9090-b8adbf536767-original-pull-secret\") pod \"global-pull-secret-syncer-bjprg\" (UID: \"ec74b46f-3165-4772-9090-b8adbf536767\") " pod="kube-system/global-pull-secret-syncer-bjprg" Jul 23 03:56:26.247948 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:26.247825 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73cf013c-7d0b-4139-bf40-8d5d528474a3-metrics-certs\") pod \"network-metrics-daemon-pq2xh\" (UID: \"73cf013c-7d0b-4139-bf40-8d5d528474a3\") " pod="openshift-multus/network-metrics-daemon-pq2xh" Jul 23 03:56:26.302611 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:26.302575 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l57h5\"" Jul 23 03:56:26.310508 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:26.310474 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pq2xh" Jul 23 03:56:26.436513 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:26.436476 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zrms\" (UniqueName: \"kubernetes.io/projected/1e14ea08-0647-4056-9f4a-e639fb16c928-kube-api-access-4zrms\") pod \"network-check-target-w99tk\" (UID: \"1e14ea08-0647-4056-9f4a-e639fb16c928\") " pod="openshift-network-diagnostics/network-check-target-w99tk" Jul 23 03:56:26.439297 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:26.439273 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 23 03:56:26.449966 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:26.449942 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 23 03:56:26.461057 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:26.461023 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zrms\" (UniqueName: \"kubernetes.io/projected/1e14ea08-0647-4056-9f4a-e639fb16c928-kube-api-access-4zrms\") pod \"network-check-target-w99tk\" (UID: \"1e14ea08-0647-4056-9f4a-e639fb16c928\") " pod="openshift-network-diagnostics/network-check-target-w99tk" Jul 23 03:56:26.498240 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:26.498184 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjprg" Jul 23 03:56:26.612900 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:26.612788 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zlbwn\"" Jul 23 03:56:26.620555 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:26.620297 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w99tk" Jul 23 03:56:26.713544 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:26.713510 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pq2xh"] Jul 23 03:56:26.717300 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:56:26.717266 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73cf013c_7d0b_4139_bf40_8d5d528474a3.slice/crio-7af01c26d76a203716ade4150e86313c0d44cd59f875cbe66fe88798a89619de WatchSource:0}: Error finding container 7af01c26d76a203716ade4150e86313c0d44cd59f875cbe66fe88798a89619de: Status 404 returned error can't find the container with id 7af01c26d76a203716ade4150e86313c0d44cd59f875cbe66fe88798a89619de Jul 23 03:56:26.732065 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:26.731998 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bjprg"] Jul 23 03:56:26.735459 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:56:26.735409 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec74b46f_3165_4772_9090_b8adbf536767.slice/crio-be6ca977a046195797a5dec5132d47a8edff13a759c1f242b07469c54313cb3f WatchSource:0}: Error finding container be6ca977a046195797a5dec5132d47a8edff13a759c1f242b07469c54313cb3f: Status 404 returned error can't find the container with id be6ca977a046195797a5dec5132d47a8edff13a759c1f242b07469c54313cb3f Jul 23 03:56:26.778261 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:26.778184 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-w99tk"] Jul 23 03:56:26.781585 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:26.781552 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-h6ksc" event={"ID":"b6d9f954-7eb4-4d0d-9961-ad00ed214cd3","Type":"ContainerStarted","Data":"ffde37ef6e91abaab259971c3639f121bc99ac59790a37105d8b04a4c200d57c"} Jul 23 03:56:26.783273 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:56:26.783249 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e14ea08_0647_4056_9f4a_e639fb16c928.slice/crio-71afbaf49f4f29fe5a760203563125470644177ff05716cdcebeb34fcc209be6 WatchSource:0}: Error finding container 71afbaf49f4f29fe5a760203563125470644177ff05716cdcebeb34fcc209be6: Status 404 returned error can't find the container with id 71afbaf49f4f29fe5a760203563125470644177ff05716cdcebeb34fcc209be6 Jul 23 03:56:26.783795 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:26.783771 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pq2xh" event={"ID":"73cf013c-7d0b-4139-bf40-8d5d528474a3","Type":"ContainerStarted","Data":"7af01c26d76a203716ade4150e86313c0d44cd59f875cbe66fe88798a89619de"} Jul 23 03:56:26.785209 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:26.785104 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bjprg" event={"ID":"ec74b46f-3165-4772-9090-b8adbf536767","Type":"ContainerStarted","Data":"be6ca977a046195797a5dec5132d47a8edff13a759c1f242b07469c54313cb3f"} Jul 23 03:56:26.786436 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:26.786405 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wzgn9_dbd53593-e5e7-473a-b71c-8fbf9272913f/console-operator/2.log" Jul 23 03:56:26.787871 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:26.787850 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rhkfn" event={"ID":"8462a1d3-1d08-4b40-9790-9a60d9ff231e","Type":"ContainerStarted","Data":"810ebbc5551127c3f7e928cea0a72dca5987ff25185be743b5941e574a3ffeff"} Jul 23 03:56:26.803209 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:26.801139 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-h6ksc" podStartSLOduration=16.523104709 podStartE2EDuration="19.801120173s" totalImagesPullingTime="3.278015464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 03:56:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 03:56:26.799900674 +0000 UTC m=+66.705204961" watchObservedRunningTime="2026-07-23 03:56:26.801120173 +0000 UTC m=+66.706424465" Jul 23 03:56:27.794153 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:27.794111 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-w99tk" event={"ID":"1e14ea08-0647-4056-9f4a-e639fb16c928","Type":"ContainerStarted","Data":"71afbaf49f4f29fe5a760203563125470644177ff05716cdcebeb34fcc209be6"} Jul 23 03:56:27.948605 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:27.948138 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d-metrics-tls\") pod \"dns-default-w64s6\" (UID: \"2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d\") " pod="openshift-dns/dns-default-w64s6" Jul 23 03:56:27.952164 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:27.952139 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d-metrics-tls\") pod \"dns-default-w64s6\" (UID: \"2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d\") " pod="openshift-dns/dns-default-w64s6" Jul 23 03:56:28.156160 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:28.155864 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jzncz\"" Jul 23 03:56:28.163518 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:28.163390 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w64s6" Jul 23 03:56:28.336543 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:28.335969 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w64s6"] Jul 23 03:56:28.340513 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:56:28.340402 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e1f5194_79cd_4bdb_9afc_133dbeb7cf5d.slice/crio-a901f6de3f65e8bab07be3aab990c13d2c0dadd1b56e27fe8d4534c06bc60ff3 WatchSource:0}: Error finding container a901f6de3f65e8bab07be3aab990c13d2c0dadd1b56e27fe8d4534c06bc60ff3: Status 404 returned error can't find the container with id a901f6de3f65e8bab07be3aab990c13d2c0dadd1b56e27fe8d4534c06bc60ff3 Jul 23 03:56:28.802188 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:28.801331 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pq2xh" event={"ID":"73cf013c-7d0b-4139-bf40-8d5d528474a3","Type":"ContainerStarted","Data":"e32646d209be90fc43fa2eba11631828eb345d9615906d2098b384aa3b61c141"} Jul 23 03:56:28.802188 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:28.801404 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pq2xh" event={"ID":"73cf013c-7d0b-4139-bf40-8d5d528474a3","Type":"ContainerStarted","Data":"73372be13cdc8ab66f1f5e13478957ae33c5e4e27553b34f4095e87d7443de3d"} Jul 23 03:56:28.803915 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:28.803886 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w64s6" event={"ID":"2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d","Type":"ContainerStarted","Data":"a901f6de3f65e8bab07be3aab990c13d2c0dadd1b56e27fe8d4534c06bc60ff3"} Jul 23 03:56:28.825864 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:28.825188 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-pq2xh" podStartSLOduration=67.556958924 podStartE2EDuration="1m8.825134435s" totalImagesPullingTime="1.268175511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 03:55:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 03:56:28.824288268 +0000 UTC m=+68.729592584" watchObservedRunningTime="2026-07-23 03:56:28.825134435 +0000 UTC m=+68.730438730" Jul 23 03:56:29.315818 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:29.315761 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7467ff7dc-dlggz"] Jul 23 03:56:29.393681 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:29.393645 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-fb4fdbc47-ppvss"] Jul 23 03:56:29.435116 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:29.434966 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-fb4fdbc47-ppvss"] Jul 23 03:56:29.435116 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:29.435096 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fb4fdbc47-ppvss" Jul 23 03:56:29.566330 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:29.566253 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/84aef285-74a3-433d-bcdd-6a22df1e44d9-image-registry-private-configuration\") pod \"image-registry-fb4fdbc47-ppvss\" (UID: \"84aef285-74a3-433d-bcdd-6a22df1e44d9\") " pod="openshift-image-registry/image-registry-fb4fdbc47-ppvss" Jul 23 03:56:29.566330 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:29.566313 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/84aef285-74a3-433d-bcdd-6a22df1e44d9-ca-trust-extracted\") pod \"image-registry-fb4fdbc47-ppvss\" (UID: \"84aef285-74a3-433d-bcdd-6a22df1e44d9\") " pod="openshift-image-registry/image-registry-fb4fdbc47-ppvss" Jul 23 03:56:29.566566 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:29.566385 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/84aef285-74a3-433d-bcdd-6a22df1e44d9-installation-pull-secrets\") pod \"image-registry-fb4fdbc47-ppvss\" (UID: \"84aef285-74a3-433d-bcdd-6a22df1e44d9\") " pod="openshift-image-registry/image-registry-fb4fdbc47-ppvss" Jul 23 03:56:29.566566 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:29.566465 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt7rd\" (UniqueName: \"kubernetes.io/projected/84aef285-74a3-433d-bcdd-6a22df1e44d9-kube-api-access-nt7rd\") pod \"image-registry-fb4fdbc47-ppvss\" (UID: \"84aef285-74a3-433d-bcdd-6a22df1e44d9\") " pod="openshift-image-registry/image-registry-fb4fdbc47-ppvss" Jul 23 03:56:29.566566 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:29.566495 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/84aef285-74a3-433d-bcdd-6a22df1e44d9-registry-tls\") pod \"image-registry-fb4fdbc47-ppvss\" (UID: \"84aef285-74a3-433d-bcdd-6a22df1e44d9\") " pod="openshift-image-registry/image-registry-fb4fdbc47-ppvss" Jul 23 03:56:29.566566 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:29.566538 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/84aef285-74a3-433d-bcdd-6a22df1e44d9-registry-certificates\") pod \"image-registry-fb4fdbc47-ppvss\" (UID: \"84aef285-74a3-433d-bcdd-6a22df1e44d9\") " pod="openshift-image-registry/image-registry-fb4fdbc47-ppvss" Jul 23 03:56:29.566733 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:29.566599 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/84aef285-74a3-433d-bcdd-6a22df1e44d9-bound-sa-token\") pod \"image-registry-fb4fdbc47-ppvss\" (UID: \"84aef285-74a3-433d-bcdd-6a22df1e44d9\") " pod="openshift-image-registry/image-registry-fb4fdbc47-ppvss" Jul 23 03:56:29.566733 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:29.566632 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/84aef285-74a3-433d-bcdd-6a22df1e44d9-trusted-ca\") pod \"image-registry-fb4fdbc47-ppvss\" (UID: \"84aef285-74a3-433d-bcdd-6a22df1e44d9\") " pod="openshift-image-registry/image-registry-fb4fdbc47-ppvss" Jul 23 03:56:29.667514 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:29.667207 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/84aef285-74a3-433d-bcdd-6a22df1e44d9-bound-sa-token\") pod \"image-registry-fb4fdbc47-ppvss\" (UID: \"84aef285-74a3-433d-bcdd-6a22df1e44d9\") " pod="openshift-image-registry/image-registry-fb4fdbc47-ppvss" Jul 23 03:56:29.667514 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:29.667271 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/84aef285-74a3-433d-bcdd-6a22df1e44d9-trusted-ca\") pod \"image-registry-fb4fdbc47-ppvss\" (UID: \"84aef285-74a3-433d-bcdd-6a22df1e44d9\") " pod="openshift-image-registry/image-registry-fb4fdbc47-ppvss" Jul 23 03:56:29.667514 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:29.667310 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/84aef285-74a3-433d-bcdd-6a22df1e44d9-image-registry-private-configuration\") pod \"image-registry-fb4fdbc47-ppvss\" (UID: \"84aef285-74a3-433d-bcdd-6a22df1e44d9\") " pod="openshift-image-registry/image-registry-fb4fdbc47-ppvss" Jul 23 03:56:29.667514 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:29.667351 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/84aef285-74a3-433d-bcdd-6a22df1e44d9-ca-trust-extracted\") pod \"image-registry-fb4fdbc47-ppvss\" (UID: \"84aef285-74a3-433d-bcdd-6a22df1e44d9\") " pod="openshift-image-registry/image-registry-fb4fdbc47-ppvss" Jul 23 03:56:29.667514 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:29.667389 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/84aef285-74a3-433d-bcdd-6a22df1e44d9-installation-pull-secrets\") pod \"image-registry-fb4fdbc47-ppvss\" (UID: \"84aef285-74a3-433d-bcdd-6a22df1e44d9\") " pod="openshift-image-registry/image-registry-fb4fdbc47-ppvss" Jul 23 03:56:29.667886 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:29.667770 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt7rd\" (UniqueName: \"kubernetes.io/projected/84aef285-74a3-433d-bcdd-6a22df1e44d9-kube-api-access-nt7rd\") pod \"image-registry-fb4fdbc47-ppvss\" (UID: \"84aef285-74a3-433d-bcdd-6a22df1e44d9\") " pod="openshift-image-registry/image-registry-fb4fdbc47-ppvss" Jul 23 03:56:29.667886 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:29.667816 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/84aef285-74a3-433d-bcdd-6a22df1e44d9-registry-tls\") pod \"image-registry-fb4fdbc47-ppvss\" (UID: \"84aef285-74a3-433d-bcdd-6a22df1e44d9\") " pod="openshift-image-registry/image-registry-fb4fdbc47-ppvss" Jul 23 03:56:29.667886 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:29.667866 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/84aef285-74a3-433d-bcdd-6a22df1e44d9-registry-certificates\") pod \"image-registry-fb4fdbc47-ppvss\" (UID: \"84aef285-74a3-433d-bcdd-6a22df1e44d9\") " pod="openshift-image-registry/image-registry-fb4fdbc47-ppvss" Jul 23 03:56:29.668028 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:29.667877 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/84aef285-74a3-433d-bcdd-6a22df1e44d9-ca-trust-extracted\") pod \"image-registry-fb4fdbc47-ppvss\" (UID: \"84aef285-74a3-433d-bcdd-6a22df1e44d9\") " pod="openshift-image-registry/image-registry-fb4fdbc47-ppvss" Jul 23 03:56:29.669332 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:29.669309 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/84aef285-74a3-433d-bcdd-6a22df1e44d9-trusted-ca\") pod \"image-registry-fb4fdbc47-ppvss\" (UID: \"84aef285-74a3-433d-bcdd-6a22df1e44d9\") " pod="openshift-image-registry/image-registry-fb4fdbc47-ppvss" Jul 23 03:56:29.670083 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:29.670037 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/84aef285-74a3-433d-bcdd-6a22df1e44d9-registry-certificates\") pod \"image-registry-fb4fdbc47-ppvss\" (UID: \"84aef285-74a3-433d-bcdd-6a22df1e44d9\") " pod="openshift-image-registry/image-registry-fb4fdbc47-ppvss" Jul 23 03:56:29.671157 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:29.671139 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/84aef285-74a3-433d-bcdd-6a22df1e44d9-installation-pull-secrets\") pod \"image-registry-fb4fdbc47-ppvss\" (UID: \"84aef285-74a3-433d-bcdd-6a22df1e44d9\") " pod="openshift-image-registry/image-registry-fb4fdbc47-ppvss" Jul 23 03:56:29.672246 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:29.672208 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/84aef285-74a3-433d-bcdd-6a22df1e44d9-registry-tls\") pod \"image-registry-fb4fdbc47-ppvss\" (UID: \"84aef285-74a3-433d-bcdd-6a22df1e44d9\") " pod="openshift-image-registry/image-registry-fb4fdbc47-ppvss" Jul 23 03:56:29.672629 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:29.672605 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/84aef285-74a3-433d-bcdd-6a22df1e44d9-image-registry-private-configuration\") pod \"image-registry-fb4fdbc47-ppvss\" (UID: \"84aef285-74a3-433d-bcdd-6a22df1e44d9\") " pod="openshift-image-registry/image-registry-fb4fdbc47-ppvss" Jul 23 03:56:29.682987 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:29.682950 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt7rd\" (UniqueName: \"kubernetes.io/projected/84aef285-74a3-433d-bcdd-6a22df1e44d9-kube-api-access-nt7rd\") pod \"image-registry-fb4fdbc47-ppvss\" (UID: \"84aef285-74a3-433d-bcdd-6a22df1e44d9\") " pod="openshift-image-registry/image-registry-fb4fdbc47-ppvss" Jul 23 03:56:29.683444 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:29.683405 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/84aef285-74a3-433d-bcdd-6a22df1e44d9-bound-sa-token\") pod \"image-registry-fb4fdbc47-ppvss\" (UID: \"84aef285-74a3-433d-bcdd-6a22df1e44d9\") " pod="openshift-image-registry/image-registry-fb4fdbc47-ppvss" Jul 23 03:56:29.747455 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:29.747225 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fb4fdbc47-ppvss" Jul 23 03:56:30.795135 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:30.795100 2567 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-wzgn9" Jul 23 03:56:30.795568 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:30.795548 2567 scope.go:117] "RemoveContainer" containerID="e1be40eec240c773e4fb6eee0ecfbeda570f52bb97fc11388b6e83634bdd4130" Jul 23 03:56:30.795754 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:30.795732 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-565b7b569d-wzgn9_openshift-console-operator(dbd53593-e5e7-473a-b71c-8fbf9272913f)\"" pod="openshift-console-operator/console-operator-565b7b569d-wzgn9" podUID="dbd53593-e5e7-473a-b71c-8fbf9272913f" Jul 23 03:56:33.699072 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:33.699043 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-fb4fdbc47-ppvss"] Jul 23 03:56:34.060382 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:56:34.060349 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84aef285_74a3_433d_bcdd_6a22df1e44d9.slice/crio-652709a733245d8a59ae09452d49a0b7286cfcace7a2922d9def964145ebb513 WatchSource:0}: Error finding container 652709a733245d8a59ae09452d49a0b7286cfcace7a2922d9def964145ebb513: Status 404 returned error can't find the container with id 652709a733245d8a59ae09452d49a0b7286cfcace7a2922d9def964145ebb513 Jul 23 03:56:34.827085 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:34.827042 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rhkfn" event={"ID":"8462a1d3-1d08-4b40-9790-9a60d9ff231e","Type":"ContainerStarted","Data":"cace05a7f130eb8d27e36b31921aadc6641971cb5171773634ee052e186dbc62"} Jul 23 03:56:34.828718 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:34.828658 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bjprg" event={"ID":"ec74b46f-3165-4772-9090-b8adbf536767","Type":"ContainerStarted","Data":"bd1bb22577cf7a45c411fa2e3f739508ea931c3a7bbc6254fcd31b7ae4238a46"} Jul 23 03:56:34.830319 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:34.830296 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-w99tk" event={"ID":"1e14ea08-0647-4056-9f4a-e639fb16c928","Type":"ContainerStarted","Data":"02ae3f607408c2675e481228d9c9b1bd9b7a1cec3c2b1466e13c5c46a4a86b65"} Jul 23 03:56:34.830486 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:34.830469 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-w99tk" Jul 23 03:56:34.832319 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:34.832299 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w64s6" event={"ID":"2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d","Type":"ContainerStarted","Data":"8a60e02cbe8208e4021006ef16a5141961413de419742aef187fb21b6a0afdef"} Jul 23 03:56:34.832405 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:34.832349 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-w64s6" Jul 23 03:56:34.832405 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:34.832372 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w64s6" event={"ID":"2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d","Type":"ContainerStarted","Data":"87280e8aac63cee87779d466a10e38ecfb26c103f199e640d8bf85c91fe38206"} Jul 23 03:56:34.833886 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:34.833864 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fb4fdbc47-ppvss" event={"ID":"84aef285-74a3-433d-bcdd-6a22df1e44d9","Type":"ContainerStarted","Data":"5749c05e870d46dfaa2d900f77360aba2df0d6b0bdbe0e736e82aabe6ed4fdef"} Jul 23 03:56:34.833973 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:34.833902 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fb4fdbc47-ppvss" event={"ID":"84aef285-74a3-433d-bcdd-6a22df1e44d9","Type":"ContainerStarted","Data":"652709a733245d8a59ae09452d49a0b7286cfcace7a2922d9def964145ebb513"} Jul 23 03:56:34.833973 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:34.833941 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-fb4fdbc47-ppvss" Jul 23 03:56:34.844465 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:34.844402 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-bjprg" podStartSLOduration=67.500421472 podStartE2EDuration="1m14.844386316s" totalImagesPullingTime="7.343964844s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 03:55:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 03:56:34.843037954 +0000 UTC m=+74.748342257" watchObservedRunningTime="2026-07-23 03:56:34.844386316 +0000 UTC m=+74.749690603" Jul 23 03:56:34.864922 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:34.864844 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-fb4fdbc47-ppvss" podStartSLOduration=5.864827155 podStartE2EDuration="5.864827155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 03:56:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 03:56:34.862600039 +0000 UTC m=+74.767904350" watchObservedRunningTime="2026-07-23 03:56:34.864827155 +0000 UTC m=+74.770131447" Jul 23 03:56:34.888602 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:34.888555 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-w99tk" podStartSLOduration=67.58996803 podStartE2EDuration="1m14.888538879s" totalImagesPullingTime="7.298570849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 03:55:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 03:56:34.887469982 +0000 UTC m=+74.792774299" watchObservedRunningTime="2026-07-23 03:56:34.888538879 +0000 UTC m=+74.793843189" Jul 23 03:56:34.920636 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:34.920595 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-w64s6" podStartSLOduration=34.166912953 podStartE2EDuration="39.92057668s" totalImagesPullingTime="5.753663727s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 03:55:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 03:56:34.920174112 +0000 UTC m=+74.825478426" watchObservedRunningTime="2026-07-23 03:56:34.92057668 +0000 UTC m=+74.825880972" Jul 23 03:56:37.846289 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:37.846207 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rhkfn" event={"ID":"8462a1d3-1d08-4b40-9790-9a60d9ff231e","Type":"ContainerStarted","Data":"2df0c54c098a99171bf1e074cbb1bffe90dd94983675db6c4eecaa964d18eaad"} Jul 23 03:56:37.865249 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:37.865204 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-rhkfn" podStartSLOduration=16.706902479 podStartE2EDuration="32.865188874s" totalImagesPullingTime="16.158286395s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 03:56:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 03:56:37.8645831 +0000 UTC m=+77.769887393" watchObservedRunningTime="2026-07-23 03:56:37.865188874 +0000 UTC m=+77.770493165" Jul 23 03:56:39.321763 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:39.321729 2567 patch_prober.go:28] interesting pod/image-registry-7467ff7dc-dlggz container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 23 03:56:39.322123 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:39.321782 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7467ff7dc-dlggz" podUID="1db0e3eb-7604-4de1-ac10-2e11ebe20a2a" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 23 03:56:41.487569 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:41.487458 2567 scope.go:117] "RemoveContainer" containerID="e1be40eec240c773e4fb6eee0ecfbeda570f52bb97fc11388b6e83634bdd4130" Jul 23 03:56:41.487945 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:41.487648 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-565b7b569d-wzgn9_openshift-console-operator(dbd53593-e5e7-473a-b71c-8fbf9272913f)\"" pod="openshift-console-operator/console-operator-565b7b569d-wzgn9" podUID="dbd53593-e5e7-473a-b71c-8fbf9272913f" Jul 23 03:56:44.839499 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:44.839470 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-w64s6" Jul 23 03:56:45.142700 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:45.142622 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-xsjwx"] Jul 23 03:56:45.147050 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:45.147031 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xsjwx" Jul 23 03:56:45.149415 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:45.149393 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 23 03:56:45.149955 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:45.149938 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 23 03:56:45.150178 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:45.150163 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 23 03:56:45.150653 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:45.150634 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 23 03:56:45.150736 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:45.150686 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 23 03:56:45.150787 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:45.150686 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9z7rs\"" Jul 23 03:56:45.150787 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:45.150748 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 23 03:56:45.288515 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:45.288486 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f39f698b-feb8-429d-942e-bc0de10e2860-node-exporter-wtmp\") pod \"node-exporter-xsjwx\" (UID: \"f39f698b-feb8-429d-942e-bc0de10e2860\") " pod="openshift-monitoring/node-exporter-xsjwx" Jul 23 03:56:45.288515 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:45.288516 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f39f698b-feb8-429d-942e-bc0de10e2860-root\") pod \"node-exporter-xsjwx\" (UID: \"f39f698b-feb8-429d-942e-bc0de10e2860\") " pod="openshift-monitoring/node-exporter-xsjwx" Jul 23 03:56:45.288709 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:45.288538 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f39f698b-feb8-429d-942e-bc0de10e2860-node-exporter-textfile\") pod \"node-exporter-xsjwx\" (UID: \"f39f698b-feb8-429d-942e-bc0de10e2860\") " pod="openshift-monitoring/node-exporter-xsjwx" Jul 23 03:56:45.288709 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:45.288598 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f39f698b-feb8-429d-942e-bc0de10e2860-node-exporter-accelerators-collector-config\") pod \"node-exporter-xsjwx\" (UID: \"f39f698b-feb8-429d-942e-bc0de10e2860\") " pod="openshift-monitoring/node-exporter-xsjwx" Jul 23 03:56:45.288709 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:45.288674 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f39f698b-feb8-429d-942e-bc0de10e2860-sys\") pod \"node-exporter-xsjwx\" (UID: \"f39f698b-feb8-429d-942e-bc0de10e2860\") " pod="openshift-monitoring/node-exporter-xsjwx" Jul 23 03:56:45.288709 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:45.288692 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f39f698b-feb8-429d-942e-bc0de10e2860-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xsjwx\" (UID: \"f39f698b-feb8-429d-942e-bc0de10e2860\") " pod="openshift-monitoring/node-exporter-xsjwx" Jul 23 03:56:45.288850 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:45.288720 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f39f698b-feb8-429d-942e-bc0de10e2860-node-exporter-tls\") pod \"node-exporter-xsjwx\" (UID: \"f39f698b-feb8-429d-942e-bc0de10e2860\") " pod="openshift-monitoring/node-exporter-xsjwx" Jul 23 03:56:45.288850 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:45.288754 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwqs7\" (UniqueName: \"kubernetes.io/projected/f39f698b-feb8-429d-942e-bc0de10e2860-kube-api-access-pwqs7\") pod \"node-exporter-xsjwx\" (UID: \"f39f698b-feb8-429d-942e-bc0de10e2860\") " pod="openshift-monitoring/node-exporter-xsjwx" Jul 23 03:56:45.288850 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:45.288799 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f39f698b-feb8-429d-942e-bc0de10e2860-metrics-client-ca\") pod \"node-exporter-xsjwx\" (UID: \"f39f698b-feb8-429d-942e-bc0de10e2860\") " pod="openshift-monitoring/node-exporter-xsjwx" Jul 23 03:56:45.389637 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:45.389608 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f39f698b-feb8-429d-942e-bc0de10e2860-node-exporter-accelerators-collector-config\") pod \"node-exporter-xsjwx\" (UID: \"f39f698b-feb8-429d-942e-bc0de10e2860\") " pod="openshift-monitoring/node-exporter-xsjwx" Jul 23 03:56:45.389777 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:45.389649 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f39f698b-feb8-429d-942e-bc0de10e2860-sys\") pod \"node-exporter-xsjwx\" (UID: \"f39f698b-feb8-429d-942e-bc0de10e2860\") " pod="openshift-monitoring/node-exporter-xsjwx" Jul 23 03:56:45.389777 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:45.389669 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f39f698b-feb8-429d-942e-bc0de10e2860-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xsjwx\" (UID: \"f39f698b-feb8-429d-942e-bc0de10e2860\") " pod="openshift-monitoring/node-exporter-xsjwx" Jul 23 03:56:45.389777 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:45.389703 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f39f698b-feb8-429d-942e-bc0de10e2860-node-exporter-tls\") pod \"node-exporter-xsjwx\" (UID: \"f39f698b-feb8-429d-942e-bc0de10e2860\") " pod="openshift-monitoring/node-exporter-xsjwx" Jul 23 03:56:45.389777 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:45.389741 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwqs7\" (UniqueName: \"kubernetes.io/projected/f39f698b-feb8-429d-942e-bc0de10e2860-kube-api-access-pwqs7\") pod \"node-exporter-xsjwx\" (UID: \"f39f698b-feb8-429d-942e-bc0de10e2860\") " pod="openshift-monitoring/node-exporter-xsjwx" Jul 23 03:56:45.389777 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:45.389775 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f39f698b-feb8-429d-942e-bc0de10e2860-metrics-client-ca\") pod \"node-exporter-xsjwx\" (UID: \"f39f698b-feb8-429d-942e-bc0de10e2860\") " pod="openshift-monitoring/node-exporter-xsjwx" Jul 23 03:56:45.390027 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:45.389738 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f39f698b-feb8-429d-942e-bc0de10e2860-sys\") pod \"node-exporter-xsjwx\" (UID: \"f39f698b-feb8-429d-942e-bc0de10e2860\") " pod="openshift-monitoring/node-exporter-xsjwx" Jul 23 03:56:45.390027 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:45.389806 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f39f698b-feb8-429d-942e-bc0de10e2860-node-exporter-wtmp\") pod \"node-exporter-xsjwx\" (UID: \"f39f698b-feb8-429d-942e-bc0de10e2860\") " pod="openshift-monitoring/node-exporter-xsjwx" Jul 23 03:56:45.390027 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:45.389843 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f39f698b-feb8-429d-942e-bc0de10e2860-root\") pod \"node-exporter-xsjwx\" (UID: \"f39f698b-feb8-429d-942e-bc0de10e2860\") " pod="openshift-monitoring/node-exporter-xsjwx" Jul 23 03:56:45.390027 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:45.389871 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f39f698b-feb8-429d-942e-bc0de10e2860-node-exporter-textfile\") pod \"node-exporter-xsjwx\" (UID: \"f39f698b-feb8-429d-942e-bc0de10e2860\") " pod="openshift-monitoring/node-exporter-xsjwx" Jul 23 03:56:45.390027 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:45.389995 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f39f698b-feb8-429d-942e-bc0de10e2860-node-exporter-wtmp\") pod \"node-exporter-xsjwx\" (UID: \"f39f698b-feb8-429d-942e-bc0de10e2860\") " pod="openshift-monitoring/node-exporter-xsjwx" Jul 23 03:56:45.390244 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:45.390069 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f39f698b-feb8-429d-942e-bc0de10e2860-root\") pod \"node-exporter-xsjwx\" (UID: \"f39f698b-feb8-429d-942e-bc0de10e2860\") " pod="openshift-monitoring/node-exporter-xsjwx" Jul 23 03:56:45.390244 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:45.390156 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f39f698b-feb8-429d-942e-bc0de10e2860-node-exporter-textfile\") pod \"node-exporter-xsjwx\" (UID: \"f39f698b-feb8-429d-942e-bc0de10e2860\") " pod="openshift-monitoring/node-exporter-xsjwx" Jul 23 03:56:45.390369 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:45.390281 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f39f698b-feb8-429d-942e-bc0de10e2860-node-exporter-accelerators-collector-config\") pod \"node-exporter-xsjwx\" (UID: \"f39f698b-feb8-429d-942e-bc0de10e2860\") " pod="openshift-monitoring/node-exporter-xsjwx" Jul 23 03:56:45.390369 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:45.390308 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f39f698b-feb8-429d-942e-bc0de10e2860-metrics-client-ca\") pod \"node-exporter-xsjwx\" (UID: \"f39f698b-feb8-429d-942e-bc0de10e2860\") " pod="openshift-monitoring/node-exporter-xsjwx" Jul 23 03:56:45.392100 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:45.392081 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f39f698b-feb8-429d-942e-bc0de10e2860-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xsjwx\" (UID: \"f39f698b-feb8-429d-942e-bc0de10e2860\") " pod="openshift-monitoring/node-exporter-xsjwx" Jul 23 03:56:45.392254 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:45.392237 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f39f698b-feb8-429d-942e-bc0de10e2860-node-exporter-tls\") pod \"node-exporter-xsjwx\" (UID: \"f39f698b-feb8-429d-942e-bc0de10e2860\") " pod="openshift-monitoring/node-exporter-xsjwx" Jul 23 03:56:45.398538 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:45.398479 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwqs7\" (UniqueName: \"kubernetes.io/projected/f39f698b-feb8-429d-942e-bc0de10e2860-kube-api-access-pwqs7\") pod \"node-exporter-xsjwx\" (UID: \"f39f698b-feb8-429d-942e-bc0de10e2860\") " pod="openshift-monitoring/node-exporter-xsjwx" Jul 23 03:56:45.456116 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:45.456091 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xsjwx" Jul 23 03:56:45.466446 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:56:45.466410 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf39f698b_feb8_429d_942e_bc0de10e2860.slice/crio-d02427d416c1dad1d6ffc3f02a7c4d682822136652b07f3cef4c98571162a0db WatchSource:0}: Error finding container d02427d416c1dad1d6ffc3f02a7c4d682822136652b07f3cef4c98571162a0db: Status 404 returned error can't find the container with id d02427d416c1dad1d6ffc3f02a7c4d682822136652b07f3cef4c98571162a0db Jul 23 03:56:45.873590 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:45.873554 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xsjwx" event={"ID":"f39f698b-feb8-429d-942e-bc0de10e2860","Type":"ContainerStarted","Data":"d02427d416c1dad1d6ffc3f02a7c4d682822136652b07f3cef4c98571162a0db"} Jul 23 03:56:46.878216 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:46.878135 2567 generic.go:356] "Generic (PLEG): container finished" podID="f39f698b-feb8-429d-942e-bc0de10e2860" containerID="c48d0cadd5fbe8f95b6cea0511a497fbcc2dc366005f0149c040d048dbd74cfa" exitCode=0 Jul 23 03:56:46.878216 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:46.878209 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xsjwx" event={"ID":"f39f698b-feb8-429d-942e-bc0de10e2860","Type":"ContainerDied","Data":"c48d0cadd5fbe8f95b6cea0511a497fbcc2dc366005f0149c040d048dbd74cfa"} Jul 23 03:56:47.887943 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:47.887907 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xsjwx" event={"ID":"f39f698b-feb8-429d-942e-bc0de10e2860","Type":"ContainerStarted","Data":"523342cbaf853319d8c02061c5168af67cf9af16fd20c48102be706f6646dfad"} Jul 23 03:56:47.888361 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:47.887979 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xsjwx" event={"ID":"f39f698b-feb8-429d-942e-bc0de10e2860","Type":"ContainerStarted","Data":"e2a0153bb149db7ec1c4b8f2767a6dbf8ce2660de8f557694442167770ef85ee"} Jul 23 03:56:47.913948 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:47.913898 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-xsjwx" podStartSLOduration=1.8503119319999999 podStartE2EDuration="2.913883458s" totalImagesPullingTime="994.802018ms" totalInitContainerRuntime="68.769508ms" isStatefulPod=true podCreationTimestamp="2026-07-23 03:56:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 03:56:47.913551759 +0000 UTC m=+87.818856071" watchObservedRunningTime="2026-07-23 03:56:47.913883458 +0000 UTC m=+87.819187749" Jul 23 03:56:49.320716 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:49.320686 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7467ff7dc-dlggz" Jul 23 03:56:53.487051 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:53.487023 2567 scope.go:117] "RemoveContainer" containerID="e1be40eec240c773e4fb6eee0ecfbeda570f52bb97fc11388b6e83634bdd4130" Jul 23 03:56:53.911384 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:53.911315 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wzgn9_dbd53593-e5e7-473a-b71c-8fbf9272913f/console-operator/2.log" Jul 23 03:56:53.911545 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:53.911406 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-wzgn9" event={"ID":"dbd53593-e5e7-473a-b71c-8fbf9272913f","Type":"ContainerStarted","Data":"f8eb059358674ebc059c0716c8c931f5cda1a18bc533e7f3579caf5bc921ded8"} Jul 23 03:56:53.911824 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:53.911797 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-wzgn9" Jul 23 03:56:54.150604 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:54.150575 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-565b7b569d-wzgn9" Jul 23 03:56:54.346333 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:54.346294 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7467ff7dc-dlggz" podUID="1db0e3eb-7604-4de1-ac10-2e11ebe20a2a" containerName="registry" containerID="cri-o://338bca6cd89b02b1560c6c1c5b5c881b6d97a3ac877fbd7b251d3c8b5180b820" gracePeriod=30 Jul 23 03:56:54.586645 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:54.586620 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7467ff7dc-dlggz" Jul 23 03:56:54.754171 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:54.754142 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-trusted-ca\") pod \"1db0e3eb-7604-4de1-ac10-2e11ebe20a2a\" (UID: \"1db0e3eb-7604-4de1-ac10-2e11ebe20a2a\") " Jul 23 03:56:54.754326 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:54.754184 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-bound-sa-token\") pod \"1db0e3eb-7604-4de1-ac10-2e11ebe20a2a\" (UID: \"1db0e3eb-7604-4de1-ac10-2e11ebe20a2a\") " Jul 23 03:56:54.754326 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:54.754216 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8pgqc\" (UniqueName: \"kubernetes.io/projected/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-kube-api-access-8pgqc\") pod \"1db0e3eb-7604-4de1-ac10-2e11ebe20a2a\" (UID: \"1db0e3eb-7604-4de1-ac10-2e11ebe20a2a\") " Jul 23 03:56:54.754326 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:54.754241 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-image-registry-private-configuration\") pod \"1db0e3eb-7604-4de1-ac10-2e11ebe20a2a\" (UID: \"1db0e3eb-7604-4de1-ac10-2e11ebe20a2a\") " Jul 23 03:56:54.754326 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:54.754283 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-registry-tls\") pod \"1db0e3eb-7604-4de1-ac10-2e11ebe20a2a\" (UID: \"1db0e3eb-7604-4de1-ac10-2e11ebe20a2a\") " Jul 23 03:56:54.754326 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:54.754312 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-installation-pull-secrets\") pod \"1db0e3eb-7604-4de1-ac10-2e11ebe20a2a\" (UID: \"1db0e3eb-7604-4de1-ac10-2e11ebe20a2a\") " Jul 23 03:56:54.754590 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:54.754349 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-ca-trust-extracted\") pod \"1db0e3eb-7604-4de1-ac10-2e11ebe20a2a\" (UID: \"1db0e3eb-7604-4de1-ac10-2e11ebe20a2a\") " Jul 23 03:56:54.754590 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:54.754404 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-registry-certificates\") pod \"1db0e3eb-7604-4de1-ac10-2e11ebe20a2a\" (UID: \"1db0e3eb-7604-4de1-ac10-2e11ebe20a2a\") " Jul 23 03:56:54.754745 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:54.754606 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "1db0e3eb-7604-4de1-ac10-2e11ebe20a2a" (UID: "1db0e3eb-7604-4de1-ac10-2e11ebe20a2a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 03:56:54.755530 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:54.755487 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "1db0e3eb-7604-4de1-ac10-2e11ebe20a2a" (UID: "1db0e3eb-7604-4de1-ac10-2e11ebe20a2a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 03:56:54.757032 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:54.757000 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "1db0e3eb-7604-4de1-ac10-2e11ebe20a2a" (UID: "1db0e3eb-7604-4de1-ac10-2e11ebe20a2a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 03:56:54.757229 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:54.757180 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "1db0e3eb-7604-4de1-ac10-2e11ebe20a2a" (UID: "1db0e3eb-7604-4de1-ac10-2e11ebe20a2a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 03:56:54.757296 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:54.757240 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "1db0e3eb-7604-4de1-ac10-2e11ebe20a2a" (UID: "1db0e3eb-7604-4de1-ac10-2e11ebe20a2a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 03:56:54.757334 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:54.757281 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "1db0e3eb-7604-4de1-ac10-2e11ebe20a2a" (UID: "1db0e3eb-7604-4de1-ac10-2e11ebe20a2a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 03:56:54.757396 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:54.757375 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-kube-api-access-8pgqc" (OuterVolumeSpecName: "kube-api-access-8pgqc") pod "1db0e3eb-7604-4de1-ac10-2e11ebe20a2a" (UID: "1db0e3eb-7604-4de1-ac10-2e11ebe20a2a"). InnerVolumeSpecName "kube-api-access-8pgqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 03:56:54.762667 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:54.762641 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "1db0e3eb-7604-4de1-ac10-2e11ebe20a2a" (UID: "1db0e3eb-7604-4de1-ac10-2e11ebe20a2a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 03:56:54.855459 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:54.855406 2567 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-ca-trust-extracted\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 03:56:54.855459 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:54.855459 2567 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-registry-certificates\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 03:56:54.855647 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:54.855474 2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-trusted-ca\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 03:56:54.855647 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:54.855485 2567 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-bound-sa-token\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 03:56:54.855647 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:54.855497 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pgqc\" (UniqueName: \"kubernetes.io/projected/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-kube-api-access-8pgqc\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 03:56:54.855647 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:54.855509 2567 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-image-registry-private-configuration\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 03:56:54.855647 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:54.855522 2567 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-registry-tls\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 03:56:54.855647 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:54.855533 2567 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a-installation-pull-secrets\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 03:56:54.916915 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:54.916886 2567 generic.go:356] "Generic (PLEG): container finished" podID="1db0e3eb-7604-4de1-ac10-2e11ebe20a2a" containerID="338bca6cd89b02b1560c6c1c5b5c881b6d97a3ac877fbd7b251d3c8b5180b820" exitCode=0 Jul 23 03:56:54.917038 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:54.916931 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7467ff7dc-dlggz" Jul 23 03:56:54.917038 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:54.916977 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7467ff7dc-dlggz" event={"ID":"1db0e3eb-7604-4de1-ac10-2e11ebe20a2a","Type":"ContainerDied","Data":"338bca6cd89b02b1560c6c1c5b5c881b6d97a3ac877fbd7b251d3c8b5180b820"} Jul 23 03:56:54.917038 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:54.917026 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7467ff7dc-dlggz" event={"ID":"1db0e3eb-7604-4de1-ac10-2e11ebe20a2a","Type":"ContainerDied","Data":"c507a6b4851d9d3f2bc3342fc616d38e038d944b9c0bc5a121a9df8c9298a4d2"} Jul 23 03:56:54.917139 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:54.917048 2567 scope.go:117] "RemoveContainer" containerID="338bca6cd89b02b1560c6c1c5b5c881b6d97a3ac877fbd7b251d3c8b5180b820" Jul 23 03:56:54.927255 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:54.927240 2567 scope.go:117] "RemoveContainer" containerID="338bca6cd89b02b1560c6c1c5b5c881b6d97a3ac877fbd7b251d3c8b5180b820" Jul 23 03:56:54.927579 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:56:54.927555 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"338bca6cd89b02b1560c6c1c5b5c881b6d97a3ac877fbd7b251d3c8b5180b820\": container with ID starting with 338bca6cd89b02b1560c6c1c5b5c881b6d97a3ac877fbd7b251d3c8b5180b820 not found: ID does not exist" containerID="338bca6cd89b02b1560c6c1c5b5c881b6d97a3ac877fbd7b251d3c8b5180b820" Jul 23 03:56:54.927657 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:54.927585 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"338bca6cd89b02b1560c6c1c5b5c881b6d97a3ac877fbd7b251d3c8b5180b820"} err="failed to get container status \"338bca6cd89b02b1560c6c1c5b5c881b6d97a3ac877fbd7b251d3c8b5180b820\": rpc error: code = NotFound desc = could not find container \"338bca6cd89b02b1560c6c1c5b5c881b6d97a3ac877fbd7b251d3c8b5180b820\": container with ID starting with 338bca6cd89b02b1560c6c1c5b5c881b6d97a3ac877fbd7b251d3c8b5180b820 not found: ID does not exist" Jul 23 03:56:54.940136 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:54.940117 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7467ff7dc-dlggz"] Jul 23 03:56:54.946058 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:54.946038 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7467ff7dc-dlggz"] Jul 23 03:56:55.841553 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:55.841523 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-fb4fdbc47-ppvss" Jul 23 03:56:56.490688 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:56:56.490658 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1db0e3eb-7604-4de1-ac10-2e11ebe20a2a" path="/var/lib/kubelet/pods/1db0e3eb-7604-4de1-ac10-2e11ebe20a2a/volumes" Jul 23 03:57:04.788695 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:04.788661 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-985dfd8b7-sq25s"] Jul 23 03:57:04.789125 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:04.788891 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1db0e3eb-7604-4de1-ac10-2e11ebe20a2a" containerName="registry" Jul 23 03:57:04.789125 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:04.788901 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1db0e3eb-7604-4de1-ac10-2e11ebe20a2a" containerName="registry" Jul 23 03:57:04.789125 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:04.788967 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1db0e3eb-7604-4de1-ac10-2e11ebe20a2a" containerName="registry" Jul 23 03:57:04.946193 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:04.946162 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-985dfd8b7-sq25s"] Jul 23 03:57:04.946344 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:04.946250 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-985dfd8b7-sq25s" Jul 23 03:57:04.948928 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:04.948899 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 23 03:57:04.949081 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:04.948903 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 23 03:57:04.949081 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:04.948975 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 23 03:57:04.949284 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:04.949269 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 23 03:57:04.950135 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:04.950104 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 23 03:57:04.950265 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:04.950164 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 23 03:57:04.950265 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:04.950161 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 23 03:57:04.950265 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:04.950212 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-hgztq\"" Jul 23 03:57:05.131501 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:05.131404 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d5d208c-645c-439a-a6a0-8bb7dda5f700-console-serving-cert\") pod \"console-985dfd8b7-sq25s\" (UID: \"1d5d208c-645c-439a-a6a0-8bb7dda5f700\") " pod="openshift-console/console-985dfd8b7-sq25s" Jul 23 03:57:05.131501 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:05.131477 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d5d208c-645c-439a-a6a0-8bb7dda5f700-console-oauth-config\") pod \"console-985dfd8b7-sq25s\" (UID: \"1d5d208c-645c-439a-a6a0-8bb7dda5f700\") " pod="openshift-console/console-985dfd8b7-sq25s" Jul 23 03:57:05.131697 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:05.131507 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d5d208c-645c-439a-a6a0-8bb7dda5f700-service-ca\") pod \"console-985dfd8b7-sq25s\" (UID: \"1d5d208c-645c-439a-a6a0-8bb7dda5f700\") " pod="openshift-console/console-985dfd8b7-sq25s" Jul 23 03:57:05.131697 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:05.131536 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d5d208c-645c-439a-a6a0-8bb7dda5f700-oauth-serving-cert\") pod \"console-985dfd8b7-sq25s\" (UID: \"1d5d208c-645c-439a-a6a0-8bb7dda5f700\") " pod="openshift-console/console-985dfd8b7-sq25s" Jul 23 03:57:05.131697 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:05.131553 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d5d208c-645c-439a-a6a0-8bb7dda5f700-console-config\") pod \"console-985dfd8b7-sq25s\" (UID: \"1d5d208c-645c-439a-a6a0-8bb7dda5f700\") " pod="openshift-console/console-985dfd8b7-sq25s" Jul 23 03:57:05.131697 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:05.131571 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wltx5\" (UniqueName: \"kubernetes.io/projected/1d5d208c-645c-439a-a6a0-8bb7dda5f700-kube-api-access-wltx5\") pod \"console-985dfd8b7-sq25s\" (UID: \"1d5d208c-645c-439a-a6a0-8bb7dda5f700\") " pod="openshift-console/console-985dfd8b7-sq25s" Jul 23 03:57:05.232306 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:05.232276 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d5d208c-645c-439a-a6a0-8bb7dda5f700-console-serving-cert\") pod \"console-985dfd8b7-sq25s\" (UID: \"1d5d208c-645c-439a-a6a0-8bb7dda5f700\") " pod="openshift-console/console-985dfd8b7-sq25s" Jul 23 03:57:05.232306 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:05.232314 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d5d208c-645c-439a-a6a0-8bb7dda5f700-console-oauth-config\") pod \"console-985dfd8b7-sq25s\" (UID: \"1d5d208c-645c-439a-a6a0-8bb7dda5f700\") " pod="openshift-console/console-985dfd8b7-sq25s" Jul 23 03:57:05.232590 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:05.232334 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d5d208c-645c-439a-a6a0-8bb7dda5f700-service-ca\") pod \"console-985dfd8b7-sq25s\" (UID: \"1d5d208c-645c-439a-a6a0-8bb7dda5f700\") " pod="openshift-console/console-985dfd8b7-sq25s" Jul 23 03:57:05.232590 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:05.232358 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d5d208c-645c-439a-a6a0-8bb7dda5f700-oauth-serving-cert\") pod \"console-985dfd8b7-sq25s\" (UID: \"1d5d208c-645c-439a-a6a0-8bb7dda5f700\") " pod="openshift-console/console-985dfd8b7-sq25s" Jul 23 03:57:05.232590 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:05.232375 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d5d208c-645c-439a-a6a0-8bb7dda5f700-console-config\") pod \"console-985dfd8b7-sq25s\" (UID: \"1d5d208c-645c-439a-a6a0-8bb7dda5f700\") " pod="openshift-console/console-985dfd8b7-sq25s" Jul 23 03:57:05.232590 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:05.232498 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wltx5\" (UniqueName: \"kubernetes.io/projected/1d5d208c-645c-439a-a6a0-8bb7dda5f700-kube-api-access-wltx5\") pod \"console-985dfd8b7-sq25s\" (UID: \"1d5d208c-645c-439a-a6a0-8bb7dda5f700\") " pod="openshift-console/console-985dfd8b7-sq25s" Jul 23 03:57:05.233141 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:05.233053 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d5d208c-645c-439a-a6a0-8bb7dda5f700-service-ca\") pod \"console-985dfd8b7-sq25s\" (UID: \"1d5d208c-645c-439a-a6a0-8bb7dda5f700\") " pod="openshift-console/console-985dfd8b7-sq25s" Jul 23 03:57:05.233141 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:05.233097 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d5d208c-645c-439a-a6a0-8bb7dda5f700-console-config\") pod \"console-985dfd8b7-sq25s\" (UID: \"1d5d208c-645c-439a-a6a0-8bb7dda5f700\") " pod="openshift-console/console-985dfd8b7-sq25s" Jul 23 03:57:05.233418 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:05.233310 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d5d208c-645c-439a-a6a0-8bb7dda5f700-oauth-serving-cert\") pod \"console-985dfd8b7-sq25s\" (UID: \"1d5d208c-645c-439a-a6a0-8bb7dda5f700\") " pod="openshift-console/console-985dfd8b7-sq25s" Jul 23 03:57:05.235003 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:05.234979 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d5d208c-645c-439a-a6a0-8bb7dda5f700-console-oauth-config\") pod \"console-985dfd8b7-sq25s\" (UID: \"1d5d208c-645c-439a-a6a0-8bb7dda5f700\") " pod="openshift-console/console-985dfd8b7-sq25s" Jul 23 03:57:05.235103 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:05.235042 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d5d208c-645c-439a-a6a0-8bb7dda5f700-console-serving-cert\") pod \"console-985dfd8b7-sq25s\" (UID: \"1d5d208c-645c-439a-a6a0-8bb7dda5f700\") " pod="openshift-console/console-985dfd8b7-sq25s" Jul 23 03:57:05.243604 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:05.243583 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wltx5\" (UniqueName: \"kubernetes.io/projected/1d5d208c-645c-439a-a6a0-8bb7dda5f700-kube-api-access-wltx5\") pod \"console-985dfd8b7-sq25s\" (UID: \"1d5d208c-645c-439a-a6a0-8bb7dda5f700\") " pod="openshift-console/console-985dfd8b7-sq25s" Jul 23 03:57:05.256457 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:05.256421 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-985dfd8b7-sq25s" Jul 23 03:57:05.420420 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:05.420392 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-985dfd8b7-sq25s"] Jul 23 03:57:05.422697 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:57:05.422670 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d5d208c_645c_439a_a6a0_8bb7dda5f700.slice/crio-a9b740bd5ba9e1c4f2bd0168cda9378b4e5adb5b7c39a2878cc8609875895535 WatchSource:0}: Error finding container a9b740bd5ba9e1c4f2bd0168cda9378b4e5adb5b7c39a2878cc8609875895535: Status 404 returned error can't find the container with id a9b740bd5ba9e1c4f2bd0168cda9378b4e5adb5b7c39a2878cc8609875895535 Jul 23 03:57:05.840003 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:05.839932 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-w99tk" Jul 23 03:57:05.955606 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:05.955566 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-985dfd8b7-sq25s" event={"ID":"1d5d208c-645c-439a-a6a0-8bb7dda5f700","Type":"ContainerStarted","Data":"a9b740bd5ba9e1c4f2bd0168cda9378b4e5adb5b7c39a2878cc8609875895535"} Jul 23 03:57:08.968799 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:08.968721 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-985dfd8b7-sq25s" event={"ID":"1d5d208c-645c-439a-a6a0-8bb7dda5f700","Type":"ContainerStarted","Data":"c0c88cd495da00f76bbf3d52c4b74821e50e4dd61426ff40218f0ec17ed93473"} Jul 23 03:57:08.989744 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:08.989695 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-985dfd8b7-sq25s" podStartSLOduration=1.57819249 podStartE2EDuration="4.989678336s" totalImagesPullingTime="3.411485846s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 03:57:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 03:57:08.987813773 +0000 UTC m=+108.893118066" watchObservedRunningTime="2026-07-23 03:57:08.989678336 +0000 UTC m=+108.894982627" Jul 23 03:57:15.257365 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:15.257327 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-985dfd8b7-sq25s" Jul 23 03:57:15.257921 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:15.257413 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-985dfd8b7-sq25s" Jul 23 03:57:15.263104 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:15.263080 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-985dfd8b7-sq25s" Jul 23 03:57:15.996092 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:15.996065 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-985dfd8b7-sq25s" Jul 23 03:57:23.778698 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:23.778666 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-985dfd8b7-sq25s"] Jul 23 03:57:26.026178 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:26.026142 2567 generic.go:356] "Generic (PLEG): container finished" podID="83d126c6-3769-45ac-8b1e-8e04b8f775b3" containerID="1f6f3fc43d84d3161be47546d70ebb60601aae871a28688d130cf948dd062b02" exitCode=0 Jul 23 03:57:26.026562 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:26.026216 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2nm67" event={"ID":"83d126c6-3769-45ac-8b1e-8e04b8f775b3","Type":"ContainerDied","Data":"1f6f3fc43d84d3161be47546d70ebb60601aae871a28688d130cf948dd062b02"} Jul 23 03:57:26.026562 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:26.026536 2567 scope.go:117] "RemoveContainer" containerID="1f6f3fc43d84d3161be47546d70ebb60601aae871a28688d130cf948dd062b02" Jul 23 03:57:27.031236 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:27.031200 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-2nm67" event={"ID":"83d126c6-3769-45ac-8b1e-8e04b8f775b3","Type":"ContainerStarted","Data":"797fc6dd8ff382f3058e036aa1b7e0bee8ee2bc778260e4bae658022dd3f9705"} Jul 23 03:57:48.801763 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:48.801728 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-985dfd8b7-sq25s" podUID="1d5d208c-645c-439a-a6a0-8bb7dda5f700" containerName="console" containerID="cri-o://c0c88cd495da00f76bbf3d52c4b74821e50e4dd61426ff40218f0ec17ed93473" gracePeriod=15 Jul 23 03:57:49.034365 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:49.034344 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-985dfd8b7-sq25s" Jul 23 03:57:49.107136 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:49.107073 2567 generic.go:356] "Generic (PLEG): container finished" podID="1d5d208c-645c-439a-a6a0-8bb7dda5f700" containerID="c0c88cd495da00f76bbf3d52c4b74821e50e4dd61426ff40218f0ec17ed93473" exitCode=0 Jul 23 03:57:49.107136 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:49.107117 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-985dfd8b7-sq25s" Jul 23 03:57:49.107279 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:49.107152 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-985dfd8b7-sq25s" event={"ID":"1d5d208c-645c-439a-a6a0-8bb7dda5f700","Type":"ContainerDied","Data":"c0c88cd495da00f76bbf3d52c4b74821e50e4dd61426ff40218f0ec17ed93473"} Jul 23 03:57:49.107279 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:49.107190 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-985dfd8b7-sq25s" event={"ID":"1d5d208c-645c-439a-a6a0-8bb7dda5f700","Type":"ContainerDied","Data":"a9b740bd5ba9e1c4f2bd0168cda9378b4e5adb5b7c39a2878cc8609875895535"} Jul 23 03:57:49.107279 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:49.107209 2567 scope.go:117] "RemoveContainer" containerID="c0c88cd495da00f76bbf3d52c4b74821e50e4dd61426ff40218f0ec17ed93473" Jul 23 03:57:49.115364 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:49.115338 2567 scope.go:117] "RemoveContainer" containerID="c0c88cd495da00f76bbf3d52c4b74821e50e4dd61426ff40218f0ec17ed93473" Jul 23 03:57:49.115758 ip-10-0-140-6 kubenswrapper[2567]: E0723 03:57:49.115716 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0c88cd495da00f76bbf3d52c4b74821e50e4dd61426ff40218f0ec17ed93473\": container with ID starting with c0c88cd495da00f76bbf3d52c4b74821e50e4dd61426ff40218f0ec17ed93473 not found: ID does not exist" containerID="c0c88cd495da00f76bbf3d52c4b74821e50e4dd61426ff40218f0ec17ed93473" Jul 23 03:57:49.115856 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:49.115773 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0c88cd495da00f76bbf3d52c4b74821e50e4dd61426ff40218f0ec17ed93473"} err="failed to get container status \"c0c88cd495da00f76bbf3d52c4b74821e50e4dd61426ff40218f0ec17ed93473\": rpc error: code = NotFound desc = could not find container \"c0c88cd495da00f76bbf3d52c4b74821e50e4dd61426ff40218f0ec17ed93473\": container with ID starting with c0c88cd495da00f76bbf3d52c4b74821e50e4dd61426ff40218f0ec17ed93473 not found: ID does not exist" Jul 23 03:57:49.146542 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:49.146523 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wltx5\" (UniqueName: \"kubernetes.io/projected/1d5d208c-645c-439a-a6a0-8bb7dda5f700-kube-api-access-wltx5\") pod \"1d5d208c-645c-439a-a6a0-8bb7dda5f700\" (UID: \"1d5d208c-645c-439a-a6a0-8bb7dda5f700\") " Jul 23 03:57:49.146651 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:49.146557 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d5d208c-645c-439a-a6a0-8bb7dda5f700-oauth-serving-cert\") pod \"1d5d208c-645c-439a-a6a0-8bb7dda5f700\" (UID: \"1d5d208c-645c-439a-a6a0-8bb7dda5f700\") " Jul 23 03:57:49.146651 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:49.146573 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d5d208c-645c-439a-a6a0-8bb7dda5f700-service-ca\") pod \"1d5d208c-645c-439a-a6a0-8bb7dda5f700\" (UID: \"1d5d208c-645c-439a-a6a0-8bb7dda5f700\") " Jul 23 03:57:49.146651 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:49.146598 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d5d208c-645c-439a-a6a0-8bb7dda5f700-console-config\") pod \"1d5d208c-645c-439a-a6a0-8bb7dda5f700\" (UID: \"1d5d208c-645c-439a-a6a0-8bb7dda5f700\") " Jul 23 03:57:49.146651 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:49.146628 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d5d208c-645c-439a-a6a0-8bb7dda5f700-console-serving-cert\") pod \"1d5d208c-645c-439a-a6a0-8bb7dda5f700\" (UID: \"1d5d208c-645c-439a-a6a0-8bb7dda5f700\") " Jul 23 03:57:49.146857 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:49.146738 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d5d208c-645c-439a-a6a0-8bb7dda5f700-console-oauth-config\") pod \"1d5d208c-645c-439a-a6a0-8bb7dda5f700\" (UID: \"1d5d208c-645c-439a-a6a0-8bb7dda5f700\") " Jul 23 03:57:49.146988 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:49.146950 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d5d208c-645c-439a-a6a0-8bb7dda5f700-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1d5d208c-645c-439a-a6a0-8bb7dda5f700" (UID: "1d5d208c-645c-439a-a6a0-8bb7dda5f700"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 03:57:49.146988 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:49.146957 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d5d208c-645c-439a-a6a0-8bb7dda5f700-service-ca" (OuterVolumeSpecName: "service-ca") pod "1d5d208c-645c-439a-a6a0-8bb7dda5f700" (UID: "1d5d208c-645c-439a-a6a0-8bb7dda5f700"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 03:57:49.147128 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:49.147037 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d5d208c-645c-439a-a6a0-8bb7dda5f700-console-config" (OuterVolumeSpecName: "console-config") pod "1d5d208c-645c-439a-a6a0-8bb7dda5f700" (UID: "1d5d208c-645c-439a-a6a0-8bb7dda5f700"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 03:57:49.148829 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:49.148805 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d5d208c-645c-439a-a6a0-8bb7dda5f700-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1d5d208c-645c-439a-a6a0-8bb7dda5f700" (UID: "1d5d208c-645c-439a-a6a0-8bb7dda5f700"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 03:57:49.148903 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:49.148884 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d5d208c-645c-439a-a6a0-8bb7dda5f700-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1d5d208c-645c-439a-a6a0-8bb7dda5f700" (UID: "1d5d208c-645c-439a-a6a0-8bb7dda5f700"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 03:57:49.148939 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:49.148895 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d5d208c-645c-439a-a6a0-8bb7dda5f700-kube-api-access-wltx5" (OuterVolumeSpecName: "kube-api-access-wltx5") pod "1d5d208c-645c-439a-a6a0-8bb7dda5f700" (UID: "1d5d208c-645c-439a-a6a0-8bb7dda5f700"). InnerVolumeSpecName "kube-api-access-wltx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 03:57:49.247622 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:49.247589 2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d5d208c-645c-439a-a6a0-8bb7dda5f700-console-oauth-config\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 03:57:49.247622 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:49.247616 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wltx5\" (UniqueName: \"kubernetes.io/projected/1d5d208c-645c-439a-a6a0-8bb7dda5f700-kube-api-access-wltx5\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 03:57:49.247622 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:49.247625 2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d5d208c-645c-439a-a6a0-8bb7dda5f700-oauth-serving-cert\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 03:57:49.247849 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:49.247635 2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d5d208c-645c-439a-a6a0-8bb7dda5f700-service-ca\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 03:57:49.247849 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:49.247644 2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d5d208c-645c-439a-a6a0-8bb7dda5f700-console-config\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 03:57:49.247849 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:49.247652 2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d5d208c-645c-439a-a6a0-8bb7dda5f700-console-serving-cert\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 03:57:49.429683 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:49.429652 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-985dfd8b7-sq25s"] Jul 23 03:57:49.433415 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:49.433389 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-985dfd8b7-sq25s"] Jul 23 03:57:50.491291 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:57:50.491263 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d5d208c-645c-439a-a6a0-8bb7dda5f700" path="/var/lib/kubelet/pods/1d5d208c-645c-439a-a6a0-8bb7dda5f700/volumes" Jul 23 03:58:50.656066 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:58:50.656037 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754058zx9"] Jul 23 03:58:50.656520 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:58:50.656259 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d5d208c-645c-439a-a6a0-8bb7dda5f700" containerName="console" Jul 23 03:58:50.656520 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:58:50.656269 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d5d208c-645c-439a-a6a0-8bb7dda5f700" containerName="console" Jul 23 03:58:50.656520 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:58:50.656315 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d5d208c-645c-439a-a6a0-8bb7dda5f700" containerName="console" Jul 23 03:58:50.659157 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:58:50.659140 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754058zx9" Jul 23 03:58:50.663501 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:58:50.663474 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 23 03:58:50.665697 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:58:50.665683 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 23 03:58:50.665761 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:58:50.665699 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-99zzx\"" Jul 23 03:58:50.671238 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:58:50.671218 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754058zx9"] Jul 23 03:58:50.759988 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:58:50.759956 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f9efa8b9-c3e8-45fc-b2e5-9a6ec6ad06ab-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754058zx9\" (UID: \"f9efa8b9-c3e8-45fc-b2e5-9a6ec6ad06ab\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754058zx9" Jul 23 03:58:50.759988 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:58:50.760000 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmns4\" (UniqueName: \"kubernetes.io/projected/f9efa8b9-c3e8-45fc-b2e5-9a6ec6ad06ab-kube-api-access-qmns4\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754058zx9\" (UID: \"f9efa8b9-c3e8-45fc-b2e5-9a6ec6ad06ab\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754058zx9" Jul 23 03:58:50.760213 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:58:50.760024 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f9efa8b9-c3e8-45fc-b2e5-9a6ec6ad06ab-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754058zx9\" (UID: \"f9efa8b9-c3e8-45fc-b2e5-9a6ec6ad06ab\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754058zx9" Jul 23 03:58:50.861143 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:58:50.861111 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f9efa8b9-c3e8-45fc-b2e5-9a6ec6ad06ab-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754058zx9\" (UID: \"f9efa8b9-c3e8-45fc-b2e5-9a6ec6ad06ab\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754058zx9" Jul 23 03:58:50.861327 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:58:50.861153 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmns4\" (UniqueName: \"kubernetes.io/projected/f9efa8b9-c3e8-45fc-b2e5-9a6ec6ad06ab-kube-api-access-qmns4\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754058zx9\" (UID: \"f9efa8b9-c3e8-45fc-b2e5-9a6ec6ad06ab\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754058zx9" Jul 23 03:58:50.861327 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:58:50.861187 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f9efa8b9-c3e8-45fc-b2e5-9a6ec6ad06ab-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754058zx9\" (UID: \"f9efa8b9-c3e8-45fc-b2e5-9a6ec6ad06ab\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754058zx9" Jul 23 03:58:50.861597 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:58:50.861576 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f9efa8b9-c3e8-45fc-b2e5-9a6ec6ad06ab-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754058zx9\" (UID: \"f9efa8b9-c3e8-45fc-b2e5-9a6ec6ad06ab\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754058zx9" Jul 23 03:58:50.861648 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:58:50.861593 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f9efa8b9-c3e8-45fc-b2e5-9a6ec6ad06ab-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754058zx9\" (UID: \"f9efa8b9-c3e8-45fc-b2e5-9a6ec6ad06ab\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754058zx9" Jul 23 03:58:50.873205 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:58:50.873181 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmns4\" (UniqueName: \"kubernetes.io/projected/f9efa8b9-c3e8-45fc-b2e5-9a6ec6ad06ab-kube-api-access-qmns4\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754058zx9\" (UID: \"f9efa8b9-c3e8-45fc-b2e5-9a6ec6ad06ab\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754058zx9" Jul 23 03:58:50.967766 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:58:50.967734 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754058zx9" Jul 23 03:58:51.106154 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:58:51.106005 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754058zx9"] Jul 23 03:58:51.108196 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:58:51.108170 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9efa8b9_c3e8_45fc_b2e5_9a6ec6ad06ab.slice/crio-45bd2d07a57a35ba5d59ed0711fcf81d37e022f781a52a8e450fd4b20a5e9ca2 WatchSource:0}: Error finding container 45bd2d07a57a35ba5d59ed0711fcf81d37e022f781a52a8e450fd4b20a5e9ca2: Status 404 returned error can't find the container with id 45bd2d07a57a35ba5d59ed0711fcf81d37e022f781a52a8e450fd4b20a5e9ca2 Jul 23 03:58:51.313249 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:58:51.313167 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754058zx9" event={"ID":"f9efa8b9-c3e8-45fc-b2e5-9a6ec6ad06ab","Type":"ContainerStarted","Data":"45bd2d07a57a35ba5d59ed0711fcf81d37e022f781a52a8e450fd4b20a5e9ca2"} Jul 23 03:58:58.346008 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:58:58.345970 2567 generic.go:356] "Generic (PLEG): container finished" podID="f9efa8b9-c3e8-45fc-b2e5-9a6ec6ad06ab" containerID="4426ed1d5e13f5eee8d9fce94f1e14b1dcac32372a2ec99336f4c48a0ffe11d8" exitCode=0 Jul 23 03:58:58.346375 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:58:58.346024 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754058zx9" event={"ID":"f9efa8b9-c3e8-45fc-b2e5-9a6ec6ad06ab","Type":"ContainerDied","Data":"4426ed1d5e13f5eee8d9fce94f1e14b1dcac32372a2ec99336f4c48a0ffe11d8"} Jul 23 03:59:01.357998 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:01.357957 2567 generic.go:356] "Generic (PLEG): container finished" podID="f9efa8b9-c3e8-45fc-b2e5-9a6ec6ad06ab" containerID="08cbd48f7556826b6f0b09822cda7a8f73c9fc4a87b757eda4a164ef9bc45191" exitCode=0 Jul 23 03:59:01.358395 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:01.358039 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754058zx9" event={"ID":"f9efa8b9-c3e8-45fc-b2e5-9a6ec6ad06ab","Type":"ContainerDied","Data":"08cbd48f7556826b6f0b09822cda7a8f73c9fc4a87b757eda4a164ef9bc45191"} Jul 23 03:59:01.486965 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:01.486938 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-655585db4-cxbcp"] Jul 23 03:59:01.489969 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:01.489955 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-655585db4-cxbcp" Jul 23 03:59:01.497931 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:01.497913 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 23 03:59:01.498010 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:01.497959 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 23 03:59:01.498653 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:01.498639 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 23 03:59:01.518480 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:01.518459 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-655585db4-cxbcp"] Jul 23 03:59:01.526462 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:01.526411 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 23 03:59:01.541144 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:01.541125 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7qll\" (UniqueName: \"kubernetes.io/projected/cdac988b-5c5d-438f-a7ff-c99bbedad668-kube-api-access-f7qll\") pod \"klusterlet-addon-workmgr-655585db4-cxbcp\" (UID: \"cdac988b-5c5d-438f-a7ff-c99bbedad668\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-655585db4-cxbcp" Jul 23 03:59:01.541229 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:01.541157 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cdac988b-5c5d-438f-a7ff-c99bbedad668-tmp\") pod \"klusterlet-addon-workmgr-655585db4-cxbcp\" (UID: \"cdac988b-5c5d-438f-a7ff-c99bbedad668\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-655585db4-cxbcp" Jul 23 03:59:01.541229 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:01.541178 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/cdac988b-5c5d-438f-a7ff-c99bbedad668-klusterlet-config\") pod \"klusterlet-addon-workmgr-655585db4-cxbcp\" (UID: \"cdac988b-5c5d-438f-a7ff-c99bbedad668\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-655585db4-cxbcp" Jul 23 03:59:01.642092 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:01.642027 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7qll\" (UniqueName: \"kubernetes.io/projected/cdac988b-5c5d-438f-a7ff-c99bbedad668-kube-api-access-f7qll\") pod \"klusterlet-addon-workmgr-655585db4-cxbcp\" (UID: \"cdac988b-5c5d-438f-a7ff-c99bbedad668\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-655585db4-cxbcp" Jul 23 03:59:01.642092 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:01.642076 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cdac988b-5c5d-438f-a7ff-c99bbedad668-tmp\") pod \"klusterlet-addon-workmgr-655585db4-cxbcp\" (UID: \"cdac988b-5c5d-438f-a7ff-c99bbedad668\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-655585db4-cxbcp" Jul 23 03:59:01.642271 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:01.642110 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/cdac988b-5c5d-438f-a7ff-c99bbedad668-klusterlet-config\") pod \"klusterlet-addon-workmgr-655585db4-cxbcp\" (UID: \"cdac988b-5c5d-438f-a7ff-c99bbedad668\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-655585db4-cxbcp" Jul 23 03:59:01.642508 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:01.642490 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cdac988b-5c5d-438f-a7ff-c99bbedad668-tmp\") pod \"klusterlet-addon-workmgr-655585db4-cxbcp\" (UID: \"cdac988b-5c5d-438f-a7ff-c99bbedad668\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-655585db4-cxbcp" Jul 23 03:59:01.644771 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:01.644749 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/cdac988b-5c5d-438f-a7ff-c99bbedad668-klusterlet-config\") pod \"klusterlet-addon-workmgr-655585db4-cxbcp\" (UID: \"cdac988b-5c5d-438f-a7ff-c99bbedad668\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-655585db4-cxbcp" Jul 23 03:59:01.651883 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:01.651865 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7qll\" (UniqueName: \"kubernetes.io/projected/cdac988b-5c5d-438f-a7ff-c99bbedad668-kube-api-access-f7qll\") pod \"klusterlet-addon-workmgr-655585db4-cxbcp\" (UID: \"cdac988b-5c5d-438f-a7ff-c99bbedad668\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-655585db4-cxbcp" Jul 23 03:59:01.799049 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:01.799017 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-655585db4-cxbcp" Jul 23 03:59:01.934907 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:01.934862 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-655585db4-cxbcp"] Jul 23 03:59:01.936906 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:59:01.936867 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdac988b_5c5d_438f_a7ff_c99bbedad668.slice/crio-c734f72dc981982228c3d54d2f80dee1e9aa08549968907f662380148e049fb6 WatchSource:0}: Error finding container c734f72dc981982228c3d54d2f80dee1e9aa08549968907f662380148e049fb6: Status 404 returned error can't find the container with id c734f72dc981982228c3d54d2f80dee1e9aa08549968907f662380148e049fb6 Jul 23 03:59:02.363678 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:02.363592 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-655585db4-cxbcp" event={"ID":"cdac988b-5c5d-438f-a7ff-c99bbedad668","Type":"ContainerStarted","Data":"c734f72dc981982228c3d54d2f80dee1e9aa08549968907f662380148e049fb6"} Jul 23 03:59:06.382176 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:06.382142 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-655585db4-cxbcp" event={"ID":"cdac988b-5c5d-438f-a7ff-c99bbedad668","Type":"ContainerStarted","Data":"d2794ff2ffe24621f47eebbd32fb99192a9dc2b33b4c8cb5d19172c7af8100e4"} Jul 23 03:59:06.382745 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:06.382692 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-655585db4-cxbcp" Jul 23 03:59:06.384477 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:06.384450 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-655585db4-cxbcp" Jul 23 03:59:06.406707 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:06.406655 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-655585db4-cxbcp" podStartSLOduration=1.569009777 podStartE2EDuration="5.40663748s" totalImagesPullingTime="3.837627703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 03:59:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 03:59:06.404892248 +0000 UTC m=+226.310196540" watchObservedRunningTime="2026-07-23 03:59:06.40663748 +0000 UTC m=+226.311941802" Jul 23 03:59:09.395772 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:09.395732 2567 generic.go:356] "Generic (PLEG): container finished" podID="f9efa8b9-c3e8-45fc-b2e5-9a6ec6ad06ab" containerID="071ffd3a38d0af88a8dce36f70160f7b2b7b0eac921cafaef97f664f5f7d1974" exitCode=0 Jul 23 03:59:09.396136 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:09.395820 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754058zx9" event={"ID":"f9efa8b9-c3e8-45fc-b2e5-9a6ec6ad06ab","Type":"ContainerDied","Data":"071ffd3a38d0af88a8dce36f70160f7b2b7b0eac921cafaef97f664f5f7d1974"} Jul 23 03:59:10.521504 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:10.521483 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754058zx9" Jul 23 03:59:10.610349 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:10.610311 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmns4\" (UniqueName: \"kubernetes.io/projected/f9efa8b9-c3e8-45fc-b2e5-9a6ec6ad06ab-kube-api-access-qmns4\") pod \"f9efa8b9-c3e8-45fc-b2e5-9a6ec6ad06ab\" (UID: \"f9efa8b9-c3e8-45fc-b2e5-9a6ec6ad06ab\") " Jul 23 03:59:10.610541 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:10.610366 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f9efa8b9-c3e8-45fc-b2e5-9a6ec6ad06ab-util\") pod \"f9efa8b9-c3e8-45fc-b2e5-9a6ec6ad06ab\" (UID: \"f9efa8b9-c3e8-45fc-b2e5-9a6ec6ad06ab\") " Jul 23 03:59:10.610541 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:10.610398 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f9efa8b9-c3e8-45fc-b2e5-9a6ec6ad06ab-bundle\") pod \"f9efa8b9-c3e8-45fc-b2e5-9a6ec6ad06ab\" (UID: \"f9efa8b9-c3e8-45fc-b2e5-9a6ec6ad06ab\") " Jul 23 03:59:10.611198 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:10.611171 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9efa8b9-c3e8-45fc-b2e5-9a6ec6ad06ab-bundle" (OuterVolumeSpecName: "bundle") pod "f9efa8b9-c3e8-45fc-b2e5-9a6ec6ad06ab" (UID: "f9efa8b9-c3e8-45fc-b2e5-9a6ec6ad06ab"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 03:59:10.612605 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:10.612584 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9efa8b9-c3e8-45fc-b2e5-9a6ec6ad06ab-kube-api-access-qmns4" (OuterVolumeSpecName: "kube-api-access-qmns4") pod "f9efa8b9-c3e8-45fc-b2e5-9a6ec6ad06ab" (UID: "f9efa8b9-c3e8-45fc-b2e5-9a6ec6ad06ab"). InnerVolumeSpecName "kube-api-access-qmns4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 03:59:10.614942 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:10.614923 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9efa8b9-c3e8-45fc-b2e5-9a6ec6ad06ab-util" (OuterVolumeSpecName: "util") pod "f9efa8b9-c3e8-45fc-b2e5-9a6ec6ad06ab" (UID: "f9efa8b9-c3e8-45fc-b2e5-9a6ec6ad06ab"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 03:59:10.710858 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:10.710829 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmns4\" (UniqueName: \"kubernetes.io/projected/f9efa8b9-c3e8-45fc-b2e5-9a6ec6ad06ab-kube-api-access-qmns4\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 03:59:10.710858 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:10.710854 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f9efa8b9-c3e8-45fc-b2e5-9a6ec6ad06ab-util\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 03:59:10.710858 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:10.710863 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f9efa8b9-c3e8-45fc-b2e5-9a6ec6ad06ab-bundle\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 03:59:11.405770 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:11.405737 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754058zx9" Jul 23 03:59:11.405930 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:11.405741 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754058zx9" event={"ID":"f9efa8b9-c3e8-45fc-b2e5-9a6ec6ad06ab","Type":"ContainerDied","Data":"45bd2d07a57a35ba5d59ed0711fcf81d37e022f781a52a8e450fd4b20a5e9ca2"} Jul 23 03:59:11.405930 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:11.405855 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45bd2d07a57a35ba5d59ed0711fcf81d37e022f781a52a8e450fd4b20a5e9ca2" Jul 23 03:59:19.717875 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:19.717839 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ngww8"] Jul 23 03:59:19.718269 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:19.718087 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9efa8b9-c3e8-45fc-b2e5-9a6ec6ad06ab" containerName="pull" Jul 23 03:59:19.718269 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:19.718097 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9efa8b9-c3e8-45fc-b2e5-9a6ec6ad06ab" containerName="pull" Jul 23 03:59:19.718269 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:19.718106 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9efa8b9-c3e8-45fc-b2e5-9a6ec6ad06ab" containerName="extract" Jul 23 03:59:19.718269 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:19.718111 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9efa8b9-c3e8-45fc-b2e5-9a6ec6ad06ab" containerName="extract" Jul 23 03:59:19.718269 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:19.718136 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9efa8b9-c3e8-45fc-b2e5-9a6ec6ad06ab" containerName="util" Jul 23 03:59:19.718269 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:19.718142 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9efa8b9-c3e8-45fc-b2e5-9a6ec6ad06ab" containerName="util" Jul 23 03:59:19.718269 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:19.718181 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9efa8b9-c3e8-45fc-b2e5-9a6ec6ad06ab" containerName="extract" Jul 23 03:59:19.724714 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:19.724692 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ngww8" Jul 23 03:59:19.727680 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:19.727655 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 23 03:59:19.728140 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:19.728122 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-5qvg9\"" Jul 23 03:59:19.728256 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:19.728237 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 23 03:59:19.739884 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:19.739862 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ngww8"] Jul 23 03:59:19.778462 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:19.778415 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4vnz\" (UniqueName: \"kubernetes.io/projected/5e2c865d-f60f-431d-b438-f9bb5a6c3329-kube-api-access-t4vnz\") pod \"cert-manager-operator-controller-manager-55c444bf86-ngww8\" (UID: \"5e2c865d-f60f-431d-b438-f9bb5a6c3329\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ngww8" Jul 23 03:59:19.778462 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:19.778471 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e2c865d-f60f-431d-b438-f9bb5a6c3329-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-ngww8\" (UID: \"5e2c865d-f60f-431d-b438-f9bb5a6c3329\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ngww8" Jul 23 03:59:19.879358 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:19.879324 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4vnz\" (UniqueName: \"kubernetes.io/projected/5e2c865d-f60f-431d-b438-f9bb5a6c3329-kube-api-access-t4vnz\") pod \"cert-manager-operator-controller-manager-55c444bf86-ngww8\" (UID: \"5e2c865d-f60f-431d-b438-f9bb5a6c3329\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ngww8" Jul 23 03:59:19.879358 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:19.879360 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e2c865d-f60f-431d-b438-f9bb5a6c3329-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-ngww8\" (UID: \"5e2c865d-f60f-431d-b438-f9bb5a6c3329\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ngww8" Jul 23 03:59:19.879710 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:19.879694 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e2c865d-f60f-431d-b438-f9bb5a6c3329-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-ngww8\" (UID: \"5e2c865d-f60f-431d-b438-f9bb5a6c3329\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ngww8" Jul 23 03:59:19.888888 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:19.888864 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4vnz\" (UniqueName: \"kubernetes.io/projected/5e2c865d-f60f-431d-b438-f9bb5a6c3329-kube-api-access-t4vnz\") pod \"cert-manager-operator-controller-manager-55c444bf86-ngww8\" (UID: \"5e2c865d-f60f-431d-b438-f9bb5a6c3329\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ngww8" Jul 23 03:59:20.033662 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:20.033582 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ngww8" Jul 23 03:59:20.173213 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:20.173191 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ngww8"] Jul 23 03:59:20.175383 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:59:20.175353 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e2c865d_f60f_431d_b438_f9bb5a6c3329.slice/crio-60e73a10cf9b4b669125a98369771f287a54125ebafa2e959d7470000f35117b WatchSource:0}: Error finding container 60e73a10cf9b4b669125a98369771f287a54125ebafa2e959d7470000f35117b: Status 404 returned error can't find the container with id 60e73a10cf9b4b669125a98369771f287a54125ebafa2e959d7470000f35117b Jul 23 03:59:20.437760 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:20.437727 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ngww8" event={"ID":"5e2c865d-f60f-431d-b438-f9bb5a6c3329","Type":"ContainerStarted","Data":"60e73a10cf9b4b669125a98369771f287a54125ebafa2e959d7470000f35117b"} Jul 23 03:59:22.448334 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:22.448295 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ngww8" event={"ID":"5e2c865d-f60f-431d-b438-f9bb5a6c3329","Type":"ContainerStarted","Data":"10dd348f5b34f70389addc751bac3ebbda9c95568c5516ea4352c9442fac8dc9"} Jul 23 03:59:22.472864 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:22.472821 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ngww8" podStartSLOduration=1.615230195 podStartE2EDuration="3.47280531s" totalImagesPullingTime="1.857575115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 03:59:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 03:59:22.470371311 +0000 UTC m=+242.375675603" watchObservedRunningTime="2026-07-23 03:59:22.47280531 +0000 UTC m=+242.378109598" Jul 23 03:59:23.349871 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:23.349838 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fns4mq"] Jul 23 03:59:23.353155 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:23.353134 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fns4mq" Jul 23 03:59:23.355775 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:23.355756 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-99zzx\"" Jul 23 03:59:23.356061 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:23.356023 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 23 03:59:23.356211 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:23.356188 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 23 03:59:23.377789 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:23.377765 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fns4mq"] Jul 23 03:59:23.408694 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:23.408658 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5957d4ab-71a8-4a84-8bd4-05639f5c661f-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fns4mq\" (UID: \"5957d4ab-71a8-4a84-8bd4-05639f5c661f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fns4mq" Jul 23 03:59:23.408857 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:23.408765 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h27zm\" (UniqueName: \"kubernetes.io/projected/5957d4ab-71a8-4a84-8bd4-05639f5c661f-kube-api-access-h27zm\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fns4mq\" (UID: \"5957d4ab-71a8-4a84-8bd4-05639f5c661f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fns4mq" Jul 23 03:59:23.408857 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:23.408838 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5957d4ab-71a8-4a84-8bd4-05639f5c661f-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fns4mq\" (UID: \"5957d4ab-71a8-4a84-8bd4-05639f5c661f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fns4mq" Jul 23 03:59:23.509843 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:23.509808 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5957d4ab-71a8-4a84-8bd4-05639f5c661f-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fns4mq\" (UID: \"5957d4ab-71a8-4a84-8bd4-05639f5c661f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fns4mq" Jul 23 03:59:23.510221 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:23.509858 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5957d4ab-71a8-4a84-8bd4-05639f5c661f-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fns4mq\" (UID: \"5957d4ab-71a8-4a84-8bd4-05639f5c661f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fns4mq" Jul 23 03:59:23.510221 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:23.509906 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h27zm\" (UniqueName: \"kubernetes.io/projected/5957d4ab-71a8-4a84-8bd4-05639f5c661f-kube-api-access-h27zm\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fns4mq\" (UID: \"5957d4ab-71a8-4a84-8bd4-05639f5c661f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fns4mq" Jul 23 03:59:23.510221 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:23.510199 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5957d4ab-71a8-4a84-8bd4-05639f5c661f-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fns4mq\" (UID: \"5957d4ab-71a8-4a84-8bd4-05639f5c661f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fns4mq" Jul 23 03:59:23.510324 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:23.510250 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5957d4ab-71a8-4a84-8bd4-05639f5c661f-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fns4mq\" (UID: \"5957d4ab-71a8-4a84-8bd4-05639f5c661f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fns4mq" Jul 23 03:59:23.534085 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:23.534053 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h27zm\" (UniqueName: \"kubernetes.io/projected/5957d4ab-71a8-4a84-8bd4-05639f5c661f-kube-api-access-h27zm\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fns4mq\" (UID: \"5957d4ab-71a8-4a84-8bd4-05639f5c661f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fns4mq" Jul 23 03:59:23.662630 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:23.662542 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fns4mq" Jul 23 03:59:23.799463 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:23.799410 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fns4mq"] Jul 23 03:59:23.800368 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:59:23.800341 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5957d4ab_71a8_4a84_8bd4_05639f5c661f.slice/crio-bc8c36823a5550c2b12a6f42efd742ae6e8d11ac1b33d549620c77949fbb0dc4 WatchSource:0}: Error finding container bc8c36823a5550c2b12a6f42efd742ae6e8d11ac1b33d549620c77949fbb0dc4: Status 404 returned error can't find the container with id bc8c36823a5550c2b12a6f42efd742ae6e8d11ac1b33d549620c77949fbb0dc4 Jul 23 03:59:24.458186 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:24.458153 2567 generic.go:356] "Generic (PLEG): container finished" podID="5957d4ab-71a8-4a84-8bd4-05639f5c661f" containerID="988b5abebd61224850c97b28d57071f8a88188944e36216a6a4ae359d1925efa" exitCode=0 Jul 23 03:59:24.458354 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:24.458215 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fns4mq" event={"ID":"5957d4ab-71a8-4a84-8bd4-05639f5c661f","Type":"ContainerDied","Data":"988b5abebd61224850c97b28d57071f8a88188944e36216a6a4ae359d1925efa"} Jul 23 03:59:24.458354 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:24.458253 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fns4mq" event={"ID":"5957d4ab-71a8-4a84-8bd4-05639f5c661f","Type":"ContainerStarted","Data":"bc8c36823a5550c2b12a6f42efd742ae6e8d11ac1b33d549620c77949fbb0dc4"} Jul 23 03:59:26.468686 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:26.468607 2567 generic.go:356] "Generic (PLEG): container finished" podID="5957d4ab-71a8-4a84-8bd4-05639f5c661f" containerID="7d5fa8ff219686cb1268598dbc622a25f9bc8baa2c5001f036b6dc5fb3820a00" exitCode=0 Jul 23 03:59:26.468686 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:26.468669 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fns4mq" event={"ID":"5957d4ab-71a8-4a84-8bd4-05639f5c661f","Type":"ContainerDied","Data":"7d5fa8ff219686cb1268598dbc622a25f9bc8baa2c5001f036b6dc5fb3820a00"} Jul 23 03:59:27.474568 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:27.474534 2567 generic.go:356] "Generic (PLEG): container finished" podID="5957d4ab-71a8-4a84-8bd4-05639f5c661f" containerID="6c2e7f1f4cfe7137959f5fad0de43c435836aefeec7f01756291e62a7f65ae8b" exitCode=0 Jul 23 03:59:27.474971 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:27.474608 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fns4mq" event={"ID":"5957d4ab-71a8-4a84-8bd4-05639f5c661f","Type":"ContainerDied","Data":"6c2e7f1f4cfe7137959f5fad0de43c435836aefeec7f01756291e62a7f65ae8b"} Jul 23 03:59:28.600183 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:28.600159 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fns4mq" Jul 23 03:59:28.651076 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:28.651047 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5957d4ab-71a8-4a84-8bd4-05639f5c661f-bundle\") pod \"5957d4ab-71a8-4a84-8bd4-05639f5c661f\" (UID: \"5957d4ab-71a8-4a84-8bd4-05639f5c661f\") " Jul 23 03:59:28.651213 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:28.651104 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5957d4ab-71a8-4a84-8bd4-05639f5c661f-util\") pod \"5957d4ab-71a8-4a84-8bd4-05639f5c661f\" (UID: \"5957d4ab-71a8-4a84-8bd4-05639f5c661f\") " Jul 23 03:59:28.651213 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:28.651165 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h27zm\" (UniqueName: \"kubernetes.io/projected/5957d4ab-71a8-4a84-8bd4-05639f5c661f-kube-api-access-h27zm\") pod \"5957d4ab-71a8-4a84-8bd4-05639f5c661f\" (UID: \"5957d4ab-71a8-4a84-8bd4-05639f5c661f\") " Jul 23 03:59:28.651465 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:28.651411 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5957d4ab-71a8-4a84-8bd4-05639f5c661f-bundle" (OuterVolumeSpecName: "bundle") pod "5957d4ab-71a8-4a84-8bd4-05639f5c661f" (UID: "5957d4ab-71a8-4a84-8bd4-05639f5c661f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 03:59:28.653191 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:28.653169 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5957d4ab-71a8-4a84-8bd4-05639f5c661f-kube-api-access-h27zm" (OuterVolumeSpecName: "kube-api-access-h27zm") pod "5957d4ab-71a8-4a84-8bd4-05639f5c661f" (UID: "5957d4ab-71a8-4a84-8bd4-05639f5c661f"). InnerVolumeSpecName "kube-api-access-h27zm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 03:59:28.657119 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:28.657098 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5957d4ab-71a8-4a84-8bd4-05639f5c661f-util" (OuterVolumeSpecName: "util") pod "5957d4ab-71a8-4a84-8bd4-05639f5c661f" (UID: "5957d4ab-71a8-4a84-8bd4-05639f5c661f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 03:59:28.751631 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:28.751569 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h27zm\" (UniqueName: \"kubernetes.io/projected/5957d4ab-71a8-4a84-8bd4-05639f5c661f-kube-api-access-h27zm\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 03:59:28.751631 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:28.751591 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5957d4ab-71a8-4a84-8bd4-05639f5c661f-bundle\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 03:59:28.751631 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:28.751600 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5957d4ab-71a8-4a84-8bd4-05639f5c661f-util\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 03:59:29.483613 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:29.483584 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fns4mq" Jul 23 03:59:29.483778 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:29.483585 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fns4mq" event={"ID":"5957d4ab-71a8-4a84-8bd4-05639f5c661f","Type":"ContainerDied","Data":"bc8c36823a5550c2b12a6f42efd742ae6e8d11ac1b33d549620c77949fbb0dc4"} Jul 23 03:59:29.483778 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:29.483703 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc8c36823a5550c2b12a6f42efd742ae6e8d11ac1b33d549620c77949fbb0dc4" Jul 23 03:59:29.846789 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:29.846715 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-xbfvw"] Jul 23 03:59:29.847133 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:29.846958 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5957d4ab-71a8-4a84-8bd4-05639f5c661f" containerName="util" Jul 23 03:59:29.847133 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:29.846968 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="5957d4ab-71a8-4a84-8bd4-05639f5c661f" containerName="util" Jul 23 03:59:29.847133 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:29.846981 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5957d4ab-71a8-4a84-8bd4-05639f5c661f" containerName="extract" Jul 23 03:59:29.847133 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:29.846987 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="5957d4ab-71a8-4a84-8bd4-05639f5c661f" containerName="extract" Jul 23 03:59:29.847133 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:29.846995 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5957d4ab-71a8-4a84-8bd4-05639f5c661f" containerName="pull" Jul 23 03:59:29.847133 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:29.847000 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="5957d4ab-71a8-4a84-8bd4-05639f5c661f" containerName="pull" Jul 23 03:59:29.847133 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:29.847055 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="5957d4ab-71a8-4a84-8bd4-05639f5c661f" containerName="extract" Jul 23 03:59:29.849902 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:29.849884 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-xbfvw" Jul 23 03:59:29.853754 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:29.853731 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 23 03:59:29.853874 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:29.853767 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-2xpxx\"" Jul 23 03:59:29.853874 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:29.853826 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 23 03:59:29.862047 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:29.862025 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-xbfvw"] Jul 23 03:59:29.959463 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:29.959408 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgcx7\" (UniqueName: \"kubernetes.io/projected/d4a0f7b2-9cef-4575-8dad-966a1aeb5002-kube-api-access-rgcx7\") pod \"cert-manager-cainjector-559cdd8c87-xbfvw\" (UID: \"d4a0f7b2-9cef-4575-8dad-966a1aeb5002\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xbfvw" Jul 23 03:59:29.959638 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:29.959501 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d4a0f7b2-9cef-4575-8dad-966a1aeb5002-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-xbfvw\" (UID: \"d4a0f7b2-9cef-4575-8dad-966a1aeb5002\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xbfvw" Jul 23 03:59:30.059952 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:30.059919 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgcx7\" (UniqueName: \"kubernetes.io/projected/d4a0f7b2-9cef-4575-8dad-966a1aeb5002-kube-api-access-rgcx7\") pod \"cert-manager-cainjector-559cdd8c87-xbfvw\" (UID: \"d4a0f7b2-9cef-4575-8dad-966a1aeb5002\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xbfvw" Jul 23 03:59:30.060125 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:30.059973 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d4a0f7b2-9cef-4575-8dad-966a1aeb5002-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-xbfvw\" (UID: \"d4a0f7b2-9cef-4575-8dad-966a1aeb5002\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xbfvw" Jul 23 03:59:30.078146 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:30.078112 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d4a0f7b2-9cef-4575-8dad-966a1aeb5002-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-xbfvw\" (UID: \"d4a0f7b2-9cef-4575-8dad-966a1aeb5002\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xbfvw" Jul 23 03:59:30.078542 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:30.078525 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgcx7\" (UniqueName: \"kubernetes.io/projected/d4a0f7b2-9cef-4575-8dad-966a1aeb5002-kube-api-access-rgcx7\") pod \"cert-manager-cainjector-559cdd8c87-xbfvw\" (UID: \"d4a0f7b2-9cef-4575-8dad-966a1aeb5002\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xbfvw" Jul 23 03:59:30.175116 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:30.175091 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-xbfvw" Jul 23 03:59:30.303816 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:30.303789 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-xbfvw"] Jul 23 03:59:30.307766 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:59:30.307734 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4a0f7b2_9cef_4575_8dad_966a1aeb5002.slice/crio-f0a5f25561bfa528d7c44703989ecf026301aa83edad1f3b5b36ca8d75e5658b WatchSource:0}: Error finding container f0a5f25561bfa528d7c44703989ecf026301aa83edad1f3b5b36ca8d75e5658b: Status 404 returned error can't find the container with id f0a5f25561bfa528d7c44703989ecf026301aa83edad1f3b5b36ca8d75e5658b Jul 23 03:59:30.491001 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:30.490915 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-xbfvw" event={"ID":"d4a0f7b2-9cef-4575-8dad-966a1aeb5002","Type":"ContainerStarted","Data":"f0a5f25561bfa528d7c44703989ecf026301aa83edad1f3b5b36ca8d75e5658b"} Jul 23 03:59:33.504236 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:33.504201 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-xbfvw" event={"ID":"d4a0f7b2-9cef-4575-8dad-966a1aeb5002","Type":"ContainerStarted","Data":"9961d95184534cbe19b41f33b9d939bdb496bc167243d0b3aff2ec13e8b65b9a"} Jul 23 03:59:33.525555 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:33.525506 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-xbfvw" podStartSLOduration=1.880019185 podStartE2EDuration="4.525488847s" totalImagesPullingTime="2.645469662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 03:59:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 03:59:33.524889432 +0000 UTC m=+253.430193723" watchObservedRunningTime="2026-07-23 03:59:33.525488847 +0000 UTC m=+253.430793139" Jul 23 03:59:39.387540 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:39.387504 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ljbh7"] Jul 23 03:59:39.405636 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:39.405604 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ljbh7"] Jul 23 03:59:39.405787 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:39.405698 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ljbh7" Jul 23 03:59:39.412255 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:39.412220 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-22bpg\"" Jul 23 03:59:39.413147 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:39.413129 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 23 03:59:39.416276 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:39.416254 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 23 03:59:39.537967 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:39.537938 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e7e16755-e2e6-42b6-9ed1-e63706e73cd3-tmp\") pod \"openshift-lws-operator-fd8ccff4c-ljbh7\" (UID: \"e7e16755-e2e6-42b6-9ed1-e63706e73cd3\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ljbh7" Jul 23 03:59:39.538142 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:39.538010 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnjsm\" (UniqueName: \"kubernetes.io/projected/e7e16755-e2e6-42b6-9ed1-e63706e73cd3-kube-api-access-pnjsm\") pod \"openshift-lws-operator-fd8ccff4c-ljbh7\" (UID: \"e7e16755-e2e6-42b6-9ed1-e63706e73cd3\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ljbh7" Jul 23 03:59:39.638445 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:39.638349 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnjsm\" (UniqueName: \"kubernetes.io/projected/e7e16755-e2e6-42b6-9ed1-e63706e73cd3-kube-api-access-pnjsm\") pod \"openshift-lws-operator-fd8ccff4c-ljbh7\" (UID: \"e7e16755-e2e6-42b6-9ed1-e63706e73cd3\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ljbh7" Jul 23 03:59:39.638445 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:39.638390 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e7e16755-e2e6-42b6-9ed1-e63706e73cd3-tmp\") pod \"openshift-lws-operator-fd8ccff4c-ljbh7\" (UID: \"e7e16755-e2e6-42b6-9ed1-e63706e73cd3\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ljbh7" Jul 23 03:59:39.638826 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:39.638807 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e7e16755-e2e6-42b6-9ed1-e63706e73cd3-tmp\") pod \"openshift-lws-operator-fd8ccff4c-ljbh7\" (UID: \"e7e16755-e2e6-42b6-9ed1-e63706e73cd3\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ljbh7" Jul 23 03:59:39.650823 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:39.650790 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnjsm\" (UniqueName: \"kubernetes.io/projected/e7e16755-e2e6-42b6-9ed1-e63706e73cd3-kube-api-access-pnjsm\") pod \"openshift-lws-operator-fd8ccff4c-ljbh7\" (UID: \"e7e16755-e2e6-42b6-9ed1-e63706e73cd3\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ljbh7" Jul 23 03:59:39.715307 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:39.715279 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ljbh7" Jul 23 03:59:39.886349 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:39.886324 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ljbh7"] Jul 23 03:59:39.888779 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:59:39.888706 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7e16755_e2e6_42b6_9ed1_e63706e73cd3.slice/crio-d498202ff3efd932f203c21e0633214f160a80717f921255854246079052eb10 WatchSource:0}: Error finding container d498202ff3efd932f203c21e0633214f160a80717f921255854246079052eb10: Status 404 returned error can't find the container with id d498202ff3efd932f203c21e0633214f160a80717f921255854246079052eb10 Jul 23 03:59:40.532645 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:40.532604 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ljbh7" event={"ID":"e7e16755-e2e6-42b6-9ed1-e63706e73cd3","Type":"ContainerStarted","Data":"d498202ff3efd932f203c21e0633214f160a80717f921255854246079052eb10"} Jul 23 03:59:42.541583 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:42.541548 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ljbh7" event={"ID":"e7e16755-e2e6-42b6-9ed1-e63706e73cd3","Type":"ContainerStarted","Data":"7c752a3cad3b9ce1483757eedfe88d2527189ad7cc4adfb9507157b24942cd90"} Jul 23 03:59:42.560847 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:42.560794 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ljbh7" podStartSLOduration=1.881268329 podStartE2EDuration="3.560778859s" totalImagesPullingTime="1.67951053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 03:59:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 03:59:42.559805631 +0000 UTC m=+262.465109926" watchObservedRunningTime="2026-07-23 03:59:42.560778859 +0000 UTC m=+262.466083151" Jul 23 03:59:45.094003 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:45.093970 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhv2dk"] Jul 23 03:59:45.097171 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:45.097155 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhv2dk" Jul 23 03:59:45.099986 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:45.099962 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 23 03:59:45.100114 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:45.099964 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 23 03:59:45.101112 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:45.101068 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-99zzx\"" Jul 23 03:59:45.106409 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:45.106390 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhv2dk"] Jul 23 03:59:45.182146 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:45.182115 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5661b6ce-2483-4d3d-a3dd-4e78278775e5-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhv2dk\" (UID: \"5661b6ce-2483-4d3d-a3dd-4e78278775e5\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhv2dk" Jul 23 03:59:45.182276 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:45.182154 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5661b6ce-2483-4d3d-a3dd-4e78278775e5-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhv2dk\" (UID: \"5661b6ce-2483-4d3d-a3dd-4e78278775e5\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhv2dk" Jul 23 03:59:45.182276 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:45.182230 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpb2r\" (UniqueName: \"kubernetes.io/projected/5661b6ce-2483-4d3d-a3dd-4e78278775e5-kube-api-access-gpb2r\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhv2dk\" (UID: \"5661b6ce-2483-4d3d-a3dd-4e78278775e5\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhv2dk" Jul 23 03:59:45.282935 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:45.282905 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5661b6ce-2483-4d3d-a3dd-4e78278775e5-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhv2dk\" (UID: \"5661b6ce-2483-4d3d-a3dd-4e78278775e5\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhv2dk" Jul 23 03:59:45.283124 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:45.282947 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5661b6ce-2483-4d3d-a3dd-4e78278775e5-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhv2dk\" (UID: \"5661b6ce-2483-4d3d-a3dd-4e78278775e5\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhv2dk" Jul 23 03:59:45.283124 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:45.282988 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpb2r\" (UniqueName: \"kubernetes.io/projected/5661b6ce-2483-4d3d-a3dd-4e78278775e5-kube-api-access-gpb2r\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhv2dk\" (UID: \"5661b6ce-2483-4d3d-a3dd-4e78278775e5\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhv2dk" Jul 23 03:59:45.283321 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:45.283300 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5661b6ce-2483-4d3d-a3dd-4e78278775e5-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhv2dk\" (UID: \"5661b6ce-2483-4d3d-a3dd-4e78278775e5\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhv2dk" Jul 23 03:59:45.283358 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:45.283340 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5661b6ce-2483-4d3d-a3dd-4e78278775e5-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhv2dk\" (UID: \"5661b6ce-2483-4d3d-a3dd-4e78278775e5\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhv2dk" Jul 23 03:59:45.294253 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:45.294225 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpb2r\" (UniqueName: \"kubernetes.io/projected/5661b6ce-2483-4d3d-a3dd-4e78278775e5-kube-api-access-gpb2r\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhv2dk\" (UID: \"5661b6ce-2483-4d3d-a3dd-4e78278775e5\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhv2dk" Jul 23 03:59:45.405930 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:45.405833 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhv2dk" Jul 23 03:59:45.538171 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:45.538076 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhv2dk"] Jul 23 03:59:45.540631 ip-10-0-140-6 kubenswrapper[2567]: W0723 03:59:45.540602 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5661b6ce_2483_4d3d_a3dd_4e78278775e5.slice/crio-b2f9ddfbc21519d586d086a807f7e99a59f0d219fea093775ddb869e0e140ad6 WatchSource:0}: Error finding container b2f9ddfbc21519d586d086a807f7e99a59f0d219fea093775ddb869e0e140ad6: Status 404 returned error can't find the container with id b2f9ddfbc21519d586d086a807f7e99a59f0d219fea093775ddb869e0e140ad6 Jul 23 03:59:45.558404 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:45.558377 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhv2dk" event={"ID":"5661b6ce-2483-4d3d-a3dd-4e78278775e5","Type":"ContainerStarted","Data":"b2f9ddfbc21519d586d086a807f7e99a59f0d219fea093775ddb869e0e140ad6"} Jul 23 03:59:46.563503 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:46.563391 2567 generic.go:356] "Generic (PLEG): container finished" podID="5661b6ce-2483-4d3d-a3dd-4e78278775e5" containerID="1c31b82153628ff87b4f718ec842d4bac52b7605f5c887c936d8de3601a356a2" exitCode=0 Jul 23 03:59:46.563503 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:46.563456 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhv2dk" event={"ID":"5661b6ce-2483-4d3d-a3dd-4e78278775e5","Type":"ContainerDied","Data":"1c31b82153628ff87b4f718ec842d4bac52b7605f5c887c936d8de3601a356a2"} Jul 23 03:59:47.568602 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:47.568565 2567 generic.go:356] "Generic (PLEG): container finished" podID="5661b6ce-2483-4d3d-a3dd-4e78278775e5" containerID="f7cefec0c0f1bf960cbec1cd2042a66136db81b380b058de5651b0ae49881fa2" exitCode=0 Jul 23 03:59:47.569035 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:47.568633 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhv2dk" event={"ID":"5661b6ce-2483-4d3d-a3dd-4e78278775e5","Type":"ContainerDied","Data":"f7cefec0c0f1bf960cbec1cd2042a66136db81b380b058de5651b0ae49881fa2"} Jul 23 03:59:48.574283 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:48.574248 2567 generic.go:356] "Generic (PLEG): container finished" podID="5661b6ce-2483-4d3d-a3dd-4e78278775e5" containerID="5751a71523cfdec2b8f7395e5a52b99a99d5992bf230ee6d580229ef00c74bde" exitCode=0 Jul 23 03:59:48.574662 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:48.574296 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhv2dk" event={"ID":"5661b6ce-2483-4d3d-a3dd-4e78278775e5","Type":"ContainerDied","Data":"5751a71523cfdec2b8f7395e5a52b99a99d5992bf230ee6d580229ef00c74bde"} Jul 23 03:59:49.700696 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:49.700673 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhv2dk" Jul 23 03:59:49.819262 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:49.819229 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gpb2r\" (UniqueName: \"kubernetes.io/projected/5661b6ce-2483-4d3d-a3dd-4e78278775e5-kube-api-access-gpb2r\") pod \"5661b6ce-2483-4d3d-a3dd-4e78278775e5\" (UID: \"5661b6ce-2483-4d3d-a3dd-4e78278775e5\") " Jul 23 03:59:49.819417 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:49.819283 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5661b6ce-2483-4d3d-a3dd-4e78278775e5-util\") pod \"5661b6ce-2483-4d3d-a3dd-4e78278775e5\" (UID: \"5661b6ce-2483-4d3d-a3dd-4e78278775e5\") " Jul 23 03:59:49.819417 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:49.819319 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5661b6ce-2483-4d3d-a3dd-4e78278775e5-bundle\") pod \"5661b6ce-2483-4d3d-a3dd-4e78278775e5\" (UID: \"5661b6ce-2483-4d3d-a3dd-4e78278775e5\") " Jul 23 03:59:49.820048 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:49.820024 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5661b6ce-2483-4d3d-a3dd-4e78278775e5-bundle" (OuterVolumeSpecName: "bundle") pod "5661b6ce-2483-4d3d-a3dd-4e78278775e5" (UID: "5661b6ce-2483-4d3d-a3dd-4e78278775e5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 03:59:49.821416 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:49.821392 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5661b6ce-2483-4d3d-a3dd-4e78278775e5-kube-api-access-gpb2r" (OuterVolumeSpecName: "kube-api-access-gpb2r") pod "5661b6ce-2483-4d3d-a3dd-4e78278775e5" (UID: "5661b6ce-2483-4d3d-a3dd-4e78278775e5"). InnerVolumeSpecName "kube-api-access-gpb2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 03:59:49.827370 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:49.827326 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5661b6ce-2483-4d3d-a3dd-4e78278775e5-util" (OuterVolumeSpecName: "util") pod "5661b6ce-2483-4d3d-a3dd-4e78278775e5" (UID: "5661b6ce-2483-4d3d-a3dd-4e78278775e5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 03:59:49.920480 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:49.920455 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5661b6ce-2483-4d3d-a3dd-4e78278775e5-bundle\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 03:59:49.920480 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:49.920481 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpb2r\" (UniqueName: \"kubernetes.io/projected/5661b6ce-2483-4d3d-a3dd-4e78278775e5-kube-api-access-gpb2r\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 03:59:49.920669 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:49.920490 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5661b6ce-2483-4d3d-a3dd-4e78278775e5-util\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 03:59:50.584521 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:50.584485 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhv2dk" event={"ID":"5661b6ce-2483-4d3d-a3dd-4e78278775e5","Type":"ContainerDied","Data":"b2f9ddfbc21519d586d086a807f7e99a59f0d219fea093775ddb869e0e140ad6"} Jul 23 03:59:50.584655 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:50.584533 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhv2dk" Jul 23 03:59:50.584655 ip-10-0-140-6 kubenswrapper[2567]: I0723 03:59:50.584557 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2f9ddfbc21519d586d086a807f7e99a59f0d219fea093775ddb869e0e140ad6" Jul 23 04:00:01.672284 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:01.672242 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5775946f86-jh6jx"] Jul 23 04:00:01.672720 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:01.672571 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5661b6ce-2483-4d3d-a3dd-4e78278775e5" containerName="extract" Jul 23 04:00:01.672720 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:01.672584 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="5661b6ce-2483-4d3d-a3dd-4e78278775e5" containerName="extract" Jul 23 04:00:01.672720 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:01.672595 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5661b6ce-2483-4d3d-a3dd-4e78278775e5" containerName="pull" Jul 23 04:00:01.672720 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:01.672601 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="5661b6ce-2483-4d3d-a3dd-4e78278775e5" containerName="pull" Jul 23 04:00:01.672720 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:01.672606 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5661b6ce-2483-4d3d-a3dd-4e78278775e5" containerName="util" Jul 23 04:00:01.672720 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:01.672612 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="5661b6ce-2483-4d3d-a3dd-4e78278775e5" containerName="util" Jul 23 04:00:01.672720 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:01.672654 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="5661b6ce-2483-4d3d-a3dd-4e78278775e5" containerName="extract" Jul 23 04:00:01.680059 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:01.680039 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5775946f86-jh6jx" Jul 23 04:00:01.688038 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:01.688016 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 23 04:00:01.688198 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:01.688177 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 23 04:00:01.688519 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:01.688501 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-2clxt\"" Jul 23 04:00:01.688741 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:01.688725 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 23 04:00:01.688908 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:01.688891 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 23 04:00:01.708904 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:01.708872 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5775946f86-jh6jx"] Jul 23 04:00:01.811740 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:01.811704 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/03f43fe7-a1ff-4b1c-847a-0c3eff57597f-manifests\") pod \"opendatahub-operator-controller-manager-5775946f86-jh6jx\" (UID: \"03f43fe7-a1ff-4b1c-847a-0c3eff57597f\") " pod="opendatahub/opendatahub-operator-controller-manager-5775946f86-jh6jx" Jul 23 04:00:01.811740 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:01.811742 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/03f43fe7-a1ff-4b1c-847a-0c3eff57597f-webhook-cert\") pod \"opendatahub-operator-controller-manager-5775946f86-jh6jx\" (UID: \"03f43fe7-a1ff-4b1c-847a-0c3eff57597f\") " pod="opendatahub/opendatahub-operator-controller-manager-5775946f86-jh6jx" Jul 23 04:00:01.811964 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:01.811772 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/03f43fe7-a1ff-4b1c-847a-0c3eff57597f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5775946f86-jh6jx\" (UID: \"03f43fe7-a1ff-4b1c-847a-0c3eff57597f\") " pod="opendatahub/opendatahub-operator-controller-manager-5775946f86-jh6jx" Jul 23 04:00:01.811964 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:01.811792 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03f43fe7-a1ff-4b1c-847a-0c3eff57597f-tmp\") pod \"opendatahub-operator-controller-manager-5775946f86-jh6jx\" (UID: \"03f43fe7-a1ff-4b1c-847a-0c3eff57597f\") " pod="opendatahub/opendatahub-operator-controller-manager-5775946f86-jh6jx" Jul 23 04:00:01.811964 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:01.811848 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzgff\" (UniqueName: \"kubernetes.io/projected/03f43fe7-a1ff-4b1c-847a-0c3eff57597f-kube-api-access-fzgff\") pod \"opendatahub-operator-controller-manager-5775946f86-jh6jx\" (UID: \"03f43fe7-a1ff-4b1c-847a-0c3eff57597f\") " pod="opendatahub/opendatahub-operator-controller-manager-5775946f86-jh6jx" Jul 23 04:00:01.900531 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:01.900496 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6776597644-gzpxr"] Jul 23 04:00:01.903902 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:01.903885 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6776597644-gzpxr" Jul 23 04:00:01.906704 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:01.906682 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 23 04:00:01.906940 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:01.906905 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 23 04:00:01.906940 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:01.906935 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-r57g6\"" Jul 23 04:00:01.907157 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:01.907143 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 23 04:00:01.912260 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:01.912241 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/03f43fe7-a1ff-4b1c-847a-0c3eff57597f-manifests\") pod \"opendatahub-operator-controller-manager-5775946f86-jh6jx\" (UID: \"03f43fe7-a1ff-4b1c-847a-0c3eff57597f\") " pod="opendatahub/opendatahub-operator-controller-manager-5775946f86-jh6jx" Jul 23 04:00:01.912339 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:01.912271 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/03f43fe7-a1ff-4b1c-847a-0c3eff57597f-webhook-cert\") pod \"opendatahub-operator-controller-manager-5775946f86-jh6jx\" (UID: \"03f43fe7-a1ff-4b1c-847a-0c3eff57597f\") " pod="opendatahub/opendatahub-operator-controller-manager-5775946f86-jh6jx" Jul 23 04:00:01.912339 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:01.912298 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/03f43fe7-a1ff-4b1c-847a-0c3eff57597f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5775946f86-jh6jx\" (UID: \"03f43fe7-a1ff-4b1c-847a-0c3eff57597f\") " pod="opendatahub/opendatahub-operator-controller-manager-5775946f86-jh6jx" Jul 23 04:00:01.912339 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:01.912316 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03f43fe7-a1ff-4b1c-847a-0c3eff57597f-tmp\") pod \"opendatahub-operator-controller-manager-5775946f86-jh6jx\" (UID: \"03f43fe7-a1ff-4b1c-847a-0c3eff57597f\") " pod="opendatahub/opendatahub-operator-controller-manager-5775946f86-jh6jx" Jul 23 04:00:01.912489 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:01.912357 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzgff\" (UniqueName: \"kubernetes.io/projected/03f43fe7-a1ff-4b1c-847a-0c3eff57597f-kube-api-access-fzgff\") pod \"opendatahub-operator-controller-manager-5775946f86-jh6jx\" (UID: \"03f43fe7-a1ff-4b1c-847a-0c3eff57597f\") " pod="opendatahub/opendatahub-operator-controller-manager-5775946f86-jh6jx" Jul 23 04:00:01.912653 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:01.912633 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/03f43fe7-a1ff-4b1c-847a-0c3eff57597f-manifests\") pod \"opendatahub-operator-controller-manager-5775946f86-jh6jx\" (UID: \"03f43fe7-a1ff-4b1c-847a-0c3eff57597f\") " pod="opendatahub/opendatahub-operator-controller-manager-5775946f86-jh6jx" Jul 23 04:00:01.914725 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:01.914706 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03f43fe7-a1ff-4b1c-847a-0c3eff57597f-tmp\") pod \"opendatahub-operator-controller-manager-5775946f86-jh6jx\" (UID: \"03f43fe7-a1ff-4b1c-847a-0c3eff57597f\") " pod="opendatahub/opendatahub-operator-controller-manager-5775946f86-jh6jx" Jul 23 04:00:01.914927 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:01.914910 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/03f43fe7-a1ff-4b1c-847a-0c3eff57597f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5775946f86-jh6jx\" (UID: \"03f43fe7-a1ff-4b1c-847a-0c3eff57597f\") " pod="opendatahub/opendatahub-operator-controller-manager-5775946f86-jh6jx" Jul 23 04:00:01.914970 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:01.914914 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/03f43fe7-a1ff-4b1c-847a-0c3eff57597f-webhook-cert\") pod \"opendatahub-operator-controller-manager-5775946f86-jh6jx\" (UID: \"03f43fe7-a1ff-4b1c-847a-0c3eff57597f\") " pod="opendatahub/opendatahub-operator-controller-manager-5775946f86-jh6jx" Jul 23 04:00:01.919247 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:01.919224 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6776597644-gzpxr"] Jul 23 04:00:01.925590 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:01.925533 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzgff\" (UniqueName: \"kubernetes.io/projected/03f43fe7-a1ff-4b1c-847a-0c3eff57597f-kube-api-access-fzgff\") pod \"opendatahub-operator-controller-manager-5775946f86-jh6jx\" (UID: \"03f43fe7-a1ff-4b1c-847a-0c3eff57597f\") " pod="opendatahub/opendatahub-operator-controller-manager-5775946f86-jh6jx" Jul 23 04:00:01.990590 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:01.990556 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5775946f86-jh6jx" Jul 23 04:00:02.013264 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:02.013231 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a2d1aef1-4986-452b-b52d-7e35369e1a01-manager-config\") pod \"lws-controller-manager-6776597644-gzpxr\" (UID: \"a2d1aef1-4986-452b-b52d-7e35369e1a01\") " pod="openshift-lws-operator/lws-controller-manager-6776597644-gzpxr" Jul 23 04:00:02.013408 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:02.013269 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2d1aef1-4986-452b-b52d-7e35369e1a01-cert\") pod \"lws-controller-manager-6776597644-gzpxr\" (UID: \"a2d1aef1-4986-452b-b52d-7e35369e1a01\") " pod="openshift-lws-operator/lws-controller-manager-6776597644-gzpxr" Jul 23 04:00:02.013408 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:02.013300 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a2d1aef1-4986-452b-b52d-7e35369e1a01-metrics-cert\") pod \"lws-controller-manager-6776597644-gzpxr\" (UID: \"a2d1aef1-4986-452b-b52d-7e35369e1a01\") " pod="openshift-lws-operator/lws-controller-manager-6776597644-gzpxr" Jul 23 04:00:02.013408 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:02.013376 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wkdd\" (UniqueName: \"kubernetes.io/projected/a2d1aef1-4986-452b-b52d-7e35369e1a01-kube-api-access-6wkdd\") pod \"lws-controller-manager-6776597644-gzpxr\" (UID: \"a2d1aef1-4986-452b-b52d-7e35369e1a01\") " pod="openshift-lws-operator/lws-controller-manager-6776597644-gzpxr" Jul 23 04:00:02.114673 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:02.114641 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a2d1aef1-4986-452b-b52d-7e35369e1a01-metrics-cert\") pod \"lws-controller-manager-6776597644-gzpxr\" (UID: \"a2d1aef1-4986-452b-b52d-7e35369e1a01\") " pod="openshift-lws-operator/lws-controller-manager-6776597644-gzpxr" Jul 23 04:00:02.114812 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:02.114701 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wkdd\" (UniqueName: \"kubernetes.io/projected/a2d1aef1-4986-452b-b52d-7e35369e1a01-kube-api-access-6wkdd\") pod \"lws-controller-manager-6776597644-gzpxr\" (UID: \"a2d1aef1-4986-452b-b52d-7e35369e1a01\") " pod="openshift-lws-operator/lws-controller-manager-6776597644-gzpxr" Jul 23 04:00:02.114812 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:02.114772 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a2d1aef1-4986-452b-b52d-7e35369e1a01-manager-config\") pod \"lws-controller-manager-6776597644-gzpxr\" (UID: \"a2d1aef1-4986-452b-b52d-7e35369e1a01\") " pod="openshift-lws-operator/lws-controller-manager-6776597644-gzpxr" Jul 23 04:00:02.114812 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:02.114795 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2d1aef1-4986-452b-b52d-7e35369e1a01-cert\") pod \"lws-controller-manager-6776597644-gzpxr\" (UID: \"a2d1aef1-4986-452b-b52d-7e35369e1a01\") " pod="openshift-lws-operator/lws-controller-manager-6776597644-gzpxr" Jul 23 04:00:02.115527 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:02.115508 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a2d1aef1-4986-452b-b52d-7e35369e1a01-manager-config\") pod \"lws-controller-manager-6776597644-gzpxr\" (UID: \"a2d1aef1-4986-452b-b52d-7e35369e1a01\") " pod="openshift-lws-operator/lws-controller-manager-6776597644-gzpxr" Jul 23 04:00:02.117315 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:02.117296 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2d1aef1-4986-452b-b52d-7e35369e1a01-cert\") pod \"lws-controller-manager-6776597644-gzpxr\" (UID: \"a2d1aef1-4986-452b-b52d-7e35369e1a01\") " pod="openshift-lws-operator/lws-controller-manager-6776597644-gzpxr" Jul 23 04:00:02.117359 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:02.117332 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a2d1aef1-4986-452b-b52d-7e35369e1a01-metrics-cert\") pod \"lws-controller-manager-6776597644-gzpxr\" (UID: \"a2d1aef1-4986-452b-b52d-7e35369e1a01\") " pod="openshift-lws-operator/lws-controller-manager-6776597644-gzpxr" Jul 23 04:00:02.138465 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:02.138415 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5775946f86-jh6jx"] Jul 23 04:00:02.140952 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:02.140934 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wkdd\" (UniqueName: \"kubernetes.io/projected/a2d1aef1-4986-452b-b52d-7e35369e1a01-kube-api-access-6wkdd\") pod \"lws-controller-manager-6776597644-gzpxr\" (UID: \"a2d1aef1-4986-452b-b52d-7e35369e1a01\") " pod="openshift-lws-operator/lws-controller-manager-6776597644-gzpxr" Jul 23 04:00:02.142040 ip-10-0-140-6 kubenswrapper[2567]: W0723 04:00:02.142016 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03f43fe7_a1ff_4b1c_847a_0c3eff57597f.slice/crio-b8db03d3dd174255ff9e034ed8ac1f839093efa42b931f736e1d82a3775685ad WatchSource:0}: Error finding container b8db03d3dd174255ff9e034ed8ac1f839093efa42b931f736e1d82a3775685ad: Status 404 returned error can't find the container with id b8db03d3dd174255ff9e034ed8ac1f839093efa42b931f736e1d82a3775685ad Jul 23 04:00:02.212762 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:02.212672 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6776597644-gzpxr" Jul 23 04:00:02.355501 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:02.355463 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6776597644-gzpxr"] Jul 23 04:00:02.356988 ip-10-0-140-6 kubenswrapper[2567]: W0723 04:00:02.356951 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2d1aef1_4986_452b_b52d_7e35369e1a01.slice/crio-6540bf617a1ef2f6de31c9084780dc3a65758b5f4ff1d815e884691bfb0a25ba WatchSource:0}: Error finding container 6540bf617a1ef2f6de31c9084780dc3a65758b5f4ff1d815e884691bfb0a25ba: Status 404 returned error can't find the container with id 6540bf617a1ef2f6de31c9084780dc3a65758b5f4ff1d815e884691bfb0a25ba Jul 23 04:00:02.638016 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:02.637925 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5775946f86-jh6jx" event={"ID":"03f43fe7-a1ff-4b1c-847a-0c3eff57597f","Type":"ContainerStarted","Data":"b8db03d3dd174255ff9e034ed8ac1f839093efa42b931f736e1d82a3775685ad"} Jul 23 04:00:02.639054 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:02.639027 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6776597644-gzpxr" event={"ID":"a2d1aef1-4986-452b-b52d-7e35369e1a01","Type":"ContainerStarted","Data":"6540bf617a1ef2f6de31c9084780dc3a65758b5f4ff1d815e884691bfb0a25ba"} Jul 23 04:00:06.667044 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:06.667004 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5775946f86-jh6jx" event={"ID":"03f43fe7-a1ff-4b1c-847a-0c3eff57597f","Type":"ContainerStarted","Data":"1c07b06dfb395a25f6c11aabd151b2419e0e75ad19a7717d4be85bd042b91041"} Jul 23 04:00:07.672653 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:07.672614 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6776597644-gzpxr" event={"ID":"a2d1aef1-4986-452b-b52d-7e35369e1a01","Type":"ContainerStarted","Data":"aa851b5e2a17b6d518130f5951ca8c97fb18ba02a9e9639351b95dc122cee369"} Jul 23 04:00:07.673120 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:07.672769 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6776597644-gzpxr" Jul 23 04:00:07.674160 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:07.674132 2567 generic.go:356] "Generic (PLEG): container finished" podID="03f43fe7-a1ff-4b1c-847a-0c3eff57597f" containerID="1c07b06dfb395a25f6c11aabd151b2419e0e75ad19a7717d4be85bd042b91041" exitCode=0 Jul 23 04:00:07.674271 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:07.674192 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5775946f86-jh6jx" event={"ID":"03f43fe7-a1ff-4b1c-847a-0c3eff57597f","Type":"ContainerDied","Data":"1c07b06dfb395a25f6c11aabd151b2419e0e75ad19a7717d4be85bd042b91041"} Jul 23 04:00:07.697384 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:07.697330 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6776597644-gzpxr" podStartSLOduration=2.206587587 podStartE2EDuration="6.697311871s" totalImagesPullingTime="4.490724284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 04:00:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 04:00:07.69569383 +0000 UTC m=+287.600998151" watchObservedRunningTime="2026-07-23 04:00:07.697311871 +0000 UTC m=+287.602616165" Jul 23 04:00:08.680142 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:08.680110 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5775946f86-jh6jx" event={"ID":"03f43fe7-a1ff-4b1c-847a-0c3eff57597f","Type":"ContainerStarted","Data":"b6b4e0b0cbb571ec7e62abe7fd653a52aaefefd7cf663f3d4c488200365a3d42"} Jul 23 04:00:08.680540 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:08.680275 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5775946f86-jh6jx" Jul 23 04:00:08.705071 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:08.705020 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5775946f86-jh6jx" podStartSLOduration=2.493012262 podStartE2EDuration="7.705004872s" totalImagesPullingTime="3.968056502s" totalInitContainerRuntime="1.243936108s" isStatefulPod=false podCreationTimestamp="2026-07-23 04:00:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 04:00:08.703420303 +0000 UTC m=+288.608724596" watchObservedRunningTime="2026-07-23 04:00:08.705004872 +0000 UTC m=+288.610309163" Jul 23 04:00:18.682390 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:18.682361 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6776597644-gzpxr" Jul 23 04:00:19.686456 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:19.686409 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5775946f86-jh6jx" Jul 23 04:00:20.409130 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:20.409099 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wzgn9_dbd53593-e5e7-473a-b71c-8fbf9272913f/console-operator/2.log" Jul 23 04:00:20.415390 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:20.415363 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wzgn9_dbd53593-e5e7-473a-b71c-8fbf9272913f/console-operator/2.log" Jul 23 04:00:20.421134 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:20.421113 2567 kubelet.go:1648] "Image garbage collection succeeded" Jul 23 04:00:29.849387 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:29.849351 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mc5m7"] Jul 23 04:00:29.852689 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:29.852674 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mc5m7" Jul 23 04:00:29.855722 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:29.855703 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 23 04:00:29.855839 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:29.855782 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-fs47x\"" Jul 23 04:00:29.867608 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:29.867581 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mc5m7"] Jul 23 04:00:29.938562 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:29.938528 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb5a035e-cabb-4796-9094-30073eef0c21-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mc5m7\" (UID: \"bb5a035e-cabb-4796-9094-30073eef0c21\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mc5m7" Jul 23 04:00:29.938562 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:29.938559 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb5a035e-cabb-4796-9094-30073eef0c21-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mc5m7\" (UID: \"bb5a035e-cabb-4796-9094-30073eef0c21\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mc5m7" Jul 23 04:00:29.938795 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:29.938578 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb5a035e-cabb-4796-9094-30073eef0c21-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mc5m7\" (UID: \"bb5a035e-cabb-4796-9094-30073eef0c21\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mc5m7" Jul 23 04:00:29.938795 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:29.938683 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb5a035e-cabb-4796-9094-30073eef0c21-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mc5m7\" (UID: \"bb5a035e-cabb-4796-9094-30073eef0c21\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mc5m7" Jul 23 04:00:29.938795 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:29.938762 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb5a035e-cabb-4796-9094-30073eef0c21-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mc5m7\" (UID: \"bb5a035e-cabb-4796-9094-30073eef0c21\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mc5m7" Jul 23 04:00:29.938952 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:29.938800 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb5a035e-cabb-4796-9094-30073eef0c21-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mc5m7\" (UID: \"bb5a035e-cabb-4796-9094-30073eef0c21\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mc5m7" Jul 23 04:00:29.938952 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:29.938860 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb5a035e-cabb-4796-9094-30073eef0c21-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mc5m7\" (UID: \"bb5a035e-cabb-4796-9094-30073eef0c21\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mc5m7" Jul 23 04:00:29.938952 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:29.938890 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb5a035e-cabb-4796-9094-30073eef0c21-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mc5m7\" (UID: \"bb5a035e-cabb-4796-9094-30073eef0c21\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mc5m7" Jul 23 04:00:29.939061 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:29.938956 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x24n9\" (UniqueName: \"kubernetes.io/projected/bb5a035e-cabb-4796-9094-30073eef0c21-kube-api-access-x24n9\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mc5m7\" (UID: \"bb5a035e-cabb-4796-9094-30073eef0c21\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mc5m7" Jul 23 04:00:30.039336 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:30.039304 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb5a035e-cabb-4796-9094-30073eef0c21-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mc5m7\" (UID: \"bb5a035e-cabb-4796-9094-30073eef0c21\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mc5m7" Jul 23 04:00:30.039561 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:30.039351 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb5a035e-cabb-4796-9094-30073eef0c21-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mc5m7\" (UID: \"bb5a035e-cabb-4796-9094-30073eef0c21\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mc5m7" Jul 23 04:00:30.039561 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:30.039382 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb5a035e-cabb-4796-9094-30073eef0c21-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mc5m7\" (UID: \"bb5a035e-cabb-4796-9094-30073eef0c21\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mc5m7" Jul 23 04:00:30.039561 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:30.039403 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb5a035e-cabb-4796-9094-30073eef0c21-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mc5m7\" (UID: \"bb5a035e-cabb-4796-9094-30073eef0c21\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mc5m7" Jul 23 04:00:30.039561 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:30.039447 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb5a035e-cabb-4796-9094-30073eef0c21-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mc5m7\" (UID: \"bb5a035e-cabb-4796-9094-30073eef0c21\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mc5m7" Jul 23 04:00:30.039561 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:30.039471 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb5a035e-cabb-4796-9094-30073eef0c21-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mc5m7\" (UID: \"bb5a035e-cabb-4796-9094-30073eef0c21\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mc5m7" Jul 23 04:00:30.039561 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:30.039508 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x24n9\" (UniqueName: \"kubernetes.io/projected/bb5a035e-cabb-4796-9094-30073eef0c21-kube-api-access-x24n9\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mc5m7\" (UID: \"bb5a035e-cabb-4796-9094-30073eef0c21\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mc5m7" Jul 23 04:00:30.039561 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:30.039527 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb5a035e-cabb-4796-9094-30073eef0c21-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mc5m7\" (UID: \"bb5a035e-cabb-4796-9094-30073eef0c21\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mc5m7" Jul 23 04:00:30.039561 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:30.039547 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb5a035e-cabb-4796-9094-30073eef0c21-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mc5m7\" (UID: \"bb5a035e-cabb-4796-9094-30073eef0c21\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mc5m7" Jul 23 04:00:30.039831 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:30.039787 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb5a035e-cabb-4796-9094-30073eef0c21-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mc5m7\" (UID: \"bb5a035e-cabb-4796-9094-30073eef0c21\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mc5m7" Jul 23 04:00:30.039831 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:30.039800 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb5a035e-cabb-4796-9094-30073eef0c21-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mc5m7\" (UID: \"bb5a035e-cabb-4796-9094-30073eef0c21\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mc5m7" Jul 23 04:00:30.040019 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:30.040002 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb5a035e-cabb-4796-9094-30073eef0c21-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mc5m7\" (UID: \"bb5a035e-cabb-4796-9094-30073eef0c21\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mc5m7" Jul 23 04:00:30.040019 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:30.040006 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb5a035e-cabb-4796-9094-30073eef0c21-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mc5m7\" (UID: \"bb5a035e-cabb-4796-9094-30073eef0c21\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mc5m7" Jul 23 04:00:30.040474 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:30.040418 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb5a035e-cabb-4796-9094-30073eef0c21-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mc5m7\" (UID: \"bb5a035e-cabb-4796-9094-30073eef0c21\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mc5m7" Jul 23 04:00:30.042079 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:30.042058 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb5a035e-cabb-4796-9094-30073eef0c21-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mc5m7\" (UID: \"bb5a035e-cabb-4796-9094-30073eef0c21\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mc5m7" Jul 23 04:00:30.042202 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:30.042187 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb5a035e-cabb-4796-9094-30073eef0c21-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mc5m7\" (UID: \"bb5a035e-cabb-4796-9094-30073eef0c21\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mc5m7" Jul 23 04:00:30.049608 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:30.049585 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb5a035e-cabb-4796-9094-30073eef0c21-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mc5m7\" (UID: \"bb5a035e-cabb-4796-9094-30073eef0c21\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mc5m7" Jul 23 04:00:30.049748 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:30.049598 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x24n9\" (UniqueName: \"kubernetes.io/projected/bb5a035e-cabb-4796-9094-30073eef0c21-kube-api-access-x24n9\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mc5m7\" (UID: \"bb5a035e-cabb-4796-9094-30073eef0c21\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mc5m7" Jul 23 04:00:30.162846 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:30.162764 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mc5m7" Jul 23 04:00:30.342168 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:30.342131 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mc5m7"] Jul 23 04:00:30.345189 ip-10-0-140-6 kubenswrapper[2567]: W0723 04:00:30.345164 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb5a035e_cabb_4796_9094_30073eef0c21.slice/crio-214db3457729249fea5e68ef6ce0a86dbc26c3199ac242e76e51a15f830783df WatchSource:0}: Error finding container 214db3457729249fea5e68ef6ce0a86dbc26c3199ac242e76e51a15f830783df: Status 404 returned error can't find the container with id 214db3457729249fea5e68ef6ce0a86dbc26c3199ac242e76e51a15f830783df Jul 23 04:00:30.347107 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:30.347092 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 04:00:30.771992 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:30.771963 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mc5m7" event={"ID":"bb5a035e-cabb-4796-9094-30073eef0c21","Type":"ContainerStarted","Data":"214db3457729249fea5e68ef6ce0a86dbc26c3199ac242e76e51a15f830783df"} Jul 23 04:00:33.143268 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:33.143226 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 04:00:33.143696 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:33.143321 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 04:00:33.143696 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:33.143370 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 04:00:33.582964 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:33.582933 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-rzrj5"] Jul 23 04:00:33.586112 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:33.586092 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-rzrj5" Jul 23 04:00:33.590730 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:33.590711 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 23 04:00:33.591977 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:33.591958 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-p4xrv\"" Jul 23 04:00:33.596024 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:33.596003 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-mwvqq"] Jul 23 04:00:33.598977 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:33.598964 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mwvqq" Jul 23 04:00:33.601601 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:33.601579 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-w9qwk\"" Jul 23 04:00:33.601805 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:33.601792 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 23 04:00:33.612124 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:33.612104 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-rzrj5"] Jul 23 04:00:33.613866 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:33.613844 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-mwvqq"] Jul 23 04:00:33.675993 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:33.675963 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8fdae660-c7ad-4c99-9c82-f2881f07b883-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-mwvqq\" (UID: \"8fdae660-c7ad-4c99-9c82-f2881f07b883\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mwvqq" Jul 23 04:00:33.676122 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:33.676008 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d60f90ba-ef41-4ac7-80d6-d03eae7b361a-cert\") pod \"kserve-controller-manager-d6f4b65b8-rzrj5\" (UID: \"d60f90ba-ef41-4ac7-80d6-d03eae7b361a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-rzrj5" Jul 23 04:00:33.676122 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:33.676093 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hctrv\" (UniqueName: \"kubernetes.io/projected/8fdae660-c7ad-4c99-9c82-f2881f07b883-kube-api-access-hctrv\") pod \"llmisvc-controller-manager-777c6bd7d5-mwvqq\" (UID: \"8fdae660-c7ad-4c99-9c82-f2881f07b883\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mwvqq" Jul 23 04:00:33.676227 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:33.676154 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlpgs\" (UniqueName: \"kubernetes.io/projected/d60f90ba-ef41-4ac7-80d6-d03eae7b361a-kube-api-access-hlpgs\") pod \"kserve-controller-manager-d6f4b65b8-rzrj5\" (UID: \"d60f90ba-ef41-4ac7-80d6-d03eae7b361a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-rzrj5" Jul 23 04:00:33.777393 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:33.777360 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8fdae660-c7ad-4c99-9c82-f2881f07b883-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-mwvqq\" (UID: \"8fdae660-c7ad-4c99-9c82-f2881f07b883\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mwvqq" Jul 23 04:00:33.777393 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:33.777394 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d60f90ba-ef41-4ac7-80d6-d03eae7b361a-cert\") pod \"kserve-controller-manager-d6f4b65b8-rzrj5\" (UID: \"d60f90ba-ef41-4ac7-80d6-d03eae7b361a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-rzrj5" Jul 23 04:00:33.777602 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:33.777421 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hctrv\" (UniqueName: \"kubernetes.io/projected/8fdae660-c7ad-4c99-9c82-f2881f07b883-kube-api-access-hctrv\") pod \"llmisvc-controller-manager-777c6bd7d5-mwvqq\" (UID: \"8fdae660-c7ad-4c99-9c82-f2881f07b883\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mwvqq" Jul 23 04:00:33.777602 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:33.777466 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlpgs\" (UniqueName: \"kubernetes.io/projected/d60f90ba-ef41-4ac7-80d6-d03eae7b361a-kube-api-access-hlpgs\") pod \"kserve-controller-manager-d6f4b65b8-rzrj5\" (UID: \"d60f90ba-ef41-4ac7-80d6-d03eae7b361a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-rzrj5" Jul 23 04:00:33.777602 ip-10-0-140-6 kubenswrapper[2567]: E0723 04:00:33.777544 2567 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 23 04:00:33.777694 ip-10-0-140-6 kubenswrapper[2567]: E0723 04:00:33.777613 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d60f90ba-ef41-4ac7-80d6-d03eae7b361a-cert podName:d60f90ba-ef41-4ac7-80d6-d03eae7b361a nodeName:}" failed. No retries permitted until 2026-07-23 04:00:34.277597791 +0000 UTC m=+314.182902061 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d60f90ba-ef41-4ac7-80d6-d03eae7b361a-cert") pod "kserve-controller-manager-d6f4b65b8-rzrj5" (UID: "d60f90ba-ef41-4ac7-80d6-d03eae7b361a") : secret "kserve-webhook-server-cert" not found Jul 23 04:00:33.779995 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:33.779974 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8fdae660-c7ad-4c99-9c82-f2881f07b883-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-mwvqq\" (UID: \"8fdae660-c7ad-4c99-9c82-f2881f07b883\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mwvqq" Jul 23 04:00:33.792497 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:33.792462 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mc5m7" event={"ID":"bb5a035e-cabb-4796-9094-30073eef0c21","Type":"ContainerStarted","Data":"9a0aa50ba89255c5f6690e546dbb8c68ec7366cc9e21ec710dc363c45e74a484"} Jul 23 04:00:33.798041 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:33.793499 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hctrv\" (UniqueName: \"kubernetes.io/projected/8fdae660-c7ad-4c99-9c82-f2881f07b883-kube-api-access-hctrv\") pod \"llmisvc-controller-manager-777c6bd7d5-mwvqq\" (UID: \"8fdae660-c7ad-4c99-9c82-f2881f07b883\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mwvqq" Jul 23 04:00:33.799018 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:33.799000 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlpgs\" (UniqueName: \"kubernetes.io/projected/d60f90ba-ef41-4ac7-80d6-d03eae7b361a-kube-api-access-hlpgs\") pod \"kserve-controller-manager-d6f4b65b8-rzrj5\" (UID: \"d60f90ba-ef41-4ac7-80d6-d03eae7b361a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-rzrj5" Jul 23 04:00:33.820631 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:33.820555 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mc5m7" podStartSLOduration=2.024767218 podStartE2EDuration="4.820538269s" totalImagesPullingTime="2.795771051s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 04:00:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 04:00:33.819767974 +0000 UTC m=+313.725072277" watchObservedRunningTime="2026-07-23 04:00:33.820538269 +0000 UTC m=+313.725842561" Jul 23 04:00:33.907286 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:33.907217 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mwvqq" Jul 23 04:00:34.054495 ip-10-0-140-6 kubenswrapper[2567]: W0723 04:00:34.054469 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8fdae660_c7ad_4c99_9c82_f2881f07b883.slice/crio-4a02f93a132a5d133204f6d184ad712ddc83ea554ee07d2fc7073f946e39b8b7 WatchSource:0}: Error finding container 4a02f93a132a5d133204f6d184ad712ddc83ea554ee07d2fc7073f946e39b8b7: Status 404 returned error can't find the container with id 4a02f93a132a5d133204f6d184ad712ddc83ea554ee07d2fc7073f946e39b8b7 Jul 23 04:00:34.055868 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:34.055848 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-mwvqq"] Jul 23 04:00:34.163461 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:34.163371 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mc5m7" Jul 23 04:00:34.168192 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:34.168173 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mc5m7" Jul 23 04:00:34.281742 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:34.281713 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d60f90ba-ef41-4ac7-80d6-d03eae7b361a-cert\") pod \"kserve-controller-manager-d6f4b65b8-rzrj5\" (UID: \"d60f90ba-ef41-4ac7-80d6-d03eae7b361a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-rzrj5" Jul 23 04:00:34.284362 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:34.284311 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d60f90ba-ef41-4ac7-80d6-d03eae7b361a-cert\") pod \"kserve-controller-manager-d6f4b65b8-rzrj5\" (UID: \"d60f90ba-ef41-4ac7-80d6-d03eae7b361a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-rzrj5" Jul 23 04:00:34.496375 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:34.496340 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-rzrj5" Jul 23 04:00:34.640378 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:34.640357 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-rzrj5"] Jul 23 04:00:34.642420 ip-10-0-140-6 kubenswrapper[2567]: W0723 04:00:34.642387 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd60f90ba_ef41_4ac7_80d6_d03eae7b361a.slice/crio-c090491871437207b2498be59a547ccaceedcd3d5d4791fdb79664a523b92edf WatchSource:0}: Error finding container c090491871437207b2498be59a547ccaceedcd3d5d4791fdb79664a523b92edf: Status 404 returned error can't find the container with id c090491871437207b2498be59a547ccaceedcd3d5d4791fdb79664a523b92edf Jul 23 04:00:34.797881 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:34.797783 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mwvqq" event={"ID":"8fdae660-c7ad-4c99-9c82-f2881f07b883","Type":"ContainerStarted","Data":"4a02f93a132a5d133204f6d184ad712ddc83ea554ee07d2fc7073f946e39b8b7"} Jul 23 04:00:34.798916 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:34.798892 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-rzrj5" event={"ID":"d60f90ba-ef41-4ac7-80d6-d03eae7b361a","Type":"ContainerStarted","Data":"c090491871437207b2498be59a547ccaceedcd3d5d4791fdb79664a523b92edf"} Jul 23 04:00:34.799090 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:34.799077 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mc5m7" Jul 23 04:00:34.799965 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:34.799949 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mc5m7" Jul 23 04:00:38.821686 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:38.821647 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mwvqq" event={"ID":"8fdae660-c7ad-4c99-9c82-f2881f07b883","Type":"ContainerStarted","Data":"7ab96e5c78ee30ad7a3c7ca9319d8c7f28f28d1adce6a36d4e921fc1e6aad823"} Jul 23 04:00:38.822160 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:38.821743 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mwvqq" Jul 23 04:00:38.823273 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:38.823249 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-rzrj5" event={"ID":"d60f90ba-ef41-4ac7-80d6-d03eae7b361a","Type":"ContainerStarted","Data":"bd682bbd212e007481e1e02de3673476ddf76e571966b09df68ff67c640ae5a1"} Jul 23 04:00:38.823396 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:38.823384 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-rzrj5" Jul 23 04:00:38.844963 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:38.844919 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mwvqq" podStartSLOduration=1.82848232 podStartE2EDuration="5.844901773s" totalImagesPullingTime="4.016419453s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 04:00:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 04:00:38.843344277 +0000 UTC m=+318.748648570" watchObservedRunningTime="2026-07-23 04:00:38.844901773 +0000 UTC m=+318.750206425" Jul 23 04:00:38.869498 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:00:38.869445 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-rzrj5" podStartSLOduration=3.024351417 podStartE2EDuration="5.86941625s" totalImagesPullingTime="2.845064833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 04:00:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 04:00:38.867999295 +0000 UTC m=+318.773303586" watchObservedRunningTime="2026-07-23 04:00:38.86941625 +0000 UTC m=+318.774720561" Jul 23 04:01:09.829741 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:09.829707 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mwvqq" Jul 23 04:01:09.833136 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:09.833114 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-rzrj5" Jul 23 04:01:38.588552 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:38.588508 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sqh2r"] Jul 23 04:01:38.591950 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:38.591936 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sqh2r" Jul 23 04:01:38.595086 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:38.595063 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-pbwj7\"" Jul 23 04:01:38.595213 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:38.595094 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 23 04:01:38.595213 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:38.595067 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 23 04:01:38.600266 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:38.600245 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sqh2r"] Jul 23 04:01:38.716223 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:38.716193 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-749cm\" (UniqueName: \"kubernetes.io/projected/12c78b78-909c-46b3-8c5c-b69ee1437002-kube-api-access-749cm\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sqh2r\" (UID: \"12c78b78-909c-46b3-8c5c-b69ee1437002\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sqh2r" Jul 23 04:01:38.716363 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:38.716272 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12c78b78-909c-46b3-8c5c-b69ee1437002-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sqh2r\" (UID: \"12c78b78-909c-46b3-8c5c-b69ee1437002\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sqh2r" Jul 23 04:01:38.716363 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:38.716297 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12c78b78-909c-46b3-8c5c-b69ee1437002-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sqh2r\" (UID: \"12c78b78-909c-46b3-8c5c-b69ee1437002\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sqh2r" Jul 23 04:01:38.816739 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:38.816699 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12c78b78-909c-46b3-8c5c-b69ee1437002-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sqh2r\" (UID: \"12c78b78-909c-46b3-8c5c-b69ee1437002\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sqh2r" Jul 23 04:01:38.816739 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:38.816742 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12c78b78-909c-46b3-8c5c-b69ee1437002-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sqh2r\" (UID: \"12c78b78-909c-46b3-8c5c-b69ee1437002\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sqh2r" Jul 23 04:01:38.816924 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:38.816789 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-749cm\" (UniqueName: \"kubernetes.io/projected/12c78b78-909c-46b3-8c5c-b69ee1437002-kube-api-access-749cm\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sqh2r\" (UID: \"12c78b78-909c-46b3-8c5c-b69ee1437002\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sqh2r" Jul 23 04:01:38.817061 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:38.817042 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12c78b78-909c-46b3-8c5c-b69ee1437002-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sqh2r\" (UID: \"12c78b78-909c-46b3-8c5c-b69ee1437002\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sqh2r" Jul 23 04:01:38.817097 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:38.817069 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12c78b78-909c-46b3-8c5c-b69ee1437002-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sqh2r\" (UID: \"12c78b78-909c-46b3-8c5c-b69ee1437002\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sqh2r" Jul 23 04:01:38.826490 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:38.826470 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-749cm\" (UniqueName: \"kubernetes.io/projected/12c78b78-909c-46b3-8c5c-b69ee1437002-kube-api-access-749cm\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sqh2r\" (UID: \"12c78b78-909c-46b3-8c5c-b69ee1437002\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sqh2r" Jul 23 04:01:38.901478 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:38.901404 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sqh2r" Jul 23 04:01:39.049729 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:39.048727 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sqh2r"] Jul 23 04:01:39.050824 ip-10-0-140-6 kubenswrapper[2567]: W0723 04:01:39.050791 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12c78b78_909c_46b3_8c5c_b69ee1437002.slice/crio-9aab77bf4d1e29c7c34750ea3bccdafe6f2730b5bfc1c1eb598994f1be523329 WatchSource:0}: Error finding container 9aab77bf4d1e29c7c34750ea3bccdafe6f2730b5bfc1c1eb598994f1be523329: Status 404 returned error can't find the container with id 9aab77bf4d1e29c7c34750ea3bccdafe6f2730b5bfc1c1eb598994f1be523329 Jul 23 04:01:39.081918 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:39.081887 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sqh2r" event={"ID":"12c78b78-909c-46b3-8c5c-b69ee1437002","Type":"ContainerStarted","Data":"9aab77bf4d1e29c7c34750ea3bccdafe6f2730b5bfc1c1eb598994f1be523329"} Jul 23 04:01:39.082488 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:39.082460 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vb9vz"] Jul 23 04:01:39.086015 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:39.085999 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vb9vz" Jul 23 04:01:39.097584 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:39.097557 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vb9vz"] Jul 23 04:01:39.219868 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:39.219835 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d344803-fa33-4428-b5e6-2accaa3a9637-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vb9vz\" (UID: \"7d344803-fa33-4428-b5e6-2accaa3a9637\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vb9vz" Jul 23 04:01:39.219868 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:39.219869 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d344803-fa33-4428-b5e6-2accaa3a9637-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vb9vz\" (UID: \"7d344803-fa33-4428-b5e6-2accaa3a9637\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vb9vz" Jul 23 04:01:39.220039 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:39.219939 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fwxk\" (UniqueName: \"kubernetes.io/projected/7d344803-fa33-4428-b5e6-2accaa3a9637-kube-api-access-7fwxk\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vb9vz\" (UID: \"7d344803-fa33-4428-b5e6-2accaa3a9637\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vb9vz" Jul 23 04:01:39.320771 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:39.320736 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fwxk\" (UniqueName: \"kubernetes.io/projected/7d344803-fa33-4428-b5e6-2accaa3a9637-kube-api-access-7fwxk\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vb9vz\" (UID: \"7d344803-fa33-4428-b5e6-2accaa3a9637\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vb9vz" Jul 23 04:01:39.320928 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:39.320836 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d344803-fa33-4428-b5e6-2accaa3a9637-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vb9vz\" (UID: \"7d344803-fa33-4428-b5e6-2accaa3a9637\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vb9vz" Jul 23 04:01:39.320928 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:39.320864 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d344803-fa33-4428-b5e6-2accaa3a9637-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vb9vz\" (UID: \"7d344803-fa33-4428-b5e6-2accaa3a9637\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vb9vz" Jul 23 04:01:39.321211 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:39.321193 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d344803-fa33-4428-b5e6-2accaa3a9637-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vb9vz\" (UID: \"7d344803-fa33-4428-b5e6-2accaa3a9637\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vb9vz" Jul 23 04:01:39.321248 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:39.321221 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d344803-fa33-4428-b5e6-2accaa3a9637-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vb9vz\" (UID: \"7d344803-fa33-4428-b5e6-2accaa3a9637\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vb9vz" Jul 23 04:01:39.329685 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:39.329662 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fwxk\" (UniqueName: \"kubernetes.io/projected/7d344803-fa33-4428-b5e6-2accaa3a9637-kube-api-access-7fwxk\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vb9vz\" (UID: \"7d344803-fa33-4428-b5e6-2accaa3a9637\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vb9vz" Jul 23 04:01:39.391998 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:39.391947 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r7lhw"] Jul 23 04:01:39.395597 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:39.395582 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r7lhw" Jul 23 04:01:39.397720 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:39.397703 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vb9vz" Jul 23 04:01:39.406169 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:39.406114 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r7lhw"] Jul 23 04:01:39.522309 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:39.522278 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/06ed2906-f99f-4fb8-9359-f6258a7f44af-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r7lhw\" (UID: \"06ed2906-f99f-4fb8-9359-f6258a7f44af\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r7lhw" Jul 23 04:01:39.522488 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:39.522317 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/06ed2906-f99f-4fb8-9359-f6258a7f44af-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r7lhw\" (UID: \"06ed2906-f99f-4fb8-9359-f6258a7f44af\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r7lhw" Jul 23 04:01:39.522488 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:39.522347 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl7q5\" (UniqueName: \"kubernetes.io/projected/06ed2906-f99f-4fb8-9359-f6258a7f44af-kube-api-access-bl7q5\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r7lhw\" (UID: \"06ed2906-f99f-4fb8-9359-f6258a7f44af\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r7lhw" Jul 23 04:01:39.528997 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:39.528836 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vb9vz"] Jul 23 04:01:39.531025 ip-10-0-140-6 kubenswrapper[2567]: W0723 04:01:39.531002 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d344803_fa33_4428_b5e6_2accaa3a9637.slice/crio-892c47bf049c8b0124283b2d9f801270a4fdf41f3199770db9bb0dfced1acb23 WatchSource:0}: Error finding container 892c47bf049c8b0124283b2d9f801270a4fdf41f3199770db9bb0dfced1acb23: Status 404 returned error can't find the container with id 892c47bf049c8b0124283b2d9f801270a4fdf41f3199770db9bb0dfced1acb23 Jul 23 04:01:39.623042 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:39.623019 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/06ed2906-f99f-4fb8-9359-f6258a7f44af-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r7lhw\" (UID: \"06ed2906-f99f-4fb8-9359-f6258a7f44af\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r7lhw" Jul 23 04:01:39.623455 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:39.623076 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/06ed2906-f99f-4fb8-9359-f6258a7f44af-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r7lhw\" (UID: \"06ed2906-f99f-4fb8-9359-f6258a7f44af\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r7lhw" Jul 23 04:01:39.623455 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:39.623244 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl7q5\" (UniqueName: \"kubernetes.io/projected/06ed2906-f99f-4fb8-9359-f6258a7f44af-kube-api-access-bl7q5\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r7lhw\" (UID: \"06ed2906-f99f-4fb8-9359-f6258a7f44af\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r7lhw" Jul 23 04:01:39.623626 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:39.623468 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/06ed2906-f99f-4fb8-9359-f6258a7f44af-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r7lhw\" (UID: \"06ed2906-f99f-4fb8-9359-f6258a7f44af\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r7lhw" Jul 23 04:01:39.623626 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:39.623468 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/06ed2906-f99f-4fb8-9359-f6258a7f44af-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r7lhw\" (UID: \"06ed2906-f99f-4fb8-9359-f6258a7f44af\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r7lhw" Jul 23 04:01:39.632152 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:39.632128 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl7q5\" (UniqueName: \"kubernetes.io/projected/06ed2906-f99f-4fb8-9359-f6258a7f44af-kube-api-access-bl7q5\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r7lhw\" (UID: \"06ed2906-f99f-4fb8-9359-f6258a7f44af\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r7lhw" Jul 23 04:01:39.704399 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:39.704372 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r7lhw" Jul 23 04:01:39.790131 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:39.790102 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fq5lh"] Jul 23 04:01:39.793803 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:39.793780 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fq5lh" Jul 23 04:01:39.804272 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:39.804246 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fq5lh"] Jul 23 04:01:39.836191 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:39.836164 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r7lhw"] Jul 23 04:01:39.841168 ip-10-0-140-6 kubenswrapper[2567]: W0723 04:01:39.841139 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06ed2906_f99f_4fb8_9359_f6258a7f44af.slice/crio-54908605cecfcc7c3e6f02c2ebbf23e72779a1a52b17599f13103ed227465b9f WatchSource:0}: Error finding container 54908605cecfcc7c3e6f02c2ebbf23e72779a1a52b17599f13103ed227465b9f: Status 404 returned error can't find the container with id 54908605cecfcc7c3e6f02c2ebbf23e72779a1a52b17599f13103ed227465b9f Jul 23 04:01:39.926415 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:39.926385 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qknrm\" (UniqueName: \"kubernetes.io/projected/1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b-kube-api-access-qknrm\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fq5lh\" (UID: \"1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fq5lh" Jul 23 04:01:39.926577 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:39.926440 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fq5lh\" (UID: \"1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fq5lh" Jul 23 04:01:39.926577 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:39.926541 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fq5lh\" (UID: \"1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fq5lh" Jul 23 04:01:40.027396 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:40.027323 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fq5lh\" (UID: \"1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fq5lh" Jul 23 04:01:40.027548 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:40.027403 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qknrm\" (UniqueName: \"kubernetes.io/projected/1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b-kube-api-access-qknrm\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fq5lh\" (UID: \"1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fq5lh" Jul 23 04:01:40.027548 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:40.027456 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fq5lh\" (UID: \"1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fq5lh" Jul 23 04:01:40.027744 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:40.027724 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fq5lh\" (UID: \"1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fq5lh" Jul 23 04:01:40.027817 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:40.027798 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fq5lh\" (UID: \"1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fq5lh" Jul 23 04:01:40.036421 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:40.036401 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qknrm\" (UniqueName: \"kubernetes.io/projected/1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b-kube-api-access-qknrm\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fq5lh\" (UID: \"1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fq5lh" Jul 23 04:01:40.087656 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:40.087630 2567 generic.go:356] "Generic (PLEG): container finished" podID="7d344803-fa33-4428-b5e6-2accaa3a9637" containerID="acfff3ecbad77e2c7a8b899235439d23a2784400324d52d1ca9c001502c86ed6" exitCode=0 Jul 23 04:01:40.087778 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:40.087707 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vb9vz" event={"ID":"7d344803-fa33-4428-b5e6-2accaa3a9637","Type":"ContainerDied","Data":"acfff3ecbad77e2c7a8b899235439d23a2784400324d52d1ca9c001502c86ed6"} Jul 23 04:01:40.087778 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:40.087749 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vb9vz" event={"ID":"7d344803-fa33-4428-b5e6-2accaa3a9637","Type":"ContainerStarted","Data":"892c47bf049c8b0124283b2d9f801270a4fdf41f3199770db9bb0dfced1acb23"} Jul 23 04:01:40.089277 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:40.089247 2567 generic.go:356] "Generic (PLEG): container finished" podID="06ed2906-f99f-4fb8-9359-f6258a7f44af" containerID="4e30ea7583647035f36d79e064c16e11bbba709ddd3d8d599a6efaf1c9dc96fe" exitCode=0 Jul 23 04:01:40.089354 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:40.089331 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r7lhw" event={"ID":"06ed2906-f99f-4fb8-9359-f6258a7f44af","Type":"ContainerDied","Data":"4e30ea7583647035f36d79e064c16e11bbba709ddd3d8d599a6efaf1c9dc96fe"} Jul 23 04:01:40.089400 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:40.089373 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r7lhw" event={"ID":"06ed2906-f99f-4fb8-9359-f6258a7f44af","Type":"ContainerStarted","Data":"54908605cecfcc7c3e6f02c2ebbf23e72779a1a52b17599f13103ed227465b9f"} Jul 23 04:01:40.090882 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:40.090864 2567 generic.go:356] "Generic (PLEG): container finished" podID="12c78b78-909c-46b3-8c5c-b69ee1437002" containerID="04bdaef3449d76c0abeeea742cf6c8c3504436c4c59d9aabe90a9f463323de6e" exitCode=0 Jul 23 04:01:40.090958 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:40.090922 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sqh2r" event={"ID":"12c78b78-909c-46b3-8c5c-b69ee1437002","Type":"ContainerDied","Data":"04bdaef3449d76c0abeeea742cf6c8c3504436c4c59d9aabe90a9f463323de6e"} Jul 23 04:01:40.106838 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:40.106819 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fq5lh" Jul 23 04:01:40.239576 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:40.239555 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fq5lh"] Jul 23 04:01:40.240966 ip-10-0-140-6 kubenswrapper[2567]: W0723 04:01:40.240945 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1799d3f5_a3e3_4fb5_ab9e_8be8b4bfff2b.slice/crio-f81c6b41236a4fad3b622b376a8074c6418c5000b95074e0860960ddf709249f WatchSource:0}: Error finding container f81c6b41236a4fad3b622b376a8074c6418c5000b95074e0860960ddf709249f: Status 404 returned error can't find the container with id f81c6b41236a4fad3b622b376a8074c6418c5000b95074e0860960ddf709249f Jul 23 04:01:41.097325 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:41.097291 2567 generic.go:356] "Generic (PLEG): container finished" podID="1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b" containerID="ae4e11db98b5f1405f546952100dfb29057656b79d668cc2546ec8887fc1b33a" exitCode=0 Jul 23 04:01:41.097701 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:41.097349 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fq5lh" event={"ID":"1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b","Type":"ContainerDied","Data":"ae4e11db98b5f1405f546952100dfb29057656b79d668cc2546ec8887fc1b33a"} Jul 23 04:01:41.097701 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:41.097413 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fq5lh" event={"ID":"1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b","Type":"ContainerStarted","Data":"f81c6b41236a4fad3b622b376a8074c6418c5000b95074e0860960ddf709249f"} Jul 23 04:01:42.105018 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:42.104932 2567 generic.go:356] "Generic (PLEG): container finished" podID="7d344803-fa33-4428-b5e6-2accaa3a9637" containerID="4af63a31f2b989dd47316e0d265bdec6c88656e39015005a8e2935ff8866cf97" exitCode=0 Jul 23 04:01:42.105397 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:42.105024 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vb9vz" event={"ID":"7d344803-fa33-4428-b5e6-2accaa3a9637","Type":"ContainerDied","Data":"4af63a31f2b989dd47316e0d265bdec6c88656e39015005a8e2935ff8866cf97"} Jul 23 04:01:42.106923 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:42.106902 2567 generic.go:356] "Generic (PLEG): container finished" podID="06ed2906-f99f-4fb8-9359-f6258a7f44af" containerID="2cf4ecddd4463896dc6a20ec64f5858beea0bbc2a95c26b84f01197019109a4c" exitCode=0 Jul 23 04:01:42.107026 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:42.106974 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r7lhw" event={"ID":"06ed2906-f99f-4fb8-9359-f6258a7f44af","Type":"ContainerDied","Data":"2cf4ecddd4463896dc6a20ec64f5858beea0bbc2a95c26b84f01197019109a4c"} Jul 23 04:01:42.108783 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:42.108760 2567 generic.go:356] "Generic (PLEG): container finished" podID="1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b" containerID="335029fa9e39890ed3e80a62d174985a2f208c4875002c0ad0193a7b0070df9a" exitCode=0 Jul 23 04:01:42.108884 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:42.108833 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fq5lh" event={"ID":"1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b","Type":"ContainerDied","Data":"335029fa9e39890ed3e80a62d174985a2f208c4875002c0ad0193a7b0070df9a"} Jul 23 04:01:42.110835 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:42.110817 2567 generic.go:356] "Generic (PLEG): container finished" podID="12c78b78-909c-46b3-8c5c-b69ee1437002" containerID="687c8cd10e0d2804904471e67ec753153886a8b7002c0b935faca1a6c2dca82b" exitCode=0 Jul 23 04:01:42.110904 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:42.110864 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sqh2r" event={"ID":"12c78b78-909c-46b3-8c5c-b69ee1437002","Type":"ContainerDied","Data":"687c8cd10e0d2804904471e67ec753153886a8b7002c0b935faca1a6c2dca82b"} Jul 23 04:01:43.117496 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:43.117393 2567 generic.go:356] "Generic (PLEG): container finished" podID="7d344803-fa33-4428-b5e6-2accaa3a9637" containerID="87e3bd45c89de41888a74028ee4b540a5ab99352c98084223dbf47de6d259fa7" exitCode=0 Jul 23 04:01:43.117496 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:43.117467 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vb9vz" event={"ID":"7d344803-fa33-4428-b5e6-2accaa3a9637","Type":"ContainerDied","Data":"87e3bd45c89de41888a74028ee4b540a5ab99352c98084223dbf47de6d259fa7"} Jul 23 04:01:43.119493 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:43.119462 2567 generic.go:356] "Generic (PLEG): container finished" podID="06ed2906-f99f-4fb8-9359-f6258a7f44af" containerID="fc804dd510506950cfd15fda045be090ebce6ae77dee5e1163baf122172e8bd7" exitCode=0 Jul 23 04:01:43.119626 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:43.119543 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r7lhw" event={"ID":"06ed2906-f99f-4fb8-9359-f6258a7f44af","Type":"ContainerDied","Data":"fc804dd510506950cfd15fda045be090ebce6ae77dee5e1163baf122172e8bd7"} Jul 23 04:01:43.121536 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:43.121513 2567 generic.go:356] "Generic (PLEG): container finished" podID="1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b" containerID="c7170c2b45b894994c9d138ae6c8643f6c7b80b10003cc0db01bbe570e37d5fd" exitCode=0 Jul 23 04:01:43.121665 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:43.121593 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fq5lh" event={"ID":"1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b","Type":"ContainerDied","Data":"c7170c2b45b894994c9d138ae6c8643f6c7b80b10003cc0db01bbe570e37d5fd"} Jul 23 04:01:43.123626 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:43.123606 2567 generic.go:356] "Generic (PLEG): container finished" podID="12c78b78-909c-46b3-8c5c-b69ee1437002" containerID="a1f8e9e6bb43b879f988b960572b290aa54fb1f7f16b540bb31c6e8a7ffed0d1" exitCode=0 Jul 23 04:01:43.123735 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:43.123668 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sqh2r" event={"ID":"12c78b78-909c-46b3-8c5c-b69ee1437002","Type":"ContainerDied","Data":"a1f8e9e6bb43b879f988b960572b290aa54fb1f7f16b540bb31c6e8a7ffed0d1"} Jul 23 04:01:44.291705 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:44.291679 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r7lhw" Jul 23 04:01:44.320786 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:44.320764 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sqh2r" Jul 23 04:01:44.326409 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:44.326385 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vb9vz" Jul 23 04:01:44.336478 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:44.336460 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fq5lh" Jul 23 04:01:44.462646 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:44.462615 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12c78b78-909c-46b3-8c5c-b69ee1437002-util\") pod \"12c78b78-909c-46b3-8c5c-b69ee1437002\" (UID: \"12c78b78-909c-46b3-8c5c-b69ee1437002\") " Jul 23 04:01:44.462646 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:44.462647 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bl7q5\" (UniqueName: \"kubernetes.io/projected/06ed2906-f99f-4fb8-9359-f6258a7f44af-kube-api-access-bl7q5\") pod \"06ed2906-f99f-4fb8-9359-f6258a7f44af\" (UID: \"06ed2906-f99f-4fb8-9359-f6258a7f44af\") " Jul 23 04:01:44.462878 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:44.462678 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d344803-fa33-4428-b5e6-2accaa3a9637-util\") pod \"7d344803-fa33-4428-b5e6-2accaa3a9637\" (UID: \"7d344803-fa33-4428-b5e6-2accaa3a9637\") " Jul 23 04:01:44.462878 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:44.462699 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b-bundle\") pod \"1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b\" (UID: \"1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b\") " Jul 23 04:01:44.462878 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:44.462716 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/06ed2906-f99f-4fb8-9359-f6258a7f44af-util\") pod \"06ed2906-f99f-4fb8-9359-f6258a7f44af\" (UID: \"06ed2906-f99f-4fb8-9359-f6258a7f44af\") " Jul 23 04:01:44.462878 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:44.462747 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-749cm\" (UniqueName: \"kubernetes.io/projected/12c78b78-909c-46b3-8c5c-b69ee1437002-kube-api-access-749cm\") pod \"12c78b78-909c-46b3-8c5c-b69ee1437002\" (UID: \"12c78b78-909c-46b3-8c5c-b69ee1437002\") " Jul 23 04:01:44.462878 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:44.462772 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d344803-fa33-4428-b5e6-2accaa3a9637-bundle\") pod \"7d344803-fa33-4428-b5e6-2accaa3a9637\" (UID: \"7d344803-fa33-4428-b5e6-2accaa3a9637\") " Jul 23 04:01:44.462878 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:44.462804 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qknrm\" (UniqueName: \"kubernetes.io/projected/1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b-kube-api-access-qknrm\") pod \"1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b\" (UID: \"1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b\") " Jul 23 04:01:44.462878 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:44.462836 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/06ed2906-f99f-4fb8-9359-f6258a7f44af-bundle\") pod \"06ed2906-f99f-4fb8-9359-f6258a7f44af\" (UID: \"06ed2906-f99f-4fb8-9359-f6258a7f44af\") " Jul 23 04:01:44.462878 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:44.462879 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b-util\") pod \"1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b\" (UID: \"1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b\") " Jul 23 04:01:44.463402 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:44.462919 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12c78b78-909c-46b3-8c5c-b69ee1437002-bundle\") pod \"12c78b78-909c-46b3-8c5c-b69ee1437002\" (UID: \"12c78b78-909c-46b3-8c5c-b69ee1437002\") " Jul 23 04:01:44.463402 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:44.462963 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7fwxk\" (UniqueName: \"kubernetes.io/projected/7d344803-fa33-4428-b5e6-2accaa3a9637-kube-api-access-7fwxk\") pod \"7d344803-fa33-4428-b5e6-2accaa3a9637\" (UID: \"7d344803-fa33-4428-b5e6-2accaa3a9637\") " Jul 23 04:01:44.463402 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:44.463282 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b-bundle" (OuterVolumeSpecName: "bundle") pod "1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b" (UID: "1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 04:01:44.464818 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:44.463999 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d344803-fa33-4428-b5e6-2accaa3a9637-bundle" (OuterVolumeSpecName: "bundle") pod "7d344803-fa33-4428-b5e6-2accaa3a9637" (UID: "7d344803-fa33-4428-b5e6-2accaa3a9637"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 04:01:44.464818 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:44.464583 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12c78b78-909c-46b3-8c5c-b69ee1437002-bundle" (OuterVolumeSpecName: "bundle") pod "12c78b78-909c-46b3-8c5c-b69ee1437002" (UID: "12c78b78-909c-46b3-8c5c-b69ee1437002"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 04:01:44.465620 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:44.465587 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06ed2906-f99f-4fb8-9359-f6258a7f44af-bundle" (OuterVolumeSpecName: "bundle") pod "06ed2906-f99f-4fb8-9359-f6258a7f44af" (UID: "06ed2906-f99f-4fb8-9359-f6258a7f44af"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 04:01:44.465620 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:44.465607 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06ed2906-f99f-4fb8-9359-f6258a7f44af-kube-api-access-bl7q5" (OuterVolumeSpecName: "kube-api-access-bl7q5") pod "06ed2906-f99f-4fb8-9359-f6258a7f44af" (UID: "06ed2906-f99f-4fb8-9359-f6258a7f44af"). InnerVolumeSpecName "kube-api-access-bl7q5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 04:01:44.465783 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:44.465677 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12c78b78-909c-46b3-8c5c-b69ee1437002-kube-api-access-749cm" (OuterVolumeSpecName: "kube-api-access-749cm") pod "12c78b78-909c-46b3-8c5c-b69ee1437002" (UID: "12c78b78-909c-46b3-8c5c-b69ee1437002"). InnerVolumeSpecName "kube-api-access-749cm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 04:01:44.465837 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:44.465815 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d344803-fa33-4428-b5e6-2accaa3a9637-kube-api-access-7fwxk" (OuterVolumeSpecName: "kube-api-access-7fwxk") pod "7d344803-fa33-4428-b5e6-2accaa3a9637" (UID: "7d344803-fa33-4428-b5e6-2accaa3a9637"). InnerVolumeSpecName "kube-api-access-7fwxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 04:01:44.467411 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:44.467381 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b-kube-api-access-qknrm" (OuterVolumeSpecName: "kube-api-access-qknrm") pod "1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b" (UID: "1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b"). InnerVolumeSpecName "kube-api-access-qknrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 04:01:44.468767 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:44.468736 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12c78b78-909c-46b3-8c5c-b69ee1437002-util" (OuterVolumeSpecName: "util") pod "12c78b78-909c-46b3-8c5c-b69ee1437002" (UID: "12c78b78-909c-46b3-8c5c-b69ee1437002"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 04:01:44.469644 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:44.469624 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d344803-fa33-4428-b5e6-2accaa3a9637-util" (OuterVolumeSpecName: "util") pod "7d344803-fa33-4428-b5e6-2accaa3a9637" (UID: "7d344803-fa33-4428-b5e6-2accaa3a9637"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 04:01:44.469919 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:44.469901 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b-util" (OuterVolumeSpecName: "util") pod "1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b" (UID: "1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 04:01:44.470805 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:44.470782 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06ed2906-f99f-4fb8-9359-f6258a7f44af-util" (OuterVolumeSpecName: "util") pod "06ed2906-f99f-4fb8-9359-f6258a7f44af" (UID: "06ed2906-f99f-4fb8-9359-f6258a7f44af"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 04:01:44.564333 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:44.564308 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b-bundle\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:01:44.564333 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:44.564334 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/06ed2906-f99f-4fb8-9359-f6258a7f44af-util\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:01:44.564517 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:44.564347 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-749cm\" (UniqueName: \"kubernetes.io/projected/12c78b78-909c-46b3-8c5c-b69ee1437002-kube-api-access-749cm\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:01:44.564517 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:44.564361 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d344803-fa33-4428-b5e6-2accaa3a9637-bundle\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:01:44.564517 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:44.564373 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qknrm\" (UniqueName: \"kubernetes.io/projected/1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b-kube-api-access-qknrm\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:01:44.564517 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:44.564384 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/06ed2906-f99f-4fb8-9359-f6258a7f44af-bundle\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:01:44.564517 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:44.564395 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b-util\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:01:44.564517 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:44.564405 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12c78b78-909c-46b3-8c5c-b69ee1437002-bundle\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:01:44.564517 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:44.564417 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fwxk\" (UniqueName: \"kubernetes.io/projected/7d344803-fa33-4428-b5e6-2accaa3a9637-kube-api-access-7fwxk\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:01:44.564517 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:44.564446 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12c78b78-909c-46b3-8c5c-b69ee1437002-util\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:01:44.564517 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:44.564458 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl7q5\" (UniqueName: \"kubernetes.io/projected/06ed2906-f99f-4fb8-9359-f6258a7f44af-kube-api-access-bl7q5\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:01:44.564517 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:44.564470 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d344803-fa33-4428-b5e6-2accaa3a9637-util\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:01:45.135389 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:45.135359 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vb9vz" Jul 23 04:01:45.135614 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:45.135357 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vb9vz" event={"ID":"7d344803-fa33-4428-b5e6-2accaa3a9637","Type":"ContainerDied","Data":"892c47bf049c8b0124283b2d9f801270a4fdf41f3199770db9bb0dfced1acb23"} Jul 23 04:01:45.135614 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:45.135486 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="892c47bf049c8b0124283b2d9f801270a4fdf41f3199770db9bb0dfced1acb23" Jul 23 04:01:45.137359 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:45.137342 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r7lhw" Jul 23 04:01:45.137584 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:45.137345 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r7lhw" event={"ID":"06ed2906-f99f-4fb8-9359-f6258a7f44af","Type":"ContainerDied","Data":"54908605cecfcc7c3e6f02c2ebbf23e72779a1a52b17599f13103ed227465b9f"} Jul 23 04:01:45.137673 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:45.137624 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54908605cecfcc7c3e6f02c2ebbf23e72779a1a52b17599f13103ed227465b9f" Jul 23 04:01:45.139334 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:45.139311 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fq5lh" Jul 23 04:01:45.139458 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:45.139348 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fq5lh" event={"ID":"1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b","Type":"ContainerDied","Data":"f81c6b41236a4fad3b622b376a8074c6418c5000b95074e0860960ddf709249f"} Jul 23 04:01:45.139458 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:45.139390 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f81c6b41236a4fad3b622b376a8074c6418c5000b95074e0860960ddf709249f" Jul 23 04:01:45.141590 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:45.141572 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sqh2r" event={"ID":"12c78b78-909c-46b3-8c5c-b69ee1437002","Type":"ContainerDied","Data":"9aab77bf4d1e29c7c34750ea3bccdafe6f2730b5bfc1c1eb598994f1be523329"} Jul 23 04:01:45.141685 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:45.141607 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9aab77bf4d1e29c7c34750ea3bccdafe6f2730b5bfc1c1eb598994f1be523329" Jul 23 04:01:45.141685 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:01:45.141609 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sqh2r" Jul 23 04:02:01.810882 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:01.810848 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-56h6j"] Jul 23 04:02:01.811285 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:01.811129 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06ed2906-f99f-4fb8-9359-f6258a7f44af" containerName="extract" Jul 23 04:02:01.811285 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:01.811139 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="06ed2906-f99f-4fb8-9359-f6258a7f44af" containerName="extract" Jul 23 04:02:01.811285 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:01.811149 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b" containerName="pull" Jul 23 04:02:01.811285 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:01.811154 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b" containerName="pull" Jul 23 04:02:01.811285 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:01.811163 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12c78b78-909c-46b3-8c5c-b69ee1437002" containerName="extract" Jul 23 04:02:01.811285 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:01.811169 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="12c78b78-909c-46b3-8c5c-b69ee1437002" containerName="extract" Jul 23 04:02:01.811285 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:01.811176 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12c78b78-909c-46b3-8c5c-b69ee1437002" containerName="pull" Jul 23 04:02:01.811285 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:01.811182 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="12c78b78-909c-46b3-8c5c-b69ee1437002" containerName="pull" Jul 23 04:02:01.811285 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:01.811188 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06ed2906-f99f-4fb8-9359-f6258a7f44af" containerName="util" Jul 23 04:02:01.811285 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:01.811195 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="06ed2906-f99f-4fb8-9359-f6258a7f44af" containerName="util" Jul 23 04:02:01.811285 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:01.811203 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d344803-fa33-4428-b5e6-2accaa3a9637" containerName="extract" Jul 23 04:02:01.811285 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:01.811208 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d344803-fa33-4428-b5e6-2accaa3a9637" containerName="extract" Jul 23 04:02:01.811285 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:01.811217 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12c78b78-909c-46b3-8c5c-b69ee1437002" containerName="util" Jul 23 04:02:01.811285 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:01.811222 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="12c78b78-909c-46b3-8c5c-b69ee1437002" containerName="util" Jul 23 04:02:01.811285 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:01.811228 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06ed2906-f99f-4fb8-9359-f6258a7f44af" containerName="pull" Jul 23 04:02:01.811285 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:01.811233 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="06ed2906-f99f-4fb8-9359-f6258a7f44af" containerName="pull" Jul 23 04:02:01.811285 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:01.811242 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b" containerName="util" Jul 23 04:02:01.811285 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:01.811248 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b" containerName="util" Jul 23 04:02:01.811285 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:01.811258 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d344803-fa33-4428-b5e6-2accaa3a9637" containerName="util" Jul 23 04:02:01.811285 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:01.811265 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d344803-fa33-4428-b5e6-2accaa3a9637" containerName="util" Jul 23 04:02:01.811285 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:01.811273 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b" containerName="extract" Jul 23 04:02:01.811285 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:01.811280 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b" containerName="extract" Jul 23 04:02:01.811285 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:01.811290 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d344803-fa33-4428-b5e6-2accaa3a9637" containerName="pull" Jul 23 04:02:01.812074 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:01.811298 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d344803-fa33-4428-b5e6-2accaa3a9637" containerName="pull" Jul 23 04:02:01.812074 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:01.811351 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d344803-fa33-4428-b5e6-2accaa3a9637" containerName="extract" Jul 23 04:02:01.812074 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:01.811358 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="06ed2906-f99f-4fb8-9359-f6258a7f44af" containerName="extract" Jul 23 04:02:01.812074 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:01.811365 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b" containerName="extract" Jul 23 04:02:01.812074 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:01.811371 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="12c78b78-909c-46b3-8c5c-b69ee1437002" containerName="extract" Jul 23 04:02:01.816153 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:01.816136 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-56h6j" Jul 23 04:02:01.819770 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:01.819749 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 23 04:02:01.820795 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:01.820773 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-pklsv\"" Jul 23 04:02:01.820896 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:01.820784 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 23 04:02:01.836319 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:01.836298 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-56h6j"] Jul 23 04:02:01.899630 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:01.899593 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86xfz\" (UniqueName: \"kubernetes.io/projected/d9ee86b7-6d1a-4b01-80d8-6054cbc019dc-kube-api-access-86xfz\") pod \"authorino-operator-75cc8d5cb7-56h6j\" (UID: \"d9ee86b7-6d1a-4b01-80d8-6054cbc019dc\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-56h6j" Jul 23 04:02:02.000713 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:02.000674 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86xfz\" (UniqueName: \"kubernetes.io/projected/d9ee86b7-6d1a-4b01-80d8-6054cbc019dc-kube-api-access-86xfz\") pod \"authorino-operator-75cc8d5cb7-56h6j\" (UID: \"d9ee86b7-6d1a-4b01-80d8-6054cbc019dc\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-56h6j" Jul 23 04:02:02.017148 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:02.017124 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86xfz\" (UniqueName: \"kubernetes.io/projected/d9ee86b7-6d1a-4b01-80d8-6054cbc019dc-kube-api-access-86xfz\") pod \"authorino-operator-75cc8d5cb7-56h6j\" (UID: \"d9ee86b7-6d1a-4b01-80d8-6054cbc019dc\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-56h6j" Jul 23 04:02:02.125857 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:02.125774 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-56h6j" Jul 23 04:02:02.266937 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:02.266908 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-56h6j"] Jul 23 04:02:02.270362 ip-10-0-140-6 kubenswrapper[2567]: W0723 04:02:02.270339 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9ee86b7_6d1a_4b01_80d8_6054cbc019dc.slice/crio-3c483fceeb6bdaa8e37feb66b766bc09266a8db78a23199303d3603b56f72f4d WatchSource:0}: Error finding container 3c483fceeb6bdaa8e37feb66b766bc09266a8db78a23199303d3603b56f72f4d: Status 404 returned error can't find the container with id 3c483fceeb6bdaa8e37feb66b766bc09266a8db78a23199303d3603b56f72f4d Jul 23 04:02:03.233835 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:03.233795 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-56h6j" event={"ID":"d9ee86b7-6d1a-4b01-80d8-6054cbc019dc","Type":"ContainerStarted","Data":"3c483fceeb6bdaa8e37feb66b766bc09266a8db78a23199303d3603b56f72f4d"} Jul 23 04:02:05.244934 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:05.244893 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-56h6j" event={"ID":"d9ee86b7-6d1a-4b01-80d8-6054cbc019dc","Type":"ContainerStarted","Data":"205b1efc5a4dd18689274c1b1a4dfeaa3bbd8f58c9d3f4c845c21adb023f7f80"} Jul 23 04:02:05.245279 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:05.244974 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-56h6j" Jul 23 04:02:05.274509 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:05.274469 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-56h6j" podStartSLOduration=1.999392372 podStartE2EDuration="4.274455664s" totalImagesPullingTime="2.275063292s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 04:02:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 04:02:05.271775776 +0000 UTC m=+405.177080059" watchObservedRunningTime="2026-07-23 04:02:05.274455664 +0000 UTC m=+405.179759950" Jul 23 04:02:16.251836 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:16.251803 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-56h6j" Jul 23 04:02:25.697708 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:25.697676 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6qglq"] Jul 23 04:02:25.701159 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:25.701137 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6qglq" Jul 23 04:02:25.703983 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:25.703943 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-fsm4b\"" Jul 23 04:02:25.714054 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:25.714029 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6qglq"] Jul 23 04:02:25.797284 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:25.797259 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m97pd\" (UniqueName: \"kubernetes.io/projected/040429aa-732f-498d-bee1-3ee87e763646-kube-api-access-m97pd\") pod \"limitador-operator-controller-manager-859d7b78bd-6qglq\" (UID: \"040429aa-732f-498d-bee1-3ee87e763646\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6qglq" Jul 23 04:02:25.898467 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:25.898417 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m97pd\" (UniqueName: \"kubernetes.io/projected/040429aa-732f-498d-bee1-3ee87e763646-kube-api-access-m97pd\") pod \"limitador-operator-controller-manager-859d7b78bd-6qglq\" (UID: \"040429aa-732f-498d-bee1-3ee87e763646\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6qglq" Jul 23 04:02:25.911271 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:25.911239 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m97pd\" (UniqueName: \"kubernetes.io/projected/040429aa-732f-498d-bee1-3ee87e763646-kube-api-access-m97pd\") pod \"limitador-operator-controller-manager-859d7b78bd-6qglq\" (UID: \"040429aa-732f-498d-bee1-3ee87e763646\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6qglq" Jul 23 04:02:26.012023 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:26.011955 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6qglq" Jul 23 04:02:26.148273 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:26.148247 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6qglq"] Jul 23 04:02:26.150378 ip-10-0-140-6 kubenswrapper[2567]: W0723 04:02:26.150354 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod040429aa_732f_498d_bee1_3ee87e763646.slice/crio-fee1106e07b5170a64f1d5b780aa978df29ed714b10980b7022073141d96eabb WatchSource:0}: Error finding container fee1106e07b5170a64f1d5b780aa978df29ed714b10980b7022073141d96eabb: Status 404 returned error can't find the container with id fee1106e07b5170a64f1d5b780aa978df29ed714b10980b7022073141d96eabb Jul 23 04:02:26.348804 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:26.348714 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6qglq" event={"ID":"040429aa-732f-498d-bee1-3ee87e763646","Type":"ContainerStarted","Data":"fee1106e07b5170a64f1d5b780aa978df29ed714b10980b7022073141d96eabb"} Jul 23 04:02:28.362592 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:28.362562 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6qglq" event={"ID":"040429aa-732f-498d-bee1-3ee87e763646","Type":"ContainerStarted","Data":"4eb40ae846279aaa638f59ee644980726c3ffa12307b558a0e1cde78082268d4"} Jul 23 04:02:28.363059 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:28.362683 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6qglq" Jul 23 04:02:28.419104 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:28.419054 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6qglq" podStartSLOduration=1.753089706 podStartE2EDuration="3.419041228s" totalImagesPullingTime="1.665951522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 04:02:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 04:02:28.417056485 +0000 UTC m=+428.322360776" watchObservedRunningTime="2026-07-23 04:02:28.419041228 +0000 UTC m=+428.324345520" Jul 23 04:02:39.369931 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:02:39.369898 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6qglq" Jul 23 04:03:15.227407 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:15.227372 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-5wwsm"] Jul 23 04:03:15.230732 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:15.230713 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-5wwsm" Jul 23 04:03:15.233761 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:15.233736 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-mrzjq\"" Jul 23 04:03:15.239542 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:15.239510 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-5wwsm"] Jul 23 04:03:15.283855 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:15.283828 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cj6c\" (UniqueName: \"kubernetes.io/projected/c37feaa4-5dcc-4ac9-9505-9eebb5f9196b-kube-api-access-7cj6c\") pod \"authorino-58cd955675-5wwsm\" (UID: \"c37feaa4-5dcc-4ac9-9505-9eebb5f9196b\") " pod="kuadrant-system/authorino-58cd955675-5wwsm" Jul 23 04:03:15.384709 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:15.384679 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cj6c\" (UniqueName: \"kubernetes.io/projected/c37feaa4-5dcc-4ac9-9505-9eebb5f9196b-kube-api-access-7cj6c\") pod \"authorino-58cd955675-5wwsm\" (UID: \"c37feaa4-5dcc-4ac9-9505-9eebb5f9196b\") " pod="kuadrant-system/authorino-58cd955675-5wwsm" Jul 23 04:03:15.394168 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:15.394140 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cj6c\" (UniqueName: \"kubernetes.io/projected/c37feaa4-5dcc-4ac9-9505-9eebb5f9196b-kube-api-access-7cj6c\") pod \"authorino-58cd955675-5wwsm\" (UID: \"c37feaa4-5dcc-4ac9-9505-9eebb5f9196b\") " pod="kuadrant-system/authorino-58cd955675-5wwsm" Jul 23 04:03:15.539471 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:15.539369 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-5wwsm" Jul 23 04:03:15.670288 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:15.670253 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-5wwsm"] Jul 23 04:03:15.672161 ip-10-0-140-6 kubenswrapper[2567]: W0723 04:03:15.672131 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc37feaa4_5dcc_4ac9_9505_9eebb5f9196b.slice/crio-94eb24b29640a9b2c350994b28df22f395b70597baf628a08b41121f39945e33 WatchSource:0}: Error finding container 94eb24b29640a9b2c350994b28df22f395b70597baf628a08b41121f39945e33: Status 404 returned error can't find the container with id 94eb24b29640a9b2c350994b28df22f395b70597baf628a08b41121f39945e33 Jul 23 04:03:16.586726 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:16.586697 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-5wwsm" event={"ID":"c37feaa4-5dcc-4ac9-9505-9eebb5f9196b","Type":"ContainerStarted","Data":"94eb24b29640a9b2c350994b28df22f395b70597baf628a08b41121f39945e33"} Jul 23 04:03:19.605415 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:19.605335 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-5wwsm" event={"ID":"c37feaa4-5dcc-4ac9-9505-9eebb5f9196b","Type":"ContainerStarted","Data":"8395e1042cf064c53edf39e748e4beb5667ede3e2a94f74901bcaae163570fa9"} Jul 23 04:03:19.623022 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:19.622974 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-5wwsm" podStartSLOduration=1.080174433 podStartE2EDuration="4.622959266s" totalImagesPullingTime="3.542784833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 04:03:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 04:03:19.621508987 +0000 UTC m=+479.526813281" watchObservedRunningTime="2026-07-23 04:03:19.622959266 +0000 UTC m=+479.528263558" Jul 23 04:03:20.144063 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:20.144028 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-5wwsm"] Jul 23 04:03:21.614476 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:21.614417 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-5wwsm" podUID="c37feaa4-5dcc-4ac9-9505-9eebb5f9196b" containerName="authorino" containerID="cri-o://8395e1042cf064c53edf39e748e4beb5667ede3e2a94f74901bcaae163570fa9" gracePeriod=30 Jul 23 04:03:21.859654 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:21.859631 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-5wwsm" Jul 23 04:03:21.942451 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:21.942404 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7cj6c\" (UniqueName: \"kubernetes.io/projected/c37feaa4-5dcc-4ac9-9505-9eebb5f9196b-kube-api-access-7cj6c\") pod \"c37feaa4-5dcc-4ac9-9505-9eebb5f9196b\" (UID: \"c37feaa4-5dcc-4ac9-9505-9eebb5f9196b\") " Jul 23 04:03:21.944459 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:21.944420 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c37feaa4-5dcc-4ac9-9505-9eebb5f9196b-kube-api-access-7cj6c" (OuterVolumeSpecName: "kube-api-access-7cj6c") pod "c37feaa4-5dcc-4ac9-9505-9eebb5f9196b" (UID: "c37feaa4-5dcc-4ac9-9505-9eebb5f9196b"). InnerVolumeSpecName "kube-api-access-7cj6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 04:03:22.043489 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:22.043459 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cj6c\" (UniqueName: \"kubernetes.io/projected/c37feaa4-5dcc-4ac9-9505-9eebb5f9196b-kube-api-access-7cj6c\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:03:22.619936 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:22.619908 2567 generic.go:356] "Generic (PLEG): container finished" podID="c37feaa4-5dcc-4ac9-9505-9eebb5f9196b" containerID="8395e1042cf064c53edf39e748e4beb5667ede3e2a94f74901bcaae163570fa9" exitCode=0 Jul 23 04:03:22.620320 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:22.619946 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-5wwsm" Jul 23 04:03:22.620320 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:22.620008 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-5wwsm" event={"ID":"c37feaa4-5dcc-4ac9-9505-9eebb5f9196b","Type":"ContainerDied","Data":"8395e1042cf064c53edf39e748e4beb5667ede3e2a94f74901bcaae163570fa9"} Jul 23 04:03:22.620320 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:22.620046 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-5wwsm" event={"ID":"c37feaa4-5dcc-4ac9-9505-9eebb5f9196b","Type":"ContainerDied","Data":"94eb24b29640a9b2c350994b28df22f395b70597baf628a08b41121f39945e33"} Jul 23 04:03:22.620320 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:22.620063 2567 scope.go:117] "RemoveContainer" containerID="8395e1042cf064c53edf39e748e4beb5667ede3e2a94f74901bcaae163570fa9" Jul 23 04:03:22.628597 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:22.628581 2567 scope.go:117] "RemoveContainer" containerID="8395e1042cf064c53edf39e748e4beb5667ede3e2a94f74901bcaae163570fa9" Jul 23 04:03:22.628850 ip-10-0-140-6 kubenswrapper[2567]: E0723 04:03:22.628833 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8395e1042cf064c53edf39e748e4beb5667ede3e2a94f74901bcaae163570fa9\": container with ID starting with 8395e1042cf064c53edf39e748e4beb5667ede3e2a94f74901bcaae163570fa9 not found: ID does not exist" containerID="8395e1042cf064c53edf39e748e4beb5667ede3e2a94f74901bcaae163570fa9" Jul 23 04:03:22.628910 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:22.628862 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8395e1042cf064c53edf39e748e4beb5667ede3e2a94f74901bcaae163570fa9"} err="failed to get container status \"8395e1042cf064c53edf39e748e4beb5667ede3e2a94f74901bcaae163570fa9\": rpc error: code = NotFound desc = could not find container \"8395e1042cf064c53edf39e748e4beb5667ede3e2a94f74901bcaae163570fa9\": container with ID starting with 8395e1042cf064c53edf39e748e4beb5667ede3e2a94f74901bcaae163570fa9 not found: ID does not exist" Jul 23 04:03:22.637498 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:22.637473 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-5wwsm"] Jul 23 04:03:22.641484 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:22.641463 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-5wwsm"] Jul 23 04:03:24.494157 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:24.494112 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c37feaa4-5dcc-4ac9-9505-9eebb5f9196b" path="/var/lib/kubelet/pods/c37feaa4-5dcc-4ac9-9505-9eebb5f9196b/volumes" Jul 23 04:03:42.888856 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:42.888823 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-djgf2"] Jul 23 04:03:42.889267 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:42.889123 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c37feaa4-5dcc-4ac9-9505-9eebb5f9196b" containerName="authorino" Jul 23 04:03:42.889267 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:42.889133 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c37feaa4-5dcc-4ac9-9505-9eebb5f9196b" containerName="authorino" Jul 23 04:03:42.889267 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:42.889185 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c37feaa4-5dcc-4ac9-9505-9eebb5f9196b" containerName="authorino" Jul 23 04:03:42.891927 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:42.891908 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-djgf2" Jul 23 04:03:42.894748 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:42.894730 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-mrzjq\"" Jul 23 04:03:42.905143 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:42.905119 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-djgf2"] Jul 23 04:03:43.022586 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:43.022548 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsjht\" (UniqueName: \"kubernetes.io/projected/cf169784-ec2a-46b6-881a-7006bf881c33-kube-api-access-fsjht\") pod \"authorino-5d4b67c446-djgf2\" (UID: \"cf169784-ec2a-46b6-881a-7006bf881c33\") " pod="kuadrant-system/authorino-5d4b67c446-djgf2" Jul 23 04:03:43.119448 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:43.119399 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-djgf2"] Jul 23 04:03:43.119662 ip-10-0-140-6 kubenswrapper[2567]: E0723 04:03:43.119641 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-fsjht], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-djgf2" podUID="cf169784-ec2a-46b6-881a-7006bf881c33" Jul 23 04:03:43.123746 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:43.123713 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsjht\" (UniqueName: \"kubernetes.io/projected/cf169784-ec2a-46b6-881a-7006bf881c33-kube-api-access-fsjht\") pod \"authorino-5d4b67c446-djgf2\" (UID: \"cf169784-ec2a-46b6-881a-7006bf881c33\") " pod="kuadrant-system/authorino-5d4b67c446-djgf2" Jul 23 04:03:43.135305 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:43.135273 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsjht\" (UniqueName: \"kubernetes.io/projected/cf169784-ec2a-46b6-881a-7006bf881c33-kube-api-access-fsjht\") pod \"authorino-5d4b67c446-djgf2\" (UID: \"cf169784-ec2a-46b6-881a-7006bf881c33\") " pod="kuadrant-system/authorino-5d4b67c446-djgf2" Jul 23 04:03:43.149097 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:43.149035 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58967bf94c-s52pr"] Jul 23 04:03:43.152296 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:43.152281 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58967bf94c-s52pr" Jul 23 04:03:43.164182 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:43.164159 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58967bf94c-s52pr"] Jul 23 04:03:43.325105 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:43.325066 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sbh4\" (UniqueName: \"kubernetes.io/projected/f06f35a1-9322-4714-a6d1-ac655231f50b-kube-api-access-8sbh4\") pod \"authorino-58967bf94c-s52pr\" (UID: \"f06f35a1-9322-4714-a6d1-ac655231f50b\") " pod="kuadrant-system/authorino-58967bf94c-s52pr" Jul 23 04:03:43.426276 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:43.426243 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sbh4\" (UniqueName: \"kubernetes.io/projected/f06f35a1-9322-4714-a6d1-ac655231f50b-kube-api-access-8sbh4\") pod \"authorino-58967bf94c-s52pr\" (UID: \"f06f35a1-9322-4714-a6d1-ac655231f50b\") " pod="kuadrant-system/authorino-58967bf94c-s52pr" Jul 23 04:03:43.436205 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:43.436179 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sbh4\" (UniqueName: \"kubernetes.io/projected/f06f35a1-9322-4714-a6d1-ac655231f50b-kube-api-access-8sbh4\") pod \"authorino-58967bf94c-s52pr\" (UID: \"f06f35a1-9322-4714-a6d1-ac655231f50b\") " pod="kuadrant-system/authorino-58967bf94c-s52pr" Jul 23 04:03:43.460979 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:43.460951 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58967bf94c-s52pr" Jul 23 04:03:43.589693 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:43.589651 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58967bf94c-s52pr"] Jul 23 04:03:43.619809 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:43.619778 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-765664f98c-hckf9"] Jul 23 04:03:43.624384 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:43.624363 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-765664f98c-hckf9" Jul 23 04:03:43.625498 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:43.625478 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58967bf94c-s52pr"] Jul 23 04:03:43.627183 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:43.627046 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 23 04:03:43.629562 ip-10-0-140-6 kubenswrapper[2567]: W0723 04:03:43.629540 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf06f35a1_9322_4714_a6d1_ac655231f50b.slice/crio-57f6ae4891cbc79eeae83c49577c96039ecbe456b9ee771df2403857d8619d8d WatchSource:0}: Error finding container 57f6ae4891cbc79eeae83c49577c96039ecbe456b9ee771df2403857d8619d8d: Status 404 returned error can't find the container with id 57f6ae4891cbc79eeae83c49577c96039ecbe456b9ee771df2403857d8619d8d Jul 23 04:03:43.635392 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:43.635371 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-765664f98c-hckf9"] Jul 23 04:03:43.723361 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:43.723291 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-djgf2" Jul 23 04:03:43.723547 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:43.723293 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58967bf94c-s52pr" event={"ID":"f06f35a1-9322-4714-a6d1-ac655231f50b","Type":"ContainerStarted","Data":"57f6ae4891cbc79eeae83c49577c96039ecbe456b9ee771df2403857d8619d8d"} Jul 23 04:03:43.728703 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:43.728682 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d5487907-a311-4c19-a65d-aa9b6528e045-tls-cert\") pod \"authorino-765664f98c-hckf9\" (UID: \"d5487907-a311-4c19-a65d-aa9b6528e045\") " pod="kuadrant-system/authorino-765664f98c-hckf9" Jul 23 04:03:43.728804 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:43.728718 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsxgk\" (UniqueName: \"kubernetes.io/projected/d5487907-a311-4c19-a65d-aa9b6528e045-kube-api-access-tsxgk\") pod \"authorino-765664f98c-hckf9\" (UID: \"d5487907-a311-4c19-a65d-aa9b6528e045\") " pod="kuadrant-system/authorino-765664f98c-hckf9" Jul 23 04:03:43.729650 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:43.729634 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-djgf2" Jul 23 04:03:43.829949 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:43.829926 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fsjht\" (UniqueName: \"kubernetes.io/projected/cf169784-ec2a-46b6-881a-7006bf881c33-kube-api-access-fsjht\") pod \"cf169784-ec2a-46b6-881a-7006bf881c33\" (UID: \"cf169784-ec2a-46b6-881a-7006bf881c33\") " Jul 23 04:03:43.830068 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:43.830040 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d5487907-a311-4c19-a65d-aa9b6528e045-tls-cert\") pod \"authorino-765664f98c-hckf9\" (UID: \"d5487907-a311-4c19-a65d-aa9b6528e045\") " pod="kuadrant-system/authorino-765664f98c-hckf9" Jul 23 04:03:43.830118 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:43.830073 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsxgk\" (UniqueName: \"kubernetes.io/projected/d5487907-a311-4c19-a65d-aa9b6528e045-kube-api-access-tsxgk\") pod \"authorino-765664f98c-hckf9\" (UID: \"d5487907-a311-4c19-a65d-aa9b6528e045\") " pod="kuadrant-system/authorino-765664f98c-hckf9" Jul 23 04:03:43.832159 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:43.832122 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf169784-ec2a-46b6-881a-7006bf881c33-kube-api-access-fsjht" (OuterVolumeSpecName: "kube-api-access-fsjht") pod "cf169784-ec2a-46b6-881a-7006bf881c33" (UID: "cf169784-ec2a-46b6-881a-7006bf881c33"). InnerVolumeSpecName "kube-api-access-fsjht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 04:03:43.832616 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:43.832600 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d5487907-a311-4c19-a65d-aa9b6528e045-tls-cert\") pod \"authorino-765664f98c-hckf9\" (UID: \"d5487907-a311-4c19-a65d-aa9b6528e045\") " pod="kuadrant-system/authorino-765664f98c-hckf9" Jul 23 04:03:43.839986 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:43.839966 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsxgk\" (UniqueName: \"kubernetes.io/projected/d5487907-a311-4c19-a65d-aa9b6528e045-kube-api-access-tsxgk\") pod \"authorino-765664f98c-hckf9\" (UID: \"d5487907-a311-4c19-a65d-aa9b6528e045\") " pod="kuadrant-system/authorino-765664f98c-hckf9" Jul 23 04:03:43.931049 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:43.931024 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsjht\" (UniqueName: \"kubernetes.io/projected/cf169784-ec2a-46b6-881a-7006bf881c33-kube-api-access-fsjht\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:03:43.942942 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:43.942907 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-765664f98c-hckf9" Jul 23 04:03:44.079389 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:44.079317 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-765664f98c-hckf9"] Jul 23 04:03:44.081402 ip-10-0-140-6 kubenswrapper[2567]: W0723 04:03:44.081378 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5487907_a311_4c19_a65d_aa9b6528e045.slice/crio-1c7b1c889e4053c230d6cbda2abe688eb42cf52b6e9b68e245d56478b943309e WatchSource:0}: Error finding container 1c7b1c889e4053c230d6cbda2abe688eb42cf52b6e9b68e245d56478b943309e: Status 404 returned error can't find the container with id 1c7b1c889e4053c230d6cbda2abe688eb42cf52b6e9b68e245d56478b943309e Jul 23 04:03:44.729896 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:44.729814 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58967bf94c-s52pr" event={"ID":"f06f35a1-9322-4714-a6d1-ac655231f50b","Type":"ContainerStarted","Data":"078ba4287414fd01ad7c26d5ebb3d1005a77a22b51aa8cb3cd0bf9825b583aa0"} Jul 23 04:03:44.729896 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:44.729865 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58967bf94c-s52pr" podUID="f06f35a1-9322-4714-a6d1-ac655231f50b" containerName="authorino" containerID="cri-o://078ba4287414fd01ad7c26d5ebb3d1005a77a22b51aa8cb3cd0bf9825b583aa0" gracePeriod=30 Jul 23 04:03:44.731553 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:44.731515 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-765664f98c-hckf9" event={"ID":"d5487907-a311-4c19-a65d-aa9b6528e045","Type":"ContainerStarted","Data":"642ffdcea14e68472ca380bb40deb05756e6590235ab7801a7746bbaebd8547e"} Jul 23 04:03:44.731553 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:44.731556 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-765664f98c-hckf9" event={"ID":"d5487907-a311-4c19-a65d-aa9b6528e045","Type":"ContainerStarted","Data":"1c7b1c889e4053c230d6cbda2abe688eb42cf52b6e9b68e245d56478b943309e"} Jul 23 04:03:44.731734 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:44.731603 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-djgf2" Jul 23 04:03:44.750853 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:44.750811 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58967bf94c-s52pr" podStartSLOduration=1.2511295040000001 podStartE2EDuration="1.750798031s" totalImagesPullingTime="499.668527ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 04:03:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 04:03:44.750309003 +0000 UTC m=+504.655613297" watchObservedRunningTime="2026-07-23 04:03:44.750798031 +0000 UTC m=+504.656102323" Jul 23 04:03:44.786734 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:44.786706 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-djgf2"] Jul 23 04:03:44.791169 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:44.791139 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-djgf2"] Jul 23 04:03:44.821038 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:44.820991 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-765664f98c-hckf9" podStartSLOduration=1.45821325 podStartE2EDuration="1.820978055s" totalImagesPullingTime="362.764805ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 04:03:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 04:03:44.819361921 +0000 UTC m=+504.724666213" watchObservedRunningTime="2026-07-23 04:03:44.820978055 +0000 UTC m=+504.726282347" Jul 23 04:03:44.984481 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:44.984407 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58967bf94c-s52pr" Jul 23 04:03:45.141019 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:45.140990 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8sbh4\" (UniqueName: \"kubernetes.io/projected/f06f35a1-9322-4714-a6d1-ac655231f50b-kube-api-access-8sbh4\") pod \"f06f35a1-9322-4714-a6d1-ac655231f50b\" (UID: \"f06f35a1-9322-4714-a6d1-ac655231f50b\") " Jul 23 04:03:45.143192 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:45.143160 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f06f35a1-9322-4714-a6d1-ac655231f50b-kube-api-access-8sbh4" (OuterVolumeSpecName: "kube-api-access-8sbh4") pod "f06f35a1-9322-4714-a6d1-ac655231f50b" (UID: "f06f35a1-9322-4714-a6d1-ac655231f50b"). InnerVolumeSpecName "kube-api-access-8sbh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 04:03:45.241942 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:45.241865 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sbh4\" (UniqueName: \"kubernetes.io/projected/f06f35a1-9322-4714-a6d1-ac655231f50b-kube-api-access-8sbh4\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:03:45.736979 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:45.736943 2567 generic.go:356] "Generic (PLEG): container finished" podID="f06f35a1-9322-4714-a6d1-ac655231f50b" containerID="078ba4287414fd01ad7c26d5ebb3d1005a77a22b51aa8cb3cd0bf9825b583aa0" exitCode=0 Jul 23 04:03:45.736979 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:45.736982 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58967bf94c-s52pr" Jul 23 04:03:45.737217 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:45.737016 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58967bf94c-s52pr" event={"ID":"f06f35a1-9322-4714-a6d1-ac655231f50b","Type":"ContainerDied","Data":"078ba4287414fd01ad7c26d5ebb3d1005a77a22b51aa8cb3cd0bf9825b583aa0"} Jul 23 04:03:45.737217 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:45.737047 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58967bf94c-s52pr" event={"ID":"f06f35a1-9322-4714-a6d1-ac655231f50b","Type":"ContainerDied","Data":"57f6ae4891cbc79eeae83c49577c96039ecbe456b9ee771df2403857d8619d8d"} Jul 23 04:03:45.737217 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:45.737063 2567 scope.go:117] "RemoveContainer" containerID="078ba4287414fd01ad7c26d5ebb3d1005a77a22b51aa8cb3cd0bf9825b583aa0" Jul 23 04:03:45.745485 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:45.745465 2567 scope.go:117] "RemoveContainer" containerID="078ba4287414fd01ad7c26d5ebb3d1005a77a22b51aa8cb3cd0bf9825b583aa0" Jul 23 04:03:45.745736 ip-10-0-140-6 kubenswrapper[2567]: E0723 04:03:45.745715 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"078ba4287414fd01ad7c26d5ebb3d1005a77a22b51aa8cb3cd0bf9825b583aa0\": container with ID starting with 078ba4287414fd01ad7c26d5ebb3d1005a77a22b51aa8cb3cd0bf9825b583aa0 not found: ID does not exist" containerID="078ba4287414fd01ad7c26d5ebb3d1005a77a22b51aa8cb3cd0bf9825b583aa0" Jul 23 04:03:45.745797 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:45.745743 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"078ba4287414fd01ad7c26d5ebb3d1005a77a22b51aa8cb3cd0bf9825b583aa0"} err="failed to get container status \"078ba4287414fd01ad7c26d5ebb3d1005a77a22b51aa8cb3cd0bf9825b583aa0\": rpc error: code = NotFound desc = could not find container \"078ba4287414fd01ad7c26d5ebb3d1005a77a22b51aa8cb3cd0bf9825b583aa0\": container with ID starting with 078ba4287414fd01ad7c26d5ebb3d1005a77a22b51aa8cb3cd0bf9825b583aa0 not found: ID does not exist" Jul 23 04:03:45.759802 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:45.759774 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58967bf94c-s52pr"] Jul 23 04:03:45.764504 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:45.764484 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58967bf94c-s52pr"] Jul 23 04:03:46.492471 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:46.492415 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf169784-ec2a-46b6-881a-7006bf881c33" path="/var/lib/kubelet/pods/cf169784-ec2a-46b6-881a-7006bf881c33/volumes" Jul 23 04:03:46.492916 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:03:46.492663 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f06f35a1-9322-4714-a6d1-ac655231f50b" path="/var/lib/kubelet/pods/f06f35a1-9322-4714-a6d1-ac655231f50b/volumes" Jul 23 04:05:20.446070 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:05:20.446044 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wzgn9_dbd53593-e5e7-473a-b71c-8fbf9272913f/console-operator/2.log" Jul 23 04:05:20.453212 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:05:20.453188 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wzgn9_dbd53593-e5e7-473a-b71c-8fbf9272913f/console-operator/2.log" Jul 23 04:06:29.483771 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:06:29.483722 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-9f544c994-gctq6"] Jul 23 04:06:29.484355 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:06:29.484213 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f06f35a1-9322-4714-a6d1-ac655231f50b" containerName="authorino" Jul 23 04:06:29.484355 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:06:29.484234 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f06f35a1-9322-4714-a6d1-ac655231f50b" containerName="authorino" Jul 23 04:06:29.484355 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:06:29.484318 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f06f35a1-9322-4714-a6d1-ac655231f50b" containerName="authorino" Jul 23 04:06:29.487252 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:06:29.487231 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-9f544c994-gctq6" Jul 23 04:06:29.497805 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:06:29.497784 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-9f544c994-gctq6"] Jul 23 04:06:29.681410 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:06:29.681367 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg8pl\" (UniqueName: \"kubernetes.io/projected/fa78f912-a721-4e62-91b4-0101d53b3f1c-kube-api-access-tg8pl\") pod \"authorino-9f544c994-gctq6\" (UID: \"fa78f912-a721-4e62-91b4-0101d53b3f1c\") " pod="kuadrant-system/authorino-9f544c994-gctq6" Jul 23 04:06:29.681604 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:06:29.681418 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fa78f912-a721-4e62-91b4-0101d53b3f1c-tls-cert\") pod \"authorino-9f544c994-gctq6\" (UID: \"fa78f912-a721-4e62-91b4-0101d53b3f1c\") " pod="kuadrant-system/authorino-9f544c994-gctq6" Jul 23 04:06:29.782280 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:06:29.782199 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg8pl\" (UniqueName: \"kubernetes.io/projected/fa78f912-a721-4e62-91b4-0101d53b3f1c-kube-api-access-tg8pl\") pod \"authorino-9f544c994-gctq6\" (UID: \"fa78f912-a721-4e62-91b4-0101d53b3f1c\") " pod="kuadrant-system/authorino-9f544c994-gctq6" Jul 23 04:06:29.782280 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:06:29.782243 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fa78f912-a721-4e62-91b4-0101d53b3f1c-tls-cert\") pod \"authorino-9f544c994-gctq6\" (UID: \"fa78f912-a721-4e62-91b4-0101d53b3f1c\") " pod="kuadrant-system/authorino-9f544c994-gctq6" Jul 23 04:06:29.784811 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:06:29.784786 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fa78f912-a721-4e62-91b4-0101d53b3f1c-tls-cert\") pod \"authorino-9f544c994-gctq6\" (UID: \"fa78f912-a721-4e62-91b4-0101d53b3f1c\") " pod="kuadrant-system/authorino-9f544c994-gctq6" Jul 23 04:06:29.791755 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:06:29.791732 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg8pl\" (UniqueName: \"kubernetes.io/projected/fa78f912-a721-4e62-91b4-0101d53b3f1c-kube-api-access-tg8pl\") pod \"authorino-9f544c994-gctq6\" (UID: \"fa78f912-a721-4e62-91b4-0101d53b3f1c\") " pod="kuadrant-system/authorino-9f544c994-gctq6" Jul 23 04:06:29.795508 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:06:29.795469 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-9f544c994-gctq6" Jul 23 04:06:29.935187 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:06:29.935162 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-9f544c994-gctq6"] Jul 23 04:06:29.937300 ip-10-0-140-6 kubenswrapper[2567]: W0723 04:06:29.937277 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa78f912_a721_4e62_91b4_0101d53b3f1c.slice/crio-e9e2c48536f0ddf6968dd977c44f724279fc71cdf3cad953addf7db61e7afe39 WatchSource:0}: Error finding container e9e2c48536f0ddf6968dd977c44f724279fc71cdf3cad953addf7db61e7afe39: Status 404 returned error can't find the container with id e9e2c48536f0ddf6968dd977c44f724279fc71cdf3cad953addf7db61e7afe39 Jul 23 04:06:29.938929 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:06:29.938911 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 04:06:30.535899 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:06:30.535861 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-9f544c994-gctq6" event={"ID":"fa78f912-a721-4e62-91b4-0101d53b3f1c","Type":"ContainerStarted","Data":"e9e2c48536f0ddf6968dd977c44f724279fc71cdf3cad953addf7db61e7afe39"} Jul 23 04:06:31.544285 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:06:31.544239 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-9f544c994-gctq6" event={"ID":"fa78f912-a721-4e62-91b4-0101d53b3f1c","Type":"ContainerStarted","Data":"5cf438d56167628f116d8c9d3f895c15a637ec219131f9594a08481dcd889e4e"} Jul 23 04:06:31.569185 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:06:31.569120 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-9f544c994-gctq6" podStartSLOduration=1.92189394 podStartE2EDuration="2.56909739s" totalImagesPullingTime="647.20345ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 04:06:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 04:06:31.56824611 +0000 UTC m=+671.473550403" watchObservedRunningTime="2026-07-23 04:06:31.56909739 +0000 UTC m=+671.474401683" Jul 23 04:06:31.620542 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:06:31.620509 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-765664f98c-hckf9"] Jul 23 04:06:31.620768 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:06:31.620735 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-765664f98c-hckf9" podUID="d5487907-a311-4c19-a65d-aa9b6528e045" containerName="authorino" containerID="cri-o://642ffdcea14e68472ca380bb40deb05756e6590235ab7801a7746bbaebd8547e" gracePeriod=30 Jul 23 04:06:31.879796 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:06:31.879768 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-765664f98c-hckf9" Jul 23 04:06:31.901229 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:06:31.901195 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d5487907-a311-4c19-a65d-aa9b6528e045-tls-cert\") pod \"d5487907-a311-4c19-a65d-aa9b6528e045\" (UID: \"d5487907-a311-4c19-a65d-aa9b6528e045\") " Jul 23 04:06:31.901355 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:06:31.901247 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tsxgk\" (UniqueName: \"kubernetes.io/projected/d5487907-a311-4c19-a65d-aa9b6528e045-kube-api-access-tsxgk\") pod \"d5487907-a311-4c19-a65d-aa9b6528e045\" (UID: \"d5487907-a311-4c19-a65d-aa9b6528e045\") " Jul 23 04:06:31.903474 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:06:31.903418 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5487907-a311-4c19-a65d-aa9b6528e045-kube-api-access-tsxgk" (OuterVolumeSpecName: "kube-api-access-tsxgk") pod "d5487907-a311-4c19-a65d-aa9b6528e045" (UID: "d5487907-a311-4c19-a65d-aa9b6528e045"). InnerVolumeSpecName "kube-api-access-tsxgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 04:06:31.911485 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:06:31.911449 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5487907-a311-4c19-a65d-aa9b6528e045-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "d5487907-a311-4c19-a65d-aa9b6528e045" (UID: "d5487907-a311-4c19-a65d-aa9b6528e045"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 04:06:32.001875 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:06:32.001841 2567 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d5487907-a311-4c19-a65d-aa9b6528e045-tls-cert\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:06:32.001875 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:06:32.001870 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsxgk\" (UniqueName: \"kubernetes.io/projected/d5487907-a311-4c19-a65d-aa9b6528e045-kube-api-access-tsxgk\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:06:32.550232 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:06:32.550199 2567 generic.go:356] "Generic (PLEG): container finished" podID="d5487907-a311-4c19-a65d-aa9b6528e045" containerID="642ffdcea14e68472ca380bb40deb05756e6590235ab7801a7746bbaebd8547e" exitCode=0 Jul 23 04:06:32.550682 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:06:32.550247 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-765664f98c-hckf9" Jul 23 04:06:32.550682 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:06:32.550293 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-765664f98c-hckf9" event={"ID":"d5487907-a311-4c19-a65d-aa9b6528e045","Type":"ContainerDied","Data":"642ffdcea14e68472ca380bb40deb05756e6590235ab7801a7746bbaebd8547e"} Jul 23 04:06:32.550682 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:06:32.550332 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-765664f98c-hckf9" event={"ID":"d5487907-a311-4c19-a65d-aa9b6528e045","Type":"ContainerDied","Data":"1c7b1c889e4053c230d6cbda2abe688eb42cf52b6e9b68e245d56478b943309e"} Jul 23 04:06:32.550682 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:06:32.550352 2567 scope.go:117] "RemoveContainer" containerID="642ffdcea14e68472ca380bb40deb05756e6590235ab7801a7746bbaebd8547e" Jul 23 04:06:32.558876 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:06:32.558854 2567 scope.go:117] "RemoveContainer" containerID="642ffdcea14e68472ca380bb40deb05756e6590235ab7801a7746bbaebd8547e" Jul 23 04:06:32.559157 ip-10-0-140-6 kubenswrapper[2567]: E0723 04:06:32.559137 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"642ffdcea14e68472ca380bb40deb05756e6590235ab7801a7746bbaebd8547e\": container with ID starting with 642ffdcea14e68472ca380bb40deb05756e6590235ab7801a7746bbaebd8547e not found: ID does not exist" containerID="642ffdcea14e68472ca380bb40deb05756e6590235ab7801a7746bbaebd8547e" Jul 23 04:06:32.559209 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:06:32.559170 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"642ffdcea14e68472ca380bb40deb05756e6590235ab7801a7746bbaebd8547e"} err="failed to get container status \"642ffdcea14e68472ca380bb40deb05756e6590235ab7801a7746bbaebd8547e\": rpc error: code = NotFound desc = could not find container \"642ffdcea14e68472ca380bb40deb05756e6590235ab7801a7746bbaebd8547e\": container with ID starting with 642ffdcea14e68472ca380bb40deb05756e6590235ab7801a7746bbaebd8547e not found: ID does not exist" Jul 23 04:06:32.607131 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:06:32.607104 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-765664f98c-hckf9"] Jul 23 04:06:32.627212 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:06:32.627187 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-765664f98c-hckf9"] Jul 23 04:06:34.493095 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:06:34.493059 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5487907-a311-4c19-a65d-aa9b6528e045" path="/var/lib/kubelet/pods/d5487907-a311-4c19-a65d-aa9b6528e045/volumes" Jul 23 04:10:20.478815 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:10:20.478788 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wzgn9_dbd53593-e5e7-473a-b71c-8fbf9272913f/console-operator/2.log" Jul 23 04:10:20.482541 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:10:20.482519 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wzgn9_dbd53593-e5e7-473a-b71c-8fbf9272913f/console-operator/2.log" Jul 23 04:15:00.159212 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:15:00.159172 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29746335-mgprt"] Jul 23 04:15:00.159783 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:15:00.159652 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5487907-a311-4c19-a65d-aa9b6528e045" containerName="authorino" Jul 23 04:15:00.159783 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:15:00.159673 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5487907-a311-4c19-a65d-aa9b6528e045" containerName="authorino" Jul 23 04:15:00.159783 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:15:00.159784 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5487907-a311-4c19-a65d-aa9b6528e045" containerName="authorino" Jul 23 04:15:00.162817 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:15:00.162792 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746335-mgprt" Jul 23 04:15:00.165570 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:15:00.165544 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dpftb\"" Jul 23 04:15:00.173620 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:15:00.173596 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29746335-mgprt"] Jul 23 04:15:00.174183 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:15:00.174165 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746335-mgprt" Jul 23 04:15:00.533794 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:15:00.533762 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29746335-mgprt"] Jul 23 04:15:00.537167 ip-10-0-140-6 kubenswrapper[2567]: W0723 04:15:00.537142 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7327303b_30a9_4589_9b5a_b5f65072eabd.slice/crio-576cc4e45fae1a8d42865bf8f3237b137f9c6b5430be9ed5df7980f8b8175d3e WatchSource:0}: Error finding container 576cc4e45fae1a8d42865bf8f3237b137f9c6b5430be9ed5df7980f8b8175d3e: Status 404 returned error can't find the container with id 576cc4e45fae1a8d42865bf8f3237b137f9c6b5430be9ed5df7980f8b8175d3e Jul 23 04:15:00.539067 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:15:00.539050 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 04:15:01.015859 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:15:01.015791 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746335-mgprt" event={"ID":"7327303b-30a9-4589-9b5a-b5f65072eabd","Type":"ContainerStarted","Data":"576cc4e45fae1a8d42865bf8f3237b137f9c6b5430be9ed5df7980f8b8175d3e"} Jul 23 04:15:04.032736 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:15:04.032648 2567 generic.go:356] "Generic (PLEG): container finished" podID="7327303b-30a9-4589-9b5a-b5f65072eabd" containerID="db64df97e2b087a5b04acaa4c8494e96ffb658e0b6962fb8ee5102692c249e0b" exitCode=0 Jul 23 04:15:04.033090 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:15:04.032740 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746335-mgprt" event={"ID":"7327303b-30a9-4589-9b5a-b5f65072eabd","Type":"ContainerDied","Data":"db64df97e2b087a5b04acaa4c8494e96ffb658e0b6962fb8ee5102692c249e0b"} Jul 23 04:15:05.160781 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:15:05.160758 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746335-mgprt" Jul 23 04:15:06.042778 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:15:06.042748 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746335-mgprt" Jul 23 04:15:06.042946 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:15:06.042778 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746335-mgprt" event={"ID":"7327303b-30a9-4589-9b5a-b5f65072eabd","Type":"ContainerDied","Data":"576cc4e45fae1a8d42865bf8f3237b137f9c6b5430be9ed5df7980f8b8175d3e"} Jul 23 04:15:06.042946 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:15:06.042825 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="576cc4e45fae1a8d42865bf8f3237b137f9c6b5430be9ed5df7980f8b8175d3e" Jul 23 04:15:20.516182 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:15:20.516149 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wzgn9_dbd53593-e5e7-473a-b71c-8fbf9272913f/console-operator/2.log" Jul 23 04:15:20.517650 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:15:20.517630 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wzgn9_dbd53593-e5e7-473a-b71c-8fbf9272913f/console-operator/2.log" Jul 23 04:20:20.549478 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:20:20.549449 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wzgn9_dbd53593-e5e7-473a-b71c-8fbf9272913f/console-operator/2.log" Jul 23 04:20:20.550017 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:20:20.549860 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wzgn9_dbd53593-e5e7-473a-b71c-8fbf9272913f/console-operator/2.log" Jul 23 04:21:26.030493 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:26.030462 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl"] Jul 23 04:21:26.030949 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:26.030776 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7327303b-30a9-4589-9b5a-b5f65072eabd" containerName="cleanup" Jul 23 04:21:26.030949 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:26.030786 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7327303b-30a9-4589-9b5a-b5f65072eabd" containerName="cleanup" Jul 23 04:21:26.030949 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:26.030830 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="7327303b-30a9-4589-9b5a-b5f65072eabd" containerName="cleanup" Jul 23 04:21:26.033798 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:26.033783 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl" Jul 23 04:21:26.037154 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:26.037127 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-0ba06924-openshift-default-dockercfg-lx4s5\"" Jul 23 04:21:26.062394 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:26.062367 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl"] Jul 23 04:21:26.104239 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:26.104214 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d8f3129-7302-4bf9-87e4-61101b15ec86-istio-envoy\") pod \"e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl\" (UID: \"4d8f3129-7302-4bf9-87e4-61101b15ec86\") " pod="openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl" Jul 23 04:21:26.104336 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:26.104249 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d8f3129-7302-4bf9-87e4-61101b15ec86-credential-socket\") pod \"e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl\" (UID: \"4d8f3129-7302-4bf9-87e4-61101b15ec86\") " pod="openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl" Jul 23 04:21:26.104336 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:26.104270 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnkfz\" (UniqueName: \"kubernetes.io/projected/4d8f3129-7302-4bf9-87e4-61101b15ec86-kube-api-access-qnkfz\") pod \"e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl\" (UID: \"4d8f3129-7302-4bf9-87e4-61101b15ec86\") " pod="openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl" Jul 23 04:21:26.104336 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:26.104302 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d8f3129-7302-4bf9-87e4-61101b15ec86-istio-data\") pod \"e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl\" (UID: \"4d8f3129-7302-4bf9-87e4-61101b15ec86\") " pod="openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl" Jul 23 04:21:26.104336 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:26.104329 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d8f3129-7302-4bf9-87e4-61101b15ec86-istio-token\") pod \"e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl\" (UID: \"4d8f3129-7302-4bf9-87e4-61101b15ec86\") " pod="openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl" Jul 23 04:21:26.104530 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:26.104356 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d8f3129-7302-4bf9-87e4-61101b15ec86-istio-podinfo\") pod \"e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl\" (UID: \"4d8f3129-7302-4bf9-87e4-61101b15ec86\") " pod="openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl" Jul 23 04:21:26.104530 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:26.104412 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d8f3129-7302-4bf9-87e4-61101b15ec86-workload-socket\") pod \"e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl\" (UID: \"4d8f3129-7302-4bf9-87e4-61101b15ec86\") " pod="openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl" Jul 23 04:21:26.104530 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:26.104464 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d8f3129-7302-4bf9-87e4-61101b15ec86-workload-certs\") pod \"e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl\" (UID: \"4d8f3129-7302-4bf9-87e4-61101b15ec86\") " pod="openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl" Jul 23 04:21:26.104530 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:26.104514 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d8f3129-7302-4bf9-87e4-61101b15ec86-istiod-ca-cert\") pod \"e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl\" (UID: \"4d8f3129-7302-4bf9-87e4-61101b15ec86\") " pod="openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl" Jul 23 04:21:26.205830 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:26.205796 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d8f3129-7302-4bf9-87e4-61101b15ec86-istio-envoy\") pod \"e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl\" (UID: \"4d8f3129-7302-4bf9-87e4-61101b15ec86\") " pod="openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl" Jul 23 04:21:26.205991 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:26.205840 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d8f3129-7302-4bf9-87e4-61101b15ec86-credential-socket\") pod \"e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl\" (UID: \"4d8f3129-7302-4bf9-87e4-61101b15ec86\") " pod="openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl" Jul 23 04:21:26.205991 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:26.205867 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnkfz\" (UniqueName: \"kubernetes.io/projected/4d8f3129-7302-4bf9-87e4-61101b15ec86-kube-api-access-qnkfz\") pod \"e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl\" (UID: \"4d8f3129-7302-4bf9-87e4-61101b15ec86\") " pod="openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl" Jul 23 04:21:26.205991 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:26.205890 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d8f3129-7302-4bf9-87e4-61101b15ec86-istio-data\") pod \"e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl\" (UID: \"4d8f3129-7302-4bf9-87e4-61101b15ec86\") " pod="openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl" Jul 23 04:21:26.205991 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:26.205935 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d8f3129-7302-4bf9-87e4-61101b15ec86-istio-token\") pod \"e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl\" (UID: \"4d8f3129-7302-4bf9-87e4-61101b15ec86\") " pod="openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl" Jul 23 04:21:26.205991 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:26.205984 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d8f3129-7302-4bf9-87e4-61101b15ec86-istio-podinfo\") pod \"e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl\" (UID: \"4d8f3129-7302-4bf9-87e4-61101b15ec86\") " pod="openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl" Jul 23 04:21:26.206262 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:26.206015 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d8f3129-7302-4bf9-87e4-61101b15ec86-workload-socket\") pod \"e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl\" (UID: \"4d8f3129-7302-4bf9-87e4-61101b15ec86\") " pod="openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl" Jul 23 04:21:26.206262 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:26.206042 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d8f3129-7302-4bf9-87e4-61101b15ec86-workload-certs\") pod \"e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl\" (UID: \"4d8f3129-7302-4bf9-87e4-61101b15ec86\") " pod="openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl" Jul 23 04:21:26.206262 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:26.206099 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d8f3129-7302-4bf9-87e4-61101b15ec86-istiod-ca-cert\") pod \"e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl\" (UID: \"4d8f3129-7302-4bf9-87e4-61101b15ec86\") " pod="openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl" Jul 23 04:21:26.206421 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:26.206312 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d8f3129-7302-4bf9-87e4-61101b15ec86-istio-data\") pod \"e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl\" (UID: \"4d8f3129-7302-4bf9-87e4-61101b15ec86\") " pod="openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl" Jul 23 04:21:26.206421 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:26.206371 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d8f3129-7302-4bf9-87e4-61101b15ec86-workload-certs\") pod \"e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl\" (UID: \"4d8f3129-7302-4bf9-87e4-61101b15ec86\") " pod="openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl" Jul 23 04:21:26.206559 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:26.206465 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d8f3129-7302-4bf9-87e4-61101b15ec86-workload-socket\") pod \"e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl\" (UID: \"4d8f3129-7302-4bf9-87e4-61101b15ec86\") " pod="openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl" Jul 23 04:21:26.206559 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:26.206488 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d8f3129-7302-4bf9-87e4-61101b15ec86-credential-socket\") pod \"e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl\" (UID: \"4d8f3129-7302-4bf9-87e4-61101b15ec86\") " pod="openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl" Jul 23 04:21:26.206791 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:26.206771 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d8f3129-7302-4bf9-87e4-61101b15ec86-istiod-ca-cert\") pod \"e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl\" (UID: \"4d8f3129-7302-4bf9-87e4-61101b15ec86\") " pod="openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl" Jul 23 04:21:26.208264 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:26.208240 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d8f3129-7302-4bf9-87e4-61101b15ec86-istio-envoy\") pod \"e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl\" (UID: \"4d8f3129-7302-4bf9-87e4-61101b15ec86\") " pod="openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl" Jul 23 04:21:26.208634 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:26.208613 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d8f3129-7302-4bf9-87e4-61101b15ec86-istio-podinfo\") pod \"e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl\" (UID: \"4d8f3129-7302-4bf9-87e4-61101b15ec86\") " pod="openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl" Jul 23 04:21:26.214155 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:26.214125 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d8f3129-7302-4bf9-87e4-61101b15ec86-istio-token\") pod \"e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl\" (UID: \"4d8f3129-7302-4bf9-87e4-61101b15ec86\") " pod="openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl" Jul 23 04:21:26.214235 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:26.214197 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnkfz\" (UniqueName: \"kubernetes.io/projected/4d8f3129-7302-4bf9-87e4-61101b15ec86-kube-api-access-qnkfz\") pod \"e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl\" (UID: \"4d8f3129-7302-4bf9-87e4-61101b15ec86\") " pod="openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl" Jul 23 04:21:26.343698 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:26.343633 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl" Jul 23 04:21:26.483257 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:26.483095 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl"] Jul 23 04:21:26.485931 ip-10-0-140-6 kubenswrapper[2567]: W0723 04:21:26.485906 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d8f3129_7302_4bf9_87e4_61101b15ec86.slice/crio-2b43fb91aad4e2890762456573e8e8fbed88dddc0a3919462541a485c296bb7e WatchSource:0}: Error finding container 2b43fb91aad4e2890762456573e8e8fbed88dddc0a3919462541a485c296bb7e: Status 404 returned error can't find the container with id 2b43fb91aad4e2890762456573e8e8fbed88dddc0a3919462541a485c296bb7e Jul 23 04:21:26.488108 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:26.488076 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 04:21:26.488218 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:26.488140 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 04:21:26.488218 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:26.488170 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 04:21:26.903890 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:26.903854 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl" event={"ID":"4d8f3129-7302-4bf9-87e4-61101b15ec86","Type":"ContainerStarted","Data":"721315b4109e06f7527af02ef6db14ef1db3e415b21a2d41c39cf36c295cceae"} Jul 23 04:21:26.904032 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:26.903912 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl" event={"ID":"4d8f3129-7302-4bf9-87e4-61101b15ec86","Type":"ContainerStarted","Data":"2b43fb91aad4e2890762456573e8e8fbed88dddc0a3919462541a485c296bb7e"} Jul 23 04:21:26.926011 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:26.925963 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl" podStartSLOduration=0.92594718 podStartE2EDuration="925.94718ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 04:21:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 04:21:26.92450098 +0000 UTC m=+1566.829805305" watchObservedRunningTime="2026-07-23 04:21:26.92594718 +0000 UTC m=+1566.831251472" Jul 23 04:21:27.344495 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:27.344459 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl" Jul 23 04:21:27.349618 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:27.349592 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl" Jul 23 04:21:27.908495 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:27.908463 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl" Jul 23 04:21:27.909510 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:27.909488 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl" Jul 23 04:21:43.287807 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:43.286537 2567 patch_prober.go:28] interesting pod/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": read tcp 10.133.0.2:51994->10.133.0.41:15021: read: connection reset by peer" start-of-body= Jul 23 04:21:43.287807 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:43.286616 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl" podUID="4d8f3129-7302-4bf9-87e4-61101b15ec86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": read tcp 10.133.0.2:51994->10.133.0.41:15021: read: connection reset by peer" Jul 23 04:21:43.998887 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:43.998850 2567 generic.go:356] "Generic (PLEG): container finished" podID="4d8f3129-7302-4bf9-87e4-61101b15ec86" containerID="721315b4109e06f7527af02ef6db14ef1db3e415b21a2d41c39cf36c295cceae" exitCode=0 Jul 23 04:21:43.999050 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:43.998923 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl" event={"ID":"4d8f3129-7302-4bf9-87e4-61101b15ec86","Type":"ContainerDied","Data":"721315b4109e06f7527af02ef6db14ef1db3e415b21a2d41c39cf36c295cceae"} Jul 23 04:21:43.999261 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:43.999248 2567 scope.go:117] "RemoveContainer" containerID="721315b4109e06f7527af02ef6db14ef1db3e415b21a2d41c39cf36c295cceae" Jul 23 04:21:44.000210 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:44.000183 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 04:21:44.000267 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:44.000255 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 04:21:44.000302 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:44.000285 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 04:21:45.005944 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:45.005905 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl" event={"ID":"4d8f3129-7302-4bf9-87e4-61101b15ec86","Type":"ContainerStarted","Data":"617bf0d48d367fd63cf8bf6caea962d66d2730fbb750f333fc5145c13e5b9cde"} Jul 23 04:21:45.344727 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:45.344635 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl" Jul 23 04:21:45.345048 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:45.345025 2567 patch_prober.go:28] interesting pod/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 23 04:21:45.345201 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:45.345178 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl" podUID="4d8f3129-7302-4bf9-87e4-61101b15ec86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 23 04:21:46.345162 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:46.344629 2567 patch_prober.go:28] interesting pod/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 23 04:21:46.345162 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:46.344703 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl" podUID="4d8f3129-7302-4bf9-87e4-61101b15ec86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 23 04:21:47.345172 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:47.345125 2567 patch_prober.go:28] interesting pod/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 23 04:21:47.345691 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:47.345195 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl" podUID="4d8f3129-7302-4bf9-87e4-61101b15ec86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 23 04:21:48.344411 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:48.344373 2567 patch_prober.go:28] interesting pod/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 23 04:21:48.344702 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:48.344675 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl" podUID="4d8f3129-7302-4bf9-87e4-61101b15ec86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 23 04:21:49.347687 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:49.347657 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl" Jul 23 04:21:49.348088 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:49.347939 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl" Jul 23 04:21:49.348685 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:49.348667 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl" Jul 23 04:21:53.861252 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:53.861215 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ba06924-mmr8v"] Jul 23 04:21:53.864268 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:53.864251 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ba06924-mmr8v" Jul 23 04:21:53.866837 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:53.866811 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 04:21:53.866932 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:53.866878 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dpftb\"" Jul 23 04:21:53.873472 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:53.873416 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ba06924-mmr8v"] Jul 23 04:21:53.937686 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:53.937656 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/311bafdf-e074-4918-b690-cf62eb11004a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0ba06924-mmr8v\" (UID: \"311bafdf-e074-4918-b690-cf62eb11004a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ba06924-mmr8v" Jul 23 04:21:54.038329 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:54.038290 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/311bafdf-e074-4918-b690-cf62eb11004a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0ba06924-mmr8v\" (UID: \"311bafdf-e074-4918-b690-cf62eb11004a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ba06924-mmr8v" Jul 23 04:21:54.039021 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:54.039001 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/311bafdf-e074-4918-b690-cf62eb11004a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0ba06924-mmr8v\" (UID: \"311bafdf-e074-4918-b690-cf62eb11004a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ba06924-mmr8v" Jul 23 04:21:54.175745 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:54.175716 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ba06924-mmr8v" Jul 23 04:21:54.310314 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:54.310290 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ba06924-mmr8v"] Jul 23 04:21:54.312236 ip-10-0-140-6 kubenswrapper[2567]: W0723 04:21:54.312212 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod311bafdf_e074_4918_b690_cf62eb11004a.slice/crio-ccdabee6a4412d2f0581f580b1f8ed75070559aa463e86e1c2ded0b9e22f7ab2 WatchSource:0}: Error finding container ccdabee6a4412d2f0581f580b1f8ed75070559aa463e86e1c2ded0b9e22f7ab2: Status 404 returned error can't find the container with id ccdabee6a4412d2f0581f580b1f8ed75070559aa463e86e1c2ded0b9e22f7ab2 Jul 23 04:21:55.056343 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:55.056314 2567 generic.go:356] "Generic (PLEG): container finished" podID="311bafdf-e074-4918-b690-cf62eb11004a" containerID="5f896ec224f29cd69016725f21bb12be0c4194f3c10278d21b9b8fc6b27378b3" exitCode=0 Jul 23 04:21:55.056722 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:55.056398 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ba06924-mmr8v" event={"ID":"311bafdf-e074-4918-b690-cf62eb11004a","Type":"ContainerDied","Data":"5f896ec224f29cd69016725f21bb12be0c4194f3c10278d21b9b8fc6b27378b3"} Jul 23 04:21:55.056722 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:55.056473 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ba06924-mmr8v" event={"ID":"311bafdf-e074-4918-b690-cf62eb11004a","Type":"ContainerStarted","Data":"ccdabee6a4412d2f0581f580b1f8ed75070559aa463e86e1c2ded0b9e22f7ab2"} Jul 23 04:21:56.190026 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:56.190002 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ba06924-mmr8v" Jul 23 04:21:56.258294 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:56.258269 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/311bafdf-e074-4918-b690-cf62eb11004a-maas-api-service-ca\") pod \"311bafdf-e074-4918-b690-cf62eb11004a\" (UID: \"311bafdf-e074-4918-b690-cf62eb11004a\") " Jul 23 04:21:56.258644 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:56.258622 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/311bafdf-e074-4918-b690-cf62eb11004a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "311bafdf-e074-4918-b690-cf62eb11004a" (UID: "311bafdf-e074-4918-b690-cf62eb11004a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 04:21:56.359836 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:56.359776 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/311bafdf-e074-4918-b690-cf62eb11004a-maas-api-service-ca\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:21:57.067809 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:57.067778 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ba06924-mmr8v" Jul 23 04:21:57.067978 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:57.067778 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ba06924-mmr8v" event={"ID":"311bafdf-e074-4918-b690-cf62eb11004a","Type":"ContainerDied","Data":"ccdabee6a4412d2f0581f580b1f8ed75070559aa463e86e1c2ded0b9e22f7ab2"} Jul 23 04:21:57.067978 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:21:57.067905 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ccdabee6a4412d2f0581f580b1f8ed75070559aa463e86e1c2ded0b9e22f7ab2" Jul 23 04:22:08.950550 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:08.950496 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ba06924-mmr8v"] Jul 23 04:22:08.953496 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:08.953464 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ba06924-mmr8v"] Jul 23 04:22:09.214015 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:09.213934 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl"] Jul 23 04:22:09.214234 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:09.214190 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl" podUID="4d8f3129-7302-4bf9-87e4-61101b15ec86" containerName="istio-proxy" containerID="cri-o://617bf0d48d367fd63cf8bf6caea962d66d2730fbb750f333fc5145c13e5b9cde" gracePeriod=30 Jul 23 04:22:10.273575 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:10.273531 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc"] Jul 23 04:22:10.273948 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:10.273847 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="311bafdf-e074-4918-b690-cf62eb11004a" containerName="revoke-keys" Jul 23 04:22:10.273948 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:10.273858 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="311bafdf-e074-4918-b690-cf62eb11004a" containerName="revoke-keys" Jul 23 04:22:10.273948 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:10.273914 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="311bafdf-e074-4918-b690-cf62eb11004a" containerName="revoke-keys" Jul 23 04:22:10.277055 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:10.277036 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc" Jul 23 04:22:10.279651 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:10.279627 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-8238e07c-gw-openshift-default-dockercfg-kn52n\"" Jul 23 04:22:10.296281 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:10.296255 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc"] Jul 23 04:22:10.378198 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:10.378164 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lmjw\" (UniqueName: \"kubernetes.io/projected/22c81c29-eb1b-441c-b77a-568d7dae87df-kube-api-access-8lmjw\") pod \"e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc\" (UID: \"22c81c29-eb1b-441c-b77a-568d7dae87df\") " pod="openshift-ingress/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc" Jul 23 04:22:10.378198 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:10.378201 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22c81c29-eb1b-441c-b77a-568d7dae87df-istiod-ca-cert\") pod \"e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc\" (UID: \"22c81c29-eb1b-441c-b77a-568d7dae87df\") " pod="openshift-ingress/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc" Jul 23 04:22:10.378489 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:10.378287 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22c81c29-eb1b-441c-b77a-568d7dae87df-credential-socket\") pod \"e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc\" (UID: \"22c81c29-eb1b-441c-b77a-568d7dae87df\") " pod="openshift-ingress/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc" Jul 23 04:22:10.378489 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:10.378322 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22c81c29-eb1b-441c-b77a-568d7dae87df-workload-certs\") pod \"e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc\" (UID: \"22c81c29-eb1b-441c-b77a-568d7dae87df\") " pod="openshift-ingress/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc" Jul 23 04:22:10.378489 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:10.378357 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22c81c29-eb1b-441c-b77a-568d7dae87df-istio-envoy\") pod \"e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc\" (UID: \"22c81c29-eb1b-441c-b77a-568d7dae87df\") " pod="openshift-ingress/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc" Jul 23 04:22:10.378489 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:10.378469 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22c81c29-eb1b-441c-b77a-568d7dae87df-istio-podinfo\") pod \"e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc\" (UID: \"22c81c29-eb1b-441c-b77a-568d7dae87df\") " pod="openshift-ingress/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc" Jul 23 04:22:10.378689 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:10.378510 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22c81c29-eb1b-441c-b77a-568d7dae87df-istio-data\") pod \"e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc\" (UID: \"22c81c29-eb1b-441c-b77a-568d7dae87df\") " pod="openshift-ingress/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc" Jul 23 04:22:10.378689 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:10.378534 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22c81c29-eb1b-441c-b77a-568d7dae87df-istio-token\") pod \"e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc\" (UID: \"22c81c29-eb1b-441c-b77a-568d7dae87df\") " pod="openshift-ingress/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc" Jul 23 04:22:10.378689 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:10.378590 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22c81c29-eb1b-441c-b77a-568d7dae87df-workload-socket\") pod \"e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc\" (UID: \"22c81c29-eb1b-441c-b77a-568d7dae87df\") " pod="openshift-ingress/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc" Jul 23 04:22:10.479986 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:10.479954 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22c81c29-eb1b-441c-b77a-568d7dae87df-istiod-ca-cert\") pod \"e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc\" (UID: \"22c81c29-eb1b-441c-b77a-568d7dae87df\") " pod="openshift-ingress/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc" Jul 23 04:22:10.480149 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:10.480013 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22c81c29-eb1b-441c-b77a-568d7dae87df-credential-socket\") pod \"e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc\" (UID: \"22c81c29-eb1b-441c-b77a-568d7dae87df\") " pod="openshift-ingress/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc" Jul 23 04:22:10.480149 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:10.480038 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22c81c29-eb1b-441c-b77a-568d7dae87df-workload-certs\") pod \"e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc\" (UID: \"22c81c29-eb1b-441c-b77a-568d7dae87df\") " pod="openshift-ingress/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc" Jul 23 04:22:10.480149 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:10.480065 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22c81c29-eb1b-441c-b77a-568d7dae87df-istio-envoy\") pod \"e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc\" (UID: \"22c81c29-eb1b-441c-b77a-568d7dae87df\") " pod="openshift-ingress/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc" Jul 23 04:22:10.480149 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:10.480118 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22c81c29-eb1b-441c-b77a-568d7dae87df-istio-podinfo\") pod \"e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc\" (UID: \"22c81c29-eb1b-441c-b77a-568d7dae87df\") " pod="openshift-ingress/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc" Jul 23 04:22:10.480335 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:10.480152 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22c81c29-eb1b-441c-b77a-568d7dae87df-istio-data\") pod \"e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc\" (UID: \"22c81c29-eb1b-441c-b77a-568d7dae87df\") " pod="openshift-ingress/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc" Jul 23 04:22:10.480335 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:10.480282 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22c81c29-eb1b-441c-b77a-568d7dae87df-istio-token\") pod \"e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc\" (UID: \"22c81c29-eb1b-441c-b77a-568d7dae87df\") " pod="openshift-ingress/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc" Jul 23 04:22:10.480452 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:10.480365 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22c81c29-eb1b-441c-b77a-568d7dae87df-workload-socket\") pod \"e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc\" (UID: \"22c81c29-eb1b-441c-b77a-568d7dae87df\") " pod="openshift-ingress/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc" Jul 23 04:22:10.480452 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:10.480398 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lmjw\" (UniqueName: \"kubernetes.io/projected/22c81c29-eb1b-441c-b77a-568d7dae87df-kube-api-access-8lmjw\") pod \"e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc\" (UID: \"22c81c29-eb1b-441c-b77a-568d7dae87df\") " pod="openshift-ingress/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc" Jul 23 04:22:10.480566 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:10.480493 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22c81c29-eb1b-441c-b77a-568d7dae87df-credential-socket\") pod \"e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc\" (UID: \"22c81c29-eb1b-441c-b77a-568d7dae87df\") " pod="openshift-ingress/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc" Jul 23 04:22:10.480566 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:10.480539 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22c81c29-eb1b-441c-b77a-568d7dae87df-workload-certs\") pod \"e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc\" (UID: \"22c81c29-eb1b-441c-b77a-568d7dae87df\") " pod="openshift-ingress/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc" Jul 23 04:22:10.480676 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:10.480631 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22c81c29-eb1b-441c-b77a-568d7dae87df-istio-data\") pod \"e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc\" (UID: \"22c81c29-eb1b-441c-b77a-568d7dae87df\") " pod="openshift-ingress/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc" Jul 23 04:22:10.480795 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:10.480777 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22c81c29-eb1b-441c-b77a-568d7dae87df-workload-socket\") pod \"e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc\" (UID: \"22c81c29-eb1b-441c-b77a-568d7dae87df\") " pod="openshift-ingress/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc" Jul 23 04:22:10.481064 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:10.481047 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22c81c29-eb1b-441c-b77a-568d7dae87df-istiod-ca-cert\") pod \"e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc\" (UID: \"22c81c29-eb1b-441c-b77a-568d7dae87df\") " pod="openshift-ingress/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc" Jul 23 04:22:10.483096 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:10.483074 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22c81c29-eb1b-441c-b77a-568d7dae87df-istio-envoy\") pod \"e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc\" (UID: \"22c81c29-eb1b-441c-b77a-568d7dae87df\") " pod="openshift-ingress/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc" Jul 23 04:22:10.483294 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:10.483275 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22c81c29-eb1b-441c-b77a-568d7dae87df-istio-podinfo\") pod \"e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc\" (UID: \"22c81c29-eb1b-441c-b77a-568d7dae87df\") " pod="openshift-ingress/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc" Jul 23 04:22:10.489481 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:10.489460 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lmjw\" (UniqueName: \"kubernetes.io/projected/22c81c29-eb1b-441c-b77a-568d7dae87df-kube-api-access-8lmjw\") pod \"e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc\" (UID: \"22c81c29-eb1b-441c-b77a-568d7dae87df\") " pod="openshift-ingress/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc" Jul 23 04:22:10.489930 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:10.489911 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22c81c29-eb1b-441c-b77a-568d7dae87df-istio-token\") pod \"e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc\" (UID: \"22c81c29-eb1b-441c-b77a-568d7dae87df\") " pod="openshift-ingress/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc" Jul 23 04:22:10.493098 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:10.493076 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="311bafdf-e074-4918-b690-cf62eb11004a" path="/var/lib/kubelet/pods/311bafdf-e074-4918-b690-cf62eb11004a/volumes" Jul 23 04:22:10.586544 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:10.586476 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc" Jul 23 04:22:10.724215 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:10.724185 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc"] Jul 23 04:22:10.726440 ip-10-0-140-6 kubenswrapper[2567]: W0723 04:22:10.726398 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22c81c29_eb1b_441c_b77a_568d7dae87df.slice/crio-8c5318b8e354e899e467e39b9997771b2857763c27adc9b29f8d4ee0e8604a84 WatchSource:0}: Error finding container 8c5318b8e354e899e467e39b9997771b2857763c27adc9b29f8d4ee0e8604a84: Status 404 returned error can't find the container with id 8c5318b8e354e899e467e39b9997771b2857763c27adc9b29f8d4ee0e8604a84 Jul 23 04:22:10.728713 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:10.728680 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 04:22:10.728789 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:10.728755 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 04:22:10.728789 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:10.728786 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 04:22:11.138221 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:11.138182 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc" event={"ID":"22c81c29-eb1b-441c-b77a-568d7dae87df","Type":"ContainerStarted","Data":"a2d6aa319500697d7ef36464b2db4811eaa3a915e7cfdac51f9fa85b689e13a5"} Jul 23 04:22:11.138446 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:11.138245 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc" event={"ID":"22c81c29-eb1b-441c-b77a-568d7dae87df","Type":"ContainerStarted","Data":"8c5318b8e354e899e467e39b9997771b2857763c27adc9b29f8d4ee0e8604a84"} Jul 23 04:22:11.161414 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:11.161370 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc" podStartSLOduration=1.16135501 podStartE2EDuration="1.16135501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 04:22:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 04:22:11.159815485 +0000 UTC m=+1611.065119793" watchObservedRunningTime="2026-07-23 04:22:11.16135501 +0000 UTC m=+1611.066659302" Jul 23 04:22:11.587372 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:11.587338 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc" Jul 23 04:22:11.592097 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:11.592070 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc" Jul 23 04:22:12.143638 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:12.143604 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc" Jul 23 04:22:12.144601 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:12.144578 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc" Jul 23 04:22:14.464635 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:14.464610 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl" Jul 23 04:22:14.510652 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:14.510586 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d8f3129-7302-4bf9-87e4-61101b15ec86-workload-socket\") pod \"4d8f3129-7302-4bf9-87e4-61101b15ec86\" (UID: \"4d8f3129-7302-4bf9-87e4-61101b15ec86\") " Jul 23 04:22:14.510652 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:14.510624 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d8f3129-7302-4bf9-87e4-61101b15ec86-istio-podinfo\") pod \"4d8f3129-7302-4bf9-87e4-61101b15ec86\" (UID: \"4d8f3129-7302-4bf9-87e4-61101b15ec86\") " Jul 23 04:22:14.510652 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:14.510644 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d8f3129-7302-4bf9-87e4-61101b15ec86-istio-envoy\") pod \"4d8f3129-7302-4bf9-87e4-61101b15ec86\" (UID: \"4d8f3129-7302-4bf9-87e4-61101b15ec86\") " Jul 23 04:22:14.510903 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:14.510673 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d8f3129-7302-4bf9-87e4-61101b15ec86-workload-certs\") pod \"4d8f3129-7302-4bf9-87e4-61101b15ec86\" (UID: \"4d8f3129-7302-4bf9-87e4-61101b15ec86\") " Jul 23 04:22:14.510903 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:14.510702 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d8f3129-7302-4bf9-87e4-61101b15ec86-istio-data\") pod \"4d8f3129-7302-4bf9-87e4-61101b15ec86\" (UID: \"4d8f3129-7302-4bf9-87e4-61101b15ec86\") " Jul 23 04:22:14.510903 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:14.510727 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qnkfz\" (UniqueName: \"kubernetes.io/projected/4d8f3129-7302-4bf9-87e4-61101b15ec86-kube-api-access-qnkfz\") pod \"4d8f3129-7302-4bf9-87e4-61101b15ec86\" (UID: \"4d8f3129-7302-4bf9-87e4-61101b15ec86\") " Jul 23 04:22:14.510903 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:14.510753 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d8f3129-7302-4bf9-87e4-61101b15ec86-istiod-ca-cert\") pod \"4d8f3129-7302-4bf9-87e4-61101b15ec86\" (UID: \"4d8f3129-7302-4bf9-87e4-61101b15ec86\") " Jul 23 04:22:14.510903 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:14.510784 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d8f3129-7302-4bf9-87e4-61101b15ec86-credential-socket\") pod \"4d8f3129-7302-4bf9-87e4-61101b15ec86\" (UID: \"4d8f3129-7302-4bf9-87e4-61101b15ec86\") " Jul 23 04:22:14.510903 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:14.510828 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d8f3129-7302-4bf9-87e4-61101b15ec86-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4d8f3129-7302-4bf9-87e4-61101b15ec86" (UID: "4d8f3129-7302-4bf9-87e4-61101b15ec86"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 04:22:14.510903 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:14.510846 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d8f3129-7302-4bf9-87e4-61101b15ec86-istio-token\") pod \"4d8f3129-7302-4bf9-87e4-61101b15ec86\" (UID: \"4d8f3129-7302-4bf9-87e4-61101b15ec86\") " Jul 23 04:22:14.511257 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:14.510992 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d8f3129-7302-4bf9-87e4-61101b15ec86-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4d8f3129-7302-4bf9-87e4-61101b15ec86" (UID: "4d8f3129-7302-4bf9-87e4-61101b15ec86"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 04:22:14.511257 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:14.511126 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d8f3129-7302-4bf9-87e4-61101b15ec86-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4d8f3129-7302-4bf9-87e4-61101b15ec86" (UID: "4d8f3129-7302-4bf9-87e4-61101b15ec86"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 04:22:14.511257 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:14.511134 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d8f3129-7302-4bf9-87e4-61101b15ec86-workload-certs\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:22:14.511257 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:14.511167 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d8f3129-7302-4bf9-87e4-61101b15ec86-workload-socket\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:22:14.511842 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:14.511813 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d8f3129-7302-4bf9-87e4-61101b15ec86-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4d8f3129-7302-4bf9-87e4-61101b15ec86" (UID: "4d8f3129-7302-4bf9-87e4-61101b15ec86"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 04:22:14.511986 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:14.511931 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d8f3129-7302-4bf9-87e4-61101b15ec86-istio-data" (OuterVolumeSpecName: "istio-data") pod "4d8f3129-7302-4bf9-87e4-61101b15ec86" (UID: "4d8f3129-7302-4bf9-87e4-61101b15ec86"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 04:22:14.513251 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:14.513229 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d8f3129-7302-4bf9-87e4-61101b15ec86-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4d8f3129-7302-4bf9-87e4-61101b15ec86" (UID: "4d8f3129-7302-4bf9-87e4-61101b15ec86"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 04:22:14.513382 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:14.513257 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d8f3129-7302-4bf9-87e4-61101b15ec86-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4d8f3129-7302-4bf9-87e4-61101b15ec86" (UID: "4d8f3129-7302-4bf9-87e4-61101b15ec86"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 04:22:14.513382 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:14.513344 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d8f3129-7302-4bf9-87e4-61101b15ec86-istio-token" (OuterVolumeSpecName: "istio-token") pod "4d8f3129-7302-4bf9-87e4-61101b15ec86" (UID: "4d8f3129-7302-4bf9-87e4-61101b15ec86"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 04:22:14.513702 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:14.513679 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d8f3129-7302-4bf9-87e4-61101b15ec86-kube-api-access-qnkfz" (OuterVolumeSpecName: "kube-api-access-qnkfz") pod "4d8f3129-7302-4bf9-87e4-61101b15ec86" (UID: "4d8f3129-7302-4bf9-87e4-61101b15ec86"). InnerVolumeSpecName "kube-api-access-qnkfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 04:22:14.612315 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:14.612280 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d8f3129-7302-4bf9-87e4-61101b15ec86-istio-envoy\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:22:14.612315 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:14.612309 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d8f3129-7302-4bf9-87e4-61101b15ec86-istio-data\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:22:14.612526 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:14.612322 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnkfz\" (UniqueName: \"kubernetes.io/projected/4d8f3129-7302-4bf9-87e4-61101b15ec86-kube-api-access-qnkfz\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:22:14.612526 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:14.612336 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d8f3129-7302-4bf9-87e4-61101b15ec86-istiod-ca-cert\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:22:14.612526 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:14.612348 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d8f3129-7302-4bf9-87e4-61101b15ec86-credential-socket\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:22:14.612526 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:14.612358 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d8f3129-7302-4bf9-87e4-61101b15ec86-istio-token\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:22:14.612526 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:14.612369 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d8f3129-7302-4bf9-87e4-61101b15ec86-istio-podinfo\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:22:15.161258 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:15.161226 2567 generic.go:356] "Generic (PLEG): container finished" podID="4d8f3129-7302-4bf9-87e4-61101b15ec86" containerID="617bf0d48d367fd63cf8bf6caea962d66d2730fbb750f333fc5145c13e5b9cde" exitCode=0 Jul 23 04:22:15.161419 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:15.161271 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl" Jul 23 04:22:15.161419 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:15.161318 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl" event={"ID":"4d8f3129-7302-4bf9-87e4-61101b15ec86","Type":"ContainerDied","Data":"617bf0d48d367fd63cf8bf6caea962d66d2730fbb750f333fc5145c13e5b9cde"} Jul 23 04:22:15.161419 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:15.161375 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl" event={"ID":"4d8f3129-7302-4bf9-87e4-61101b15ec86","Type":"ContainerDied","Data":"2b43fb91aad4e2890762456573e8e8fbed88dddc0a3919462541a485c296bb7e"} Jul 23 04:22:15.161419 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:15.161404 2567 scope.go:117] "RemoveContainer" containerID="617bf0d48d367fd63cf8bf6caea962d66d2730fbb750f333fc5145c13e5b9cde" Jul 23 04:22:15.171125 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:15.171111 2567 scope.go:117] "RemoveContainer" containerID="721315b4109e06f7527af02ef6db14ef1db3e415b21a2d41c39cf36c295cceae" Jul 23 04:22:15.178974 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:15.178954 2567 scope.go:117] "RemoveContainer" containerID="617bf0d48d367fd63cf8bf6caea962d66d2730fbb750f333fc5145c13e5b9cde" Jul 23 04:22:15.179218 ip-10-0-140-6 kubenswrapper[2567]: E0723 04:22:15.179202 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"617bf0d48d367fd63cf8bf6caea962d66d2730fbb750f333fc5145c13e5b9cde\": container with ID starting with 617bf0d48d367fd63cf8bf6caea962d66d2730fbb750f333fc5145c13e5b9cde not found: ID does not exist" containerID="617bf0d48d367fd63cf8bf6caea962d66d2730fbb750f333fc5145c13e5b9cde" Jul 23 04:22:15.179263 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:15.179226 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"617bf0d48d367fd63cf8bf6caea962d66d2730fbb750f333fc5145c13e5b9cde"} err="failed to get container status \"617bf0d48d367fd63cf8bf6caea962d66d2730fbb750f333fc5145c13e5b9cde\": rpc error: code = NotFound desc = could not find container \"617bf0d48d367fd63cf8bf6caea962d66d2730fbb750f333fc5145c13e5b9cde\": container with ID starting with 617bf0d48d367fd63cf8bf6caea962d66d2730fbb750f333fc5145c13e5b9cde not found: ID does not exist" Jul 23 04:22:15.179263 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:15.179242 2567 scope.go:117] "RemoveContainer" containerID="721315b4109e06f7527af02ef6db14ef1db3e415b21a2d41c39cf36c295cceae" Jul 23 04:22:15.179504 ip-10-0-140-6 kubenswrapper[2567]: E0723 04:22:15.179485 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"721315b4109e06f7527af02ef6db14ef1db3e415b21a2d41c39cf36c295cceae\": container with ID starting with 721315b4109e06f7527af02ef6db14ef1db3e415b21a2d41c39cf36c295cceae not found: ID does not exist" containerID="721315b4109e06f7527af02ef6db14ef1db3e415b21a2d41c39cf36c295cceae" Jul 23 04:22:15.179565 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:15.179511 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"721315b4109e06f7527af02ef6db14ef1db3e415b21a2d41c39cf36c295cceae"} err="failed to get container status \"721315b4109e06f7527af02ef6db14ef1db3e415b21a2d41c39cf36c295cceae\": rpc error: code = NotFound desc = could not find container \"721315b4109e06f7527af02ef6db14ef1db3e415b21a2d41c39cf36c295cceae\": container with ID starting with 721315b4109e06f7527af02ef6db14ef1db3e415b21a2d41c39cf36c295cceae not found: ID does not exist" Jul 23 04:22:15.184352 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:15.184332 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl"] Jul 23 04:22:15.187853 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:15.187833 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-0ba06924-openshift-default-86bbdf47d7-c2qsl"] Jul 23 04:22:16.492487 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:16.492452 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d8f3129-7302-4bf9-87e4-61101b15ec86" path="/var/lib/kubelet/pods/4d8f3129-7302-4bf9-87e4-61101b15ec86/volumes" Jul 23 04:22:24.207099 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:24.207066 2567 generic.go:356] "Generic (PLEG): container finished" podID="22c81c29-eb1b-441c-b77a-568d7dae87df" containerID="a2d6aa319500697d7ef36464b2db4811eaa3a915e7cfdac51f9fa85b689e13a5" exitCode=0 Jul 23 04:22:24.207491 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:24.207124 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc" event={"ID":"22c81c29-eb1b-441c-b77a-568d7dae87df","Type":"ContainerDied","Data":"a2d6aa319500697d7ef36464b2db4811eaa3a915e7cfdac51f9fa85b689e13a5"} Jul 23 04:22:24.207551 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:24.207497 2567 scope.go:117] "RemoveContainer" containerID="a2d6aa319500697d7ef36464b2db4811eaa3a915e7cfdac51f9fa85b689e13a5" Jul 23 04:22:24.208607 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:24.208577 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 04:22:24.208709 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:24.208661 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 04:22:24.208768 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:24.208709 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 04:22:25.214124 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:25.214086 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc" event={"ID":"22c81c29-eb1b-441c-b77a-568d7dae87df","Type":"ContainerStarted","Data":"35d739ba4667cf4ed0ba3fcb910be0922922ebda0ca9cf3e53eebb76d3a40668"} Jul 23 04:22:25.587732 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:25.587638 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc" Jul 23 04:22:25.588874 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:25.588842 2567 patch_prober.go:28] interesting pod/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 23 04:22:25.589001 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:25.588914 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc" podUID="22c81c29-eb1b-441c-b77a-568d7dae87df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 23 04:22:26.587687 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:26.587650 2567 patch_prober.go:28] interesting pod/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 23 04:22:26.588091 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:26.587723 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc" podUID="22c81c29-eb1b-441c-b77a-568d7dae87df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 23 04:22:27.143736 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:27.143692 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc" Jul 23 04:22:27.587060 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:27.587021 2567 patch_prober.go:28] interesting pod/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 23 04:22:27.587239 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:27.587080 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc" podUID="22c81c29-eb1b-441c-b77a-568d7dae87df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 23 04:22:28.587150 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:28.587115 2567 patch_prober.go:28] interesting pod/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 23 04:22:28.587559 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:28.587174 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc" podUID="22c81c29-eb1b-441c-b77a-568d7dae87df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 23 04:22:29.590438 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:29.590390 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc" Jul 23 04:22:29.591309 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:29.591291 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc" Jul 23 04:22:37.638944 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:37.638908 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8238e07c-kgkm7"] Jul 23 04:22:37.640090 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:37.640059 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d8f3129-7302-4bf9-87e4-61101b15ec86" containerName="istio-proxy" Jul 23 04:22:37.640256 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:37.640244 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d8f3129-7302-4bf9-87e4-61101b15ec86" containerName="istio-proxy" Jul 23 04:22:37.640572 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:37.640556 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d8f3129-7302-4bf9-87e4-61101b15ec86" containerName="istio-proxy" Jul 23 04:22:37.640675 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:37.640665 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d8f3129-7302-4bf9-87e4-61101b15ec86" containerName="istio-proxy" Jul 23 04:22:37.645093 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:37.645073 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8238e07c-kgkm7" Jul 23 04:22:37.648315 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:37.648291 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dpftb\"" Jul 23 04:22:37.648315 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:37.648302 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 04:22:37.653362 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:37.653325 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8238e07c-kgkm7"] Jul 23 04:22:37.704587 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:37.704556 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/121a4e55-429c-41be-b4da-12c276936e26-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8238e07c-kgkm7\" (UID: \"121a4e55-429c-41be-b4da-12c276936e26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8238e07c-kgkm7" Jul 23 04:22:37.805193 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:37.805167 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/121a4e55-429c-41be-b4da-12c276936e26-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8238e07c-kgkm7\" (UID: \"121a4e55-429c-41be-b4da-12c276936e26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8238e07c-kgkm7" Jul 23 04:22:37.805751 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:37.805732 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/121a4e55-429c-41be-b4da-12c276936e26-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8238e07c-kgkm7\" (UID: \"121a4e55-429c-41be-b4da-12c276936e26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8238e07c-kgkm7" Jul 23 04:22:37.957410 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:37.957375 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8238e07c-kgkm7" Jul 23 04:22:38.088925 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:38.088896 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8238e07c-kgkm7"] Jul 23 04:22:38.090181 ip-10-0-140-6 kubenswrapper[2567]: W0723 04:22:38.090157 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod121a4e55_429c_41be_b4da_12c276936e26.slice/crio-afed135e999e0484b570290dfed8a8d47ecb0731de64d59141d89672ff2d60f9 WatchSource:0}: Error finding container afed135e999e0484b570290dfed8a8d47ecb0731de64d59141d89672ff2d60f9: Status 404 returned error can't find the container with id afed135e999e0484b570290dfed8a8d47ecb0731de64d59141d89672ff2d60f9 Jul 23 04:22:38.281305 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:38.281270 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8238e07c-kgkm7" event={"ID":"121a4e55-429c-41be-b4da-12c276936e26","Type":"ContainerStarted","Data":"afed135e999e0484b570290dfed8a8d47ecb0731de64d59141d89672ff2d60f9"} Jul 23 04:22:39.287085 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:39.287048 2567 generic.go:356] "Generic (PLEG): container finished" podID="121a4e55-429c-41be-b4da-12c276936e26" containerID="6dcb49e523ffcb07874ebb35ef1118d4f85fbfa0ab2ed7b1a7039c4bea6da817" exitCode=0 Jul 23 04:22:39.287558 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:39.287135 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8238e07c-kgkm7" event={"ID":"121a4e55-429c-41be-b4da-12c276936e26","Type":"ContainerDied","Data":"6dcb49e523ffcb07874ebb35ef1118d4f85fbfa0ab2ed7b1a7039c4bea6da817"} Jul 23 04:22:40.425920 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:40.425896 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8238e07c-kgkm7" Jul 23 04:22:40.532047 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:40.532016 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/121a4e55-429c-41be-b4da-12c276936e26-maas-api-service-ca\") pod \"121a4e55-429c-41be-b4da-12c276936e26\" (UID: \"121a4e55-429c-41be-b4da-12c276936e26\") " Jul 23 04:22:40.532376 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:40.532356 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/121a4e55-429c-41be-b4da-12c276936e26-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "121a4e55-429c-41be-b4da-12c276936e26" (UID: "121a4e55-429c-41be-b4da-12c276936e26"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 04:22:40.633014 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:40.632946 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/121a4e55-429c-41be-b4da-12c276936e26-maas-api-service-ca\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:22:41.297861 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:41.297835 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8238e07c-kgkm7" Jul 23 04:22:41.298029 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:41.297834 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8238e07c-kgkm7" event={"ID":"121a4e55-429c-41be-b4da-12c276936e26","Type":"ContainerDied","Data":"afed135e999e0484b570290dfed8a8d47ecb0731de64d59141d89672ff2d60f9"} Jul 23 04:22:41.298029 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:41.297960 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afed135e999e0484b570290dfed8a8d47ecb0731de64d59141d89672ff2d60f9" Jul 23 04:22:52.716462 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:52.716411 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8238e07c-kgkm7"] Jul 23 04:22:52.718643 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:52.718621 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8238e07c-kgkm7"] Jul 23 04:22:52.855713 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:52.855682 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc"] Jul 23 04:22:52.855914 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:52.855886 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc" podUID="22c81c29-eb1b-441c-b77a-568d7dae87df" containerName="istio-proxy" containerID="cri-o://35d739ba4667cf4ed0ba3fcb910be0922922ebda0ca9cf3e53eebb76d3a40668" gracePeriod=30 Jul 23 04:22:53.653849 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:53.653819 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg"] Jul 23 04:22:53.654130 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:53.654118 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d8f3129-7302-4bf9-87e4-61101b15ec86" containerName="istio-proxy" Jul 23 04:22:53.654171 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:53.654131 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d8f3129-7302-4bf9-87e4-61101b15ec86" containerName="istio-proxy" Jul 23 04:22:53.654171 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:53.654146 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="121a4e55-429c-41be-b4da-12c276936e26" containerName="revoke-keys" Jul 23 04:22:53.654171 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:53.654151 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="121a4e55-429c-41be-b4da-12c276936e26" containerName="revoke-keys" Jul 23 04:22:53.654266 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:53.654231 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="121a4e55-429c-41be-b4da-12c276936e26" containerName="revoke-keys" Jul 23 04:22:53.657114 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:53.657095 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg" Jul 23 04:22:53.659697 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:53.659667 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-5116e3b8-openshift-default-dockercfg-ks5sn\"" Jul 23 04:22:53.677118 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:53.677089 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg"] Jul 23 04:22:53.742014 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:53.741976 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65d32934-31c9-4c28-b555-2fffc9955420-istio-envoy\") pod \"e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg\" (UID: \"65d32934-31c9-4c28-b555-2fffc9955420\") " pod="openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg" Jul 23 04:22:53.742504 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:53.742033 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d32934-31c9-4c28-b555-2fffc9955420-workload-socket\") pod \"e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg\" (UID: \"65d32934-31c9-4c28-b555-2fffc9955420\") " pod="openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg" Jul 23 04:22:53.742504 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:53.742116 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65d32934-31c9-4c28-b555-2fffc9955420-istiod-ca-cert\") pod \"e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg\" (UID: \"65d32934-31c9-4c28-b555-2fffc9955420\") " pod="openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg" Jul 23 04:22:53.742504 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:53.742212 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65d32934-31c9-4c28-b555-2fffc9955420-workload-certs\") pod \"e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg\" (UID: \"65d32934-31c9-4c28-b555-2fffc9955420\") " pod="openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg" Jul 23 04:22:53.742504 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:53.742269 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65d32934-31c9-4c28-b555-2fffc9955420-istio-data\") pod \"e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg\" (UID: \"65d32934-31c9-4c28-b555-2fffc9955420\") " pod="openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg" Jul 23 04:22:53.742504 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:53.742315 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d32934-31c9-4c28-b555-2fffc9955420-credential-socket\") pod \"e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg\" (UID: \"65d32934-31c9-4c28-b555-2fffc9955420\") " pod="openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg" Jul 23 04:22:53.742504 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:53.742340 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65d32934-31c9-4c28-b555-2fffc9955420-istio-podinfo\") pod \"e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg\" (UID: \"65d32934-31c9-4c28-b555-2fffc9955420\") " pod="openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg" Jul 23 04:22:53.742504 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:53.742363 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mghjb\" (UniqueName: \"kubernetes.io/projected/65d32934-31c9-4c28-b555-2fffc9955420-kube-api-access-mghjb\") pod \"e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg\" (UID: \"65d32934-31c9-4c28-b555-2fffc9955420\") " pod="openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg" Jul 23 04:22:53.742504 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:53.742464 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65d32934-31c9-4c28-b555-2fffc9955420-istio-token\") pod \"e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg\" (UID: \"65d32934-31c9-4c28-b555-2fffc9955420\") " pod="openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg" Jul 23 04:22:53.843441 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:53.843380 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65d32934-31c9-4c28-b555-2fffc9955420-istio-envoy\") pod \"e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg\" (UID: \"65d32934-31c9-4c28-b555-2fffc9955420\") " pod="openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg" Jul 23 04:22:53.843618 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:53.843480 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d32934-31c9-4c28-b555-2fffc9955420-workload-socket\") pod \"e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg\" (UID: \"65d32934-31c9-4c28-b555-2fffc9955420\") " pod="openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg" Jul 23 04:22:53.843618 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:53.843522 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65d32934-31c9-4c28-b555-2fffc9955420-istiod-ca-cert\") pod \"e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg\" (UID: \"65d32934-31c9-4c28-b555-2fffc9955420\") " pod="openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg" Jul 23 04:22:53.843618 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:53.843568 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65d32934-31c9-4c28-b555-2fffc9955420-workload-certs\") pod \"e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg\" (UID: \"65d32934-31c9-4c28-b555-2fffc9955420\") " pod="openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg" Jul 23 04:22:53.843775 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:53.843618 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65d32934-31c9-4c28-b555-2fffc9955420-istio-data\") pod \"e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg\" (UID: \"65d32934-31c9-4c28-b555-2fffc9955420\") " pod="openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg" Jul 23 04:22:53.843775 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:53.843657 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d32934-31c9-4c28-b555-2fffc9955420-credential-socket\") pod \"e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg\" (UID: \"65d32934-31c9-4c28-b555-2fffc9955420\") " pod="openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg" Jul 23 04:22:53.843775 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:53.843684 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65d32934-31c9-4c28-b555-2fffc9955420-istio-podinfo\") pod \"e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg\" (UID: \"65d32934-31c9-4c28-b555-2fffc9955420\") " pod="openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg" Jul 23 04:22:53.843775 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:53.843709 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mghjb\" (UniqueName: \"kubernetes.io/projected/65d32934-31c9-4c28-b555-2fffc9955420-kube-api-access-mghjb\") pod \"e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg\" (UID: \"65d32934-31c9-4c28-b555-2fffc9955420\") " pod="openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg" Jul 23 04:22:53.843775 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:53.843766 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65d32934-31c9-4c28-b555-2fffc9955420-istio-token\") pod \"e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg\" (UID: \"65d32934-31c9-4c28-b555-2fffc9955420\") " pod="openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg" Jul 23 04:22:53.844018 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:53.843867 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d32934-31c9-4c28-b555-2fffc9955420-workload-socket\") pod \"e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg\" (UID: \"65d32934-31c9-4c28-b555-2fffc9955420\") " pod="openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg" Jul 23 04:22:53.844241 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:53.844216 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65d32934-31c9-4c28-b555-2fffc9955420-workload-certs\") pod \"e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg\" (UID: \"65d32934-31c9-4c28-b555-2fffc9955420\") " pod="openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg" Jul 23 04:22:53.844382 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:53.844312 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65d32934-31c9-4c28-b555-2fffc9955420-istio-data\") pod \"e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg\" (UID: \"65d32934-31c9-4c28-b555-2fffc9955420\") " pod="openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg" Jul 23 04:22:53.844475 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:53.844340 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d32934-31c9-4c28-b555-2fffc9955420-credential-socket\") pod \"e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg\" (UID: \"65d32934-31c9-4c28-b555-2fffc9955420\") " pod="openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg" Jul 23 04:22:53.844760 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:53.844735 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65d32934-31c9-4c28-b555-2fffc9955420-istiod-ca-cert\") pod \"e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg\" (UID: \"65d32934-31c9-4c28-b555-2fffc9955420\") " pod="openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg" Jul 23 04:22:53.846134 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:53.846109 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65d32934-31c9-4c28-b555-2fffc9955420-istio-envoy\") pod \"e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg\" (UID: \"65d32934-31c9-4c28-b555-2fffc9955420\") " pod="openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg" Jul 23 04:22:53.846259 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:53.846237 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65d32934-31c9-4c28-b555-2fffc9955420-istio-podinfo\") pod \"e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg\" (UID: \"65d32934-31c9-4c28-b555-2fffc9955420\") " pod="openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg" Jul 23 04:22:53.853031 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:53.853010 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mghjb\" (UniqueName: \"kubernetes.io/projected/65d32934-31c9-4c28-b555-2fffc9955420-kube-api-access-mghjb\") pod \"e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg\" (UID: \"65d32934-31c9-4c28-b555-2fffc9955420\") " pod="openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg" Jul 23 04:22:53.853116 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:53.853103 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65d32934-31c9-4c28-b555-2fffc9955420-istio-token\") pod \"e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg\" (UID: \"65d32934-31c9-4c28-b555-2fffc9955420\") " pod="openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg" Jul 23 04:22:53.967065 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:53.967033 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg" Jul 23 04:22:54.121857 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:54.121829 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg"] Jul 23 04:22:54.122575 ip-10-0-140-6 kubenswrapper[2567]: W0723 04:22:54.122551 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65d32934_31c9_4c28_b555_2fffc9955420.slice/crio-a013cdeb85996009bae68dfff8c9f8db962ae7cfd5c5fb164b9ab882d3cc6e7e WatchSource:0}: Error finding container a013cdeb85996009bae68dfff8c9f8db962ae7cfd5c5fb164b9ab882d3cc6e7e: Status 404 returned error can't find the container with id a013cdeb85996009bae68dfff8c9f8db962ae7cfd5c5fb164b9ab882d3cc6e7e Jul 23 04:22:54.124918 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:54.124885 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 04:22:54.124991 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:54.124947 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 04:22:54.124991 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:54.124979 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 04:22:54.366620 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:54.366527 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg" event={"ID":"65d32934-31c9-4c28-b555-2fffc9955420","Type":"ContainerStarted","Data":"dff38dbae6b09716c271bbec6c9588eab3cd0e00f4177b3a500d7c621cc3b448"} Jul 23 04:22:54.366775 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:54.366623 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg" event={"ID":"65d32934-31c9-4c28-b555-2fffc9955420","Type":"ContainerStarted","Data":"a013cdeb85996009bae68dfff8c9f8db962ae7cfd5c5fb164b9ab882d3cc6e7e"} Jul 23 04:22:54.387690 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:54.387645 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg" podStartSLOduration=1.387629564 podStartE2EDuration="1.387629564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 04:22:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 04:22:54.387173451 +0000 UTC m=+1654.292477763" watchObservedRunningTime="2026-07-23 04:22:54.387629564 +0000 UTC m=+1654.292933864" Jul 23 04:22:54.492387 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:54.492357 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="121a4e55-429c-41be-b4da-12c276936e26" path="/var/lib/kubelet/pods/121a4e55-429c-41be-b4da-12c276936e26/volumes" Jul 23 04:22:54.968162 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:54.968128 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg" Jul 23 04:22:55.972367 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:55.972338 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg" Jul 23 04:22:56.376944 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:56.376862 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg" Jul 23 04:22:56.377889 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:56.377871 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg" Jul 23 04:22:58.103845 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:58.103823 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc" Jul 23 04:22:58.280481 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:58.280447 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22c81c29-eb1b-441c-b77a-568d7dae87df-istiod-ca-cert\") pod \"22c81c29-eb1b-441c-b77a-568d7dae87df\" (UID: \"22c81c29-eb1b-441c-b77a-568d7dae87df\") " Jul 23 04:22:58.280481 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:58.280488 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22c81c29-eb1b-441c-b77a-568d7dae87df-istio-envoy\") pod \"22c81c29-eb1b-441c-b77a-568d7dae87df\" (UID: \"22c81c29-eb1b-441c-b77a-568d7dae87df\") " Jul 23 04:22:58.280822 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:58.280528 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8lmjw\" (UniqueName: \"kubernetes.io/projected/22c81c29-eb1b-441c-b77a-568d7dae87df-kube-api-access-8lmjw\") pod \"22c81c29-eb1b-441c-b77a-568d7dae87df\" (UID: \"22c81c29-eb1b-441c-b77a-568d7dae87df\") " Jul 23 04:22:58.280822 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:58.280567 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22c81c29-eb1b-441c-b77a-568d7dae87df-credential-socket\") pod \"22c81c29-eb1b-441c-b77a-568d7dae87df\" (UID: \"22c81c29-eb1b-441c-b77a-568d7dae87df\") " Jul 23 04:22:58.280822 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:58.280583 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22c81c29-eb1b-441c-b77a-568d7dae87df-workload-certs\") pod \"22c81c29-eb1b-441c-b77a-568d7dae87df\" (UID: \"22c81c29-eb1b-441c-b77a-568d7dae87df\") " Jul 23 04:22:58.280822 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:58.280611 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22c81c29-eb1b-441c-b77a-568d7dae87df-istio-data\") pod \"22c81c29-eb1b-441c-b77a-568d7dae87df\" (UID: \"22c81c29-eb1b-441c-b77a-568d7dae87df\") " Jul 23 04:22:58.280822 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:58.280643 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22c81c29-eb1b-441c-b77a-568d7dae87df-istio-podinfo\") pod \"22c81c29-eb1b-441c-b77a-568d7dae87df\" (UID: \"22c81c29-eb1b-441c-b77a-568d7dae87df\") " Jul 23 04:22:58.280822 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:58.280665 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22c81c29-eb1b-441c-b77a-568d7dae87df-workload-socket\") pod \"22c81c29-eb1b-441c-b77a-568d7dae87df\" (UID: \"22c81c29-eb1b-441c-b77a-568d7dae87df\") " Jul 23 04:22:58.280822 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:58.280718 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22c81c29-eb1b-441c-b77a-568d7dae87df-istio-token\") pod \"22c81c29-eb1b-441c-b77a-568d7dae87df\" (UID: \"22c81c29-eb1b-441c-b77a-568d7dae87df\") " Jul 23 04:22:58.281190 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:58.280880 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22c81c29-eb1b-441c-b77a-568d7dae87df-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "22c81c29-eb1b-441c-b77a-568d7dae87df" (UID: "22c81c29-eb1b-441c-b77a-568d7dae87df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 04:22:58.281190 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:58.280904 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22c81c29-eb1b-441c-b77a-568d7dae87df-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "22c81c29-eb1b-441c-b77a-568d7dae87df" (UID: "22c81c29-eb1b-441c-b77a-568d7dae87df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 04:22:58.281190 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:58.280942 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22c81c29-eb1b-441c-b77a-568d7dae87df-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "22c81c29-eb1b-441c-b77a-568d7dae87df" (UID: "22c81c29-eb1b-441c-b77a-568d7dae87df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 04:22:58.281190 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:58.281033 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22c81c29-eb1b-441c-b77a-568d7dae87df-istiod-ca-cert\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:22:58.281190 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:58.281049 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22c81c29-eb1b-441c-b77a-568d7dae87df-credential-socket\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:22:58.281190 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:58.281062 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22c81c29-eb1b-441c-b77a-568d7dae87df-workload-certs\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:22:58.281190 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:58.281159 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22c81c29-eb1b-441c-b77a-568d7dae87df-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "22c81c29-eb1b-441c-b77a-568d7dae87df" (UID: "22c81c29-eb1b-441c-b77a-568d7dae87df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 04:22:58.281770 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:58.281715 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22c81c29-eb1b-441c-b77a-568d7dae87df-istio-data" (OuterVolumeSpecName: "istio-data") pod "22c81c29-eb1b-441c-b77a-568d7dae87df" (UID: "22c81c29-eb1b-441c-b77a-568d7dae87df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 04:22:58.282932 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:58.282900 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22c81c29-eb1b-441c-b77a-568d7dae87df-istio-token" (OuterVolumeSpecName: "istio-token") pod "22c81c29-eb1b-441c-b77a-568d7dae87df" (UID: "22c81c29-eb1b-441c-b77a-568d7dae87df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 04:22:58.283052 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:58.282944 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22c81c29-eb1b-441c-b77a-568d7dae87df-kube-api-access-8lmjw" (OuterVolumeSpecName: "kube-api-access-8lmjw") pod "22c81c29-eb1b-441c-b77a-568d7dae87df" (UID: "22c81c29-eb1b-441c-b77a-568d7dae87df"). InnerVolumeSpecName "kube-api-access-8lmjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 04:22:58.283052 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:58.282968 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22c81c29-eb1b-441c-b77a-568d7dae87df-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "22c81c29-eb1b-441c-b77a-568d7dae87df" (UID: "22c81c29-eb1b-441c-b77a-568d7dae87df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 04:22:58.283198 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:58.283087 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22c81c29-eb1b-441c-b77a-568d7dae87df-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "22c81c29-eb1b-441c-b77a-568d7dae87df" (UID: "22c81c29-eb1b-441c-b77a-568d7dae87df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 04:22:58.381790 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:58.381764 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22c81c29-eb1b-441c-b77a-568d7dae87df-istio-token\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:22:58.381790 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:58.381786 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22c81c29-eb1b-441c-b77a-568d7dae87df-istio-envoy\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:22:58.382009 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:58.381801 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lmjw\" (UniqueName: \"kubernetes.io/projected/22c81c29-eb1b-441c-b77a-568d7dae87df-kube-api-access-8lmjw\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:22:58.382009 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:58.381813 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22c81c29-eb1b-441c-b77a-568d7dae87df-istio-data\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:22:58.382009 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:58.381822 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22c81c29-eb1b-441c-b77a-568d7dae87df-istio-podinfo\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:22:58.382009 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:58.381829 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22c81c29-eb1b-441c-b77a-568d7dae87df-workload-socket\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:22:58.387571 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:58.387546 2567 generic.go:356] "Generic (PLEG): container finished" podID="22c81c29-eb1b-441c-b77a-568d7dae87df" containerID="35d739ba4667cf4ed0ba3fcb910be0922922ebda0ca9cf3e53eebb76d3a40668" exitCode=0 Jul 23 04:22:58.387685 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:58.387599 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc" Jul 23 04:22:58.387685 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:58.387633 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc" event={"ID":"22c81c29-eb1b-441c-b77a-568d7dae87df","Type":"ContainerDied","Data":"35d739ba4667cf4ed0ba3fcb910be0922922ebda0ca9cf3e53eebb76d3a40668"} Jul 23 04:22:58.387761 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:58.387691 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc" event={"ID":"22c81c29-eb1b-441c-b77a-568d7dae87df","Type":"ContainerDied","Data":"8c5318b8e354e899e467e39b9997771b2857763c27adc9b29f8d4ee0e8604a84"} Jul 23 04:22:58.387761 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:58.387719 2567 scope.go:117] "RemoveContainer" containerID="35d739ba4667cf4ed0ba3fcb910be0922922ebda0ca9cf3e53eebb76d3a40668" Jul 23 04:22:58.401712 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:58.401695 2567 scope.go:117] "RemoveContainer" containerID="a2d6aa319500697d7ef36464b2db4811eaa3a915e7cfdac51f9fa85b689e13a5" Jul 23 04:22:58.410014 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:58.409996 2567 scope.go:117] "RemoveContainer" containerID="35d739ba4667cf4ed0ba3fcb910be0922922ebda0ca9cf3e53eebb76d3a40668" Jul 23 04:22:58.410260 ip-10-0-140-6 kubenswrapper[2567]: E0723 04:22:58.410241 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35d739ba4667cf4ed0ba3fcb910be0922922ebda0ca9cf3e53eebb76d3a40668\": container with ID starting with 35d739ba4667cf4ed0ba3fcb910be0922922ebda0ca9cf3e53eebb76d3a40668 not found: ID does not exist" containerID="35d739ba4667cf4ed0ba3fcb910be0922922ebda0ca9cf3e53eebb76d3a40668" Jul 23 04:22:58.410330 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:58.410271 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35d739ba4667cf4ed0ba3fcb910be0922922ebda0ca9cf3e53eebb76d3a40668"} err="failed to get container status \"35d739ba4667cf4ed0ba3fcb910be0922922ebda0ca9cf3e53eebb76d3a40668\": rpc error: code = NotFound desc = could not find container \"35d739ba4667cf4ed0ba3fcb910be0922922ebda0ca9cf3e53eebb76d3a40668\": container with ID starting with 35d739ba4667cf4ed0ba3fcb910be0922922ebda0ca9cf3e53eebb76d3a40668 not found: ID does not exist" Jul 23 04:22:58.410330 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:58.410295 2567 scope.go:117] "RemoveContainer" containerID="a2d6aa319500697d7ef36464b2db4811eaa3a915e7cfdac51f9fa85b689e13a5" Jul 23 04:22:58.410539 ip-10-0-140-6 kubenswrapper[2567]: E0723 04:22:58.410521 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2d6aa319500697d7ef36464b2db4811eaa3a915e7cfdac51f9fa85b689e13a5\": container with ID starting with a2d6aa319500697d7ef36464b2db4811eaa3a915e7cfdac51f9fa85b689e13a5 not found: ID does not exist" containerID="a2d6aa319500697d7ef36464b2db4811eaa3a915e7cfdac51f9fa85b689e13a5" Jul 23 04:22:58.410606 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:58.410547 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2d6aa319500697d7ef36464b2db4811eaa3a915e7cfdac51f9fa85b689e13a5"} err="failed to get container status \"a2d6aa319500697d7ef36464b2db4811eaa3a915e7cfdac51f9fa85b689e13a5\": rpc error: code = NotFound desc = could not find container \"a2d6aa319500697d7ef36464b2db4811eaa3a915e7cfdac51f9fa85b689e13a5\": container with ID starting with a2d6aa319500697d7ef36464b2db4811eaa3a915e7cfdac51f9fa85b689e13a5 not found: ID does not exist" Jul 23 04:22:58.418775 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:58.418728 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc"] Jul 23 04:22:58.421368 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:58.421349 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-8238e07c-gw-openshift-default-58f767db6c-kj7lc"] Jul 23 04:22:58.492567 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:22:58.492538 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22c81c29-eb1b-441c-b77a-568d7dae87df" path="/var/lib/kubelet/pods/22c81c29-eb1b-441c-b77a-568d7dae87df/volumes" Jul 23 04:23:11.378329 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:11.378293 2567 patch_prober.go:28] interesting pod/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 23 04:23:11.378758 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:11.378364 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg" podUID="65d32934-31c9-4c28-b555-2fffc9955420" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 23 04:23:11.455893 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:11.455865 2567 generic.go:356] "Generic (PLEG): container finished" podID="65d32934-31c9-4c28-b555-2fffc9955420" containerID="dff38dbae6b09716c271bbec6c9588eab3cd0e00f4177b3a500d7c621cc3b448" exitCode=0 Jul 23 04:23:11.456063 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:11.455926 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg" event={"ID":"65d32934-31c9-4c28-b555-2fffc9955420","Type":"ContainerDied","Data":"dff38dbae6b09716c271bbec6c9588eab3cd0e00f4177b3a500d7c621cc3b448"} Jul 23 04:23:11.456248 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:11.456235 2567 scope.go:117] "RemoveContainer" containerID="dff38dbae6b09716c271bbec6c9588eab3cd0e00f4177b3a500d7c621cc3b448" Jul 23 04:23:11.457203 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:11.457172 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 04:23:11.457301 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:11.457262 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 04:23:11.457361 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:11.457310 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 04:23:12.465829 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:12.465786 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg" event={"ID":"65d32934-31c9-4c28-b555-2fffc9955420","Type":"ContainerStarted","Data":"55c529525643459cd2c6666b21479a08c881ebe76d307fbce9e84c941408189b"} Jul 23 04:23:12.967852 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:12.967808 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg" Jul 23 04:23:12.968077 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:12.968055 2567 patch_prober.go:28] interesting pod/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 23 04:23:12.968127 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:12.968105 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg" podUID="65d32934-31c9-4c28-b555-2fffc9955420" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 23 04:23:13.967944 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:13.967902 2567 patch_prober.go:28] interesting pod/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 23 04:23:13.968355 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:13.967960 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg" podUID="65d32934-31c9-4c28-b555-2fffc9955420" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 23 04:23:14.967682 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:14.967640 2567 patch_prober.go:28] interesting pod/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 23 04:23:14.967863 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:14.967698 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg" podUID="65d32934-31c9-4c28-b555-2fffc9955420" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 23 04:23:15.967650 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:15.967612 2567 patch_prober.go:28] interesting pod/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 23 04:23:15.968052 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:15.967678 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg" podUID="65d32934-31c9-4c28-b555-2fffc9955420" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 23 04:23:16.971847 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:16.971816 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg" Jul 23 04:23:16.972241 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:16.972147 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg" Jul 23 04:23:16.972864 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:16.972843 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg" Jul 23 04:23:21.586247 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:21.586209 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5116e3b8-c2kln"] Jul 23 04:23:21.586674 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:21.586544 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22c81c29-eb1b-441c-b77a-568d7dae87df" containerName="istio-proxy" Jul 23 04:23:21.586674 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:21.586556 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="22c81c29-eb1b-441c-b77a-568d7dae87df" containerName="istio-proxy" Jul 23 04:23:21.586674 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:21.586620 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="22c81c29-eb1b-441c-b77a-568d7dae87df" containerName="istio-proxy" Jul 23 04:23:21.586674 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:21.586629 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="22c81c29-eb1b-441c-b77a-568d7dae87df" containerName="istio-proxy" Jul 23 04:23:21.589393 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:21.589378 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5116e3b8-c2kln" Jul 23 04:23:21.592060 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:21.592042 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 04:23:21.592214 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:21.592199 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dpftb\"" Jul 23 04:23:21.604197 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:21.604167 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5116e3b8-c2kln"] Jul 23 04:23:21.669894 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:21.669861 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6b63cca-ce78-43f6-9e61-33392fd11268-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5116e3b8-c2kln\" (UID: \"c6b63cca-ce78-43f6-9e61-33392fd11268\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5116e3b8-c2kln" Jul 23 04:23:21.771149 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:21.771117 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6b63cca-ce78-43f6-9e61-33392fd11268-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5116e3b8-c2kln\" (UID: \"c6b63cca-ce78-43f6-9e61-33392fd11268\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5116e3b8-c2kln" Jul 23 04:23:21.771757 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:21.771738 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6b63cca-ce78-43f6-9e61-33392fd11268-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5116e3b8-c2kln\" (UID: \"c6b63cca-ce78-43f6-9e61-33392fd11268\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5116e3b8-c2kln" Jul 23 04:23:21.904529 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:21.904453 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5116e3b8-c2kln" Jul 23 04:23:22.035843 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:22.035814 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5116e3b8-c2kln"] Jul 23 04:23:22.037076 ip-10-0-140-6 kubenswrapper[2567]: W0723 04:23:22.037049 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6b63cca_ce78_43f6_9e61_33392fd11268.slice/crio-73925616b39a055a2e9a5ea9409ce319788fa96e4381bab38904db1e42fc3cd8 WatchSource:0}: Error finding container 73925616b39a055a2e9a5ea9409ce319788fa96e4381bab38904db1e42fc3cd8: Status 404 returned error can't find the container with id 73925616b39a055a2e9a5ea9409ce319788fa96e4381bab38904db1e42fc3cd8 Jul 23 04:23:22.521839 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:22.521754 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5116e3b8-c2kln" event={"ID":"c6b63cca-ce78-43f6-9e61-33392fd11268","Type":"ContainerStarted","Data":"02646bc46834802a282e2268de65953a1699c9b85b56bfcffccf9312dd12a697"} Jul 23 04:23:22.521839 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:22.521812 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5116e3b8-c2kln" event={"ID":"c6b63cca-ce78-43f6-9e61-33392fd11268","Type":"ContainerStarted","Data":"73925616b39a055a2e9a5ea9409ce319788fa96e4381bab38904db1e42fc3cd8"} Jul 23 04:23:22.538411 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:22.538367 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5116e3b8-c2kln" podStartSLOduration=1.538352145 podStartE2EDuration="1.538352145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 04:23:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 04:23:22.537415228 +0000 UTC m=+1682.442719517" watchObservedRunningTime="2026-07-23 04:23:22.538352145 +0000 UTC m=+1682.443656437" Jul 23 04:23:23.527922 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:23.527885 2567 generic.go:356] "Generic (PLEG): container finished" podID="c6b63cca-ce78-43f6-9e61-33392fd11268" containerID="02646bc46834802a282e2268de65953a1699c9b85b56bfcffccf9312dd12a697" exitCode=0 Jul 23 04:23:23.528268 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:23.527933 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5116e3b8-c2kln" event={"ID":"c6b63cca-ce78-43f6-9e61-33392fd11268","Type":"ContainerDied","Data":"02646bc46834802a282e2268de65953a1699c9b85b56bfcffccf9312dd12a697"} Jul 23 04:23:24.666662 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:24.666638 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5116e3b8-c2kln" Jul 23 04:23:24.694212 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:24.694188 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6b63cca-ce78-43f6-9e61-33392fd11268-maas-api-service-ca\") pod \"c6b63cca-ce78-43f6-9e61-33392fd11268\" (UID: \"c6b63cca-ce78-43f6-9e61-33392fd11268\") " Jul 23 04:23:24.694547 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:24.694523 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6b63cca-ce78-43f6-9e61-33392fd11268-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c6b63cca-ce78-43f6-9e61-33392fd11268" (UID: "c6b63cca-ce78-43f6-9e61-33392fd11268"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 04:23:24.795583 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:24.795496 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6b63cca-ce78-43f6-9e61-33392fd11268-maas-api-service-ca\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:23:25.539654 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:25.539624 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5116e3b8-c2kln" Jul 23 04:23:25.539803 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:25.539667 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5116e3b8-c2kln" event={"ID":"c6b63cca-ce78-43f6-9e61-33392fd11268","Type":"ContainerDied","Data":"73925616b39a055a2e9a5ea9409ce319788fa96e4381bab38904db1e42fc3cd8"} Jul 23 04:23:25.539803 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:25.539718 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73925616b39a055a2e9a5ea9409ce319788fa96e4381bab38904db1e42fc3cd8" Jul 23 04:23:36.671559 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:36.671525 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5116e3b8-c2kln"] Jul 23 04:23:36.675954 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:36.675930 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5116e3b8-c2kln"] Jul 23 04:23:37.890472 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:37.890440 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg"] Jul 23 04:23:37.890876 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:37.890690 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg" podUID="65d32934-31c9-4c28-b555-2fffc9955420" containerName="istio-proxy" containerID="cri-o://55c529525643459cd2c6666b21479a08c881ebe76d307fbce9e84c941408189b" gracePeriod=30 Jul 23 04:23:38.492201 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:38.492163 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6b63cca-ce78-43f6-9e61-33392fd11268" path="/var/lib/kubelet/pods/c6b63cca-ce78-43f6-9e61-33392fd11268/volumes" Jul 23 04:23:43.159077 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:43.159020 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg" Jul 23 04:23:43.252825 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:43.252788 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d32934-31c9-4c28-b555-2fffc9955420-workload-socket\") pod \"65d32934-31c9-4c28-b555-2fffc9955420\" (UID: \"65d32934-31c9-4c28-b555-2fffc9955420\") " Jul 23 04:23:43.253003 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:43.252834 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d32934-31c9-4c28-b555-2fffc9955420-credential-socket\") pod \"65d32934-31c9-4c28-b555-2fffc9955420\" (UID: \"65d32934-31c9-4c28-b555-2fffc9955420\") " Jul 23 04:23:43.253003 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:43.252878 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mghjb\" (UniqueName: \"kubernetes.io/projected/65d32934-31c9-4c28-b555-2fffc9955420-kube-api-access-mghjb\") pod \"65d32934-31c9-4c28-b555-2fffc9955420\" (UID: \"65d32934-31c9-4c28-b555-2fffc9955420\") " Jul 23 04:23:43.253003 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:43.252905 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65d32934-31c9-4c28-b555-2fffc9955420-istiod-ca-cert\") pod \"65d32934-31c9-4c28-b555-2fffc9955420\" (UID: \"65d32934-31c9-4c28-b555-2fffc9955420\") " Jul 23 04:23:43.253003 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:43.252925 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65d32934-31c9-4c28-b555-2fffc9955420-istio-token\") pod \"65d32934-31c9-4c28-b555-2fffc9955420\" (UID: \"65d32934-31c9-4c28-b555-2fffc9955420\") " Jul 23 04:23:43.253003 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:43.252950 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65d32934-31c9-4c28-b555-2fffc9955420-istio-envoy\") pod \"65d32934-31c9-4c28-b555-2fffc9955420\" (UID: \"65d32934-31c9-4c28-b555-2fffc9955420\") " Jul 23 04:23:43.253258 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:43.253006 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65d32934-31c9-4c28-b555-2fffc9955420-istio-data\") pod \"65d32934-31c9-4c28-b555-2fffc9955420\" (UID: \"65d32934-31c9-4c28-b555-2fffc9955420\") " Jul 23 04:23:43.253258 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:43.253043 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65d32934-31c9-4c28-b555-2fffc9955420-workload-certs\") pod \"65d32934-31c9-4c28-b555-2fffc9955420\" (UID: \"65d32934-31c9-4c28-b555-2fffc9955420\") " Jul 23 04:23:43.253258 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:43.253065 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65d32934-31c9-4c28-b555-2fffc9955420-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "65d32934-31c9-4c28-b555-2fffc9955420" (UID: "65d32934-31c9-4c28-b555-2fffc9955420"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 04:23:43.253258 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:43.253079 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65d32934-31c9-4c28-b555-2fffc9955420-istio-podinfo\") pod \"65d32934-31c9-4c28-b555-2fffc9955420\" (UID: \"65d32934-31c9-4c28-b555-2fffc9955420\") " Jul 23 04:23:43.253258 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:43.253228 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65d32934-31c9-4c28-b555-2fffc9955420-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "65d32934-31c9-4c28-b555-2fffc9955420" (UID: "65d32934-31c9-4c28-b555-2fffc9955420"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 04:23:43.253560 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:43.253268 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65d32934-31c9-4c28-b555-2fffc9955420-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "65d32934-31c9-4c28-b555-2fffc9955420" (UID: "65d32934-31c9-4c28-b555-2fffc9955420"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 04:23:43.253560 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:43.253407 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65d32934-31c9-4c28-b555-2fffc9955420-istiod-ca-cert\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:23:43.253560 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:43.253456 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d32934-31c9-4c28-b555-2fffc9955420-workload-socket\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:23:43.253560 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:43.253469 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d32934-31c9-4c28-b555-2fffc9955420-credential-socket\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:23:43.253560 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:43.253480 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65d32934-31c9-4c28-b555-2fffc9955420-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "65d32934-31c9-4c28-b555-2fffc9955420" (UID: "65d32934-31c9-4c28-b555-2fffc9955420"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 04:23:43.254004 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:43.253980 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65d32934-31c9-4c28-b555-2fffc9955420-istio-data" (OuterVolumeSpecName: "istio-data") pod "65d32934-31c9-4c28-b555-2fffc9955420" (UID: "65d32934-31c9-4c28-b555-2fffc9955420"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 04:23:43.255368 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:43.255346 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65d32934-31c9-4c28-b555-2fffc9955420-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "65d32934-31c9-4c28-b555-2fffc9955420" (UID: "65d32934-31c9-4c28-b555-2fffc9955420"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 04:23:43.255491 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:43.255377 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65d32934-31c9-4c28-b555-2fffc9955420-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "65d32934-31c9-4c28-b555-2fffc9955420" (UID: "65d32934-31c9-4c28-b555-2fffc9955420"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 04:23:43.255557 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:43.255540 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65d32934-31c9-4c28-b555-2fffc9955420-kube-api-access-mghjb" (OuterVolumeSpecName: "kube-api-access-mghjb") pod "65d32934-31c9-4c28-b555-2fffc9955420" (UID: "65d32934-31c9-4c28-b555-2fffc9955420"). InnerVolumeSpecName "kube-api-access-mghjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 04:23:43.255631 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:43.255608 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65d32934-31c9-4c28-b555-2fffc9955420-istio-token" (OuterVolumeSpecName: "istio-token") pod "65d32934-31c9-4c28-b555-2fffc9955420" (UID: "65d32934-31c9-4c28-b555-2fffc9955420"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 04:23:43.354861 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:43.354837 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65d32934-31c9-4c28-b555-2fffc9955420-istio-data\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:23:43.354861 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:43.354856 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65d32934-31c9-4c28-b555-2fffc9955420-workload-certs\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:23:43.355007 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:43.354871 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65d32934-31c9-4c28-b555-2fffc9955420-istio-podinfo\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:23:43.355007 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:43.354883 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mghjb\" (UniqueName: \"kubernetes.io/projected/65d32934-31c9-4c28-b555-2fffc9955420-kube-api-access-mghjb\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:23:43.355007 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:43.354891 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65d32934-31c9-4c28-b555-2fffc9955420-istio-token\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:23:43.355007 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:43.354899 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65d32934-31c9-4c28-b555-2fffc9955420-istio-envoy\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:23:43.630866 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:43.630830 2567 generic.go:356] "Generic (PLEG): container finished" podID="65d32934-31c9-4c28-b555-2fffc9955420" containerID="55c529525643459cd2c6666b21479a08c881ebe76d307fbce9e84c941408189b" exitCode=0 Jul 23 04:23:43.631056 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:43.630884 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg" Jul 23 04:23:43.631056 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:43.630922 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg" event={"ID":"65d32934-31c9-4c28-b555-2fffc9955420","Type":"ContainerDied","Data":"55c529525643459cd2c6666b21479a08c881ebe76d307fbce9e84c941408189b"} Jul 23 04:23:43.631056 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:43.630976 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg" event={"ID":"65d32934-31c9-4c28-b555-2fffc9955420","Type":"ContainerDied","Data":"a013cdeb85996009bae68dfff8c9f8db962ae7cfd5c5fb164b9ab882d3cc6e7e"} Jul 23 04:23:43.631056 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:43.630999 2567 scope.go:117] "RemoveContainer" containerID="55c529525643459cd2c6666b21479a08c881ebe76d307fbce9e84c941408189b" Jul 23 04:23:43.640450 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:43.640412 2567 scope.go:117] "RemoveContainer" containerID="dff38dbae6b09716c271bbec6c9588eab3cd0e00f4177b3a500d7c621cc3b448" Jul 23 04:23:43.649237 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:43.649057 2567 scope.go:117] "RemoveContainer" containerID="55c529525643459cd2c6666b21479a08c881ebe76d307fbce9e84c941408189b" Jul 23 04:23:43.649593 ip-10-0-140-6 kubenswrapper[2567]: E0723 04:23:43.649480 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55c529525643459cd2c6666b21479a08c881ebe76d307fbce9e84c941408189b\": container with ID starting with 55c529525643459cd2c6666b21479a08c881ebe76d307fbce9e84c941408189b not found: ID does not exist" containerID="55c529525643459cd2c6666b21479a08c881ebe76d307fbce9e84c941408189b" Jul 23 04:23:43.649593 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:43.649515 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55c529525643459cd2c6666b21479a08c881ebe76d307fbce9e84c941408189b"} err="failed to get container status \"55c529525643459cd2c6666b21479a08c881ebe76d307fbce9e84c941408189b\": rpc error: code = NotFound desc = could not find container \"55c529525643459cd2c6666b21479a08c881ebe76d307fbce9e84c941408189b\": container with ID starting with 55c529525643459cd2c6666b21479a08c881ebe76d307fbce9e84c941408189b not found: ID does not exist" Jul 23 04:23:43.649593 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:43.649536 2567 scope.go:117] "RemoveContainer" containerID="dff38dbae6b09716c271bbec6c9588eab3cd0e00f4177b3a500d7c621cc3b448" Jul 23 04:23:43.650703 ip-10-0-140-6 kubenswrapper[2567]: E0723 04:23:43.650678 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dff38dbae6b09716c271bbec6c9588eab3cd0e00f4177b3a500d7c621cc3b448\": container with ID starting with dff38dbae6b09716c271bbec6c9588eab3cd0e00f4177b3a500d7c621cc3b448 not found: ID does not exist" containerID="dff38dbae6b09716c271bbec6c9588eab3cd0e00f4177b3a500d7c621cc3b448" Jul 23 04:23:43.650807 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:43.650706 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dff38dbae6b09716c271bbec6c9588eab3cd0e00f4177b3a500d7c621cc3b448"} err="failed to get container status \"dff38dbae6b09716c271bbec6c9588eab3cd0e00f4177b3a500d7c621cc3b448\": rpc error: code = NotFound desc = could not find container \"dff38dbae6b09716c271bbec6c9588eab3cd0e00f4177b3a500d7c621cc3b448\": container with ID starting with dff38dbae6b09716c271bbec6c9588eab3cd0e00f4177b3a500d7c621cc3b448 not found: ID does not exist" Jul 23 04:23:43.657595 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:43.657575 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg"] Jul 23 04:23:43.664654 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:43.664636 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-5116e3b8-openshift-default-6c649c4bdf-t9ljg"] Jul 23 04:23:44.492177 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:23:44.492139 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65d32934-31c9-4c28-b555-2fffc9955420" path="/var/lib/kubelet/pods/65d32934-31c9-4c28-b555-2fffc9955420/volumes" Jul 23 04:24:06.390480 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:24:06.390448 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d9dfcbba-wvgrh"] Jul 23 04:24:06.390984 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:24:06.390933 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22c81c29-eb1b-441c-b77a-568d7dae87df" containerName="istio-proxy" Jul 23 04:24:06.390984 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:24:06.390951 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="22c81c29-eb1b-441c-b77a-568d7dae87df" containerName="istio-proxy" Jul 23 04:24:06.390984 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:24:06.390963 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65d32934-31c9-4c28-b555-2fffc9955420" containerName="istio-proxy" Jul 23 04:24:06.390984 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:24:06.390970 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="65d32934-31c9-4c28-b555-2fffc9955420" containerName="istio-proxy" Jul 23 04:24:06.390984 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:24:06.390983 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6b63cca-ce78-43f6-9e61-33392fd11268" containerName="revoke-keys" Jul 23 04:24:06.391245 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:24:06.390992 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6b63cca-ce78-43f6-9e61-33392fd11268" containerName="revoke-keys" Jul 23 04:24:06.391245 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:24:06.391007 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65d32934-31c9-4c28-b555-2fffc9955420" containerName="istio-proxy" Jul 23 04:24:06.391245 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:24:06.391016 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="65d32934-31c9-4c28-b555-2fffc9955420" containerName="istio-proxy" Jul 23 04:24:06.391245 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:24:06.391093 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6b63cca-ce78-43f6-9e61-33392fd11268" containerName="revoke-keys" Jul 23 04:24:06.391245 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:24:06.391111 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="65d32934-31c9-4c28-b555-2fffc9955420" containerName="istio-proxy" Jul 23 04:24:06.394067 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:24:06.394045 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d9dfcbba-wvgrh" Jul 23 04:24:06.396789 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:24:06.396769 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dpftb\"" Jul 23 04:24:06.396895 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:24:06.396771 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 04:24:06.402473 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:24:06.402449 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d9dfcbba-wvgrh"] Jul 23 04:24:06.451173 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:24:06.451143 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09508738-5604-4f4c-a6a7-328d85dd47a5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d9dfcbba-wvgrh\" (UID: \"09508738-5604-4f4c-a6a7-328d85dd47a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d9dfcbba-wvgrh" Jul 23 04:24:06.552578 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:24:06.552550 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09508738-5604-4f4c-a6a7-328d85dd47a5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d9dfcbba-wvgrh\" (UID: \"09508738-5604-4f4c-a6a7-328d85dd47a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d9dfcbba-wvgrh" Jul 23 04:24:06.553153 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:24:06.553135 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09508738-5604-4f4c-a6a7-328d85dd47a5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d9dfcbba-wvgrh\" (UID: \"09508738-5604-4f4c-a6a7-328d85dd47a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d9dfcbba-wvgrh" Jul 23 04:24:06.705043 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:24:06.705016 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d9dfcbba-wvgrh" Jul 23 04:24:06.837740 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:24:06.837711 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d9dfcbba-wvgrh"] Jul 23 04:24:06.839273 ip-10-0-140-6 kubenswrapper[2567]: W0723 04:24:06.839244 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09508738_5604_4f4c_a6a7_328d85dd47a5.slice/crio-705e40ca38e78fa15ee213e13c9a39b050db9b5ff2d8f35c06c8cdbf61e1124b WatchSource:0}: Error finding container 705e40ca38e78fa15ee213e13c9a39b050db9b5ff2d8f35c06c8cdbf61e1124b: Status 404 returned error can't find the container with id 705e40ca38e78fa15ee213e13c9a39b050db9b5ff2d8f35c06c8cdbf61e1124b Jul 23 04:24:07.753266 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:24:07.753219 2567 generic.go:356] "Generic (PLEG): container finished" podID="09508738-5604-4f4c-a6a7-328d85dd47a5" containerID="13a8db71cc9ba98b55c5f754239016fcbb81f04d4bd7f4817b659c75cc1e7ca3" exitCode=0 Jul 23 04:24:07.753675 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:24:07.753326 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d9dfcbba-wvgrh" event={"ID":"09508738-5604-4f4c-a6a7-328d85dd47a5","Type":"ContainerDied","Data":"13a8db71cc9ba98b55c5f754239016fcbb81f04d4bd7f4817b659c75cc1e7ca3"} Jul 23 04:24:07.753675 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:24:07.753399 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d9dfcbba-wvgrh" event={"ID":"09508738-5604-4f4c-a6a7-328d85dd47a5","Type":"ContainerStarted","Data":"705e40ca38e78fa15ee213e13c9a39b050db9b5ff2d8f35c06c8cdbf61e1124b"} Jul 23 04:24:08.887698 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:24:08.887676 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d9dfcbba-wvgrh" Jul 23 04:24:08.971325 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:24:08.971297 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09508738-5604-4f4c-a6a7-328d85dd47a5-maas-api-service-ca\") pod \"09508738-5604-4f4c-a6a7-328d85dd47a5\" (UID: \"09508738-5604-4f4c-a6a7-328d85dd47a5\") " Jul 23 04:24:08.971676 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:24:08.971655 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09508738-5604-4f4c-a6a7-328d85dd47a5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "09508738-5604-4f4c-a6a7-328d85dd47a5" (UID: "09508738-5604-4f4c-a6a7-328d85dd47a5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 04:24:09.072757 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:24:09.072686 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09508738-5604-4f4c-a6a7-328d85dd47a5-maas-api-service-ca\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:24:09.766919 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:24:09.766890 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d9dfcbba-wvgrh" Jul 23 04:24:09.767086 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:24:09.766920 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d9dfcbba-wvgrh" event={"ID":"09508738-5604-4f4c-a6a7-328d85dd47a5","Type":"ContainerDied","Data":"705e40ca38e78fa15ee213e13c9a39b050db9b5ff2d8f35c06c8cdbf61e1124b"} Jul 23 04:24:09.767086 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:24:09.766965 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="705e40ca38e78fa15ee213e13c9a39b050db9b5ff2d8f35c06c8cdbf61e1124b" Jul 23 04:24:21.472451 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:24:21.472396 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d9dfcbba-wvgrh"] Jul 23 04:24:21.475791 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:24:21.475764 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d9dfcbba-wvgrh"] Jul 23 04:24:22.492337 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:24:22.492292 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09508738-5604-4f4c-a6a7-328d85dd47a5" path="/var/lib/kubelet/pods/09508738-5604-4f4c-a6a7-328d85dd47a5/volumes" Jul 23 04:25:20.581237 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:25:20.581207 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wzgn9_dbd53593-e5e7-473a-b71c-8fbf9272913f/console-operator/2.log" Jul 23 04:25:20.582829 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:25:20.582808 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wzgn9_dbd53593-e5e7-473a-b71c-8fbf9272913f/console-operator/2.log" Jul 23 04:25:59.085336 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:25:59.085296 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2"] Jul 23 04:25:59.085988 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:25:59.085631 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09508738-5604-4f4c-a6a7-328d85dd47a5" containerName="revoke-keys" Jul 23 04:25:59.085988 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:25:59.085644 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="09508738-5604-4f4c-a6a7-328d85dd47a5" containerName="revoke-keys" Jul 23 04:25:59.085988 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:25:59.085713 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="09508738-5604-4f4c-a6a7-328d85dd47a5" containerName="revoke-keys" Jul 23 04:25:59.085988 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:25:59.085724 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="65d32934-31c9-4c28-b555-2fffc9955420" containerName="istio-proxy" Jul 23 04:25:59.088620 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:25:59.088605 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2" Jul 23 04:25:59.091787 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:25:59.091765 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-dc495b0d-openshift-default-dockercfg-pm4zv\"" Jul 23 04:25:59.120000 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:25:59.119971 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2"] Jul 23 04:25:59.169389 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:25:59.169366 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac188105-3461-4ded-b338-82dc36c2296a-istio-token\") pod \"e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2\" (UID: \"ac188105-3461-4ded-b338-82dc36c2296a\") " pod="openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2" Jul 23 04:25:59.169521 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:25:59.169395 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j62x\" (UniqueName: \"kubernetes.io/projected/ac188105-3461-4ded-b338-82dc36c2296a-kube-api-access-6j62x\") pod \"e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2\" (UID: \"ac188105-3461-4ded-b338-82dc36c2296a\") " pod="openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2" Jul 23 04:25:59.169521 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:25:59.169417 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac188105-3461-4ded-b338-82dc36c2296a-workload-socket\") pod \"e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2\" (UID: \"ac188105-3461-4ded-b338-82dc36c2296a\") " pod="openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2" Jul 23 04:25:59.169521 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:25:59.169459 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac188105-3461-4ded-b338-82dc36c2296a-istio-envoy\") pod \"e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2\" (UID: \"ac188105-3461-4ded-b338-82dc36c2296a\") " pod="openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2" Jul 23 04:25:59.169521 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:25:59.169480 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac188105-3461-4ded-b338-82dc36c2296a-istiod-ca-cert\") pod \"e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2\" (UID: \"ac188105-3461-4ded-b338-82dc36c2296a\") " pod="openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2" Jul 23 04:25:59.169521 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:25:59.169519 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac188105-3461-4ded-b338-82dc36c2296a-istio-data\") pod \"e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2\" (UID: \"ac188105-3461-4ded-b338-82dc36c2296a\") " pod="openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2" Jul 23 04:25:59.169757 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:25:59.169537 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac188105-3461-4ded-b338-82dc36c2296a-workload-certs\") pod \"e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2\" (UID: \"ac188105-3461-4ded-b338-82dc36c2296a\") " pod="openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2" Jul 23 04:25:59.169757 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:25:59.169553 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac188105-3461-4ded-b338-82dc36c2296a-credential-socket\") pod \"e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2\" (UID: \"ac188105-3461-4ded-b338-82dc36c2296a\") " pod="openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2" Jul 23 04:25:59.169757 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:25:59.169664 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac188105-3461-4ded-b338-82dc36c2296a-istio-podinfo\") pod \"e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2\" (UID: \"ac188105-3461-4ded-b338-82dc36c2296a\") " pod="openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2" Jul 23 04:25:59.270934 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:25:59.270908 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac188105-3461-4ded-b338-82dc36c2296a-istio-data\") pod \"e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2\" (UID: \"ac188105-3461-4ded-b338-82dc36c2296a\") " pod="openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2" Jul 23 04:25:59.271097 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:25:59.270939 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac188105-3461-4ded-b338-82dc36c2296a-workload-certs\") pod \"e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2\" (UID: \"ac188105-3461-4ded-b338-82dc36c2296a\") " pod="openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2" Jul 23 04:25:59.271097 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:25:59.270959 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac188105-3461-4ded-b338-82dc36c2296a-credential-socket\") pod \"e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2\" (UID: \"ac188105-3461-4ded-b338-82dc36c2296a\") " pod="openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2" Jul 23 04:25:59.271097 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:25:59.270987 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac188105-3461-4ded-b338-82dc36c2296a-istio-podinfo\") pod \"e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2\" (UID: \"ac188105-3461-4ded-b338-82dc36c2296a\") " pod="openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2" Jul 23 04:25:59.271097 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:25:59.271033 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac188105-3461-4ded-b338-82dc36c2296a-istio-token\") pod \"e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2\" (UID: \"ac188105-3461-4ded-b338-82dc36c2296a\") " pod="openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2" Jul 23 04:25:59.271097 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:25:59.271057 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j62x\" (UniqueName: \"kubernetes.io/projected/ac188105-3461-4ded-b338-82dc36c2296a-kube-api-access-6j62x\") pod \"e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2\" (UID: \"ac188105-3461-4ded-b338-82dc36c2296a\") " pod="openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2" Jul 23 04:25:59.271097 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:25:59.271083 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac188105-3461-4ded-b338-82dc36c2296a-workload-socket\") pod \"e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2\" (UID: \"ac188105-3461-4ded-b338-82dc36c2296a\") " pod="openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2" Jul 23 04:25:59.271386 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:25:59.271231 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac188105-3461-4ded-b338-82dc36c2296a-istio-envoy\") pod \"e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2\" (UID: \"ac188105-3461-4ded-b338-82dc36c2296a\") " pod="openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2" Jul 23 04:25:59.271386 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:25:59.271275 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac188105-3461-4ded-b338-82dc36c2296a-istiod-ca-cert\") pod \"e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2\" (UID: \"ac188105-3461-4ded-b338-82dc36c2296a\") " pod="openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2" Jul 23 04:25:59.271386 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:25:59.271346 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac188105-3461-4ded-b338-82dc36c2296a-istio-data\") pod \"e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2\" (UID: \"ac188105-3461-4ded-b338-82dc36c2296a\") " pod="openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2" Jul 23 04:25:59.271573 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:25:59.271529 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac188105-3461-4ded-b338-82dc36c2296a-credential-socket\") pod \"e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2\" (UID: \"ac188105-3461-4ded-b338-82dc36c2296a\") " pod="openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2" Jul 23 04:25:59.271621 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:25:59.271574 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac188105-3461-4ded-b338-82dc36c2296a-workload-socket\") pod \"e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2\" (UID: \"ac188105-3461-4ded-b338-82dc36c2296a\") " pod="openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2" Jul 23 04:25:59.271655 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:25:59.271620 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac188105-3461-4ded-b338-82dc36c2296a-workload-certs\") pod \"e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2\" (UID: \"ac188105-3461-4ded-b338-82dc36c2296a\") " pod="openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2" Jul 23 04:25:59.271904 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:25:59.271887 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac188105-3461-4ded-b338-82dc36c2296a-istiod-ca-cert\") pod \"e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2\" (UID: \"ac188105-3461-4ded-b338-82dc36c2296a\") " pod="openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2" Jul 23 04:25:59.273554 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:25:59.273531 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac188105-3461-4ded-b338-82dc36c2296a-istio-envoy\") pod \"e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2\" (UID: \"ac188105-3461-4ded-b338-82dc36c2296a\") " pod="openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2" Jul 23 04:25:59.273670 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:25:59.273653 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac188105-3461-4ded-b338-82dc36c2296a-istio-podinfo\") pod \"e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2\" (UID: \"ac188105-3461-4ded-b338-82dc36c2296a\") " pod="openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2" Jul 23 04:25:59.279216 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:25:59.279191 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac188105-3461-4ded-b338-82dc36c2296a-istio-token\") pod \"e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2\" (UID: \"ac188105-3461-4ded-b338-82dc36c2296a\") " pod="openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2" Jul 23 04:25:59.279401 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:25:59.279385 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j62x\" (UniqueName: \"kubernetes.io/projected/ac188105-3461-4ded-b338-82dc36c2296a-kube-api-access-6j62x\") pod \"e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2\" (UID: \"ac188105-3461-4ded-b338-82dc36c2296a\") " pod="openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2" Jul 23 04:25:59.398713 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:25:59.398632 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2" Jul 23 04:25:59.542276 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:25:59.542247 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2"] Jul 23 04:25:59.544537 ip-10-0-140-6 kubenswrapper[2567]: W0723 04:25:59.544504 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac188105_3461_4ded_b338_82dc36c2296a.slice/crio-89e4083846f354c7bf4a4449cc4229e169cfd3b5173bf7ef76323cd11b9cce79 WatchSource:0}: Error finding container 89e4083846f354c7bf4a4449cc4229e169cfd3b5173bf7ef76323cd11b9cce79: Status 404 returned error can't find the container with id 89e4083846f354c7bf4a4449cc4229e169cfd3b5173bf7ef76323cd11b9cce79 Jul 23 04:25:59.547170 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:25:59.547134 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 04:25:59.547249 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:25:59.547204 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 04:25:59.547249 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:25:59.547237 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 04:26:00.336909 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:00.336870 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2" event={"ID":"ac188105-3461-4ded-b338-82dc36c2296a","Type":"ContainerStarted","Data":"2742e4dbd2cb85b3993462e5cf2cab73ba5bde5ae0295e2c6ecd9bc9367af383"} Jul 23 04:26:00.337361 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:00.336951 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2" event={"ID":"ac188105-3461-4ded-b338-82dc36c2296a","Type":"ContainerStarted","Data":"89e4083846f354c7bf4a4449cc4229e169cfd3b5173bf7ef76323cd11b9cce79"} Jul 23 04:26:00.359252 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:00.359211 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2" podStartSLOduration=1.35919973 podStartE2EDuration="1.35919973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 04:25:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 04:26:00.357677328 +0000 UTC m=+1840.262981638" watchObservedRunningTime="2026-07-23 04:26:00.35919973 +0000 UTC m=+1840.264504021" Jul 23 04:26:00.399264 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:00.399238 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2" Jul 23 04:26:00.403992 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:00.403969 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2" Jul 23 04:26:01.341787 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:01.341754 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2" Jul 23 04:26:01.342744 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:01.342722 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2" Jul 23 04:26:14.379022 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:14.378981 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dc495b0d-qm4nq"] Jul 23 04:26:14.382713 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:14.382695 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dc495b0d-qm4nq" Jul 23 04:26:14.386779 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:14.386754 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dpftb\"" Jul 23 04:26:14.386901 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:14.386813 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 04:26:14.392450 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:14.392405 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dc495b0d-qm4nq"] Jul 23 04:26:14.497052 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:14.497014 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dd59cb6-280a-447a-9d0c-71c3187ff747-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-dc495b0d-qm4nq\" (UID: \"1dd59cb6-280a-447a-9d0c-71c3187ff747\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dc495b0d-qm4nq" Jul 23 04:26:14.598064 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:14.597983 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dd59cb6-280a-447a-9d0c-71c3187ff747-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-dc495b0d-qm4nq\" (UID: \"1dd59cb6-280a-447a-9d0c-71c3187ff747\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dc495b0d-qm4nq" Jul 23 04:26:14.598574 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:14.598554 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dd59cb6-280a-447a-9d0c-71c3187ff747-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-dc495b0d-qm4nq\" (UID: \"1dd59cb6-280a-447a-9d0c-71c3187ff747\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dc495b0d-qm4nq" Jul 23 04:26:14.695126 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:14.695088 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dc495b0d-qm4nq" Jul 23 04:26:14.853250 ip-10-0-140-6 kubenswrapper[2567]: W0723 04:26:14.853218 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dd59cb6_280a_447a_9d0c_71c3187ff747.slice/crio-6abcda937b8074d3e25bd23ad4a2e9078f92a54e87d25fba5e144319498e1224 WatchSource:0}: Error finding container 6abcda937b8074d3e25bd23ad4a2e9078f92a54e87d25fba5e144319498e1224: Status 404 returned error can't find the container with id 6abcda937b8074d3e25bd23ad4a2e9078f92a54e87d25fba5e144319498e1224 Jul 23 04:26:14.853773 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:14.853747 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dc495b0d-qm4nq"] Jul 23 04:26:15.418332 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:15.418229 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dc495b0d-qm4nq" event={"ID":"1dd59cb6-280a-447a-9d0c-71c3187ff747","Type":"ContainerStarted","Data":"a8c0b159475efc4bd2119b6ca22ea4ff7503c2645dde70d7077702a25c2565bc"} Jul 23 04:26:15.418332 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:15.418296 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dc495b0d-qm4nq" event={"ID":"1dd59cb6-280a-447a-9d0c-71c3187ff747","Type":"ContainerStarted","Data":"6abcda937b8074d3e25bd23ad4a2e9078f92a54e87d25fba5e144319498e1224"} Jul 23 04:26:15.436886 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:15.436831 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dc495b0d-qm4nq" podStartSLOduration=1.4368142320000001 podStartE2EDuration="1.436814232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 04:26:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 04:26:15.433779098 +0000 UTC m=+1855.339083391" watchObservedRunningTime="2026-07-23 04:26:15.436814232 +0000 UTC m=+1855.342118524" Jul 23 04:26:16.425944 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:16.425912 2567 generic.go:356] "Generic (PLEG): container finished" podID="1dd59cb6-280a-447a-9d0c-71c3187ff747" containerID="a8c0b159475efc4bd2119b6ca22ea4ff7503c2645dde70d7077702a25c2565bc" exitCode=0 Jul 23 04:26:16.426323 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:16.425983 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dc495b0d-qm4nq" event={"ID":"1dd59cb6-280a-447a-9d0c-71c3187ff747","Type":"ContainerDied","Data":"a8c0b159475efc4bd2119b6ca22ea4ff7503c2645dde70d7077702a25c2565bc"} Jul 23 04:26:17.103368 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:17.101683 2567 patch_prober.go:28] interesting pod/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": read tcp 10.133.0.2:43644->10.133.0.48:15021: read: connection reset by peer" start-of-body= Jul 23 04:26:17.103368 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:17.101749 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2" podUID="ac188105-3461-4ded-b338-82dc36c2296a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": read tcp 10.133.0.2:43644->10.133.0.48:15021: read: connection reset by peer" Jul 23 04:26:17.432989 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:17.432958 2567 generic.go:356] "Generic (PLEG): container finished" podID="ac188105-3461-4ded-b338-82dc36c2296a" containerID="2742e4dbd2cb85b3993462e5cf2cab73ba5bde5ae0295e2c6ecd9bc9367af383" exitCode=0 Jul 23 04:26:17.433488 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:17.433034 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2" event={"ID":"ac188105-3461-4ded-b338-82dc36c2296a","Type":"ContainerDied","Data":"2742e4dbd2cb85b3993462e5cf2cab73ba5bde5ae0295e2c6ecd9bc9367af383"} Jul 23 04:26:17.433878 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:17.433526 2567 scope.go:117] "RemoveContainer" containerID="2742e4dbd2cb85b3993462e5cf2cab73ba5bde5ae0295e2c6ecd9bc9367af383" Jul 23 04:26:17.434733 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:17.434695 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 04:26:17.434838 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:17.434818 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 04:26:17.434877 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:17.434866 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 04:26:17.574809 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:17.574781 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dc495b0d-qm4nq" Jul 23 04:26:17.629030 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:17.628997 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dd59cb6-280a-447a-9d0c-71c3187ff747-maas-api-service-ca\") pod \"1dd59cb6-280a-447a-9d0c-71c3187ff747\" (UID: \"1dd59cb6-280a-447a-9d0c-71c3187ff747\") " Jul 23 04:26:17.629345 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:17.629319 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dd59cb6-280a-447a-9d0c-71c3187ff747-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1dd59cb6-280a-447a-9d0c-71c3187ff747" (UID: "1dd59cb6-280a-447a-9d0c-71c3187ff747"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 04:26:17.730110 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:17.730035 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dd59cb6-280a-447a-9d0c-71c3187ff747-maas-api-service-ca\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:26:18.440116 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:18.440081 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2" event={"ID":"ac188105-3461-4ded-b338-82dc36c2296a","Type":"ContainerStarted","Data":"dec20019522873a52d7f239a137465e79b65781603f9251e8f6cafbb0db4a953"} Jul 23 04:26:18.441323 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:18.441301 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dc495b0d-qm4nq" Jul 23 04:26:18.441449 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:18.441323 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dc495b0d-qm4nq" event={"ID":"1dd59cb6-280a-447a-9d0c-71c3187ff747","Type":"ContainerDied","Data":"6abcda937b8074d3e25bd23ad4a2e9078f92a54e87d25fba5e144319498e1224"} Jul 23 04:26:18.441449 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:18.441363 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6abcda937b8074d3e25bd23ad4a2e9078f92a54e87d25fba5e144319498e1224" Jul 23 04:26:19.399726 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:19.399696 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2" Jul 23 04:26:19.399910 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:19.399859 2567 patch_prober.go:28] interesting pod/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 23 04:26:19.399967 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:19.399928 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2" podUID="ac188105-3461-4ded-b338-82dc36c2296a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 23 04:26:20.399621 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:20.399584 2567 patch_prober.go:28] interesting pod/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 23 04:26:20.400008 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:20.399647 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2" podUID="ac188105-3461-4ded-b338-82dc36c2296a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 23 04:26:21.399164 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:21.399130 2567 patch_prober.go:28] interesting pod/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 23 04:26:21.399346 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:21.399199 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2" podUID="ac188105-3461-4ded-b338-82dc36c2296a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 23 04:26:22.399234 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:22.399197 2567 patch_prober.go:28] interesting pod/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 23 04:26:22.399672 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:22.399250 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2" podUID="ac188105-3461-4ded-b338-82dc36c2296a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 23 04:26:23.402280 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:23.402251 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2" Jul 23 04:26:23.402723 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:23.402600 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2" Jul 23 04:26:23.403328 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:23.403307 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2" Jul 23 04:26:29.470373 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:29.470336 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dc495b0d-qm4nq"] Jul 23 04:26:29.473623 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:29.473594 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dc495b0d-qm4nq"] Jul 23 04:26:30.492402 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:30.492356 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1dd59cb6-280a-447a-9d0c-71c3187ff747" path="/var/lib/kubelet/pods/1dd59cb6-280a-447a-9d0c-71c3187ff747/volumes" Jul 23 04:26:36.157651 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:36.157619 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2"] Jul 23 04:26:36.158084 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:36.157823 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2" podUID="ac188105-3461-4ded-b338-82dc36c2296a" containerName="istio-proxy" containerID="cri-o://dec20019522873a52d7f239a137465e79b65781603f9251e8f6cafbb0db4a953" gracePeriod=30 Jul 23 04:26:38.405406 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:38.405369 2567 patch_prober.go:28] interesting pod/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 23 04:26:38.405795 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:38.405460 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2" podUID="ac188105-3461-4ded-b338-82dc36c2296a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 23 04:26:41.409921 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:41.409894 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2" Jul 23 04:26:41.527209 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:41.527134 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac188105-3461-4ded-b338-82dc36c2296a-istio-data\") pod \"ac188105-3461-4ded-b338-82dc36c2296a\" (UID: \"ac188105-3461-4ded-b338-82dc36c2296a\") " Jul 23 04:26:41.527209 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:41.527169 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac188105-3461-4ded-b338-82dc36c2296a-workload-certs\") pod \"ac188105-3461-4ded-b338-82dc36c2296a\" (UID: \"ac188105-3461-4ded-b338-82dc36c2296a\") " Jul 23 04:26:41.527209 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:41.527205 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac188105-3461-4ded-b338-82dc36c2296a-credential-socket\") pod \"ac188105-3461-4ded-b338-82dc36c2296a\" (UID: \"ac188105-3461-4ded-b338-82dc36c2296a\") " Jul 23 04:26:41.527651 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:41.527251 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac188105-3461-4ded-b338-82dc36c2296a-workload-socket\") pod \"ac188105-3461-4ded-b338-82dc36c2296a\" (UID: \"ac188105-3461-4ded-b338-82dc36c2296a\") " Jul 23 04:26:41.527651 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:41.527273 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac188105-3461-4ded-b338-82dc36c2296a-istiod-ca-cert\") pod \"ac188105-3461-4ded-b338-82dc36c2296a\" (UID: \"ac188105-3461-4ded-b338-82dc36c2296a\") " Jul 23 04:26:41.527651 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:41.527298 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac188105-3461-4ded-b338-82dc36c2296a-istio-podinfo\") pod \"ac188105-3461-4ded-b338-82dc36c2296a\" (UID: \"ac188105-3461-4ded-b338-82dc36c2296a\") " Jul 23 04:26:41.527651 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:41.527376 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac188105-3461-4ded-b338-82dc36c2296a-istio-envoy\") pod \"ac188105-3461-4ded-b338-82dc36c2296a\" (UID: \"ac188105-3461-4ded-b338-82dc36c2296a\") " Jul 23 04:26:41.527651 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:41.527419 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac188105-3461-4ded-b338-82dc36c2296a-istio-token\") pod \"ac188105-3461-4ded-b338-82dc36c2296a\" (UID: \"ac188105-3461-4ded-b338-82dc36c2296a\") " Jul 23 04:26:41.527651 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:41.527473 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6j62x\" (UniqueName: \"kubernetes.io/projected/ac188105-3461-4ded-b338-82dc36c2296a-kube-api-access-6j62x\") pod \"ac188105-3461-4ded-b338-82dc36c2296a\" (UID: \"ac188105-3461-4ded-b338-82dc36c2296a\") " Jul 23 04:26:41.527651 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:41.527470 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac188105-3461-4ded-b338-82dc36c2296a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ac188105-3461-4ded-b338-82dc36c2296a" (UID: "ac188105-3461-4ded-b338-82dc36c2296a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 04:26:41.527651 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:41.527568 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac188105-3461-4ded-b338-82dc36c2296a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ac188105-3461-4ded-b338-82dc36c2296a" (UID: "ac188105-3461-4ded-b338-82dc36c2296a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 04:26:41.527651 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:41.527586 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac188105-3461-4ded-b338-82dc36c2296a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ac188105-3461-4ded-b338-82dc36c2296a" (UID: "ac188105-3461-4ded-b338-82dc36c2296a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 04:26:41.528321 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:41.527666 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac188105-3461-4ded-b338-82dc36c2296a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ac188105-3461-4ded-b338-82dc36c2296a" (UID: "ac188105-3461-4ded-b338-82dc36c2296a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 04:26:41.528321 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:41.527756 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac188105-3461-4ded-b338-82dc36c2296a-workload-certs\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:26:41.528321 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:41.527776 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac188105-3461-4ded-b338-82dc36c2296a-credential-socket\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:26:41.528321 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:41.527795 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac188105-3461-4ded-b338-82dc36c2296a-workload-socket\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:26:41.528321 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:41.527809 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac188105-3461-4ded-b338-82dc36c2296a-istiod-ca-cert\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:26:41.528321 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:41.528074 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac188105-3461-4ded-b338-82dc36c2296a-istio-data" (OuterVolumeSpecName: "istio-data") pod "ac188105-3461-4ded-b338-82dc36c2296a" (UID: "ac188105-3461-4ded-b338-82dc36c2296a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 04:26:41.529698 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:41.529674 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac188105-3461-4ded-b338-82dc36c2296a-istio-token" (OuterVolumeSpecName: "istio-token") pod "ac188105-3461-4ded-b338-82dc36c2296a" (UID: "ac188105-3461-4ded-b338-82dc36c2296a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 04:26:41.529809 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:41.529778 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac188105-3461-4ded-b338-82dc36c2296a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ac188105-3461-4ded-b338-82dc36c2296a" (UID: "ac188105-3461-4ded-b338-82dc36c2296a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 04:26:41.530083 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:41.530058 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac188105-3461-4ded-b338-82dc36c2296a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ac188105-3461-4ded-b338-82dc36c2296a" (UID: "ac188105-3461-4ded-b338-82dc36c2296a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 04:26:41.530151 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:41.530112 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac188105-3461-4ded-b338-82dc36c2296a-kube-api-access-6j62x" (OuterVolumeSpecName: "kube-api-access-6j62x") pod "ac188105-3461-4ded-b338-82dc36c2296a" (UID: "ac188105-3461-4ded-b338-82dc36c2296a"). InnerVolumeSpecName "kube-api-access-6j62x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 04:26:41.565953 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:41.565931 2567 generic.go:356] "Generic (PLEG): container finished" podID="ac188105-3461-4ded-b338-82dc36c2296a" containerID="dec20019522873a52d7f239a137465e79b65781603f9251e8f6cafbb0db4a953" exitCode=0 Jul 23 04:26:41.566040 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:41.565962 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2" event={"ID":"ac188105-3461-4ded-b338-82dc36c2296a","Type":"ContainerDied","Data":"dec20019522873a52d7f239a137465e79b65781603f9251e8f6cafbb0db4a953"} Jul 23 04:26:41.566040 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:41.565977 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2" Jul 23 04:26:41.566040 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:41.565997 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2" event={"ID":"ac188105-3461-4ded-b338-82dc36c2296a","Type":"ContainerDied","Data":"89e4083846f354c7bf4a4449cc4229e169cfd3b5173bf7ef76323cd11b9cce79"} Jul 23 04:26:41.566040 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:41.566017 2567 scope.go:117] "RemoveContainer" containerID="dec20019522873a52d7f239a137465e79b65781603f9251e8f6cafbb0db4a953" Jul 23 04:26:41.575112 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:41.575096 2567 scope.go:117] "RemoveContainer" containerID="2742e4dbd2cb85b3993462e5cf2cab73ba5bde5ae0295e2c6ecd9bc9367af383" Jul 23 04:26:41.583782 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:41.583767 2567 scope.go:117] "RemoveContainer" containerID="dec20019522873a52d7f239a137465e79b65781603f9251e8f6cafbb0db4a953" Jul 23 04:26:41.584041 ip-10-0-140-6 kubenswrapper[2567]: E0723 04:26:41.584023 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dec20019522873a52d7f239a137465e79b65781603f9251e8f6cafbb0db4a953\": container with ID starting with dec20019522873a52d7f239a137465e79b65781603f9251e8f6cafbb0db4a953 not found: ID does not exist" containerID="dec20019522873a52d7f239a137465e79b65781603f9251e8f6cafbb0db4a953" Jul 23 04:26:41.584105 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:41.584050 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dec20019522873a52d7f239a137465e79b65781603f9251e8f6cafbb0db4a953"} err="failed to get container status \"dec20019522873a52d7f239a137465e79b65781603f9251e8f6cafbb0db4a953\": rpc error: code = NotFound desc = could not find container \"dec20019522873a52d7f239a137465e79b65781603f9251e8f6cafbb0db4a953\": container with ID starting with dec20019522873a52d7f239a137465e79b65781603f9251e8f6cafbb0db4a953 not found: ID does not exist" Jul 23 04:26:41.584105 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:41.584068 2567 scope.go:117] "RemoveContainer" containerID="2742e4dbd2cb85b3993462e5cf2cab73ba5bde5ae0295e2c6ecd9bc9367af383" Jul 23 04:26:41.584318 ip-10-0-140-6 kubenswrapper[2567]: E0723 04:26:41.584301 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2742e4dbd2cb85b3993462e5cf2cab73ba5bde5ae0295e2c6ecd9bc9367af383\": container with ID starting with 2742e4dbd2cb85b3993462e5cf2cab73ba5bde5ae0295e2c6ecd9bc9367af383 not found: ID does not exist" containerID="2742e4dbd2cb85b3993462e5cf2cab73ba5bde5ae0295e2c6ecd9bc9367af383" Jul 23 04:26:41.584358 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:41.584324 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2742e4dbd2cb85b3993462e5cf2cab73ba5bde5ae0295e2c6ecd9bc9367af383"} err="failed to get container status \"2742e4dbd2cb85b3993462e5cf2cab73ba5bde5ae0295e2c6ecd9bc9367af383\": rpc error: code = NotFound desc = could not find container \"2742e4dbd2cb85b3993462e5cf2cab73ba5bde5ae0295e2c6ecd9bc9367af383\": container with ID starting with 2742e4dbd2cb85b3993462e5cf2cab73ba5bde5ae0295e2c6ecd9bc9367af383 not found: ID does not exist" Jul 23 04:26:41.591198 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:41.591177 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2"] Jul 23 04:26:41.598335 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:41.598307 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-dc495b0d-openshift-default-7b7d65f9dc-n9vs2"] Jul 23 04:26:41.628179 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:41.628157 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac188105-3461-4ded-b338-82dc36c2296a-istio-envoy\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:26:41.628179 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:41.628181 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac188105-3461-4ded-b338-82dc36c2296a-istio-token\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:26:41.628306 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:41.628190 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j62x\" (UniqueName: \"kubernetes.io/projected/ac188105-3461-4ded-b338-82dc36c2296a-kube-api-access-6j62x\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:26:41.628306 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:41.628199 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac188105-3461-4ded-b338-82dc36c2296a-istio-data\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:26:41.628306 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:41.628208 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac188105-3461-4ded-b338-82dc36c2296a-istio-podinfo\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:26:42.492061 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:42.492029 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac188105-3461-4ded-b338-82dc36c2296a" path="/var/lib/kubelet/pods/ac188105-3461-4ded-b338-82dc36c2296a/volumes" Jul 23 04:26:56.075656 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:56.075619 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k"] Jul 23 04:26:56.076014 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:56.075924 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dd59cb6-280a-447a-9d0c-71c3187ff747" containerName="revoke-keys" Jul 23 04:26:56.076014 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:56.075935 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dd59cb6-280a-447a-9d0c-71c3187ff747" containerName="revoke-keys" Jul 23 04:26:56.076014 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:56.075949 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac188105-3461-4ded-b338-82dc36c2296a" containerName="istio-proxy" Jul 23 04:26:56.076014 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:56.075954 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac188105-3461-4ded-b338-82dc36c2296a" containerName="istio-proxy" Jul 23 04:26:56.076014 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:56.075963 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac188105-3461-4ded-b338-82dc36c2296a" containerName="istio-proxy" Jul 23 04:26:56.076014 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:56.075968 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac188105-3461-4ded-b338-82dc36c2296a" containerName="istio-proxy" Jul 23 04:26:56.076207 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:56.076019 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dd59cb6-280a-447a-9d0c-71c3187ff747" containerName="revoke-keys" Jul 23 04:26:56.076207 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:56.076026 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac188105-3461-4ded-b338-82dc36c2296a" containerName="istio-proxy" Jul 23 04:26:56.076207 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:56.076032 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac188105-3461-4ded-b338-82dc36c2296a" containerName="istio-proxy" Jul 23 04:26:56.078987 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:56.078966 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k" Jul 23 04:26:56.082141 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:56.082119 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-e0e3ac7b-openshift-default-dockercfg-nh9q8\"" Jul 23 04:26:56.091901 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:56.091881 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k"] Jul 23 04:26:56.144021 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:56.143993 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/517fb98c-3587-4984-b877-b231316c45de-istio-token\") pod \"e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k\" (UID: \"517fb98c-3587-4984-b877-b231316c45de\") " pod="openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k" Jul 23 04:26:56.144137 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:56.144030 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/517fb98c-3587-4984-b877-b231316c45de-credential-socket\") pod \"e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k\" (UID: \"517fb98c-3587-4984-b877-b231316c45de\") " pod="openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k" Jul 23 04:26:56.144137 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:56.144066 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/517fb98c-3587-4984-b877-b231316c45de-istio-podinfo\") pod \"e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k\" (UID: \"517fb98c-3587-4984-b877-b231316c45de\") " pod="openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k" Jul 23 04:26:56.144137 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:56.144115 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/517fb98c-3587-4984-b877-b231316c45de-workload-certs\") pod \"e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k\" (UID: \"517fb98c-3587-4984-b877-b231316c45de\") " pod="openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k" Jul 23 04:26:56.144306 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:56.144142 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/517fb98c-3587-4984-b877-b231316c45de-workload-socket\") pod \"e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k\" (UID: \"517fb98c-3587-4984-b877-b231316c45de\") " pod="openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k" Jul 23 04:26:56.144306 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:56.144220 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hplwh\" (UniqueName: \"kubernetes.io/projected/517fb98c-3587-4984-b877-b231316c45de-kube-api-access-hplwh\") pod \"e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k\" (UID: \"517fb98c-3587-4984-b877-b231316c45de\") " pod="openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k" Jul 23 04:26:56.144306 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:56.144287 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/517fb98c-3587-4984-b877-b231316c45de-istio-envoy\") pod \"e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k\" (UID: \"517fb98c-3587-4984-b877-b231316c45de\") " pod="openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k" Jul 23 04:26:56.144489 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:56.144340 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/517fb98c-3587-4984-b877-b231316c45de-istiod-ca-cert\") pod \"e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k\" (UID: \"517fb98c-3587-4984-b877-b231316c45de\") " pod="openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k" Jul 23 04:26:56.144489 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:56.144369 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/517fb98c-3587-4984-b877-b231316c45de-istio-data\") pod \"e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k\" (UID: \"517fb98c-3587-4984-b877-b231316c45de\") " pod="openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k" Jul 23 04:26:56.245119 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:56.245088 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/517fb98c-3587-4984-b877-b231316c45de-credential-socket\") pod \"e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k\" (UID: \"517fb98c-3587-4984-b877-b231316c45de\") " pod="openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k" Jul 23 04:26:56.245273 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:56.245128 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/517fb98c-3587-4984-b877-b231316c45de-istio-podinfo\") pod \"e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k\" (UID: \"517fb98c-3587-4984-b877-b231316c45de\") " pod="openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k" Jul 23 04:26:56.245273 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:56.245145 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/517fb98c-3587-4984-b877-b231316c45de-workload-certs\") pod \"e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k\" (UID: \"517fb98c-3587-4984-b877-b231316c45de\") " pod="openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k" Jul 23 04:26:56.245273 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:56.245163 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/517fb98c-3587-4984-b877-b231316c45de-workload-socket\") pod \"e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k\" (UID: \"517fb98c-3587-4984-b877-b231316c45de\") " pod="openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k" Jul 23 04:26:56.245273 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:56.245210 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hplwh\" (UniqueName: \"kubernetes.io/projected/517fb98c-3587-4984-b877-b231316c45de-kube-api-access-hplwh\") pod \"e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k\" (UID: \"517fb98c-3587-4984-b877-b231316c45de\") " pod="openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k" Jul 23 04:26:56.245273 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:56.245259 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/517fb98c-3587-4984-b877-b231316c45de-istio-envoy\") pod \"e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k\" (UID: \"517fb98c-3587-4984-b877-b231316c45de\") " pod="openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k" Jul 23 04:26:56.245585 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:56.245299 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/517fb98c-3587-4984-b877-b231316c45de-istiod-ca-cert\") pod \"e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k\" (UID: \"517fb98c-3587-4984-b877-b231316c45de\") " pod="openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k" Jul 23 04:26:56.245585 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:56.245325 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/517fb98c-3587-4984-b877-b231316c45de-istio-data\") pod \"e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k\" (UID: \"517fb98c-3587-4984-b877-b231316c45de\") " pod="openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k" Jul 23 04:26:56.245585 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:56.245351 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/517fb98c-3587-4984-b877-b231316c45de-istio-token\") pod \"e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k\" (UID: \"517fb98c-3587-4984-b877-b231316c45de\") " pod="openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k" Jul 23 04:26:56.245585 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:56.245563 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/517fb98c-3587-4984-b877-b231316c45de-credential-socket\") pod \"e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k\" (UID: \"517fb98c-3587-4984-b877-b231316c45de\") " pod="openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k" Jul 23 04:26:56.245826 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:56.245604 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/517fb98c-3587-4984-b877-b231316c45de-workload-certs\") pod \"e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k\" (UID: \"517fb98c-3587-4984-b877-b231316c45de\") " pod="openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k" Jul 23 04:26:56.245826 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:56.245749 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/517fb98c-3587-4984-b877-b231316c45de-istio-data\") pod \"e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k\" (UID: \"517fb98c-3587-4984-b877-b231316c45de\") " pod="openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k" Jul 23 04:26:56.246407 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:56.246385 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/517fb98c-3587-4984-b877-b231316c45de-istiod-ca-cert\") pod \"e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k\" (UID: \"517fb98c-3587-4984-b877-b231316c45de\") " pod="openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k" Jul 23 04:26:56.246482 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:56.246407 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/517fb98c-3587-4984-b877-b231316c45de-workload-socket\") pod \"e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k\" (UID: \"517fb98c-3587-4984-b877-b231316c45de\") " pod="openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k" Jul 23 04:26:56.247752 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:56.247729 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/517fb98c-3587-4984-b877-b231316c45de-istio-envoy\") pod \"e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k\" (UID: \"517fb98c-3587-4984-b877-b231316c45de\") " pod="openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k" Jul 23 04:26:56.247959 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:56.247940 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/517fb98c-3587-4984-b877-b231316c45de-istio-podinfo\") pod \"e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k\" (UID: \"517fb98c-3587-4984-b877-b231316c45de\") " pod="openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k" Jul 23 04:26:56.255315 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:56.255296 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/517fb98c-3587-4984-b877-b231316c45de-istio-token\") pod \"e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k\" (UID: \"517fb98c-3587-4984-b877-b231316c45de\") " pod="openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k" Jul 23 04:26:56.256014 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:56.255992 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hplwh\" (UniqueName: \"kubernetes.io/projected/517fb98c-3587-4984-b877-b231316c45de-kube-api-access-hplwh\") pod \"e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k\" (UID: \"517fb98c-3587-4984-b877-b231316c45de\") " pod="openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k" Jul 23 04:26:56.388835 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:56.388764 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k" Jul 23 04:26:56.537363 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:56.537334 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k"] Jul 23 04:26:56.538900 ip-10-0-140-6 kubenswrapper[2567]: W0723 04:26:56.538872 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod517fb98c_3587_4984_b877_b231316c45de.slice/crio-c37d0aa1f07b1aa3d1a7f648c3585c2efaec915436f7a083dc927ab21811eb8d WatchSource:0}: Error finding container c37d0aa1f07b1aa3d1a7f648c3585c2efaec915436f7a083dc927ab21811eb8d: Status 404 returned error can't find the container with id c37d0aa1f07b1aa3d1a7f648c3585c2efaec915436f7a083dc927ab21811eb8d Jul 23 04:26:56.541096 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:56.541054 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 04:26:56.541217 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:56.541139 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 04:26:56.541217 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:56.541183 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 04:26:56.646411 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:56.646330 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k" event={"ID":"517fb98c-3587-4984-b877-b231316c45de","Type":"ContainerStarted","Data":"b0e4fd5e24c4fe83a9a556c696a174b2551544348a7e70bd962ba8de99fd1695"} Jul 23 04:26:56.646411 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:56.646399 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k" event={"ID":"517fb98c-3587-4984-b877-b231316c45de","Type":"ContainerStarted","Data":"c37d0aa1f07b1aa3d1a7f648c3585c2efaec915436f7a083dc927ab21811eb8d"} Jul 23 04:26:56.671180 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:56.671119 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k" podStartSLOduration=0.671098486 podStartE2EDuration="671.098486ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 04:26:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 04:26:56.668239415 +0000 UTC m=+1896.573543711" watchObservedRunningTime="2026-07-23 04:26:56.671098486 +0000 UTC m=+1896.576402795" Jul 23 04:26:57.389504 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:57.389471 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k" Jul 23 04:26:57.393814 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:57.393789 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k" Jul 23 04:26:57.652343 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:57.652271 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k" Jul 23 04:26:57.653127 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:26:57.653099 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k" Jul 23 04:27:11.728623 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:27:11.728590 2567 generic.go:356] "Generic (PLEG): container finished" podID="517fb98c-3587-4984-b877-b231316c45de" containerID="b0e4fd5e24c4fe83a9a556c696a174b2551544348a7e70bd962ba8de99fd1695" exitCode=0 Jul 23 04:27:11.729036 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:27:11.728661 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k" event={"ID":"517fb98c-3587-4984-b877-b231316c45de","Type":"ContainerDied","Data":"b0e4fd5e24c4fe83a9a556c696a174b2551544348a7e70bd962ba8de99fd1695"} Jul 23 04:27:11.729036 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:27:11.728978 2567 scope.go:117] "RemoveContainer" containerID="b0e4fd5e24c4fe83a9a556c696a174b2551544348a7e70bd962ba8de99fd1695" Jul 23 04:27:11.729940 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:27:11.729913 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 04:27:11.730000 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:27:11.729980 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 04:27:11.730038 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:27:11.730010 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 04:27:12.653285 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:27:12.653221 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k" Jul 23 04:27:12.735385 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:27:12.735348 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k" event={"ID":"517fb98c-3587-4984-b877-b231316c45de","Type":"ContainerStarted","Data":"5068aae39c7124e1dd27c391ec6835e47f1bb85cb68342f348456ed627907c56"} Jul 23 04:27:13.389367 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:27:13.389328 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k" Jul 23 04:27:13.390526 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:27:13.390497 2567 patch_prober.go:28] interesting pod/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 23 04:27:13.390651 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:27:13.390549 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k" podUID="517fb98c-3587-4984-b877-b231316c45de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 23 04:27:14.389741 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:27:14.389704 2567 patch_prober.go:28] interesting pod/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 23 04:27:14.390180 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:27:14.389770 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k" podUID="517fb98c-3587-4984-b877-b231316c45de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 23 04:27:15.389364 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:27:15.389324 2567 patch_prober.go:28] interesting pod/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 23 04:27:15.389567 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:27:15.389392 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k" podUID="517fb98c-3587-4984-b877-b231316c45de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 23 04:27:16.389806 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:27:16.389770 2567 patch_prober.go:28] interesting pod/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 23 04:27:16.390276 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:27:16.389834 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k" podUID="517fb98c-3587-4984-b877-b231316c45de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 23 04:27:17.392867 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:27:17.392838 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k" Jul 23 04:27:17.393229 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:27:17.393104 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k" Jul 23 04:27:17.393774 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:27:17.393753 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k" Jul 23 04:27:36.502707 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:27:36.502671 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0e3ac7b-sdlcj"] Jul 23 04:27:36.505611 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:27:36.505594 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0e3ac7b-sdlcj" Jul 23 04:27:36.508909 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:27:36.508881 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 04:27:36.509030 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:27:36.508935 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dpftb\"" Jul 23 04:27:36.515255 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:27:36.515221 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0e3ac7b-sdlcj"] Jul 23 04:27:36.585406 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:27:36.585377 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ceb96859-62eb-4b15-887c-1611ca2b189d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e0e3ac7b-sdlcj\" (UID: \"ceb96859-62eb-4b15-887c-1611ca2b189d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0e3ac7b-sdlcj" Jul 23 04:27:36.686027 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:27:36.686006 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ceb96859-62eb-4b15-887c-1611ca2b189d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e0e3ac7b-sdlcj\" (UID: \"ceb96859-62eb-4b15-887c-1611ca2b189d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0e3ac7b-sdlcj" Jul 23 04:27:36.686630 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:27:36.686613 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ceb96859-62eb-4b15-887c-1611ca2b189d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e0e3ac7b-sdlcj\" (UID: \"ceb96859-62eb-4b15-887c-1611ca2b189d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0e3ac7b-sdlcj" Jul 23 04:27:36.817226 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:27:36.817170 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0e3ac7b-sdlcj" Jul 23 04:27:37.151894 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:27:37.151869 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0e3ac7b-sdlcj"] Jul 23 04:27:37.153960 ip-10-0-140-6 kubenswrapper[2567]: W0723 04:27:37.153931 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceb96859_62eb_4b15_887c_1611ca2b189d.slice/crio-90b8a1c2b1f51892a58f798a8475bc2300209df3eebe8a7f8717a416f75a81ba WatchSource:0}: Error finding container 90b8a1c2b1f51892a58f798a8475bc2300209df3eebe8a7f8717a416f75a81ba: Status 404 returned error can't find the container with id 90b8a1c2b1f51892a58f798a8475bc2300209df3eebe8a7f8717a416f75a81ba Jul 23 04:27:37.870720 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:27:37.870680 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0e3ac7b-sdlcj" event={"ID":"ceb96859-62eb-4b15-887c-1611ca2b189d","Type":"ContainerStarted","Data":"81169aecb77be48094e1f23de7e581da0ad6d6c0523f089fe717d78190c10392"} Jul 23 04:27:37.871111 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:27:37.870767 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0e3ac7b-sdlcj" event={"ID":"ceb96859-62eb-4b15-887c-1611ca2b189d","Type":"ContainerStarted","Data":"90b8a1c2b1f51892a58f798a8475bc2300209df3eebe8a7f8717a416f75a81ba"} Jul 23 04:27:37.889218 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:27:37.889167 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0e3ac7b-sdlcj" podStartSLOduration=1.889154776 podStartE2EDuration="1.889154776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 04:27:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 04:27:37.887798205 +0000 UTC m=+1937.793102497" watchObservedRunningTime="2026-07-23 04:27:37.889154776 +0000 UTC m=+1937.794459068" Jul 23 04:27:38.877247 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:27:38.877207 2567 generic.go:356] "Generic (PLEG): container finished" podID="ceb96859-62eb-4b15-887c-1611ca2b189d" containerID="81169aecb77be48094e1f23de7e581da0ad6d6c0523f089fe717d78190c10392" exitCode=0 Jul 23 04:27:38.877663 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:27:38.877283 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0e3ac7b-sdlcj" event={"ID":"ceb96859-62eb-4b15-887c-1611ca2b189d","Type":"ContainerDied","Data":"81169aecb77be48094e1f23de7e581da0ad6d6c0523f089fe717d78190c10392"} Jul 23 04:27:40.019217 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:27:40.019195 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0e3ac7b-sdlcj" Jul 23 04:27:40.112793 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:27:40.112768 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ceb96859-62eb-4b15-887c-1611ca2b189d-maas-api-service-ca\") pod \"ceb96859-62eb-4b15-887c-1611ca2b189d\" (UID: \"ceb96859-62eb-4b15-887c-1611ca2b189d\") " Jul 23 04:27:40.113096 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:27:40.113074 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceb96859-62eb-4b15-887c-1611ca2b189d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ceb96859-62eb-4b15-887c-1611ca2b189d" (UID: "ceb96859-62eb-4b15-887c-1611ca2b189d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 04:27:40.213623 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:27:40.213596 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ceb96859-62eb-4b15-887c-1611ca2b189d-maas-api-service-ca\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:27:40.889730 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:27:40.889642 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0e3ac7b-sdlcj" Jul 23 04:27:40.889730 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:27:40.889669 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0e3ac7b-sdlcj" event={"ID":"ceb96859-62eb-4b15-887c-1611ca2b189d","Type":"ContainerDied","Data":"90b8a1c2b1f51892a58f798a8475bc2300209df3eebe8a7f8717a416f75a81ba"} Jul 23 04:27:40.889730 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:27:40.889713 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90b8a1c2b1f51892a58f798a8475bc2300209df3eebe8a7f8717a416f75a81ba" Jul 23 04:27:56.586638 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:27:56.586603 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0e3ac7b-sdlcj"] Jul 23 04:27:56.591107 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:27:56.591083 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e0e3ac7b-sdlcj"] Jul 23 04:27:58.491994 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:27:58.491957 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ceb96859-62eb-4b15-887c-1611ca2b189d" path="/var/lib/kubelet/pods/ceb96859-62eb-4b15-887c-1611ca2b189d/volumes" Jul 23 04:28:20.675866 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:28:20.675831 2567 scope.go:117] "RemoveContainer" containerID="5f896ec224f29cd69016725f21bb12be0c4194f3c10278d21b9b8fc6b27378b3" Jul 23 04:28:47.207847 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:28:47.207810 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k"] Jul 23 04:28:47.208512 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:28:47.208114 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k" podUID="517fb98c-3587-4984-b877-b231316c45de" containerName="istio-proxy" containerID="cri-o://5068aae39c7124e1dd27c391ec6835e47f1bb85cb68342f348456ed627907c56" gracePeriod=30 Jul 23 04:28:47.395512 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:28:47.395476 2567 patch_prober.go:28] interesting pod/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 23 04:28:47.395701 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:28:47.395549 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k" podUID="517fb98c-3587-4984-b877-b231316c45de" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 23 04:28:52.452121 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:28:52.452094 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k" Jul 23 04:28:52.569735 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:28:52.569648 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/517fb98c-3587-4984-b877-b231316c45de-istio-data\") pod \"517fb98c-3587-4984-b877-b231316c45de\" (UID: \"517fb98c-3587-4984-b877-b231316c45de\") " Jul 23 04:28:52.569735 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:28:52.569693 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/517fb98c-3587-4984-b877-b231316c45de-workload-socket\") pod \"517fb98c-3587-4984-b877-b231316c45de\" (UID: \"517fb98c-3587-4984-b877-b231316c45de\") " Jul 23 04:28:52.569735 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:28:52.569711 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hplwh\" (UniqueName: \"kubernetes.io/projected/517fb98c-3587-4984-b877-b231316c45de-kube-api-access-hplwh\") pod \"517fb98c-3587-4984-b877-b231316c45de\" (UID: \"517fb98c-3587-4984-b877-b231316c45de\") " Jul 23 04:28:52.570025 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:28:52.569739 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/517fb98c-3587-4984-b877-b231316c45de-istio-token\") pod \"517fb98c-3587-4984-b877-b231316c45de\" (UID: \"517fb98c-3587-4984-b877-b231316c45de\") " Jul 23 04:28:52.570025 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:28:52.569780 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/517fb98c-3587-4984-b877-b231316c45de-istiod-ca-cert\") pod \"517fb98c-3587-4984-b877-b231316c45de\" (UID: \"517fb98c-3587-4984-b877-b231316c45de\") " Jul 23 04:28:52.570025 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:28:52.569822 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/517fb98c-3587-4984-b877-b231316c45de-credential-socket\") pod \"517fb98c-3587-4984-b877-b231316c45de\" (UID: \"517fb98c-3587-4984-b877-b231316c45de\") " Jul 23 04:28:52.570025 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:28:52.569854 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/517fb98c-3587-4984-b877-b231316c45de-istio-envoy\") pod \"517fb98c-3587-4984-b877-b231316c45de\" (UID: \"517fb98c-3587-4984-b877-b231316c45de\") " Jul 23 04:28:52.570025 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:28:52.569875 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/517fb98c-3587-4984-b877-b231316c45de-istio-podinfo\") pod \"517fb98c-3587-4984-b877-b231316c45de\" (UID: \"517fb98c-3587-4984-b877-b231316c45de\") " Jul 23 04:28:52.570025 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:28:52.569935 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/517fb98c-3587-4984-b877-b231316c45de-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "517fb98c-3587-4984-b877-b231316c45de" (UID: "517fb98c-3587-4984-b877-b231316c45de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 04:28:52.570025 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:28:52.569958 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/517fb98c-3587-4984-b877-b231316c45de-workload-certs\") pod \"517fb98c-3587-4984-b877-b231316c45de\" (UID: \"517fb98c-3587-4984-b877-b231316c45de\") " Jul 23 04:28:52.570397 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:28:52.570087 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/517fb98c-3587-4984-b877-b231316c45de-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "517fb98c-3587-4984-b877-b231316c45de" (UID: "517fb98c-3587-4984-b877-b231316c45de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 04:28:52.570397 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:28:52.570230 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/517fb98c-3587-4984-b877-b231316c45de-workload-socket\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:28:52.570397 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:28:52.570246 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/517fb98c-3587-4984-b877-b231316c45de-credential-socket\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:28:52.570397 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:28:52.570241 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/517fb98c-3587-4984-b877-b231316c45de-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "517fb98c-3587-4984-b877-b231316c45de" (UID: "517fb98c-3587-4984-b877-b231316c45de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 04:28:52.570397 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:28:52.570402 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/517fb98c-3587-4984-b877-b231316c45de-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "517fb98c-3587-4984-b877-b231316c45de" (UID: "517fb98c-3587-4984-b877-b231316c45de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 04:28:52.570830 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:28:52.570553 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/517fb98c-3587-4984-b877-b231316c45de-istio-data" (OuterVolumeSpecName: "istio-data") pod "517fb98c-3587-4984-b877-b231316c45de" (UID: "517fb98c-3587-4984-b877-b231316c45de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 04:28:52.572199 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:28:52.572171 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/517fb98c-3587-4984-b877-b231316c45de-kube-api-access-hplwh" (OuterVolumeSpecName: "kube-api-access-hplwh") pod "517fb98c-3587-4984-b877-b231316c45de" (UID: "517fb98c-3587-4984-b877-b231316c45de"). InnerVolumeSpecName "kube-api-access-hplwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 04:28:52.572466 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:28:52.572441 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/517fb98c-3587-4984-b877-b231316c45de-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "517fb98c-3587-4984-b877-b231316c45de" (UID: "517fb98c-3587-4984-b877-b231316c45de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 04:28:52.572897 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:28:52.572867 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/517fb98c-3587-4984-b877-b231316c45de-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "517fb98c-3587-4984-b877-b231316c45de" (UID: "517fb98c-3587-4984-b877-b231316c45de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 04:28:52.573057 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:28:52.573040 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/517fb98c-3587-4984-b877-b231316c45de-istio-token" (OuterVolumeSpecName: "istio-token") pod "517fb98c-3587-4984-b877-b231316c45de" (UID: "517fb98c-3587-4984-b877-b231316c45de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 04:28:52.671664 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:28:52.671632 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/517fb98c-3587-4984-b877-b231316c45de-workload-certs\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:28:52.671664 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:28:52.671659 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/517fb98c-3587-4984-b877-b231316c45de-istio-data\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:28:52.671664 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:28:52.671669 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hplwh\" (UniqueName: \"kubernetes.io/projected/517fb98c-3587-4984-b877-b231316c45de-kube-api-access-hplwh\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:28:52.671882 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:28:52.671678 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/517fb98c-3587-4984-b877-b231316c45de-istio-token\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:28:52.671882 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:28:52.671687 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/517fb98c-3587-4984-b877-b231316c45de-istiod-ca-cert\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:28:52.671882 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:28:52.671694 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/517fb98c-3587-4984-b877-b231316c45de-istio-envoy\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:28:52.671882 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:28:52.671703 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/517fb98c-3587-4984-b877-b231316c45de-istio-podinfo\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:28:53.269034 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:28:53.269002 2567 generic.go:356] "Generic (PLEG): container finished" podID="517fb98c-3587-4984-b877-b231316c45de" containerID="5068aae39c7124e1dd27c391ec6835e47f1bb85cb68342f348456ed627907c56" exitCode=0 Jul 23 04:28:53.269218 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:28:53.269055 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k" Jul 23 04:28:53.269218 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:28:53.269094 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k" event={"ID":"517fb98c-3587-4984-b877-b231316c45de","Type":"ContainerDied","Data":"5068aae39c7124e1dd27c391ec6835e47f1bb85cb68342f348456ed627907c56"} Jul 23 04:28:53.269218 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:28:53.269154 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k" event={"ID":"517fb98c-3587-4984-b877-b231316c45de","Type":"ContainerDied","Data":"c37d0aa1f07b1aa3d1a7f648c3585c2efaec915436f7a083dc927ab21811eb8d"} Jul 23 04:28:53.269218 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:28:53.269185 2567 scope.go:117] "RemoveContainer" containerID="5068aae39c7124e1dd27c391ec6835e47f1bb85cb68342f348456ed627907c56" Jul 23 04:28:53.281051 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:28:53.281034 2567 scope.go:117] "RemoveContainer" containerID="b0e4fd5e24c4fe83a9a556c696a174b2551544348a7e70bd962ba8de99fd1695" Jul 23 04:28:53.289241 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:28:53.289223 2567 scope.go:117] "RemoveContainer" containerID="5068aae39c7124e1dd27c391ec6835e47f1bb85cb68342f348456ed627907c56" Jul 23 04:28:53.289513 ip-10-0-140-6 kubenswrapper[2567]: E0723 04:28:53.289494 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5068aae39c7124e1dd27c391ec6835e47f1bb85cb68342f348456ed627907c56\": container with ID starting with 5068aae39c7124e1dd27c391ec6835e47f1bb85cb68342f348456ed627907c56 not found: ID does not exist" containerID="5068aae39c7124e1dd27c391ec6835e47f1bb85cb68342f348456ed627907c56" Jul 23 04:28:53.289575 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:28:53.289523 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5068aae39c7124e1dd27c391ec6835e47f1bb85cb68342f348456ed627907c56"} err="failed to get container status \"5068aae39c7124e1dd27c391ec6835e47f1bb85cb68342f348456ed627907c56\": rpc error: code = NotFound desc = could not find container \"5068aae39c7124e1dd27c391ec6835e47f1bb85cb68342f348456ed627907c56\": container with ID starting with 5068aae39c7124e1dd27c391ec6835e47f1bb85cb68342f348456ed627907c56 not found: ID does not exist" Jul 23 04:28:53.289575 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:28:53.289540 2567 scope.go:117] "RemoveContainer" containerID="b0e4fd5e24c4fe83a9a556c696a174b2551544348a7e70bd962ba8de99fd1695" Jul 23 04:28:53.289737 ip-10-0-140-6 kubenswrapper[2567]: E0723 04:28:53.289723 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0e4fd5e24c4fe83a9a556c696a174b2551544348a7e70bd962ba8de99fd1695\": container with ID starting with b0e4fd5e24c4fe83a9a556c696a174b2551544348a7e70bd962ba8de99fd1695 not found: ID does not exist" containerID="b0e4fd5e24c4fe83a9a556c696a174b2551544348a7e70bd962ba8de99fd1695" Jul 23 04:28:53.289779 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:28:53.289740 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0e4fd5e24c4fe83a9a556c696a174b2551544348a7e70bd962ba8de99fd1695"} err="failed to get container status \"b0e4fd5e24c4fe83a9a556c696a174b2551544348a7e70bd962ba8de99fd1695\": rpc error: code = NotFound desc = could not find container \"b0e4fd5e24c4fe83a9a556c696a174b2551544348a7e70bd962ba8de99fd1695\": container with ID starting with b0e4fd5e24c4fe83a9a556c696a174b2551544348a7e70bd962ba8de99fd1695 not found: ID does not exist" Jul 23 04:28:53.296463 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:28:53.296419 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k"] Jul 23 04:28:53.302479 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:28:53.302459 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e0e3ac7b-openshift-default-58cb5797dc-kk46k"] Jul 23 04:28:54.492794 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:28:54.492756 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="517fb98c-3587-4984-b877-b231316c45de" path="/var/lib/kubelet/pods/517fb98c-3587-4984-b877-b231316c45de/volumes" Jul 23 04:29:20.697154 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:20.697120 2567 scope.go:117] "RemoveContainer" containerID="6dcb49e523ffcb07874ebb35ef1118d4f85fbfa0ab2ed7b1a7039c4bea6da817" Jul 23 04:29:42.954446 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:42.954347 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t"] Jul 23 04:29:42.954858 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:42.954682 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="517fb98c-3587-4984-b877-b231316c45de" containerName="istio-proxy" Jul 23 04:29:42.954858 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:42.954694 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="517fb98c-3587-4984-b877-b231316c45de" containerName="istio-proxy" Jul 23 04:29:42.954858 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:42.954706 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="517fb98c-3587-4984-b877-b231316c45de" containerName="istio-proxy" Jul 23 04:29:42.954858 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:42.954713 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="517fb98c-3587-4984-b877-b231316c45de" containerName="istio-proxy" Jul 23 04:29:42.954858 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:42.954730 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ceb96859-62eb-4b15-887c-1611ca2b189d" containerName="revoke-keys" Jul 23 04:29:42.954858 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:42.954735 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ceb96859-62eb-4b15-887c-1611ca2b189d" containerName="revoke-keys" Jul 23 04:29:42.954858 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:42.954789 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="517fb98c-3587-4984-b877-b231316c45de" containerName="istio-proxy" Jul 23 04:29:42.954858 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:42.954799 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ceb96859-62eb-4b15-887c-1611ca2b189d" containerName="revoke-keys" Jul 23 04:29:42.957693 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:42.957677 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t" Jul 23 04:29:42.960223 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:42.960203 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-30964d-openshift-default-dockercfg-hcb95\"" Jul 23 04:29:42.972389 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:42.972363 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t"] Jul 23 04:29:43.065708 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:43.065675 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9796cd21-2784-44ad-86b8-b1cd1d5bf403-istio-data\") pod \"e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t\" (UID: \"9796cd21-2784-44ad-86b8-b1cd1d5bf403\") " pod="openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t" Jul 23 04:29:43.065842 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:43.065728 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9796cd21-2784-44ad-86b8-b1cd1d5bf403-workload-socket\") pod \"e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t\" (UID: \"9796cd21-2784-44ad-86b8-b1cd1d5bf403\") " pod="openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t" Jul 23 04:29:43.065842 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:43.065775 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9796cd21-2784-44ad-86b8-b1cd1d5bf403-credential-socket\") pod \"e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t\" (UID: \"9796cd21-2784-44ad-86b8-b1cd1d5bf403\") " pod="openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t" Jul 23 04:29:43.065842 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:43.065801 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9796cd21-2784-44ad-86b8-b1cd1d5bf403-workload-certs\") pod \"e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t\" (UID: \"9796cd21-2784-44ad-86b8-b1cd1d5bf403\") " pod="openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t" Jul 23 04:29:43.065842 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:43.065827 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9796cd21-2784-44ad-86b8-b1cd1d5bf403-istio-podinfo\") pod \"e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t\" (UID: \"9796cd21-2784-44ad-86b8-b1cd1d5bf403\") " pod="openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t" Jul 23 04:29:43.066066 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:43.065872 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9796cd21-2784-44ad-86b8-b1cd1d5bf403-istiod-ca-cert\") pod \"e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t\" (UID: \"9796cd21-2784-44ad-86b8-b1cd1d5bf403\") " pod="openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t" Jul 23 04:29:43.066066 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:43.065907 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55twx\" (UniqueName: \"kubernetes.io/projected/9796cd21-2784-44ad-86b8-b1cd1d5bf403-kube-api-access-55twx\") pod \"e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t\" (UID: \"9796cd21-2784-44ad-86b8-b1cd1d5bf403\") " pod="openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t" Jul 23 04:29:43.066066 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:43.065946 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9796cd21-2784-44ad-86b8-b1cd1d5bf403-istio-token\") pod \"e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t\" (UID: \"9796cd21-2784-44ad-86b8-b1cd1d5bf403\") " pod="openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t" Jul 23 04:29:43.066066 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:43.066019 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9796cd21-2784-44ad-86b8-b1cd1d5bf403-istio-envoy\") pod \"e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t\" (UID: \"9796cd21-2784-44ad-86b8-b1cd1d5bf403\") " pod="openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t" Jul 23 04:29:43.167195 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:43.167164 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9796cd21-2784-44ad-86b8-b1cd1d5bf403-istio-token\") pod \"e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t\" (UID: \"9796cd21-2784-44ad-86b8-b1cd1d5bf403\") " pod="openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t" Jul 23 04:29:43.167362 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:43.167203 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9796cd21-2784-44ad-86b8-b1cd1d5bf403-istio-envoy\") pod \"e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t\" (UID: \"9796cd21-2784-44ad-86b8-b1cd1d5bf403\") " pod="openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t" Jul 23 04:29:43.167362 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:43.167253 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9796cd21-2784-44ad-86b8-b1cd1d5bf403-istio-data\") pod \"e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t\" (UID: \"9796cd21-2784-44ad-86b8-b1cd1d5bf403\") " pod="openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t" Jul 23 04:29:43.167362 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:43.167283 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9796cd21-2784-44ad-86b8-b1cd1d5bf403-workload-socket\") pod \"e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t\" (UID: \"9796cd21-2784-44ad-86b8-b1cd1d5bf403\") " pod="openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t" Jul 23 04:29:43.167362 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:43.167302 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9796cd21-2784-44ad-86b8-b1cd1d5bf403-credential-socket\") pod \"e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t\" (UID: \"9796cd21-2784-44ad-86b8-b1cd1d5bf403\") " pod="openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t" Jul 23 04:29:43.167362 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:43.167326 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9796cd21-2784-44ad-86b8-b1cd1d5bf403-workload-certs\") pod \"e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t\" (UID: \"9796cd21-2784-44ad-86b8-b1cd1d5bf403\") " pod="openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t" Jul 23 04:29:43.167710 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:43.167354 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9796cd21-2784-44ad-86b8-b1cd1d5bf403-istio-podinfo\") pod \"e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t\" (UID: \"9796cd21-2784-44ad-86b8-b1cd1d5bf403\") " pod="openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t" Jul 23 04:29:43.167710 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:43.167406 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9796cd21-2784-44ad-86b8-b1cd1d5bf403-istiod-ca-cert\") pod \"e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t\" (UID: \"9796cd21-2784-44ad-86b8-b1cd1d5bf403\") " pod="openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t" Jul 23 04:29:43.167710 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:43.167471 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55twx\" (UniqueName: \"kubernetes.io/projected/9796cd21-2784-44ad-86b8-b1cd1d5bf403-kube-api-access-55twx\") pod \"e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t\" (UID: \"9796cd21-2784-44ad-86b8-b1cd1d5bf403\") " pod="openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t" Jul 23 04:29:43.167885 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:43.167727 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9796cd21-2784-44ad-86b8-b1cd1d5bf403-istio-data\") pod \"e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t\" (UID: \"9796cd21-2784-44ad-86b8-b1cd1d5bf403\") " pod="openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t" Jul 23 04:29:43.167885 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:43.167773 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9796cd21-2784-44ad-86b8-b1cd1d5bf403-workload-certs\") pod \"e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t\" (UID: \"9796cd21-2784-44ad-86b8-b1cd1d5bf403\") " pod="openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t" Jul 23 04:29:43.167885 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:43.167797 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9796cd21-2784-44ad-86b8-b1cd1d5bf403-workload-socket\") pod \"e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t\" (UID: \"9796cd21-2784-44ad-86b8-b1cd1d5bf403\") " pod="openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t" Jul 23 04:29:43.167885 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:43.167846 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9796cd21-2784-44ad-86b8-b1cd1d5bf403-credential-socket\") pod \"e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t\" (UID: \"9796cd21-2784-44ad-86b8-b1cd1d5bf403\") " pod="openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t" Jul 23 04:29:43.168182 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:43.168165 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9796cd21-2784-44ad-86b8-b1cd1d5bf403-istiod-ca-cert\") pod \"e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t\" (UID: \"9796cd21-2784-44ad-86b8-b1cd1d5bf403\") " pod="openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t" Jul 23 04:29:43.169852 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:43.169831 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9796cd21-2784-44ad-86b8-b1cd1d5bf403-istio-envoy\") pod \"e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t\" (UID: \"9796cd21-2784-44ad-86b8-b1cd1d5bf403\") " pod="openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t" Jul 23 04:29:43.170054 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:43.170035 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9796cd21-2784-44ad-86b8-b1cd1d5bf403-istio-podinfo\") pod \"e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t\" (UID: \"9796cd21-2784-44ad-86b8-b1cd1d5bf403\") " pod="openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t" Jul 23 04:29:43.176752 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:43.176732 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9796cd21-2784-44ad-86b8-b1cd1d5bf403-istio-token\") pod \"e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t\" (UID: \"9796cd21-2784-44ad-86b8-b1cd1d5bf403\") " pod="openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t" Jul 23 04:29:43.177050 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:43.177030 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55twx\" (UniqueName: \"kubernetes.io/projected/9796cd21-2784-44ad-86b8-b1cd1d5bf403-kube-api-access-55twx\") pod \"e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t\" (UID: \"9796cd21-2784-44ad-86b8-b1cd1d5bf403\") " pod="openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t" Jul 23 04:29:43.267636 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:43.267563 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t" Jul 23 04:29:43.406760 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:43.406733 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t"] Jul 23 04:29:43.408661 ip-10-0-140-6 kubenswrapper[2567]: W0723 04:29:43.408634 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9796cd21_2784_44ad_86b8_b1cd1d5bf403.slice/crio-a0d45138a0f4bb644e42d586048a808e89bc762a941eea26f60ff31240f169be WatchSource:0}: Error finding container a0d45138a0f4bb644e42d586048a808e89bc762a941eea26f60ff31240f169be: Status 404 returned error can't find the container with id a0d45138a0f4bb644e42d586048a808e89bc762a941eea26f60ff31240f169be Jul 23 04:29:43.411184 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:43.411146 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 04:29:43.411245 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:43.411234 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 04:29:43.411289 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:43.411275 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 04:29:43.530635 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:43.530547 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t" event={"ID":"9796cd21-2784-44ad-86b8-b1cd1d5bf403","Type":"ContainerStarted","Data":"d7f737861618af16a23ce586bb5d52e3f20c68258889b9c7e6e28967faa68272"} Jul 23 04:29:43.530635 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:43.530603 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t" event={"ID":"9796cd21-2784-44ad-86b8-b1cd1d5bf403","Type":"ContainerStarted","Data":"a0d45138a0f4bb644e42d586048a808e89bc762a941eea26f60ff31240f169be"} Jul 23 04:29:43.554656 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:43.554609 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t" podStartSLOduration=1.554590347 podStartE2EDuration="1.554590347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 04:29:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 04:29:43.552693594 +0000 UTC m=+2063.457997887" watchObservedRunningTime="2026-07-23 04:29:43.554590347 +0000 UTC m=+2063.459894695" Jul 23 04:29:44.268676 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:44.268642 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t" Jul 23 04:29:44.273677 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:44.273644 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t" Jul 23 04:29:44.536558 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:44.536457 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t" Jul 23 04:29:44.537350 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:44.537322 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t" Jul 23 04:29:59.537931 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:59.537861 2567 patch_prober.go:28] interesting pod/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 23 04:29:59.537931 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:59.537912 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t" podUID="9796cd21-2784-44ad-86b8-b1cd1d5bf403" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 23 04:29:59.619054 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:59.618987 2567 generic.go:356] "Generic (PLEG): container finished" podID="9796cd21-2784-44ad-86b8-b1cd1d5bf403" containerID="d7f737861618af16a23ce586bb5d52e3f20c68258889b9c7e6e28967faa68272" exitCode=0 Jul 23 04:29:59.619184 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:59.619063 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t" event={"ID":"9796cd21-2784-44ad-86b8-b1cd1d5bf403","Type":"ContainerDied","Data":"d7f737861618af16a23ce586bb5d52e3f20c68258889b9c7e6e28967faa68272"} Jul 23 04:29:59.619395 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:59.619381 2567 scope.go:117] "RemoveContainer" containerID="d7f737861618af16a23ce586bb5d52e3f20c68258889b9c7e6e28967faa68272" Jul 23 04:29:59.620362 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:59.620332 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 04:29:59.620452 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:59.620416 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 04:29:59.620508 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:29:59.620488 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 04:30:00.134922 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:00.134884 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-30964d-29746350-hvn4j"] Jul 23 04:30:00.135301 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:00.135287 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="517fb98c-3587-4984-b877-b231316c45de" containerName="istio-proxy" Jul 23 04:30:00.138462 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:00.138421 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-30964d-29746350-hvn4j" Jul 23 04:30:00.139619 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:00.139594 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29746350-vkldq"] Jul 23 04:30:00.141761 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:00.141742 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dpftb\"" Jul 23 04:30:00.142668 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:00.142646 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c7abdc-29746350-llk7m"] Jul 23 04:30:00.142781 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:00.142735 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746350-vkldq" Jul 23 04:30:00.146091 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:00.146054 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c7abdc-29746350-llk7m" Jul 23 04:30:00.149840 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:00.149810 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-30964d-29746350-hvn4j"] Jul 23 04:30:00.151729 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:00.151582 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-30964d-29746350-hvn4j" Jul 23 04:30:00.152910 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:00.152890 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29746350-vkldq"] Jul 23 04:30:00.160644 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:00.160624 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746350-vkldq" Jul 23 04:30:00.168448 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:00.168403 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c7abdc-29746350-llk7m"] Jul 23 04:30:00.195477 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:00.195452 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c7abdc-29746350-llk7m" Jul 23 04:30:00.378627 ip-10-0-140-6 kubenswrapper[2567]: W0723 04:30:00.375278 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb567ac7d_50f3_47e4_96ee_d2ac54ae7d5d.slice/crio-7621dc46dbb469436827fd40459e6cd0c15496e779a6fd700da2cdb8c4fa9f3f WatchSource:0}: Error finding container 7621dc46dbb469436827fd40459e6cd0c15496e779a6fd700da2cdb8c4fa9f3f: Status 404 returned error can't find the container with id 7621dc46dbb469436827fd40459e6cd0c15496e779a6fd700da2cdb8c4fa9f3f Jul 23 04:30:00.379553 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:00.379203 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c7abdc-29746350-llk7m"] Jul 23 04:30:00.522123 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:00.522096 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-30964d-29746350-hvn4j"] Jul 23 04:30:00.524312 ip-10-0-140-6 kubenswrapper[2567]: W0723 04:30:00.524286 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7739c98_90fb_4776_9a28_8d8c00ec9a5e.slice/crio-70977c399c0e08605e64f7171768ad5a89b756330d5bab0305764e51d1de6910 WatchSource:0}: Error finding container 70977c399c0e08605e64f7171768ad5a89b756330d5bab0305764e51d1de6910: Status 404 returned error can't find the container with id 70977c399c0e08605e64f7171768ad5a89b756330d5bab0305764e51d1de6910 Jul 23 04:30:00.553408 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:00.553352 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29746350-vkldq"] Jul 23 04:30:00.572564 ip-10-0-140-6 kubenswrapper[2567]: W0723 04:30:00.572532 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod908620ef_a3f9_4dee_abec_3a074d51ba5b.slice/crio-4b5f8a6b5f483c9e34c8b3d1061bbef50f3515b87d31f672c3ce69d7fd045ca4 WatchSource:0}: Error finding container 4b5f8a6b5f483c9e34c8b3d1061bbef50f3515b87d31f672c3ce69d7fd045ca4: Status 404 returned error can't find the container with id 4b5f8a6b5f483c9e34c8b3d1061bbef50f3515b87d31f672c3ce69d7fd045ca4 Jul 23 04:30:00.626254 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:00.626226 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746350-vkldq" event={"ID":"908620ef-a3f9-4dee-abec-3a074d51ba5b","Type":"ContainerStarted","Data":"4b5f8a6b5f483c9e34c8b3d1061bbef50f3515b87d31f672c3ce69d7fd045ca4"} Jul 23 04:30:00.627958 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:00.627938 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t" event={"ID":"9796cd21-2784-44ad-86b8-b1cd1d5bf403","Type":"ContainerStarted","Data":"21a78704530f4fd3b85c868c3af527460bdb1bd3875a574bdcf30e1df0c8533f"} Jul 23 04:30:00.629137 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:00.629115 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c7abdc-29746350-llk7m" event={"ID":"b567ac7d-50f3-47e4-96ee-d2ac54ae7d5d","Type":"ContainerStarted","Data":"7621dc46dbb469436827fd40459e6cd0c15496e779a6fd700da2cdb8c4fa9f3f"} Jul 23 04:30:00.630192 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:00.630170 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-30964d-29746350-hvn4j" event={"ID":"e7739c98-90fb-4776-9a28-8d8c00ec9a5e","Type":"ContainerStarted","Data":"70977c399c0e08605e64f7171768ad5a89b756330d5bab0305764e51d1de6910"} Jul 23 04:30:01.268811 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:01.268702 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t" Jul 23 04:30:01.269733 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:01.269691 2567 patch_prober.go:28] interesting pod/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 23 04:30:01.269880 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:01.269756 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t" podUID="9796cd21-2784-44ad-86b8-b1cd1d5bf403" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 23 04:30:01.637698 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:01.637593 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746350-vkldq" event={"ID":"908620ef-a3f9-4dee-abec-3a074d51ba5b","Type":"ContainerStarted","Data":"5a016263da3b877496cdc7c1cfa14e77947d113f424cdad0e7fc141f013b284e"} Jul 23 04:30:01.639130 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:01.639102 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c7abdc-29746350-llk7m" event={"ID":"b567ac7d-50f3-47e4-96ee-d2ac54ae7d5d","Type":"ContainerStarted","Data":"dc41d75ae637bb7e89cd45e5b5f043008ce6bc58847acc4c3c39ff107fc19472"} Jul 23 04:30:01.640647 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:01.640614 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-30964d-29746350-hvn4j" event={"ID":"e7739c98-90fb-4776-9a28-8d8c00ec9a5e","Type":"ContainerStarted","Data":"8b34b9a8818d9904dc8ca8a5b669c80862b376f0a151a9b81de60cb956da6e6b"} Jul 23 04:30:01.655977 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:01.655913 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746350-vkldq" podStartSLOduration=1.6558924510000002 podStartE2EDuration="1.655892451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 04:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 04:30:01.652941283 +0000 UTC m=+2081.558245578" watchObservedRunningTime="2026-07-23 04:30:01.655892451 +0000 UTC m=+2081.561196745" Jul 23 04:30:01.670472 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:01.670386 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-30964d-29746350-hvn4j" podStartSLOduration=1.670364564 podStartE2EDuration="1.670364564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 04:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 04:30:01.668484403 +0000 UTC m=+2081.573788725" watchObservedRunningTime="2026-07-23 04:30:01.670364564 +0000 UTC m=+2081.575668856" Jul 23 04:30:01.685010 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:01.684961 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c7abdc-29746350-llk7m" podStartSLOduration=1.6849451210000002 podStartE2EDuration="1.684945121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 04:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 04:30:01.684398354 +0000 UTC m=+2081.589702646" watchObservedRunningTime="2026-07-23 04:30:01.684945121 +0000 UTC m=+2081.590249413" Jul 23 04:30:02.268568 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:02.268470 2567 patch_prober.go:28] interesting pod/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 23 04:30:02.268568 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:02.268541 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t" podUID="9796cd21-2784-44ad-86b8-b1cd1d5bf403" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 23 04:30:02.652004 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:02.651895 2567 generic.go:356] "Generic (PLEG): container finished" podID="908620ef-a3f9-4dee-abec-3a074d51ba5b" containerID="5a016263da3b877496cdc7c1cfa14e77947d113f424cdad0e7fc141f013b284e" exitCode=0 Jul 23 04:30:02.652506 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:02.652034 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746350-vkldq" event={"ID":"908620ef-a3f9-4dee-abec-3a074d51ba5b","Type":"ContainerDied","Data":"5a016263da3b877496cdc7c1cfa14e77947d113f424cdad0e7fc141f013b284e"} Jul 23 04:30:02.660226 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:02.660186 2567 generic.go:356] "Generic (PLEG): container finished" podID="b567ac7d-50f3-47e4-96ee-d2ac54ae7d5d" containerID="dc41d75ae637bb7e89cd45e5b5f043008ce6bc58847acc4c3c39ff107fc19472" exitCode=0 Jul 23 04:30:02.660410 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:02.660276 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c7abdc-29746350-llk7m" event={"ID":"b567ac7d-50f3-47e4-96ee-d2ac54ae7d5d","Type":"ContainerDied","Data":"dc41d75ae637bb7e89cd45e5b5f043008ce6bc58847acc4c3c39ff107fc19472"} Jul 23 04:30:02.662450 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:02.662404 2567 generic.go:356] "Generic (PLEG): container finished" podID="e7739c98-90fb-4776-9a28-8d8c00ec9a5e" containerID="8b34b9a8818d9904dc8ca8a5b669c80862b376f0a151a9b81de60cb956da6e6b" exitCode=0 Jul 23 04:30:02.662570 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:02.662532 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-30964d-29746350-hvn4j" event={"ID":"e7739c98-90fb-4776-9a28-8d8c00ec9a5e","Type":"ContainerDied","Data":"8b34b9a8818d9904dc8ca8a5b669c80862b376f0a151a9b81de60cb956da6e6b"} Jul 23 04:30:03.268938 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:03.268892 2567 patch_prober.go:28] interesting pod/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 23 04:30:03.269111 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:03.268957 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t" podUID="9796cd21-2784-44ad-86b8-b1cd1d5bf403" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 23 04:30:03.835399 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:03.835366 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c7abdc-29746350-llk7m" Jul 23 04:30:03.903785 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:03.903761 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746350-vkldq" Jul 23 04:30:03.906551 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:03.906530 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-30964d-29746350-hvn4j" Jul 23 04:30:04.268605 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:04.268568 2567 patch_prober.go:28] interesting pod/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 23 04:30:04.268798 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:04.268622 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t" podUID="9796cd21-2784-44ad-86b8-b1cd1d5bf403" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 23 04:30:04.680497 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:04.680473 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746350-vkldq" Jul 23 04:30:04.680675 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:04.680469 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746350-vkldq" event={"ID":"908620ef-a3f9-4dee-abec-3a074d51ba5b","Type":"ContainerDied","Data":"4b5f8a6b5f483c9e34c8b3d1061bbef50f3515b87d31f672c3ce69d7fd045ca4"} Jul 23 04:30:04.680675 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:04.680604 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b5f8a6b5f483c9e34c8b3d1061bbef50f3515b87d31f672c3ce69d7fd045ca4" Jul 23 04:30:04.681827 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:04.681809 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c7abdc-29746350-llk7m" Jul 23 04:30:04.681827 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:04.681820 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c7abdc-29746350-llk7m" event={"ID":"b567ac7d-50f3-47e4-96ee-d2ac54ae7d5d","Type":"ContainerDied","Data":"7621dc46dbb469436827fd40459e6cd0c15496e779a6fd700da2cdb8c4fa9f3f"} Jul 23 04:30:04.682008 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:04.681858 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7621dc46dbb469436827fd40459e6cd0c15496e779a6fd700da2cdb8c4fa9f3f" Jul 23 04:30:04.683250 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:04.683226 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-30964d-29746350-hvn4j" event={"ID":"e7739c98-90fb-4776-9a28-8d8c00ec9a5e","Type":"ContainerDied","Data":"70977c399c0e08605e64f7171768ad5a89b756330d5bab0305764e51d1de6910"} Jul 23 04:30:04.683344 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:04.683245 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-30964d-29746350-hvn4j" Jul 23 04:30:04.683344 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:04.683259 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70977c399c0e08605e64f7171768ad5a89b756330d5bab0305764e51d1de6910" Jul 23 04:30:05.272279 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:05.272248 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t" Jul 23 04:30:05.272716 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:05.272516 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t" Jul 23 04:30:05.273154 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:05.273136 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t" Jul 23 04:30:20.614623 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:20.614591 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wzgn9_dbd53593-e5e7-473a-b71c-8fbf9272913f/console-operator/2.log" Jul 23 04:30:20.618235 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:20.618212 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wzgn9_dbd53593-e5e7-473a-b71c-8fbf9272913f/console-operator/2.log" Jul 23 04:30:20.722451 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:20.722392 2567 scope.go:117] "RemoveContainer" containerID="13a8db71cc9ba98b55c5f754239016fcbb81f04d4bd7f4817b659c75cc1e7ca3" Jul 23 04:30:20.730887 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:30:20.730867 2567 scope.go:117] "RemoveContainer" containerID="02646bc46834802a282e2268de65953a1699c9b85b56bfcffccf9312dd12a697" Jul 23 04:31:14.147216 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:14.147134 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-30964d-z97jv"] Jul 23 04:31:14.147649 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:14.147466 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7739c98-90fb-4776-9a28-8d8c00ec9a5e" containerName="cleanup" Jul 23 04:31:14.147649 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:14.147478 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7739c98-90fb-4776-9a28-8d8c00ec9a5e" containerName="cleanup" Jul 23 04:31:14.147649 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:14.147489 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b567ac7d-50f3-47e4-96ee-d2ac54ae7d5d" containerName="cleanup" Jul 23 04:31:14.147649 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:14.147495 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b567ac7d-50f3-47e4-96ee-d2ac54ae7d5d" containerName="cleanup" Jul 23 04:31:14.147649 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:14.147513 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="908620ef-a3f9-4dee-abec-3a074d51ba5b" containerName="cleanup" Jul 23 04:31:14.147649 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:14.147520 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="908620ef-a3f9-4dee-abec-3a074d51ba5b" containerName="cleanup" Jul 23 04:31:14.147649 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:14.147573 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="908620ef-a3f9-4dee-abec-3a074d51ba5b" containerName="cleanup" Jul 23 04:31:14.147649 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:14.147583 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7739c98-90fb-4776-9a28-8d8c00ec9a5e" containerName="cleanup" Jul 23 04:31:14.147649 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:14.147592 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="b567ac7d-50f3-47e4-96ee-d2ac54ae7d5d" containerName="cleanup" Jul 23 04:31:14.150680 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:14.150664 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-30964d-z97jv" Jul 23 04:31:14.153478 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:14.153456 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 04:31:14.153716 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:14.153699 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dpftb\"" Jul 23 04:31:14.163687 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:14.163663 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-30964d-z97jv"] Jul 23 04:31:14.243052 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:14.243016 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8b80baf-8bdc-42bf-a07a-f378cfa58953-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-30964d-z97jv\" (UID: \"d8b80baf-8bdc-42bf-a07a-f378cfa58953\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-30964d-z97jv" Jul 23 04:31:14.344272 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:14.344230 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8b80baf-8bdc-42bf-a07a-f378cfa58953-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-30964d-z97jv\" (UID: \"d8b80baf-8bdc-42bf-a07a-f378cfa58953\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-30964d-z97jv" Jul 23 04:31:14.344850 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:14.344830 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8b80baf-8bdc-42bf-a07a-f378cfa58953-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-30964d-z97jv\" (UID: \"d8b80baf-8bdc-42bf-a07a-f378cfa58953\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-30964d-z97jv" Jul 23 04:31:14.460405 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:14.460376 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-30964d-z97jv" Jul 23 04:31:14.595280 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:14.595249 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-30964d-z97jv"] Jul 23 04:31:14.596972 ip-10-0-140-6 kubenswrapper[2567]: W0723 04:31:14.596939 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8b80baf_8bdc_42bf_a07a_f378cfa58953.slice/crio-c6bf7ae1aa5676df3d10e0c4f7daf7e59f7d5d762268861963adf348a472582f WatchSource:0}: Error finding container c6bf7ae1aa5676df3d10e0c4f7daf7e59f7d5d762268861963adf348a472582f: Status 404 returned error can't find the container with id c6bf7ae1aa5676df3d10e0c4f7daf7e59f7d5d762268861963adf348a472582f Jul 23 04:31:15.071995 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:15.071895 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-30964d-z97jv" event={"ID":"d8b80baf-8bdc-42bf-a07a-f378cfa58953","Type":"ContainerStarted","Data":"c5a87e162ce0d734c842ad3dff116a37f3d1dd6bede2828899ca87e7f02550df"} Jul 23 04:31:15.071995 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:15.071969 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-30964d-z97jv" event={"ID":"d8b80baf-8bdc-42bf-a07a-f378cfa58953","Type":"ContainerStarted","Data":"c6bf7ae1aa5676df3d10e0c4f7daf7e59f7d5d762268861963adf348a472582f"} Jul 23 04:31:15.095551 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:15.095499 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-30964d-z97jv" podStartSLOduration=1.095482088 podStartE2EDuration="1.095482088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 04:31:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 04:31:15.094584368 +0000 UTC m=+2154.999888661" watchObservedRunningTime="2026-07-23 04:31:15.095482088 +0000 UTC m=+2155.000786379" Jul 23 04:31:16.078516 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:16.078478 2567 generic.go:356] "Generic (PLEG): container finished" podID="d8b80baf-8bdc-42bf-a07a-f378cfa58953" containerID="c5a87e162ce0d734c842ad3dff116a37f3d1dd6bede2828899ca87e7f02550df" exitCode=0 Jul 23 04:31:16.078901 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:16.078567 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-30964d-z97jv" event={"ID":"d8b80baf-8bdc-42bf-a07a-f378cfa58953","Type":"ContainerDied","Data":"c5a87e162ce0d734c842ad3dff116a37f3d1dd6bede2828899ca87e7f02550df"} Jul 23 04:31:17.215905 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:17.215878 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-30964d-z97jv" Jul 23 04:31:17.268775 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:17.268744 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8b80baf-8bdc-42bf-a07a-f378cfa58953-maas-api-service-ca\") pod \"d8b80baf-8bdc-42bf-a07a-f378cfa58953\" (UID: \"d8b80baf-8bdc-42bf-a07a-f378cfa58953\") " Jul 23 04:31:17.269064 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:17.269043 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8b80baf-8bdc-42bf-a07a-f378cfa58953-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d8b80baf-8bdc-42bf-a07a-f378cfa58953" (UID: "d8b80baf-8bdc-42bf-a07a-f378cfa58953"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 04:31:17.369636 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:17.369552 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8b80baf-8bdc-42bf-a07a-f378cfa58953-maas-api-service-ca\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:31:18.089065 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:18.089034 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-30964d-z97jv" Jul 23 04:31:18.089065 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:18.089051 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-30964d-z97jv" event={"ID":"d8b80baf-8bdc-42bf-a07a-f378cfa58953","Type":"ContainerDied","Data":"c6bf7ae1aa5676df3d10e0c4f7daf7e59f7d5d762268861963adf348a472582f"} Jul 23 04:31:18.089245 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:18.089098 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6bf7ae1aa5676df3d10e0c4f7daf7e59f7d5d762268861963adf348a472582f" Jul 23 04:31:24.646814 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:24.646781 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-30964d-29746350-hvn4j"] Jul 23 04:31:24.649079 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:24.649056 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-30964d-29746350-hvn4j"] Jul 23 04:31:26.493502 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:26.493470 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7739c98-90fb-4776-9a28-8d8c00ec9a5e" path="/var/lib/kubelet/pods/e7739c98-90fb-4776-9a28-8d8c00ec9a5e/volumes" Jul 23 04:31:29.291452 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:29.291361 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-30964d-z97jv"] Jul 23 04:31:29.293373 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:29.293351 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-30964d-z97jv"] Jul 23 04:31:30.494002 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:30.493969 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8b80baf-8bdc-42bf-a07a-f378cfa58953" path="/var/lib/kubelet/pods/d8b80baf-8bdc-42bf-a07a-f378cfa58953/volumes" Jul 23 04:31:35.834445 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:35.834404 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t"] Jul 23 04:31:35.834909 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:35.834674 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t" podUID="9796cd21-2784-44ad-86b8-b1cd1d5bf403" containerName="istio-proxy" containerID="cri-o://21a78704530f4fd3b85c868c3af527460bdb1bd3875a574bdcf30e1df0c8533f" gracePeriod=30 Jul 23 04:31:36.192005 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:36.191971 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c7abdc-9l2qs"] Jul 23 04:31:36.192308 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:36.192296 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8b80baf-8bdc-42bf-a07a-f378cfa58953" containerName="revoke-keys" Jul 23 04:31:36.192348 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:36.192310 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8b80baf-8bdc-42bf-a07a-f378cfa58953" containerName="revoke-keys" Jul 23 04:31:36.192389 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:36.192368 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8b80baf-8bdc-42bf-a07a-f378cfa58953" containerName="revoke-keys" Jul 23 04:31:36.195590 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:36.195575 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c7abdc-9l2qs" Jul 23 04:31:36.198356 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:36.198338 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dpftb\"" Jul 23 04:31:36.198472 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:36.198344 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 04:31:36.204818 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:36.204797 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c7abdc-9l2qs"] Jul 23 04:31:36.333573 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:36.333538 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d31329d0-9e45-4a5c-877d-eae413fb235c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c7abdc-9l2qs\" (UID: \"d31329d0-9e45-4a5c-877d-eae413fb235c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c7abdc-9l2qs" Jul 23 04:31:36.434704 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:36.434670 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d31329d0-9e45-4a5c-877d-eae413fb235c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c7abdc-9l2qs\" (UID: \"d31329d0-9e45-4a5c-877d-eae413fb235c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c7abdc-9l2qs" Jul 23 04:31:36.435273 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:36.435253 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d31329d0-9e45-4a5c-877d-eae413fb235c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c7abdc-9l2qs\" (UID: \"d31329d0-9e45-4a5c-877d-eae413fb235c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c7abdc-9l2qs" Jul 23 04:31:36.506807 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:36.506728 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c7abdc-9l2qs" Jul 23 04:31:36.648994 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:36.648965 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c7abdc-9l2qs"] Jul 23 04:31:36.650222 ip-10-0-140-6 kubenswrapper[2567]: W0723 04:31:36.650192 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd31329d0_9e45_4a5c_877d_eae413fb235c.slice/crio-a1a4498bf56d63a2443de2bbb3a97b5277f09a42e73d3b42d375bb3af7cdce13 WatchSource:0}: Error finding container a1a4498bf56d63a2443de2bbb3a97b5277f09a42e73d3b42d375bb3af7cdce13: Status 404 returned error can't find the container with id a1a4498bf56d63a2443de2bbb3a97b5277f09a42e73d3b42d375bb3af7cdce13 Jul 23 04:31:37.190911 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:37.190877 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c7abdc-9l2qs" event={"ID":"d31329d0-9e45-4a5c-877d-eae413fb235c","Type":"ContainerStarted","Data":"f5347a23f8cc2bb5b2179d9fdd8b76f9b6419c33d8fdcb70e024cc9abee9288e"} Jul 23 04:31:37.191310 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:37.190931 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c7abdc-9l2qs" event={"ID":"d31329d0-9e45-4a5c-877d-eae413fb235c","Type":"ContainerStarted","Data":"a1a4498bf56d63a2443de2bbb3a97b5277f09a42e73d3b42d375bb3af7cdce13"} Jul 23 04:31:37.220799 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:37.220753 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c7abdc-9l2qs" podStartSLOduration=1.220738812 podStartE2EDuration="1.220738812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 04:31:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 04:31:37.219693005 +0000 UTC m=+2177.124997297" watchObservedRunningTime="2026-07-23 04:31:37.220738812 +0000 UTC m=+2177.126043104" Jul 23 04:31:38.197975 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:38.197938 2567 generic.go:356] "Generic (PLEG): container finished" podID="d31329d0-9e45-4a5c-877d-eae413fb235c" containerID="f5347a23f8cc2bb5b2179d9fdd8b76f9b6419c33d8fdcb70e024cc9abee9288e" exitCode=0 Jul 23 04:31:38.198363 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:38.198007 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c7abdc-9l2qs" event={"ID":"d31329d0-9e45-4a5c-877d-eae413fb235c","Type":"ContainerDied","Data":"f5347a23f8cc2bb5b2179d9fdd8b76f9b6419c33d8fdcb70e024cc9abee9288e"} Jul 23 04:31:39.330141 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:39.330115 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c7abdc-9l2qs" Jul 23 04:31:39.461778 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:39.461695 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d31329d0-9e45-4a5c-877d-eae413fb235c-maas-api-service-ca\") pod \"d31329d0-9e45-4a5c-877d-eae413fb235c\" (UID: \"d31329d0-9e45-4a5c-877d-eae413fb235c\") " Jul 23 04:31:39.462091 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:39.462056 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d31329d0-9e45-4a5c-877d-eae413fb235c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d31329d0-9e45-4a5c-877d-eae413fb235c" (UID: "d31329d0-9e45-4a5c-877d-eae413fb235c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 04:31:39.562891 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:39.562861 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d31329d0-9e45-4a5c-877d-eae413fb235c-maas-api-service-ca\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:31:40.208755 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:40.208721 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c7abdc-9l2qs" Jul 23 04:31:40.209005 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:40.208763 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c7abdc-9l2qs" event={"ID":"d31329d0-9e45-4a5c-877d-eae413fb235c","Type":"ContainerDied","Data":"a1a4498bf56d63a2443de2bbb3a97b5277f09a42e73d3b42d375bb3af7cdce13"} Jul 23 04:31:40.209005 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:40.208808 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1a4498bf56d63a2443de2bbb3a97b5277f09a42e73d3b42d375bb3af7cdce13" Jul 23 04:31:41.075326 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:41.075304 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t" Jul 23 04:31:41.176937 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:41.176915 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9796cd21-2784-44ad-86b8-b1cd1d5bf403-istio-podinfo\") pod \"9796cd21-2784-44ad-86b8-b1cd1d5bf403\" (UID: \"9796cd21-2784-44ad-86b8-b1cd1d5bf403\") " Jul 23 04:31:41.177093 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:41.176958 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9796cd21-2784-44ad-86b8-b1cd1d5bf403-istiod-ca-cert\") pod \"9796cd21-2784-44ad-86b8-b1cd1d5bf403\" (UID: \"9796cd21-2784-44ad-86b8-b1cd1d5bf403\") " Jul 23 04:31:41.177093 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:41.176983 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9796cd21-2784-44ad-86b8-b1cd1d5bf403-istio-data\") pod \"9796cd21-2784-44ad-86b8-b1cd1d5bf403\" (UID: \"9796cd21-2784-44ad-86b8-b1cd1d5bf403\") " Jul 23 04:31:41.177093 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:41.177011 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55twx\" (UniqueName: \"kubernetes.io/projected/9796cd21-2784-44ad-86b8-b1cd1d5bf403-kube-api-access-55twx\") pod \"9796cd21-2784-44ad-86b8-b1cd1d5bf403\" (UID: \"9796cd21-2784-44ad-86b8-b1cd1d5bf403\") " Jul 23 04:31:41.177093 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:41.177052 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9796cd21-2784-44ad-86b8-b1cd1d5bf403-workload-certs\") pod \"9796cd21-2784-44ad-86b8-b1cd1d5bf403\" (UID: \"9796cd21-2784-44ad-86b8-b1cd1d5bf403\") " Jul 23 04:31:41.177281 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:41.177128 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9796cd21-2784-44ad-86b8-b1cd1d5bf403-workload-socket\") pod \"9796cd21-2784-44ad-86b8-b1cd1d5bf403\" (UID: \"9796cd21-2784-44ad-86b8-b1cd1d5bf403\") " Jul 23 04:31:41.177281 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:41.177155 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9796cd21-2784-44ad-86b8-b1cd1d5bf403-istio-token\") pod \"9796cd21-2784-44ad-86b8-b1cd1d5bf403\" (UID: \"9796cd21-2784-44ad-86b8-b1cd1d5bf403\") " Jul 23 04:31:41.177281 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:41.177178 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9796cd21-2784-44ad-86b8-b1cd1d5bf403-istio-envoy\") pod \"9796cd21-2784-44ad-86b8-b1cd1d5bf403\" (UID: \"9796cd21-2784-44ad-86b8-b1cd1d5bf403\") " Jul 23 04:31:41.177281 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:41.177208 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9796cd21-2784-44ad-86b8-b1cd1d5bf403-credential-socket\") pod \"9796cd21-2784-44ad-86b8-b1cd1d5bf403\" (UID: \"9796cd21-2784-44ad-86b8-b1cd1d5bf403\") " Jul 23 04:31:41.177514 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:41.177405 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9796cd21-2784-44ad-86b8-b1cd1d5bf403-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9796cd21-2784-44ad-86b8-b1cd1d5bf403" (UID: "9796cd21-2784-44ad-86b8-b1cd1d5bf403"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 04:31:41.177514 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:41.177420 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9796cd21-2784-44ad-86b8-b1cd1d5bf403-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9796cd21-2784-44ad-86b8-b1cd1d5bf403" (UID: "9796cd21-2784-44ad-86b8-b1cd1d5bf403"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 04:31:41.177514 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:41.177418 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9796cd21-2784-44ad-86b8-b1cd1d5bf403-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9796cd21-2784-44ad-86b8-b1cd1d5bf403" (UID: "9796cd21-2784-44ad-86b8-b1cd1d5bf403"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 04:31:41.177677 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:41.177645 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9796cd21-2784-44ad-86b8-b1cd1d5bf403-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9796cd21-2784-44ad-86b8-b1cd1d5bf403" (UID: "9796cd21-2784-44ad-86b8-b1cd1d5bf403"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 04:31:41.178526 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:41.178496 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9796cd21-2784-44ad-86b8-b1cd1d5bf403-istio-data" (OuterVolumeSpecName: "istio-data") pod "9796cd21-2784-44ad-86b8-b1cd1d5bf403" (UID: "9796cd21-2784-44ad-86b8-b1cd1d5bf403"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 04:31:41.179227 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:41.179202 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9796cd21-2784-44ad-86b8-b1cd1d5bf403-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9796cd21-2784-44ad-86b8-b1cd1d5bf403" (UID: "9796cd21-2784-44ad-86b8-b1cd1d5bf403"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 04:31:41.179342 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:41.179324 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9796cd21-2784-44ad-86b8-b1cd1d5bf403-istio-token" (OuterVolumeSpecName: "istio-token") pod "9796cd21-2784-44ad-86b8-b1cd1d5bf403" (UID: "9796cd21-2784-44ad-86b8-b1cd1d5bf403"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 04:31:41.179664 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:41.179639 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9796cd21-2784-44ad-86b8-b1cd1d5bf403-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9796cd21-2784-44ad-86b8-b1cd1d5bf403" (UID: "9796cd21-2784-44ad-86b8-b1cd1d5bf403"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 04:31:41.179757 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:41.179680 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9796cd21-2784-44ad-86b8-b1cd1d5bf403-kube-api-access-55twx" (OuterVolumeSpecName: "kube-api-access-55twx") pod "9796cd21-2784-44ad-86b8-b1cd1d5bf403" (UID: "9796cd21-2784-44ad-86b8-b1cd1d5bf403"). InnerVolumeSpecName "kube-api-access-55twx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 04:31:41.215991 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:41.215959 2567 generic.go:356] "Generic (PLEG): container finished" podID="9796cd21-2784-44ad-86b8-b1cd1d5bf403" containerID="21a78704530f4fd3b85c868c3af527460bdb1bd3875a574bdcf30e1df0c8533f" exitCode=0 Jul 23 04:31:41.216156 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:41.216009 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t" event={"ID":"9796cd21-2784-44ad-86b8-b1cd1d5bf403","Type":"ContainerDied","Data":"21a78704530f4fd3b85c868c3af527460bdb1bd3875a574bdcf30e1df0c8533f"} Jul 23 04:31:41.216156 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:41.216014 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t" Jul 23 04:31:41.216156 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:41.216051 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t" event={"ID":"9796cd21-2784-44ad-86b8-b1cd1d5bf403","Type":"ContainerDied","Data":"a0d45138a0f4bb644e42d586048a808e89bc762a941eea26f60ff31240f169be"} Jul 23 04:31:41.216156 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:41.216072 2567 scope.go:117] "RemoveContainer" containerID="21a78704530f4fd3b85c868c3af527460bdb1bd3875a574bdcf30e1df0c8533f" Jul 23 04:31:41.225704 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:41.225686 2567 scope.go:117] "RemoveContainer" containerID="d7f737861618af16a23ce586bb5d52e3f20c68258889b9c7e6e28967faa68272" Jul 23 04:31:41.233609 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:41.233592 2567 scope.go:117] "RemoveContainer" containerID="21a78704530f4fd3b85c868c3af527460bdb1bd3875a574bdcf30e1df0c8533f" Jul 23 04:31:41.233888 ip-10-0-140-6 kubenswrapper[2567]: E0723 04:31:41.233871 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21a78704530f4fd3b85c868c3af527460bdb1bd3875a574bdcf30e1df0c8533f\": container with ID starting with 21a78704530f4fd3b85c868c3af527460bdb1bd3875a574bdcf30e1df0c8533f not found: ID does not exist" containerID="21a78704530f4fd3b85c868c3af527460bdb1bd3875a574bdcf30e1df0c8533f" Jul 23 04:31:41.233941 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:41.233899 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21a78704530f4fd3b85c868c3af527460bdb1bd3875a574bdcf30e1df0c8533f"} err="failed to get container status \"21a78704530f4fd3b85c868c3af527460bdb1bd3875a574bdcf30e1df0c8533f\": rpc error: code = NotFound desc = could not find container \"21a78704530f4fd3b85c868c3af527460bdb1bd3875a574bdcf30e1df0c8533f\": container with ID starting with 21a78704530f4fd3b85c868c3af527460bdb1bd3875a574bdcf30e1df0c8533f not found: ID does not exist" Jul 23 04:31:41.233941 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:41.233915 2567 scope.go:117] "RemoveContainer" containerID="d7f737861618af16a23ce586bb5d52e3f20c68258889b9c7e6e28967faa68272" Jul 23 04:31:41.234151 ip-10-0-140-6 kubenswrapper[2567]: E0723 04:31:41.234134 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7f737861618af16a23ce586bb5d52e3f20c68258889b9c7e6e28967faa68272\": container with ID starting with d7f737861618af16a23ce586bb5d52e3f20c68258889b9c7e6e28967faa68272 not found: ID does not exist" containerID="d7f737861618af16a23ce586bb5d52e3f20c68258889b9c7e6e28967faa68272" Jul 23 04:31:41.234192 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:41.234156 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7f737861618af16a23ce586bb5d52e3f20c68258889b9c7e6e28967faa68272"} err="failed to get container status \"d7f737861618af16a23ce586bb5d52e3f20c68258889b9c7e6e28967faa68272\": rpc error: code = NotFound desc = could not find container \"d7f737861618af16a23ce586bb5d52e3f20c68258889b9c7e6e28967faa68272\": container with ID starting with d7f737861618af16a23ce586bb5d52e3f20c68258889b9c7e6e28967faa68272 not found: ID does not exist" Jul 23 04:31:41.240679 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:41.240658 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t"] Jul 23 04:31:41.256962 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:41.256937 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-30964d-openshift-default-cd87496f-2wv9t"] Jul 23 04:31:41.278480 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:41.278449 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9796cd21-2784-44ad-86b8-b1cd1d5bf403-workload-socket\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:31:41.278576 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:41.278488 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9796cd21-2784-44ad-86b8-b1cd1d5bf403-istio-token\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:31:41.278576 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:41.278506 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9796cd21-2784-44ad-86b8-b1cd1d5bf403-istio-envoy\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:31:41.278576 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:41.278520 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9796cd21-2784-44ad-86b8-b1cd1d5bf403-credential-socket\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:31:41.278576 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:41.278529 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9796cd21-2784-44ad-86b8-b1cd1d5bf403-istio-podinfo\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:31:41.278576 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:41.278539 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9796cd21-2784-44ad-86b8-b1cd1d5bf403-istiod-ca-cert\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:31:41.278576 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:41.278547 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9796cd21-2784-44ad-86b8-b1cd1d5bf403-istio-data\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:31:41.278576 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:41.278574 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55twx\" (UniqueName: \"kubernetes.io/projected/9796cd21-2784-44ad-86b8-b1cd1d5bf403-kube-api-access-55twx\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:31:41.278794 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:41.278583 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9796cd21-2784-44ad-86b8-b1cd1d5bf403-workload-certs\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:31:42.492681 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:42.492646 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9796cd21-2784-44ad-86b8-b1cd1d5bf403" path="/var/lib/kubelet/pods/9796cd21-2784-44ad-86b8-b1cd1d5bf403/volumes" Jul 23 04:31:46.756793 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:46.756761 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c7abdc-29746350-llk7m"] Jul 23 04:31:46.761981 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:46.761953 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c7abdc-29746350-llk7m"] Jul 23 04:31:48.491474 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:48.491440 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b567ac7d-50f3-47e4-96ee-d2ac54ae7d5d" path="/var/lib/kubelet/pods/b567ac7d-50f3-47e4-96ee-d2ac54ae7d5d/volumes" Jul 23 04:31:51.295826 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:51.295795 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c7abdc-9l2qs"] Jul 23 04:31:51.302786 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:51.302763 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c7abdc-9l2qs"] Jul 23 04:31:52.491773 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:52.491743 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d31329d0-9e45-4a5c-877d-eae413fb235c" path="/var/lib/kubelet/pods/d31329d0-9e45-4a5c-877d-eae413fb235c/volumes" Jul 23 04:31:58.813518 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:58.813484 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j"] Jul 23 04:31:58.813887 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:58.813790 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9796cd21-2784-44ad-86b8-b1cd1d5bf403" containerName="istio-proxy" Jul 23 04:31:58.813887 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:58.813801 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="9796cd21-2784-44ad-86b8-b1cd1d5bf403" containerName="istio-proxy" Jul 23 04:31:58.813887 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:58.813811 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9796cd21-2784-44ad-86b8-b1cd1d5bf403" containerName="istio-proxy" Jul 23 04:31:58.813887 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:58.813817 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="9796cd21-2784-44ad-86b8-b1cd1d5bf403" containerName="istio-proxy" Jul 23 04:31:58.813887 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:58.813828 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d31329d0-9e45-4a5c-877d-eae413fb235c" containerName="revoke-keys" Jul 23 04:31:58.813887 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:58.813834 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d31329d0-9e45-4a5c-877d-eae413fb235c" containerName="revoke-keys" Jul 23 04:31:58.814078 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:58.813904 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="9796cd21-2784-44ad-86b8-b1cd1d5bf403" containerName="istio-proxy" Jul 23 04:31:58.814078 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:58.813916 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d31329d0-9e45-4a5c-877d-eae413fb235c" containerName="revoke-keys" Jul 23 04:31:58.816864 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:58.816844 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j" Jul 23 04:31:58.819771 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:58.819753 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-b88d1b-openshift-default-dockercfg-l9hsw\"" Jul 23 04:31:58.829473 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:58.829451 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j"] Jul 23 04:31:58.917619 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:58.917590 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8002d8c4-5363-4f18-9cc4-8f7b3fce3abe-credential-socket\") pod \"e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j\" (UID: \"8002d8c4-5363-4f18-9cc4-8f7b3fce3abe\") " pod="openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j" Jul 23 04:31:58.917749 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:58.917628 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8002d8c4-5363-4f18-9cc4-8f7b3fce3abe-istio-token\") pod \"e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j\" (UID: \"8002d8c4-5363-4f18-9cc4-8f7b3fce3abe\") " pod="openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j" Jul 23 04:31:58.917749 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:58.917651 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8002d8c4-5363-4f18-9cc4-8f7b3fce3abe-istio-data\") pod \"e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j\" (UID: \"8002d8c4-5363-4f18-9cc4-8f7b3fce3abe\") " pod="openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j" Jul 23 04:31:58.917749 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:58.917678 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8002d8c4-5363-4f18-9cc4-8f7b3fce3abe-workload-certs\") pod \"e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j\" (UID: \"8002d8c4-5363-4f18-9cc4-8f7b3fce3abe\") " pod="openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j" Jul 23 04:31:58.917936 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:58.917748 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8002d8c4-5363-4f18-9cc4-8f7b3fce3abe-workload-socket\") pod \"e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j\" (UID: \"8002d8c4-5363-4f18-9cc4-8f7b3fce3abe\") " pod="openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j" Jul 23 04:31:58.917936 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:58.917785 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8002d8c4-5363-4f18-9cc4-8f7b3fce3abe-istiod-ca-cert\") pod \"e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j\" (UID: \"8002d8c4-5363-4f18-9cc4-8f7b3fce3abe\") " pod="openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j" Jul 23 04:31:58.917936 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:58.917812 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlhz6\" (UniqueName: \"kubernetes.io/projected/8002d8c4-5363-4f18-9cc4-8f7b3fce3abe-kube-api-access-vlhz6\") pod \"e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j\" (UID: \"8002d8c4-5363-4f18-9cc4-8f7b3fce3abe\") " pod="openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j" Jul 23 04:31:58.917936 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:58.917876 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8002d8c4-5363-4f18-9cc4-8f7b3fce3abe-istio-envoy\") pod \"e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j\" (UID: \"8002d8c4-5363-4f18-9cc4-8f7b3fce3abe\") " pod="openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j" Jul 23 04:31:58.917936 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:58.917916 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8002d8c4-5363-4f18-9cc4-8f7b3fce3abe-istio-podinfo\") pod \"e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j\" (UID: \"8002d8c4-5363-4f18-9cc4-8f7b3fce3abe\") " pod="openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j" Jul 23 04:31:59.018839 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:59.018806 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8002d8c4-5363-4f18-9cc4-8f7b3fce3abe-istio-envoy\") pod \"e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j\" (UID: \"8002d8c4-5363-4f18-9cc4-8f7b3fce3abe\") " pod="openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j" Jul 23 04:31:59.018996 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:59.018853 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8002d8c4-5363-4f18-9cc4-8f7b3fce3abe-istio-podinfo\") pod \"e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j\" (UID: \"8002d8c4-5363-4f18-9cc4-8f7b3fce3abe\") " pod="openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j" Jul 23 04:31:59.018996 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:59.018892 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8002d8c4-5363-4f18-9cc4-8f7b3fce3abe-credential-socket\") pod \"e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j\" (UID: \"8002d8c4-5363-4f18-9cc4-8f7b3fce3abe\") " pod="openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j" Jul 23 04:31:59.018996 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:59.018926 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8002d8c4-5363-4f18-9cc4-8f7b3fce3abe-istio-token\") pod \"e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j\" (UID: \"8002d8c4-5363-4f18-9cc4-8f7b3fce3abe\") " pod="openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j" Jul 23 04:31:59.018996 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:59.018954 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8002d8c4-5363-4f18-9cc4-8f7b3fce3abe-istio-data\") pod \"e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j\" (UID: \"8002d8c4-5363-4f18-9cc4-8f7b3fce3abe\") " pod="openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j" Jul 23 04:31:59.018996 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:59.018987 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8002d8c4-5363-4f18-9cc4-8f7b3fce3abe-workload-certs\") pod \"e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j\" (UID: \"8002d8c4-5363-4f18-9cc4-8f7b3fce3abe\") " pod="openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j" Jul 23 04:31:59.019273 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:59.019020 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8002d8c4-5363-4f18-9cc4-8f7b3fce3abe-workload-socket\") pod \"e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j\" (UID: \"8002d8c4-5363-4f18-9cc4-8f7b3fce3abe\") " pod="openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j" Jul 23 04:31:59.019273 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:59.019047 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8002d8c4-5363-4f18-9cc4-8f7b3fce3abe-istiod-ca-cert\") pod \"e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j\" (UID: \"8002d8c4-5363-4f18-9cc4-8f7b3fce3abe\") " pod="openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j" Jul 23 04:31:59.019273 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:59.019071 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlhz6\" (UniqueName: \"kubernetes.io/projected/8002d8c4-5363-4f18-9cc4-8f7b3fce3abe-kube-api-access-vlhz6\") pod \"e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j\" (UID: \"8002d8c4-5363-4f18-9cc4-8f7b3fce3abe\") " pod="openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j" Jul 23 04:31:59.019472 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:59.019417 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8002d8c4-5363-4f18-9cc4-8f7b3fce3abe-workload-certs\") pod \"e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j\" (UID: \"8002d8c4-5363-4f18-9cc4-8f7b3fce3abe\") " pod="openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j" Jul 23 04:31:59.019561 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:59.019467 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8002d8c4-5363-4f18-9cc4-8f7b3fce3abe-istio-data\") pod \"e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j\" (UID: \"8002d8c4-5363-4f18-9cc4-8f7b3fce3abe\") " pod="openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j" Jul 23 04:31:59.019561 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:59.019535 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8002d8c4-5363-4f18-9cc4-8f7b3fce3abe-workload-socket\") pod \"e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j\" (UID: \"8002d8c4-5363-4f18-9cc4-8f7b3fce3abe\") " pod="openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j" Jul 23 04:31:59.019743 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:59.019724 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8002d8c4-5363-4f18-9cc4-8f7b3fce3abe-credential-socket\") pod \"e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j\" (UID: \"8002d8c4-5363-4f18-9cc4-8f7b3fce3abe\") " pod="openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j" Jul 23 04:31:59.020153 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:59.020132 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8002d8c4-5363-4f18-9cc4-8f7b3fce3abe-istiod-ca-cert\") pod \"e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j\" (UID: \"8002d8c4-5363-4f18-9cc4-8f7b3fce3abe\") " pod="openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j" Jul 23 04:31:59.021355 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:59.021327 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8002d8c4-5363-4f18-9cc4-8f7b3fce3abe-istio-envoy\") pod \"e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j\" (UID: \"8002d8c4-5363-4f18-9cc4-8f7b3fce3abe\") " pod="openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j" Jul 23 04:31:59.021561 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:59.021544 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8002d8c4-5363-4f18-9cc4-8f7b3fce3abe-istio-podinfo\") pod \"e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j\" (UID: \"8002d8c4-5363-4f18-9cc4-8f7b3fce3abe\") " pod="openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j" Jul 23 04:31:59.028901 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:59.028882 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8002d8c4-5363-4f18-9cc4-8f7b3fce3abe-istio-token\") pod \"e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j\" (UID: \"8002d8c4-5363-4f18-9cc4-8f7b3fce3abe\") " pod="openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j" Jul 23 04:31:59.029045 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:59.029025 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlhz6\" (UniqueName: \"kubernetes.io/projected/8002d8c4-5363-4f18-9cc4-8f7b3fce3abe-kube-api-access-vlhz6\") pod \"e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j\" (UID: \"8002d8c4-5363-4f18-9cc4-8f7b3fce3abe\") " pod="openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j" Jul 23 04:31:59.127107 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:59.127040 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j" Jul 23 04:31:59.285927 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:59.285755 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j"] Jul 23 04:31:59.288715 ip-10-0-140-6 kubenswrapper[2567]: W0723 04:31:59.288687 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8002d8c4_5363_4f18_9cc4_8f7b3fce3abe.slice/crio-ecec1d9da7e48a9ec4e5a8421c59cf0e0f1e6d29186c737a98650c58806745c7 WatchSource:0}: Error finding container ecec1d9da7e48a9ec4e5a8421c59cf0e0f1e6d29186c737a98650c58806745c7: Status 404 returned error can't find the container with id ecec1d9da7e48a9ec4e5a8421c59cf0e0f1e6d29186c737a98650c58806745c7 Jul 23 04:31:59.291176 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:59.291143 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 04:31:59.291276 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:59.291209 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 04:31:59.291276 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:59.291237 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 04:31:59.309940 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:31:59.309915 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j" event={"ID":"8002d8c4-5363-4f18-9cc4-8f7b3fce3abe","Type":"ContainerStarted","Data":"ecec1d9da7e48a9ec4e5a8421c59cf0e0f1e6d29186c737a98650c58806745c7"} Jul 23 04:32:00.316975 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:00.316926 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j" event={"ID":"8002d8c4-5363-4f18-9cc4-8f7b3fce3abe","Type":"ContainerStarted","Data":"4e5352a5bf69423e72337d4cedbb0d053c46bbf568cdb7b80cb81ee0561ddf57"} Jul 23 04:32:00.340746 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:00.340680 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j" podStartSLOduration=2.340663637 podStartE2EDuration="2.340663637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 04:31:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 04:32:00.33985425 +0000 UTC m=+2200.245158544" watchObservedRunningTime="2026-07-23 04:32:00.340663637 +0000 UTC m=+2200.245967928" Jul 23 04:32:01.128203 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:01.128172 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j" Jul 23 04:32:01.133606 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:01.133581 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j" Jul 23 04:32:01.322400 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:01.322371 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j" Jul 23 04:32:01.323247 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:01.323229 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j" Jul 23 04:32:15.407684 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:15.407585 2567 generic.go:356] "Generic (PLEG): container finished" podID="8002d8c4-5363-4f18-9cc4-8f7b3fce3abe" containerID="4e5352a5bf69423e72337d4cedbb0d053c46bbf568cdb7b80cb81ee0561ddf57" exitCode=0 Jul 23 04:32:15.407684 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:15.407661 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j" event={"ID":"8002d8c4-5363-4f18-9cc4-8f7b3fce3abe","Type":"ContainerDied","Data":"4e5352a5bf69423e72337d4cedbb0d053c46bbf568cdb7b80cb81ee0561ddf57"} Jul 23 04:32:15.408217 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:15.408036 2567 scope.go:117] "RemoveContainer" containerID="4e5352a5bf69423e72337d4cedbb0d053c46bbf568cdb7b80cb81ee0561ddf57" Jul 23 04:32:15.409034 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:15.409004 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 04:32:15.409149 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:15.409073 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 04:32:15.409149 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:15.409104 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 04:32:16.322666 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:16.322634 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j" Jul 23 04:32:16.415600 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:16.415571 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j" event={"ID":"8002d8c4-5363-4f18-9cc4-8f7b3fce3abe","Type":"ContainerStarted","Data":"24f78fe22ad8d330374b6a1ce53479fa8a0692392b5b3aa4500f7f5de189c4e1"} Jul 23 04:32:17.128056 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:17.128016 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j" Jul 23 04:32:17.129185 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:17.129157 2567 patch_prober.go:28] interesting pod/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 23 04:32:17.129311 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:17.129207 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j" podUID="8002d8c4-5363-4f18-9cc4-8f7b3fce3abe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 23 04:32:18.127478 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:18.127416 2567 patch_prober.go:28] interesting pod/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 23 04:32:18.127888 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:18.127498 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j" podUID="8002d8c4-5363-4f18-9cc4-8f7b3fce3abe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 23 04:32:19.128477 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:19.128420 2567 patch_prober.go:28] interesting pod/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 23 04:32:19.128863 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:19.128514 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j" podUID="8002d8c4-5363-4f18-9cc4-8f7b3fce3abe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 23 04:32:20.127716 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:20.127677 2567 patch_prober.go:28] interesting pod/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 23 04:32:20.127910 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:20.127739 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j" podUID="8002d8c4-5363-4f18-9cc4-8f7b3fce3abe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 23 04:32:20.764621 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:20.764590 2567 scope.go:117] "RemoveContainer" containerID="a8c0b159475efc4bd2119b6ca22ea4ff7503c2645dde70d7077702a25c2565bc" Jul 23 04:32:21.131715 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:21.131626 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j" Jul 23 04:32:21.131975 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:21.131954 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j" Jul 23 04:32:21.132720 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:21.132701 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j" Jul 23 04:32:23.115113 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:23.115077 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b88d1b-x64n8"] Jul 23 04:32:23.115553 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:23.115454 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="9796cd21-2784-44ad-86b8-b1cd1d5bf403" containerName="istio-proxy" Jul 23 04:32:23.120267 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:23.120245 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b88d1b-x64n8" Jul 23 04:32:23.122926 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:23.122900 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 04:32:23.123067 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:23.122942 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dpftb\"" Jul 23 04:32:23.127112 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:23.127089 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b88d1b-x64n8"] Jul 23 04:32:23.224785 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:23.224756 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ced55be1-a918-4a60-82a6-183508b20978-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b88d1b-x64n8\" (UID: \"ced55be1-a918-4a60-82a6-183508b20978\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b88d1b-x64n8" Jul 23 04:32:23.325444 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:23.325400 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ced55be1-a918-4a60-82a6-183508b20978-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b88d1b-x64n8\" (UID: \"ced55be1-a918-4a60-82a6-183508b20978\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b88d1b-x64n8" Jul 23 04:32:23.325988 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:23.325970 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ced55be1-a918-4a60-82a6-183508b20978-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b88d1b-x64n8\" (UID: \"ced55be1-a918-4a60-82a6-183508b20978\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b88d1b-x64n8" Jul 23 04:32:23.431490 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:23.431466 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b88d1b-x64n8" Jul 23 04:32:23.562557 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:23.562531 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b88d1b-x64n8"] Jul 23 04:32:23.563974 ip-10-0-140-6 kubenswrapper[2567]: W0723 04:32:23.563947 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podced55be1_a918_4a60_82a6_183508b20978.slice/crio-82db9a75e5c481d78ceee4ab665ca091ab8ed4571f54a3cb7cad3f6b5cc45875 WatchSource:0}: Error finding container 82db9a75e5c481d78ceee4ab665ca091ab8ed4571f54a3cb7cad3f6b5cc45875: Status 404 returned error can't find the container with id 82db9a75e5c481d78ceee4ab665ca091ab8ed4571f54a3cb7cad3f6b5cc45875 Jul 23 04:32:24.460552 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:24.460512 2567 generic.go:356] "Generic (PLEG): container finished" podID="ced55be1-a918-4a60-82a6-183508b20978" containerID="c3d2d407361f39984c29b842fb1f84fd417c771bbb4f61d7da2a152461792d77" exitCode=0 Jul 23 04:32:24.460916 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:24.460603 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b88d1b-x64n8" event={"ID":"ced55be1-a918-4a60-82a6-183508b20978","Type":"ContainerDied","Data":"c3d2d407361f39984c29b842fb1f84fd417c771bbb4f61d7da2a152461792d77"} Jul 23 04:32:24.460916 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:24.460648 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b88d1b-x64n8" event={"ID":"ced55be1-a918-4a60-82a6-183508b20978","Type":"ContainerStarted","Data":"82db9a75e5c481d78ceee4ab665ca091ab8ed4571f54a3cb7cad3f6b5cc45875"} Jul 23 04:32:25.594514 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:25.594490 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b88d1b-x64n8" Jul 23 04:32:25.642676 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:25.642650 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ced55be1-a918-4a60-82a6-183508b20978-maas-api-service-ca\") pod \"ced55be1-a918-4a60-82a6-183508b20978\" (UID: \"ced55be1-a918-4a60-82a6-183508b20978\") " Jul 23 04:32:25.642998 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:25.642977 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ced55be1-a918-4a60-82a6-183508b20978-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ced55be1-a918-4a60-82a6-183508b20978" (UID: "ced55be1-a918-4a60-82a6-183508b20978"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 04:32:25.743884 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:25.743813 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ced55be1-a918-4a60-82a6-183508b20978-maas-api-service-ca\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:32:26.477627 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:26.477588 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b88d1b-x64n8" event={"ID":"ced55be1-a918-4a60-82a6-183508b20978","Type":"ContainerDied","Data":"82db9a75e5c481d78ceee4ab665ca091ab8ed4571f54a3cb7cad3f6b5cc45875"} Jul 23 04:32:26.477627 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:26.477629 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b88d1b-x64n8" Jul 23 04:32:26.477847 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:26.477668 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82db9a75e5c481d78ceee4ab665ca091ab8ed4571f54a3cb7cad3f6b5cc45875" Jul 23 04:32:43.233518 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:43.233438 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b88d1b-x64n8"] Jul 23 04:32:43.248745 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:43.248714 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b88d1b-x64n8"] Jul 23 04:32:44.492076 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:44.492040 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ced55be1-a918-4a60-82a6-183508b20978" path="/var/lib/kubelet/pods/ced55be1-a918-4a60-82a6-183508b20978/volumes" Jul 23 04:32:50.339671 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:50.339639 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j"] Jul 23 04:32:50.340049 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:50.339880 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j" podUID="8002d8c4-5363-4f18-9cc4-8f7b3fce3abe" containerName="istio-proxy" containerID="cri-o://24f78fe22ad8d330374b6a1ce53479fa8a0692392b5b3aa4500f7f5de189c4e1" gracePeriod=30 Jul 23 04:32:50.752757 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:50.752726 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a4e7cf-2w9ld"] Jul 23 04:32:50.753117 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:50.753102 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ced55be1-a918-4a60-82a6-183508b20978" containerName="revoke-keys" Jul 23 04:32:50.753117 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:50.753118 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ced55be1-a918-4a60-82a6-183508b20978" containerName="revoke-keys" Jul 23 04:32:50.753254 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:50.753193 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ced55be1-a918-4a60-82a6-183508b20978" containerName="revoke-keys" Jul 23 04:32:50.757255 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:50.757238 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a4e7cf-2w9ld" Jul 23 04:32:50.761994 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:50.761967 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dpftb\"" Jul 23 04:32:50.762112 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:50.762008 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 04:32:50.771785 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:50.771763 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a4e7cf-2w9ld"] Jul 23 04:32:50.846876 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:50.846846 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dbfda29-2626-4e02-af37-fbd013cdcdb5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a4e7cf-2w9ld\" (UID: \"1dbfda29-2626-4e02-af37-fbd013cdcdb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a4e7cf-2w9ld" Jul 23 04:32:50.948206 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:50.948179 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dbfda29-2626-4e02-af37-fbd013cdcdb5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a4e7cf-2w9ld\" (UID: \"1dbfda29-2626-4e02-af37-fbd013cdcdb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a4e7cf-2w9ld" Jul 23 04:32:50.948802 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:50.948782 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dbfda29-2626-4e02-af37-fbd013cdcdb5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a4e7cf-2w9ld\" (UID: \"1dbfda29-2626-4e02-af37-fbd013cdcdb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a4e7cf-2w9ld" Jul 23 04:32:51.067874 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:51.067817 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a4e7cf-2w9ld" Jul 23 04:32:51.135916 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:51.135625 2567 patch_prober.go:28] interesting pod/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 23 04:32:51.135916 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:51.135721 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j" podUID="8002d8c4-5363-4f18-9cc4-8f7b3fce3abe" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 23 04:32:51.211689 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:51.211657 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a4e7cf-2w9ld"] Jul 23 04:32:51.212549 ip-10-0-140-6 kubenswrapper[2567]: W0723 04:32:51.212521 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dbfda29_2626_4e02_af37_fbd013cdcdb5.slice/crio-b88a2011a6fb594348b1e0f43c7360c453334bc95af917f76fdd62b908abdf7b WatchSource:0}: Error finding container b88a2011a6fb594348b1e0f43c7360c453334bc95af917f76fdd62b908abdf7b: Status 404 returned error can't find the container with id b88a2011a6fb594348b1e0f43c7360c453334bc95af917f76fdd62b908abdf7b Jul 23 04:32:51.612146 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:51.612112 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a4e7cf-2w9ld" event={"ID":"1dbfda29-2626-4e02-af37-fbd013cdcdb5","Type":"ContainerStarted","Data":"c173875454e64c253d05c420693bdfe18a055b1ebfe506b569b119bca66ced30"} Jul 23 04:32:51.612534 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:51.612168 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a4e7cf-2w9ld" event={"ID":"1dbfda29-2626-4e02-af37-fbd013cdcdb5","Type":"ContainerStarted","Data":"b88a2011a6fb594348b1e0f43c7360c453334bc95af917f76fdd62b908abdf7b"} Jul 23 04:32:51.630677 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:51.630635 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a4e7cf-2w9ld" podStartSLOduration=1.630620263 podStartE2EDuration="1.630620263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 04:32:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 04:32:51.629656687 +0000 UTC m=+2251.534960979" watchObservedRunningTime="2026-07-23 04:32:51.630620263 +0000 UTC m=+2251.535924604" Jul 23 04:32:52.619060 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:52.619029 2567 generic.go:356] "Generic (PLEG): container finished" podID="1dbfda29-2626-4e02-af37-fbd013cdcdb5" containerID="c173875454e64c253d05c420693bdfe18a055b1ebfe506b569b119bca66ced30" exitCode=0 Jul 23 04:32:52.619448 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:52.619119 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a4e7cf-2w9ld" event={"ID":"1dbfda29-2626-4e02-af37-fbd013cdcdb5","Type":"ContainerDied","Data":"c173875454e64c253d05c420693bdfe18a055b1ebfe506b569b119bca66ced30"} Jul 23 04:32:53.755304 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:53.755277 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a4e7cf-2w9ld" Jul 23 04:32:53.870507 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:53.870477 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dbfda29-2626-4e02-af37-fbd013cdcdb5-maas-api-service-ca\") pod \"1dbfda29-2626-4e02-af37-fbd013cdcdb5\" (UID: \"1dbfda29-2626-4e02-af37-fbd013cdcdb5\") " Jul 23 04:32:53.870837 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:53.870814 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dbfda29-2626-4e02-af37-fbd013cdcdb5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1dbfda29-2626-4e02-af37-fbd013cdcdb5" (UID: "1dbfda29-2626-4e02-af37-fbd013cdcdb5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 04:32:53.971577 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:53.971545 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dbfda29-2626-4e02-af37-fbd013cdcdb5-maas-api-service-ca\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:32:54.631330 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:54.631303 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a4e7cf-2w9ld" Jul 23 04:32:54.631516 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:54.631336 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a4e7cf-2w9ld" event={"ID":"1dbfda29-2626-4e02-af37-fbd013cdcdb5","Type":"ContainerDied","Data":"b88a2011a6fb594348b1e0f43c7360c453334bc95af917f76fdd62b908abdf7b"} Jul 23 04:32:54.631516 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:54.631382 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b88a2011a6fb594348b1e0f43c7360c453334bc95af917f76fdd62b908abdf7b" Jul 23 04:32:55.595983 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:55.595961 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j" Jul 23 04:32:55.639166 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:55.639008 2567 generic.go:356] "Generic (PLEG): container finished" podID="8002d8c4-5363-4f18-9cc4-8f7b3fce3abe" containerID="24f78fe22ad8d330374b6a1ce53479fa8a0692392b5b3aa4500f7f5de189c4e1" exitCode=0 Jul 23 04:32:55.639166 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:55.639065 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j" Jul 23 04:32:55.639166 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:55.639080 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j" event={"ID":"8002d8c4-5363-4f18-9cc4-8f7b3fce3abe","Type":"ContainerDied","Data":"24f78fe22ad8d330374b6a1ce53479fa8a0692392b5b3aa4500f7f5de189c4e1"} Jul 23 04:32:55.639166 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:55.639127 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j" event={"ID":"8002d8c4-5363-4f18-9cc4-8f7b3fce3abe","Type":"ContainerDied","Data":"ecec1d9da7e48a9ec4e5a8421c59cf0e0f1e6d29186c737a98650c58806745c7"} Jul 23 04:32:55.639166 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:55.639151 2567 scope.go:117] "RemoveContainer" containerID="24f78fe22ad8d330374b6a1ce53479fa8a0692392b5b3aa4500f7f5de189c4e1" Jul 23 04:32:55.649862 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:55.649846 2567 scope.go:117] "RemoveContainer" containerID="4e5352a5bf69423e72337d4cedbb0d053c46bbf568cdb7b80cb81ee0561ddf57" Jul 23 04:32:55.657802 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:55.657783 2567 scope.go:117] "RemoveContainer" containerID="24f78fe22ad8d330374b6a1ce53479fa8a0692392b5b3aa4500f7f5de189c4e1" Jul 23 04:32:55.658063 ip-10-0-140-6 kubenswrapper[2567]: E0723 04:32:55.658043 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24f78fe22ad8d330374b6a1ce53479fa8a0692392b5b3aa4500f7f5de189c4e1\": container with ID starting with 24f78fe22ad8d330374b6a1ce53479fa8a0692392b5b3aa4500f7f5de189c4e1 not found: ID does not exist" containerID="24f78fe22ad8d330374b6a1ce53479fa8a0692392b5b3aa4500f7f5de189c4e1" Jul 23 04:32:55.658131 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:55.658072 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24f78fe22ad8d330374b6a1ce53479fa8a0692392b5b3aa4500f7f5de189c4e1"} err="failed to get container status \"24f78fe22ad8d330374b6a1ce53479fa8a0692392b5b3aa4500f7f5de189c4e1\": rpc error: code = NotFound desc = could not find container \"24f78fe22ad8d330374b6a1ce53479fa8a0692392b5b3aa4500f7f5de189c4e1\": container with ID starting with 24f78fe22ad8d330374b6a1ce53479fa8a0692392b5b3aa4500f7f5de189c4e1 not found: ID does not exist" Jul 23 04:32:55.658131 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:55.658089 2567 scope.go:117] "RemoveContainer" containerID="4e5352a5bf69423e72337d4cedbb0d053c46bbf568cdb7b80cb81ee0561ddf57" Jul 23 04:32:55.658319 ip-10-0-140-6 kubenswrapper[2567]: E0723 04:32:55.658304 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e5352a5bf69423e72337d4cedbb0d053c46bbf568cdb7b80cb81ee0561ddf57\": container with ID starting with 4e5352a5bf69423e72337d4cedbb0d053c46bbf568cdb7b80cb81ee0561ddf57 not found: ID does not exist" containerID="4e5352a5bf69423e72337d4cedbb0d053c46bbf568cdb7b80cb81ee0561ddf57" Jul 23 04:32:55.658362 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:55.658326 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e5352a5bf69423e72337d4cedbb0d053c46bbf568cdb7b80cb81ee0561ddf57"} err="failed to get container status \"4e5352a5bf69423e72337d4cedbb0d053c46bbf568cdb7b80cb81ee0561ddf57\": rpc error: code = NotFound desc = could not find container \"4e5352a5bf69423e72337d4cedbb0d053c46bbf568cdb7b80cb81ee0561ddf57\": container with ID starting with 4e5352a5bf69423e72337d4cedbb0d053c46bbf568cdb7b80cb81ee0561ddf57 not found: ID does not exist" Jul 23 04:32:55.685283 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:55.685261 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vlhz6\" (UniqueName: \"kubernetes.io/projected/8002d8c4-5363-4f18-9cc4-8f7b3fce3abe-kube-api-access-vlhz6\") pod \"8002d8c4-5363-4f18-9cc4-8f7b3fce3abe\" (UID: \"8002d8c4-5363-4f18-9cc4-8f7b3fce3abe\") " Jul 23 04:32:55.685385 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:55.685297 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8002d8c4-5363-4f18-9cc4-8f7b3fce3abe-credential-socket\") pod \"8002d8c4-5363-4f18-9cc4-8f7b3fce3abe\" (UID: \"8002d8c4-5363-4f18-9cc4-8f7b3fce3abe\") " Jul 23 04:32:55.685385 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:55.685317 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8002d8c4-5363-4f18-9cc4-8f7b3fce3abe-istio-token\") pod \"8002d8c4-5363-4f18-9cc4-8f7b3fce3abe\" (UID: \"8002d8c4-5363-4f18-9cc4-8f7b3fce3abe\") " Jul 23 04:32:55.685493 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:55.685467 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8002d8c4-5363-4f18-9cc4-8f7b3fce3abe-workload-certs\") pod \"8002d8c4-5363-4f18-9cc4-8f7b3fce3abe\" (UID: \"8002d8c4-5363-4f18-9cc4-8f7b3fce3abe\") " Jul 23 04:32:55.685545 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:55.685494 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8002d8c4-5363-4f18-9cc4-8f7b3fce3abe-istio-podinfo\") pod \"8002d8c4-5363-4f18-9cc4-8f7b3fce3abe\" (UID: \"8002d8c4-5363-4f18-9cc4-8f7b3fce3abe\") " Jul 23 04:32:55.685545 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:55.685526 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8002d8c4-5363-4f18-9cc4-8f7b3fce3abe-istio-envoy\") pod \"8002d8c4-5363-4f18-9cc4-8f7b3fce3abe\" (UID: \"8002d8c4-5363-4f18-9cc4-8f7b3fce3abe\") " Jul 23 04:32:55.685545 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:55.685526 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8002d8c4-5363-4f18-9cc4-8f7b3fce3abe-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8002d8c4-5363-4f18-9cc4-8f7b3fce3abe" (UID: "8002d8c4-5363-4f18-9cc4-8f7b3fce3abe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 04:32:55.685704 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:55.685651 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8002d8c4-5363-4f18-9cc4-8f7b3fce3abe-istiod-ca-cert\") pod \"8002d8c4-5363-4f18-9cc4-8f7b3fce3abe\" (UID: \"8002d8c4-5363-4f18-9cc4-8f7b3fce3abe\") " Jul 23 04:32:55.685758 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:55.685733 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8002d8c4-5363-4f18-9cc4-8f7b3fce3abe-istio-data\") pod \"8002d8c4-5363-4f18-9cc4-8f7b3fce3abe\" (UID: \"8002d8c4-5363-4f18-9cc4-8f7b3fce3abe\") " Jul 23 04:32:55.685758 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:55.685745 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8002d8c4-5363-4f18-9cc4-8f7b3fce3abe-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8002d8c4-5363-4f18-9cc4-8f7b3fce3abe" (UID: "8002d8c4-5363-4f18-9cc4-8f7b3fce3abe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 04:32:55.685865 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:55.685769 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8002d8c4-5363-4f18-9cc4-8f7b3fce3abe-workload-socket\") pod \"8002d8c4-5363-4f18-9cc4-8f7b3fce3abe\" (UID: \"8002d8c4-5363-4f18-9cc4-8f7b3fce3abe\") " Jul 23 04:32:55.686102 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:55.686042 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8002d8c4-5363-4f18-9cc4-8f7b3fce3abe-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8002d8c4-5363-4f18-9cc4-8f7b3fce3abe" (UID: "8002d8c4-5363-4f18-9cc4-8f7b3fce3abe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 04:32:55.686102 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:55.686089 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8002d8c4-5363-4f18-9cc4-8f7b3fce3abe-credential-socket\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:32:55.686251 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:55.686112 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8002d8c4-5363-4f18-9cc4-8f7b3fce3abe-workload-certs\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:32:55.686308 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:55.686284 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8002d8c4-5363-4f18-9cc4-8f7b3fce3abe-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8002d8c4-5363-4f18-9cc4-8f7b3fce3abe" (UID: "8002d8c4-5363-4f18-9cc4-8f7b3fce3abe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 04:32:55.686831 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:55.686799 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8002d8c4-5363-4f18-9cc4-8f7b3fce3abe-istio-data" (OuterVolumeSpecName: "istio-data") pod "8002d8c4-5363-4f18-9cc4-8f7b3fce3abe" (UID: "8002d8c4-5363-4f18-9cc4-8f7b3fce3abe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 04:32:55.687650 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:55.687626 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8002d8c4-5363-4f18-9cc4-8f7b3fce3abe-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8002d8c4-5363-4f18-9cc4-8f7b3fce3abe" (UID: "8002d8c4-5363-4f18-9cc4-8f7b3fce3abe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 04:32:55.687752 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:55.687729 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8002d8c4-5363-4f18-9cc4-8f7b3fce3abe-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8002d8c4-5363-4f18-9cc4-8f7b3fce3abe" (UID: "8002d8c4-5363-4f18-9cc4-8f7b3fce3abe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 04:32:55.687916 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:55.687896 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8002d8c4-5363-4f18-9cc4-8f7b3fce3abe-istio-token" (OuterVolumeSpecName: "istio-token") pod "8002d8c4-5363-4f18-9cc4-8f7b3fce3abe" (UID: "8002d8c4-5363-4f18-9cc4-8f7b3fce3abe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 04:32:55.687980 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:55.687967 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8002d8c4-5363-4f18-9cc4-8f7b3fce3abe-kube-api-access-vlhz6" (OuterVolumeSpecName: "kube-api-access-vlhz6") pod "8002d8c4-5363-4f18-9cc4-8f7b3fce3abe" (UID: "8002d8c4-5363-4f18-9cc4-8f7b3fce3abe"). InnerVolumeSpecName "kube-api-access-vlhz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 04:32:55.787350 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:55.787330 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8002d8c4-5363-4f18-9cc4-8f7b3fce3abe-workload-socket\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:32:55.787350 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:55.787352 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlhz6\" (UniqueName: \"kubernetes.io/projected/8002d8c4-5363-4f18-9cc4-8f7b3fce3abe-kube-api-access-vlhz6\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:32:55.787525 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:55.787362 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8002d8c4-5363-4f18-9cc4-8f7b3fce3abe-istio-token\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:32:55.787525 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:55.787370 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8002d8c4-5363-4f18-9cc4-8f7b3fce3abe-istio-podinfo\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:32:55.787525 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:55.787378 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8002d8c4-5363-4f18-9cc4-8f7b3fce3abe-istio-envoy\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:32:55.787525 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:55.787385 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8002d8c4-5363-4f18-9cc4-8f7b3fce3abe-istiod-ca-cert\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:32:55.787525 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:55.787392 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8002d8c4-5363-4f18-9cc4-8f7b3fce3abe-istio-data\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:32:55.962203 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:55.962177 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j"] Jul 23 04:32:55.968282 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:55.968257 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-b88d1b-openshift-default-77599c5cc7-j9b4j"] Jul 23 04:32:56.492355 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:32:56.492311 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8002d8c4-5363-4f18-9cc4-8f7b3fce3abe" path="/var/lib/kubelet/pods/8002d8c4-5363-4f18-9cc4-8f7b3fce3abe/volumes" Jul 23 04:33:10.813242 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:10.813209 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a4e7cf-2w9ld"] Jul 23 04:33:10.818233 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:10.818204 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a4e7cf-2w9ld"] Jul 23 04:33:12.492525 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:12.492494 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1dbfda29-2626-4e02-af37-fbd013cdcdb5" path="/var/lib/kubelet/pods/1dbfda29-2626-4e02-af37-fbd013cdcdb5/volumes" Jul 23 04:33:18.248187 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:18.248154 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt"] Jul 23 04:33:18.248706 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:18.248610 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8002d8c4-5363-4f18-9cc4-8f7b3fce3abe" containerName="istio-proxy" Jul 23 04:33:18.248706 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:18.248628 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8002d8c4-5363-4f18-9cc4-8f7b3fce3abe" containerName="istio-proxy" Jul 23 04:33:18.248706 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:18.248646 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8002d8c4-5363-4f18-9cc4-8f7b3fce3abe" containerName="istio-proxy" Jul 23 04:33:18.248706 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:18.248656 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8002d8c4-5363-4f18-9cc4-8f7b3fce3abe" containerName="istio-proxy" Jul 23 04:33:18.248706 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:18.248678 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dbfda29-2626-4e02-af37-fbd013cdcdb5" containerName="revoke-keys" Jul 23 04:33:18.248706 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:18.248687 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dbfda29-2626-4e02-af37-fbd013cdcdb5" containerName="revoke-keys" Jul 23 04:33:18.249006 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:18.248767 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8002d8c4-5363-4f18-9cc4-8f7b3fce3abe" containerName="istio-proxy" Jul 23 04:33:18.249006 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:18.248782 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dbfda29-2626-4e02-af37-fbd013cdcdb5" containerName="revoke-keys" Jul 23 04:33:18.253990 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:18.253954 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt" Jul 23 04:33:18.257511 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:18.257489 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-5fa859-openshift-default-dockercfg-wmptx\"" Jul 23 04:33:18.262928 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:18.262908 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt"] Jul 23 04:33:18.364827 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:18.364794 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68024583-771a-4ae3-88cb-fd39b8b1de1e-workload-certs\") pod \"e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt\" (UID: \"68024583-771a-4ae3-88cb-fd39b8b1de1e\") " pod="openshift-ingress/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt" Jul 23 04:33:18.364827 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:18.364826 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68024583-771a-4ae3-88cb-fd39b8b1de1e-istio-podinfo\") pod \"e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt\" (UID: \"68024583-771a-4ae3-88cb-fd39b8b1de1e\") " pod="openshift-ingress/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt" Jul 23 04:33:18.365006 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:18.364847 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68024583-771a-4ae3-88cb-fd39b8b1de1e-istiod-ca-cert\") pod \"e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt\" (UID: \"68024583-771a-4ae3-88cb-fd39b8b1de1e\") " pod="openshift-ingress/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt" Jul 23 04:33:18.365006 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:18.364863 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d798\" (UniqueName: \"kubernetes.io/projected/68024583-771a-4ae3-88cb-fd39b8b1de1e-kube-api-access-6d798\") pod \"e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt\" (UID: \"68024583-771a-4ae3-88cb-fd39b8b1de1e\") " pod="openshift-ingress/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt" Jul 23 04:33:18.365006 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:18.364911 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68024583-771a-4ae3-88cb-fd39b8b1de1e-istio-data\") pod \"e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt\" (UID: \"68024583-771a-4ae3-88cb-fd39b8b1de1e\") " pod="openshift-ingress/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt" Jul 23 04:33:18.365006 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:18.364979 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68024583-771a-4ae3-88cb-fd39b8b1de1e-workload-socket\") pod \"e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt\" (UID: \"68024583-771a-4ae3-88cb-fd39b8b1de1e\") " pod="openshift-ingress/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt" Jul 23 04:33:18.365006 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:18.365001 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68024583-771a-4ae3-88cb-fd39b8b1de1e-istio-envoy\") pod \"e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt\" (UID: \"68024583-771a-4ae3-88cb-fd39b8b1de1e\") " pod="openshift-ingress/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt" Jul 23 04:33:18.365162 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:18.365018 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68024583-771a-4ae3-88cb-fd39b8b1de1e-credential-socket\") pod \"e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt\" (UID: \"68024583-771a-4ae3-88cb-fd39b8b1de1e\") " pod="openshift-ingress/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt" Jul 23 04:33:18.365162 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:18.365041 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68024583-771a-4ae3-88cb-fd39b8b1de1e-istio-token\") pod \"e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt\" (UID: \"68024583-771a-4ae3-88cb-fd39b8b1de1e\") " pod="openshift-ingress/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt" Jul 23 04:33:18.466284 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:18.466253 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68024583-771a-4ae3-88cb-fd39b8b1de1e-istio-data\") pod \"e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt\" (UID: \"68024583-771a-4ae3-88cb-fd39b8b1de1e\") " pod="openshift-ingress/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt" Jul 23 04:33:18.466482 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:18.466350 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68024583-771a-4ae3-88cb-fd39b8b1de1e-workload-socket\") pod \"e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt\" (UID: \"68024583-771a-4ae3-88cb-fd39b8b1de1e\") " pod="openshift-ingress/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt" Jul 23 04:33:18.466482 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:18.466373 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68024583-771a-4ae3-88cb-fd39b8b1de1e-istio-envoy\") pod \"e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt\" (UID: \"68024583-771a-4ae3-88cb-fd39b8b1de1e\") " pod="openshift-ingress/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt" Jul 23 04:33:18.466482 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:18.466394 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68024583-771a-4ae3-88cb-fd39b8b1de1e-credential-socket\") pod \"e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt\" (UID: \"68024583-771a-4ae3-88cb-fd39b8b1de1e\") " pod="openshift-ingress/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt" Jul 23 04:33:18.466482 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:18.466419 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68024583-771a-4ae3-88cb-fd39b8b1de1e-istio-token\") pod \"e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt\" (UID: \"68024583-771a-4ae3-88cb-fd39b8b1de1e\") " pod="openshift-ingress/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt" Jul 23 04:33:18.466698 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:18.466522 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68024583-771a-4ae3-88cb-fd39b8b1de1e-workload-certs\") pod \"e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt\" (UID: \"68024583-771a-4ae3-88cb-fd39b8b1de1e\") " pod="openshift-ingress/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt" Jul 23 04:33:18.466698 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:18.466548 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68024583-771a-4ae3-88cb-fd39b8b1de1e-istio-podinfo\") pod \"e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt\" (UID: \"68024583-771a-4ae3-88cb-fd39b8b1de1e\") " pod="openshift-ingress/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt" Jul 23 04:33:18.466698 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:18.466577 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68024583-771a-4ae3-88cb-fd39b8b1de1e-istiod-ca-cert\") pod \"e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt\" (UID: \"68024583-771a-4ae3-88cb-fd39b8b1de1e\") " pod="openshift-ingress/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt" Jul 23 04:33:18.466698 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:18.466601 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d798\" (UniqueName: \"kubernetes.io/projected/68024583-771a-4ae3-88cb-fd39b8b1de1e-kube-api-access-6d798\") pod \"e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt\" (UID: \"68024583-771a-4ae3-88cb-fd39b8b1de1e\") " pod="openshift-ingress/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt" Jul 23 04:33:18.466893 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:18.466734 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68024583-771a-4ae3-88cb-fd39b8b1de1e-istio-data\") pod \"e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt\" (UID: \"68024583-771a-4ae3-88cb-fd39b8b1de1e\") " pod="openshift-ingress/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt" Jul 23 04:33:18.466893 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:18.466779 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68024583-771a-4ae3-88cb-fd39b8b1de1e-workload-socket\") pod \"e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt\" (UID: \"68024583-771a-4ae3-88cb-fd39b8b1de1e\") " pod="openshift-ingress/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt" Jul 23 04:33:18.466893 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:18.466839 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68024583-771a-4ae3-88cb-fd39b8b1de1e-credential-socket\") pod \"e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt\" (UID: \"68024583-771a-4ae3-88cb-fd39b8b1de1e\") " pod="openshift-ingress/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt" Jul 23 04:33:18.467542 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:18.467516 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68024583-771a-4ae3-88cb-fd39b8b1de1e-workload-certs\") pod \"e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt\" (UID: \"68024583-771a-4ae3-88cb-fd39b8b1de1e\") " pod="openshift-ingress/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt" Jul 23 04:33:18.467780 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:18.467757 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68024583-771a-4ae3-88cb-fd39b8b1de1e-istiod-ca-cert\") pod \"e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt\" (UID: \"68024583-771a-4ae3-88cb-fd39b8b1de1e\") " pod="openshift-ingress/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt" Jul 23 04:33:18.469254 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:18.469232 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68024583-771a-4ae3-88cb-fd39b8b1de1e-istio-envoy\") pod \"e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt\" (UID: \"68024583-771a-4ae3-88cb-fd39b8b1de1e\") " pod="openshift-ingress/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt" Jul 23 04:33:18.469968 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:18.469948 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68024583-771a-4ae3-88cb-fd39b8b1de1e-istio-podinfo\") pod \"e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt\" (UID: \"68024583-771a-4ae3-88cb-fd39b8b1de1e\") " pod="openshift-ingress/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt" Jul 23 04:33:18.475694 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:18.475672 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d798\" (UniqueName: \"kubernetes.io/projected/68024583-771a-4ae3-88cb-fd39b8b1de1e-kube-api-access-6d798\") pod \"e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt\" (UID: \"68024583-771a-4ae3-88cb-fd39b8b1de1e\") " pod="openshift-ingress/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt" Jul 23 04:33:18.476850 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:18.476829 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68024583-771a-4ae3-88cb-fd39b8b1de1e-istio-token\") pod \"e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt\" (UID: \"68024583-771a-4ae3-88cb-fd39b8b1de1e\") " pod="openshift-ingress/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt" Jul 23 04:33:18.565732 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:18.565666 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt" Jul 23 04:33:18.703842 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:18.703818 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt"] Jul 23 04:33:18.705739 ip-10-0-140-6 kubenswrapper[2567]: W0723 04:33:18.705712 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68024583_771a_4ae3_88cb_fd39b8b1de1e.slice/crio-26fd4fb9ca30f561142ad3e3c33bf66f40fed2d574323b9ab306e759f229cc80 WatchSource:0}: Error finding container 26fd4fb9ca30f561142ad3e3c33bf66f40fed2d574323b9ab306e759f229cc80: Status 404 returned error can't find the container with id 26fd4fb9ca30f561142ad3e3c33bf66f40fed2d574323b9ab306e759f229cc80 Jul 23 04:33:18.708266 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:18.708235 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 04:33:18.708354 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:18.708300 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 04:33:18.708354 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:18.708328 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 04:33:18.758239 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:18.758215 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt" event={"ID":"68024583-771a-4ae3-88cb-fd39b8b1de1e","Type":"ContainerStarted","Data":"26fd4fb9ca30f561142ad3e3c33bf66f40fed2d574323b9ab306e759f229cc80"} Jul 23 04:33:19.764716 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:19.764677 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt" event={"ID":"68024583-771a-4ae3-88cb-fd39b8b1de1e","Type":"ContainerStarted","Data":"8ef71f9705cb3b02869350f319d20cfbecb53c66a02d295d5fb62f71a37c8a6c"} Jul 23 04:33:19.790409 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:19.790353 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt" podStartSLOduration=1.790337084 podStartE2EDuration="1.790337084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 04:33:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 04:33:19.788337062 +0000 UTC m=+2279.693641375" watchObservedRunningTime="2026-07-23 04:33:19.790337084 +0000 UTC m=+2279.695641376" Jul 23 04:33:20.566017 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:20.565989 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt" Jul 23 04:33:20.570884 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:20.570859 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt" Jul 23 04:33:20.770131 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:20.770094 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt" Jul 23 04:33:20.770934 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:20.770914 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt" Jul 23 04:33:26.900069 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:26.900023 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7"] Jul 23 04:33:26.900621 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:26.900600 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8002d8c4-5363-4f18-9cc4-8f7b3fce3abe" containerName="istio-proxy" Jul 23 04:33:26.903402 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:26.903380 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7" Jul 23 04:33:26.907076 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:26.907044 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-ab39e0-openshift-default-dockercfg-qrxqc\"" Jul 23 04:33:26.944448 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:26.941527 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87b4f602-0ad5-4a97-885a-7a7cf730f500-credential-socket\") pod \"e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7\" (UID: \"87b4f602-0ad5-4a97-885a-7a7cf730f500\") " pod="openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7" Jul 23 04:33:26.944448 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:26.941602 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87b4f602-0ad5-4a97-885a-7a7cf730f500-workload-certs\") pod \"e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7\" (UID: \"87b4f602-0ad5-4a97-885a-7a7cf730f500\") " pod="openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7" Jul 23 04:33:26.944448 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:26.941629 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87b4f602-0ad5-4a97-885a-7a7cf730f500-istio-token\") pod \"e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7\" (UID: \"87b4f602-0ad5-4a97-885a-7a7cf730f500\") " pod="openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7" Jul 23 04:33:26.944448 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:26.941659 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87b4f602-0ad5-4a97-885a-7a7cf730f500-istio-podinfo\") pod \"e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7\" (UID: \"87b4f602-0ad5-4a97-885a-7a7cf730f500\") " pod="openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7" Jul 23 04:33:26.944448 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:26.941698 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87b4f602-0ad5-4a97-885a-7a7cf730f500-istiod-ca-cert\") pod \"e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7\" (UID: \"87b4f602-0ad5-4a97-885a-7a7cf730f500\") " pod="openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7" Jul 23 04:33:26.944448 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:26.941725 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nw7m\" (UniqueName: \"kubernetes.io/projected/87b4f602-0ad5-4a97-885a-7a7cf730f500-kube-api-access-4nw7m\") pod \"e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7\" (UID: \"87b4f602-0ad5-4a97-885a-7a7cf730f500\") " pod="openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7" Jul 23 04:33:26.944448 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:26.941781 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87b4f602-0ad5-4a97-885a-7a7cf730f500-istio-envoy\") pod \"e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7\" (UID: \"87b4f602-0ad5-4a97-885a-7a7cf730f500\") " pod="openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7" Jul 23 04:33:26.944448 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:26.941840 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87b4f602-0ad5-4a97-885a-7a7cf730f500-workload-socket\") pod \"e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7\" (UID: \"87b4f602-0ad5-4a97-885a-7a7cf730f500\") " pod="openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7" Jul 23 04:33:26.944448 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:26.941884 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87b4f602-0ad5-4a97-885a-7a7cf730f500-istio-data\") pod \"e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7\" (UID: \"87b4f602-0ad5-4a97-885a-7a7cf730f500\") " pod="openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7" Jul 23 04:33:27.000093 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:27.000052 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7"] Jul 23 04:33:27.053892 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:27.053841 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87b4f602-0ad5-4a97-885a-7a7cf730f500-istio-podinfo\") pod \"e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7\" (UID: \"87b4f602-0ad5-4a97-885a-7a7cf730f500\") " pod="openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7" Jul 23 04:33:27.054099 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:27.053901 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87b4f602-0ad5-4a97-885a-7a7cf730f500-istiod-ca-cert\") pod \"e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7\" (UID: \"87b4f602-0ad5-4a97-885a-7a7cf730f500\") " pod="openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7" Jul 23 04:33:27.054099 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:27.053936 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nw7m\" (UniqueName: \"kubernetes.io/projected/87b4f602-0ad5-4a97-885a-7a7cf730f500-kube-api-access-4nw7m\") pod \"e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7\" (UID: \"87b4f602-0ad5-4a97-885a-7a7cf730f500\") " pod="openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7" Jul 23 04:33:27.054099 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:27.053972 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87b4f602-0ad5-4a97-885a-7a7cf730f500-istio-envoy\") pod \"e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7\" (UID: \"87b4f602-0ad5-4a97-885a-7a7cf730f500\") " pod="openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7" Jul 23 04:33:27.054099 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:27.054013 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87b4f602-0ad5-4a97-885a-7a7cf730f500-workload-socket\") pod \"e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7\" (UID: \"87b4f602-0ad5-4a97-885a-7a7cf730f500\") " pod="openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7" Jul 23 04:33:27.054099 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:27.054049 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87b4f602-0ad5-4a97-885a-7a7cf730f500-istio-data\") pod \"e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7\" (UID: \"87b4f602-0ad5-4a97-885a-7a7cf730f500\") " pod="openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7" Jul 23 04:33:27.054395 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:27.054139 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87b4f602-0ad5-4a97-885a-7a7cf730f500-credential-socket\") pod \"e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7\" (UID: \"87b4f602-0ad5-4a97-885a-7a7cf730f500\") " pod="openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7" Jul 23 04:33:27.054395 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:27.054189 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87b4f602-0ad5-4a97-885a-7a7cf730f500-workload-certs\") pod \"e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7\" (UID: \"87b4f602-0ad5-4a97-885a-7a7cf730f500\") " pod="openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7" Jul 23 04:33:27.054395 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:27.054213 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87b4f602-0ad5-4a97-885a-7a7cf730f500-istio-token\") pod \"e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7\" (UID: \"87b4f602-0ad5-4a97-885a-7a7cf730f500\") " pod="openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7" Jul 23 04:33:27.055346 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:27.055314 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87b4f602-0ad5-4a97-885a-7a7cf730f500-workload-socket\") pod \"e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7\" (UID: \"87b4f602-0ad5-4a97-885a-7a7cf730f500\") " pod="openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7" Jul 23 04:33:27.055998 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:27.055970 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87b4f602-0ad5-4a97-885a-7a7cf730f500-istiod-ca-cert\") pod \"e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7\" (UID: \"87b4f602-0ad5-4a97-885a-7a7cf730f500\") " pod="openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7" Jul 23 04:33:27.056247 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:27.056224 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87b4f602-0ad5-4a97-885a-7a7cf730f500-credential-socket\") pod \"e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7\" (UID: \"87b4f602-0ad5-4a97-885a-7a7cf730f500\") " pod="openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7" Jul 23 04:33:27.056731 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:27.056711 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87b4f602-0ad5-4a97-885a-7a7cf730f500-istio-data\") pod \"e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7\" (UID: \"87b4f602-0ad5-4a97-885a-7a7cf730f500\") " pod="openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7" Jul 23 04:33:27.056965 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:27.056916 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87b4f602-0ad5-4a97-885a-7a7cf730f500-workload-certs\") pod \"e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7\" (UID: \"87b4f602-0ad5-4a97-885a-7a7cf730f500\") " pod="openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7" Jul 23 04:33:27.064962 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:27.064934 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87b4f602-0ad5-4a97-885a-7a7cf730f500-istio-podinfo\") pod \"e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7\" (UID: \"87b4f602-0ad5-4a97-885a-7a7cf730f500\") " pod="openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7" Jul 23 04:33:27.068027 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:27.067998 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87b4f602-0ad5-4a97-885a-7a7cf730f500-istio-envoy\") pod \"e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7\" (UID: \"87b4f602-0ad5-4a97-885a-7a7cf730f500\") " pod="openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7" Jul 23 04:33:27.073597 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:27.073565 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nw7m\" (UniqueName: \"kubernetes.io/projected/87b4f602-0ad5-4a97-885a-7a7cf730f500-kube-api-access-4nw7m\") pod \"e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7\" (UID: \"87b4f602-0ad5-4a97-885a-7a7cf730f500\") " pod="openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7" Jul 23 04:33:27.074365 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:27.074344 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87b4f602-0ad5-4a97-885a-7a7cf730f500-istio-token\") pod \"e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7\" (UID: \"87b4f602-0ad5-4a97-885a-7a7cf730f500\") " pod="openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7" Jul 23 04:33:27.213871 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:27.213826 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7" Jul 23 04:33:27.394476 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:27.394447 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7"] Jul 23 04:33:27.396049 ip-10-0-140-6 kubenswrapper[2567]: W0723 04:33:27.396025 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87b4f602_0ad5_4a97_885a_7a7cf730f500.slice/crio-9c8802e9f9bf7bb74712f3a6809c8932a80ce72a5750aa7c7ec557779ac799ce WatchSource:0}: Error finding container 9c8802e9f9bf7bb74712f3a6809c8932a80ce72a5750aa7c7ec557779ac799ce: Status 404 returned error can't find the container with id 9c8802e9f9bf7bb74712f3a6809c8932a80ce72a5750aa7c7ec557779ac799ce Jul 23 04:33:27.398244 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:27.398213 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 04:33:27.398310 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:27.398291 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 04:33:27.398347 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:27.398321 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 04:33:27.815723 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:27.815627 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7" event={"ID":"87b4f602-0ad5-4a97-885a-7a7cf730f500","Type":"ContainerStarted","Data":"1d9483ddf0628fffdabaaccd7eafe82ff709632a501a38e3e16516dc201f6a16"} Jul 23 04:33:27.815723 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:27.815707 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7" event={"ID":"87b4f602-0ad5-4a97-885a-7a7cf730f500","Type":"ContainerStarted","Data":"9c8802e9f9bf7bb74712f3a6809c8932a80ce72a5750aa7c7ec557779ac799ce"} Jul 23 04:33:27.854451 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:27.854312 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7" podStartSLOduration=1.854292534 podStartE2EDuration="1.854292534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 04:33:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 04:33:27.853675021 +0000 UTC m=+2287.758979344" watchObservedRunningTime="2026-07-23 04:33:27.854292534 +0000 UTC m=+2287.759596830" Jul 23 04:33:28.214143 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:28.214110 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7" Jul 23 04:33:28.219109 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:28.219080 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7" Jul 23 04:33:28.820810 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:28.820778 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7" Jul 23 04:33:28.821990 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:28.821970 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7" Jul 23 04:33:33.850038 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:33.850007 2567 generic.go:356] "Generic (PLEG): container finished" podID="68024583-771a-4ae3-88cb-fd39b8b1de1e" containerID="8ef71f9705cb3b02869350f319d20cfbecb53c66a02d295d5fb62f71a37c8a6c" exitCode=0 Jul 23 04:33:33.850407 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:33.850080 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt" event={"ID":"68024583-771a-4ae3-88cb-fd39b8b1de1e","Type":"ContainerDied","Data":"8ef71f9705cb3b02869350f319d20cfbecb53c66a02d295d5fb62f71a37c8a6c"} Jul 23 04:33:33.850476 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:33.850459 2567 scope.go:117] "RemoveContainer" containerID="8ef71f9705cb3b02869350f319d20cfbecb53c66a02d295d5fb62f71a37c8a6c" Jul 23 04:33:33.851448 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:33.851396 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 04:33:33.851534 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:33.851519 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 04:33:33.851579 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:33.851556 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 04:33:34.856802 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:34.856772 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt" event={"ID":"68024583-771a-4ae3-88cb-fd39b8b1de1e","Type":"ContainerStarted","Data":"47378560a0e016cd1a85cc2134cc7a2fcdd223c9a3a9baf6b5c4ba835a21fb79"} Jul 23 04:33:35.566242 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:35.566198 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt" Jul 23 04:33:35.567669 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:35.567644 2567 patch_prober.go:28] interesting pod/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Jul 23 04:33:35.567793 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:35.567694 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt" podUID="68024583-771a-4ae3-88cb-fd39b8b1de1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Jul 23 04:33:35.770324 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:35.770293 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt" Jul 23 04:33:36.566121 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:36.566086 2567 patch_prober.go:28] interesting pod/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Jul 23 04:33:36.566556 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:36.566145 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt" podUID="68024583-771a-4ae3-88cb-fd39b8b1de1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Jul 23 04:33:37.566335 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:37.566301 2567 patch_prober.go:28] interesting pod/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Jul 23 04:33:37.566770 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:37.566356 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt" podUID="68024583-771a-4ae3-88cb-fd39b8b1de1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Jul 23 04:33:38.566497 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:38.566454 2567 patch_prober.go:28] interesting pod/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Jul 23 04:33:38.567112 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:38.567083 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt" podUID="68024583-771a-4ae3-88cb-fd39b8b1de1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Jul 23 04:33:39.570324 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:39.570285 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt" Jul 23 04:33:39.571382 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:39.571365 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt" Jul 23 04:33:44.724498 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:44.723644 2567 patch_prober.go:28] interesting pod/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": read tcp 10.133.0.2:37392->10.133.0.62:15021: read: connection reset by peer" start-of-body= Jul 23 04:33:44.724498 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:44.723752 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7" podUID="87b4f602-0ad5-4a97-885a-7a7cf730f500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": read tcp 10.133.0.2:37392->10.133.0.62:15021: read: connection reset by peer" Jul 23 04:33:44.927805 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:44.927774 2567 generic.go:356] "Generic (PLEG): container finished" podID="87b4f602-0ad5-4a97-885a-7a7cf730f500" containerID="1d9483ddf0628fffdabaaccd7eafe82ff709632a501a38e3e16516dc201f6a16" exitCode=0 Jul 23 04:33:44.927986 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:44.927848 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7" event={"ID":"87b4f602-0ad5-4a97-885a-7a7cf730f500","Type":"ContainerDied","Data":"1d9483ddf0628fffdabaaccd7eafe82ff709632a501a38e3e16516dc201f6a16"} Jul 23 04:33:44.928260 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:44.928244 2567 scope.go:117] "RemoveContainer" containerID="1d9483ddf0628fffdabaaccd7eafe82ff709632a501a38e3e16516dc201f6a16" Jul 23 04:33:44.929232 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:44.929203 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 04:33:44.929294 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:44.929275 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 04:33:44.929333 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:44.929306 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 04:33:45.935294 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:45.935258 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7" event={"ID":"87b4f602-0ad5-4a97-885a-7a7cf730f500","Type":"ContainerStarted","Data":"7d9abdba97191e3d8c1dab57797cc7a3780fd4837bb806d15ef6dfccd049e5e8"} Jul 23 04:33:46.214259 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:46.214163 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7" Jul 23 04:33:46.214463 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:46.214375 2567 patch_prober.go:28] interesting pod/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 23 04:33:46.214543 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:46.214455 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7" podUID="87b4f602-0ad5-4a97-885a-7a7cf730f500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 23 04:33:47.214680 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:47.214644 2567 patch_prober.go:28] interesting pod/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 23 04:33:47.215089 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:47.214710 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7" podUID="87b4f602-0ad5-4a97-885a-7a7cf730f500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 23 04:33:48.214306 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:48.214268 2567 patch_prober.go:28] interesting pod/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 23 04:33:48.214514 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:48.214321 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7" podUID="87b4f602-0ad5-4a97-885a-7a7cf730f500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 23 04:33:49.215127 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:49.215090 2567 patch_prober.go:28] interesting pod/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 23 04:33:49.215577 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:49.215148 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7" podUID="87b4f602-0ad5-4a97-885a-7a7cf730f500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 23 04:33:50.217634 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:50.217605 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7" Jul 23 04:33:50.218002 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:50.217872 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7" Jul 23 04:33:50.218487 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:33:50.218470 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7" Jul 23 04:34:20.803892 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:34:20.803860 2567 scope.go:117] "RemoveContainer" containerID="81169aecb77be48094e1f23de7e581da0ad6d6c0523f089fe717d78190c10392" Jul 23 04:35:20.647641 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:35:20.647613 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wzgn9_dbd53593-e5e7-473a-b71c-8fbf9272913f/console-operator/2.log" Jul 23 04:35:20.654015 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:35:20.653994 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wzgn9_dbd53593-e5e7-473a-b71c-8fbf9272913f/console-operator/2.log" Jul 23 04:36:20.831586 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:36:20.831554 2567 scope.go:117] "RemoveContainer" containerID="dc41d75ae637bb7e89cd45e5b5f043008ce6bc58847acc4c3c39ff107fc19472" Jul 23 04:36:20.840569 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:36:20.840540 2567 scope.go:117] "RemoveContainer" containerID="8b34b9a8818d9904dc8ca8a5b669c80862b376f0a151a9b81de60cb956da6e6b" Jul 23 04:37:20.865198 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:37:20.865167 2567 scope.go:117] "RemoveContainer" containerID="c5a87e162ce0d734c842ad3dff116a37f3d1dd6bede2828899ca87e7f02550df" Jul 23 04:38:20.885979 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:38:20.885947 2567 scope.go:117] "RemoveContainer" containerID="f5347a23f8cc2bb5b2179d9fdd8b76f9b6419c33d8fdcb70e024cc9abee9288e" Jul 23 04:39:20.913509 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:20.913479 2567 scope.go:117] "RemoveContainer" containerID="c3d2d407361f39984c29b842fb1f84fd417c771bbb4f61d7da2a152461792d77" Jul 23 04:39:20.922120 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:20.922098 2567 scope.go:117] "RemoveContainer" containerID="c173875454e64c253d05c420693bdfe18a055b1ebfe506b569b119bca66ced30" Jul 23 04:39:35.294218 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:35.294179 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7642-140658098817584"] Jul 23 04:39:35.297473 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:35.297453 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7642-140658098817584" Jul 23 04:39:35.300411 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:35.300389 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-jh85x\"" Jul 23 04:39:35.305258 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:35.304938 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7642-140658098817584"] Jul 23 04:39:35.373560 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:35.373534 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h769s\" (UniqueName: \"kubernetes.io/projected/2378741d-6485-47f7-ae8f-68f22611be59-kube-api-access-h769s\") pod \"test-curl-7642-140658098817584\" (UID: \"2378741d-6485-47f7-ae8f-68f22611be59\") " pod="opendatahub/test-curl-7642-140658098817584" Jul 23 04:39:35.474691 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:35.474664 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h769s\" (UniqueName: \"kubernetes.io/projected/2378741d-6485-47f7-ae8f-68f22611be59-kube-api-access-h769s\") pod \"test-curl-7642-140658098817584\" (UID: \"2378741d-6485-47f7-ae8f-68f22611be59\") " pod="opendatahub/test-curl-7642-140658098817584" Jul 23 04:39:35.484372 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:35.484323 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h769s\" (UniqueName: \"kubernetes.io/projected/2378741d-6485-47f7-ae8f-68f22611be59-kube-api-access-h769s\") pod \"test-curl-7642-140658098817584\" (UID: \"2378741d-6485-47f7-ae8f-68f22611be59\") " pod="opendatahub/test-curl-7642-140658098817584" Jul 23 04:39:35.608231 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:35.608141 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7642-140658098817584" Jul 23 04:39:35.737682 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:35.737654 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7642-140658098817584"] Jul 23 04:39:35.739319 ip-10-0-140-6 kubenswrapper[2567]: W0723 04:39:35.739270 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2378741d_6485_47f7_ae8f_68f22611be59.slice/crio-df759495bbdfe5d2891e74a426d5dcfed901f969d0a4232bd24695ea83752e04 WatchSource:0}: Error finding container df759495bbdfe5d2891e74a426d5dcfed901f969d0a4232bd24695ea83752e04: Status 404 returned error can't find the container with id df759495bbdfe5d2891e74a426d5dcfed901f969d0a4232bd24695ea83752e04 Jul 23 04:39:35.740743 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:35.740720 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 04:39:35.777117 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:35.777094 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7642-140658098817584" event={"ID":"2378741d-6485-47f7-ae8f-68f22611be59","Type":"ContainerStarted","Data":"df759495bbdfe5d2891e74a426d5dcfed901f969d0a4232bd24695ea83752e04"} Jul 23 04:39:36.784164 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:36.784080 2567 generic.go:356] "Generic (PLEG): container finished" podID="2378741d-6485-47f7-ae8f-68f22611be59" containerID="0d73f8cdf33881baef20dff26a69baadb1a34bcecc12bfe553bb5672fc7f7820" exitCode=0 Jul 23 04:39:36.784164 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:36.784133 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7642-140658098817584" event={"ID":"2378741d-6485-47f7-ae8f-68f22611be59","Type":"ContainerDied","Data":"0d73f8cdf33881baef20dff26a69baadb1a34bcecc12bfe553bb5672fc7f7820"} Jul 23 04:39:37.916601 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:37.916567 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7642-140658098817584" Jul 23 04:39:37.955101 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:37.955074 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7642-140658098817584_2378741d-6485-47f7-ae8f-68f22611be59/test-curl-7642-140658098817584/0.log" Jul 23 04:39:37.978936 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:37.978907 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7642-140658098817584"] Jul 23 04:39:37.984809 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:37.984787 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7642-140658098817584"] Jul 23 04:39:37.994776 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:37.994760 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h769s\" (UniqueName: \"kubernetes.io/projected/2378741d-6485-47f7-ae8f-68f22611be59-kube-api-access-h769s\") pod \"2378741d-6485-47f7-ae8f-68f22611be59\" (UID: \"2378741d-6485-47f7-ae8f-68f22611be59\") " Jul 23 04:39:37.996998 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:37.996977 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2378741d-6485-47f7-ae8f-68f22611be59-kube-api-access-h769s" (OuterVolumeSpecName: "kube-api-access-h769s") pod "2378741d-6485-47f7-ae8f-68f22611be59" (UID: "2378741d-6485-47f7-ae8f-68f22611be59"). InnerVolumeSpecName "kube-api-access-h769s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 04:39:38.095758 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:38.095692 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h769s\" (UniqueName: \"kubernetes.io/projected/2378741d-6485-47f7-ae8f-68f22611be59-kube-api-access-h769s\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:39:38.106910 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:38.106885 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7642-140658098791344"] Jul 23 04:39:38.107196 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:38.107183 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2378741d-6485-47f7-ae8f-68f22611be59" containerName="test-curl-7642-140658098817584" Jul 23 04:39:38.107238 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:38.107199 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="2378741d-6485-47f7-ae8f-68f22611be59" containerName="test-curl-7642-140658098817584" Jul 23 04:39:38.107273 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:38.107258 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="2378741d-6485-47f7-ae8f-68f22611be59" containerName="test-curl-7642-140658098817584" Jul 23 04:39:38.110008 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:38.109993 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7642-140658098791344" Jul 23 04:39:38.124442 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:38.119373 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7642-140658098791344"] Jul 23 04:39:38.196366 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:38.196337 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptp9d\" (UniqueName: \"kubernetes.io/projected/27ba320a-e8c0-4771-a2a4-4343ffb8b855-kube-api-access-ptp9d\") pod \"test-curl-7642-140658098791344\" (UID: \"27ba320a-e8c0-4771-a2a4-4343ffb8b855\") " pod="opendatahub/test-curl-7642-140658098791344" Jul 23 04:39:38.297557 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:38.297521 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptp9d\" (UniqueName: \"kubernetes.io/projected/27ba320a-e8c0-4771-a2a4-4343ffb8b855-kube-api-access-ptp9d\") pod \"test-curl-7642-140658098791344\" (UID: \"27ba320a-e8c0-4771-a2a4-4343ffb8b855\") " pod="opendatahub/test-curl-7642-140658098791344" Jul 23 04:39:38.306895 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:38.306869 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptp9d\" (UniqueName: \"kubernetes.io/projected/27ba320a-e8c0-4771-a2a4-4343ffb8b855-kube-api-access-ptp9d\") pod \"test-curl-7642-140658098791344\" (UID: \"27ba320a-e8c0-4771-a2a4-4343ffb8b855\") " pod="opendatahub/test-curl-7642-140658098791344" Jul 23 04:39:38.420722 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:38.420694 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7642-140658098791344" Jul 23 04:39:38.493414 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:38.493387 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2378741d-6485-47f7-ae8f-68f22611be59" path="/var/lib/kubelet/pods/2378741d-6485-47f7-ae8f-68f22611be59/volumes" Jul 23 04:39:38.553127 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:38.553100 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7642-140658098791344"] Jul 23 04:39:38.554797 ip-10-0-140-6 kubenswrapper[2567]: W0723 04:39:38.554770 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27ba320a_e8c0_4771_a2a4_4343ffb8b855.slice/crio-903ddb09a52a9a54ca0ee4cee009a4621ac3a3ab1737e3a1ee8f1287b9b0e337 WatchSource:0}: Error finding container 903ddb09a52a9a54ca0ee4cee009a4621ac3a3ab1737e3a1ee8f1287b9b0e337: Status 404 returned error can't find the container with id 903ddb09a52a9a54ca0ee4cee009a4621ac3a3ab1737e3a1ee8f1287b9b0e337 Jul 23 04:39:38.796092 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:38.796012 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7642-140658098791344" event={"ID":"27ba320a-e8c0-4771-a2a4-4343ffb8b855","Type":"ContainerStarted","Data":"903ddb09a52a9a54ca0ee4cee009a4621ac3a3ab1737e3a1ee8f1287b9b0e337"} Jul 23 04:39:38.797169 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:38.797144 2567 scope.go:117] "RemoveContainer" containerID="0d73f8cdf33881baef20dff26a69baadb1a34bcecc12bfe553bb5672fc7f7820" Jul 23 04:39:38.797169 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:38.797159 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7642-140658098817584" Jul 23 04:39:39.805293 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:39.805259 2567 generic.go:356] "Generic (PLEG): container finished" podID="27ba320a-e8c0-4771-a2a4-4343ffb8b855" containerID="53182af910678bc8da67c7d003d8766e4ddffa38e2e850819fea4f5345bab814" exitCode=0 Jul 23 04:39:39.805803 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:39.805352 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7642-140658098791344" event={"ID":"27ba320a-e8c0-4771-a2a4-4343ffb8b855","Type":"ContainerDied","Data":"53182af910678bc8da67c7d003d8766e4ddffa38e2e850819fea4f5345bab814"} Jul 23 04:39:40.943890 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:40.943861 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7642-140658098791344" Jul 23 04:39:40.966704 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:40.966686 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7642-140658098791344_27ba320a-e8c0-4771-a2a4-4343ffb8b855/test-curl-7642-140658098791344/0.log" Jul 23 04:39:40.996834 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:40.996811 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7642-140658098791344"] Jul 23 04:39:41.001012 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:41.000993 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7642-140658098791344"] Jul 23 04:39:41.018769 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:41.018748 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptp9d\" (UniqueName: \"kubernetes.io/projected/27ba320a-e8c0-4771-a2a4-4343ffb8b855-kube-api-access-ptp9d\") pod \"27ba320a-e8c0-4771-a2a4-4343ffb8b855\" (UID: \"27ba320a-e8c0-4771-a2a4-4343ffb8b855\") " Jul 23 04:39:41.020848 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:41.020820 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27ba320a-e8c0-4771-a2a4-4343ffb8b855-kube-api-access-ptp9d" (OuterVolumeSpecName: "kube-api-access-ptp9d") pod "27ba320a-e8c0-4771-a2a4-4343ffb8b855" (UID: "27ba320a-e8c0-4771-a2a4-4343ffb8b855"). InnerVolumeSpecName "kube-api-access-ptp9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 04:39:41.119361 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:41.119288 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptp9d\" (UniqueName: \"kubernetes.io/projected/27ba320a-e8c0-4771-a2a4-4343ffb8b855-kube-api-access-ptp9d\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:39:41.165548 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:41.165519 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7642-140658098841040"] Jul 23 04:39:41.166138 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:41.166117 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27ba320a-e8c0-4771-a2a4-4343ffb8b855" containerName="test-curl-7642-140658098791344" Jul 23 04:39:41.166245 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:41.166200 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="27ba320a-e8c0-4771-a2a4-4343ffb8b855" containerName="test-curl-7642-140658098791344" Jul 23 04:39:41.166387 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:41.166372 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="27ba320a-e8c0-4771-a2a4-4343ffb8b855" containerName="test-curl-7642-140658098791344" Jul 23 04:39:41.169310 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:41.169289 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7642-140658098841040" Jul 23 04:39:41.178447 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:41.178406 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7642-140658098841040"] Jul 23 04:39:41.320768 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:41.320734 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8x44\" (UniqueName: \"kubernetes.io/projected/fcf13561-c04d-42c3-815b-11966f6ff94e-kube-api-access-h8x44\") pod \"test-curl-7642-140658098841040\" (UID: \"fcf13561-c04d-42c3-815b-11966f6ff94e\") " pod="opendatahub/test-curl-7642-140658098841040" Jul 23 04:39:41.421138 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:41.421111 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8x44\" (UniqueName: \"kubernetes.io/projected/fcf13561-c04d-42c3-815b-11966f6ff94e-kube-api-access-h8x44\") pod \"test-curl-7642-140658098841040\" (UID: \"fcf13561-c04d-42c3-815b-11966f6ff94e\") " pod="opendatahub/test-curl-7642-140658098841040" Jul 23 04:39:41.429924 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:41.429897 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8x44\" (UniqueName: \"kubernetes.io/projected/fcf13561-c04d-42c3-815b-11966f6ff94e-kube-api-access-h8x44\") pod \"test-curl-7642-140658098841040\" (UID: \"fcf13561-c04d-42c3-815b-11966f6ff94e\") " pod="opendatahub/test-curl-7642-140658098841040" Jul 23 04:39:41.480515 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:41.480487 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7642-140658098841040" Jul 23 04:39:41.611135 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:41.611112 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7642-140658098841040"] Jul 23 04:39:41.612407 ip-10-0-140-6 kubenswrapper[2567]: W0723 04:39:41.612384 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfcf13561_c04d_42c3_815b_11966f6ff94e.slice/crio-b82cf44bf38a7add729678acdb9c466a5b1f79bfa23519ce9ccd7c2f64e38639 WatchSource:0}: Error finding container b82cf44bf38a7add729678acdb9c466a5b1f79bfa23519ce9ccd7c2f64e38639: Status 404 returned error can't find the container with id b82cf44bf38a7add729678acdb9c466a5b1f79bfa23519ce9ccd7c2f64e38639 Jul 23 04:39:41.817015 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:41.816934 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7642-140658098791344" Jul 23 04:39:41.817166 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:41.816936 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="903ddb09a52a9a54ca0ee4cee009a4621ac3a3ab1737e3a1ee8f1287b9b0e337" Jul 23 04:39:41.818258 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:41.818222 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7642-140658098841040" event={"ID":"fcf13561-c04d-42c3-815b-11966f6ff94e","Type":"ContainerStarted","Data":"b82cf44bf38a7add729678acdb9c466a5b1f79bfa23519ce9ccd7c2f64e38639"} Jul 23 04:39:41.827505 ip-10-0-140-6 kubenswrapper[2567]: E0723 04:39:41.827479 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7642-140658098791344\" is forbidden: User \"system:node:ip-10-0-140-6.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-6.ec2.internal' and this object" podUID="27ba320a-e8c0-4771-a2a4-4343ffb8b855" pod="opendatahub/test-curl-7642-140658098791344" Jul 23 04:39:42.492234 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:42.492203 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27ba320a-e8c0-4771-a2a4-4343ffb8b855" path="/var/lib/kubelet/pods/27ba320a-e8c0-4771-a2a4-4343ffb8b855/volumes" Jul 23 04:39:42.824986 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:42.824890 2567 generic.go:356] "Generic (PLEG): container finished" podID="fcf13561-c04d-42c3-815b-11966f6ff94e" containerID="64e1b38b7f83dc64eeac740ef18e44bd4c0bc76d7effaa0cc48f4c0580f85816" exitCode=0 Jul 23 04:39:42.825129 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:42.824979 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7642-140658098841040" event={"ID":"fcf13561-c04d-42c3-815b-11966f6ff94e","Type":"ContainerDied","Data":"64e1b38b7f83dc64eeac740ef18e44bd4c0bc76d7effaa0cc48f4c0580f85816"} Jul 23 04:39:43.963927 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:43.963907 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7642-140658098841040" Jul 23 04:39:43.985697 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:43.985674 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7642-140658098841040_fcf13561-c04d-42c3-815b-11966f6ff94e/test-curl-7642-140658098841040/0.log" Jul 23 04:39:44.010190 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:44.010158 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7642-140658098841040"] Jul 23 04:39:44.013462 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:44.013439 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7642-140658098841040"] Jul 23 04:39:44.041361 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:44.041331 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8x44\" (UniqueName: \"kubernetes.io/projected/fcf13561-c04d-42c3-815b-11966f6ff94e-kube-api-access-h8x44\") pod \"fcf13561-c04d-42c3-815b-11966f6ff94e\" (UID: \"fcf13561-c04d-42c3-815b-11966f6ff94e\") " Jul 23 04:39:44.043593 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:44.043567 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcf13561-c04d-42c3-815b-11966f6ff94e-kube-api-access-h8x44" (OuterVolumeSpecName: "kube-api-access-h8x44") pod "fcf13561-c04d-42c3-815b-11966f6ff94e" (UID: "fcf13561-c04d-42c3-815b-11966f6ff94e"). InnerVolumeSpecName "kube-api-access-h8x44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 04:39:44.141331 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:44.141259 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7642-140658098841040"] Jul 23 04:39:44.141587 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:44.141573 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcf13561-c04d-42c3-815b-11966f6ff94e" containerName="test-curl-7642-140658098841040" Jul 23 04:39:44.141587 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:44.141586 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcf13561-c04d-42c3-815b-11966f6ff94e" containerName="test-curl-7642-140658098841040" Jul 23 04:39:44.141708 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:44.141657 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcf13561-c04d-42c3-815b-11966f6ff94e" containerName="test-curl-7642-140658098841040" Jul 23 04:39:44.142532 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:44.142509 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8x44\" (UniqueName: \"kubernetes.io/projected/fcf13561-c04d-42c3-815b-11966f6ff94e-kube-api-access-h8x44\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:39:44.144773 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:44.144755 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7642-140658098841040" Jul 23 04:39:44.152412 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:44.152387 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7642-140658098841040"] Jul 23 04:39:44.243146 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:44.243108 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnj7c\" (UniqueName: \"kubernetes.io/projected/e7b477ad-cc12-4b02-a80f-c885f3b25b8e-kube-api-access-rnj7c\") pod \"test-curl-7642-140658098841040\" (UID: \"e7b477ad-cc12-4b02-a80f-c885f3b25b8e\") " pod="opendatahub/test-curl-7642-140658098841040" Jul 23 04:39:44.343991 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:44.343961 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnj7c\" (UniqueName: \"kubernetes.io/projected/e7b477ad-cc12-4b02-a80f-c885f3b25b8e-kube-api-access-rnj7c\") pod \"test-curl-7642-140658098841040\" (UID: \"e7b477ad-cc12-4b02-a80f-c885f3b25b8e\") " pod="opendatahub/test-curl-7642-140658098841040" Jul 23 04:39:44.357212 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:44.357181 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnj7c\" (UniqueName: \"kubernetes.io/projected/e7b477ad-cc12-4b02-a80f-c885f3b25b8e-kube-api-access-rnj7c\") pod \"test-curl-7642-140658098841040\" (UID: \"e7b477ad-cc12-4b02-a80f-c885f3b25b8e\") " pod="opendatahub/test-curl-7642-140658098841040" Jul 23 04:39:44.455083 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:44.455059 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7642-140658098841040" Jul 23 04:39:44.493340 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:44.493314 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcf13561-c04d-42c3-815b-11966f6ff94e" path="/var/lib/kubelet/pods/fcf13561-c04d-42c3-815b-11966f6ff94e/volumes" Jul 23 04:39:44.602733 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:44.602706 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7642-140658098841040"] Jul 23 04:39:44.605008 ip-10-0-140-6 kubenswrapper[2567]: W0723 04:39:44.604980 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7b477ad_cc12_4b02_a80f_c885f3b25b8e.slice/crio-fb1e447147d8817528cb652cefec482201c453a8937a7a1a9bf48bc6ec7e9059 WatchSource:0}: Error finding container fb1e447147d8817528cb652cefec482201c453a8937a7a1a9bf48bc6ec7e9059: Status 404 returned error can't find the container with id fb1e447147d8817528cb652cefec482201c453a8937a7a1a9bf48bc6ec7e9059 Jul 23 04:39:44.841610 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:44.841522 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7642-140658098841040" Jul 23 04:39:44.841610 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:44.841526 2567 scope.go:117] "RemoveContainer" containerID="64e1b38b7f83dc64eeac740ef18e44bd4c0bc76d7effaa0cc48f4c0580f85816" Jul 23 04:39:44.842897 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:44.842813 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7642-140658098841040" event={"ID":"e7b477ad-cc12-4b02-a80f-c885f3b25b8e","Type":"ContainerStarted","Data":"fb1e447147d8817528cb652cefec482201c453a8937a7a1a9bf48bc6ec7e9059"} Jul 23 04:39:45.850914 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:45.850875 2567 generic.go:356] "Generic (PLEG): container finished" podID="e7b477ad-cc12-4b02-a80f-c885f3b25b8e" containerID="4794e9c5fe2f212c89baddca7fac0cecfc62d1750e62bd2273c9a0ca4a5fd30c" exitCode=0 Jul 23 04:39:45.851291 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:45.850933 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7642-140658098841040" event={"ID":"e7b477ad-cc12-4b02-a80f-c885f3b25b8e","Type":"ContainerDied","Data":"4794e9c5fe2f212c89baddca7fac0cecfc62d1750e62bd2273c9a0ca4a5fd30c"} Jul 23 04:39:46.988723 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:46.988697 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7642-140658098841040" Jul 23 04:39:47.015150 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:47.015131 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7642-140658098841040_e7b477ad-cc12-4b02-a80f-c885f3b25b8e/test-curl-7642-140658098841040/0.log" Jul 23 04:39:47.042640 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:47.042613 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7642-140658098841040"] Jul 23 04:39:47.052317 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:47.052296 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7642-140658098841040"] Jul 23 04:39:47.067397 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:47.067377 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rnj7c\" (UniqueName: \"kubernetes.io/projected/e7b477ad-cc12-4b02-a80f-c885f3b25b8e-kube-api-access-rnj7c\") pod \"e7b477ad-cc12-4b02-a80f-c885f3b25b8e\" (UID: \"e7b477ad-cc12-4b02-a80f-c885f3b25b8e\") " Jul 23 04:39:47.069495 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:47.069470 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7b477ad-cc12-4b02-a80f-c885f3b25b8e-kube-api-access-rnj7c" (OuterVolumeSpecName: "kube-api-access-rnj7c") pod "e7b477ad-cc12-4b02-a80f-c885f3b25b8e" (UID: "e7b477ad-cc12-4b02-a80f-c885f3b25b8e"). InnerVolumeSpecName "kube-api-access-rnj7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 04:39:47.168771 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:47.168750 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnj7c\" (UniqueName: \"kubernetes.io/projected/e7b477ad-cc12-4b02-a80f-c885f3b25b8e-kube-api-access-rnj7c\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:39:47.183500 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:47.183478 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7642-140658098842048"] Jul 23 04:39:47.183805 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:47.183793 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7b477ad-cc12-4b02-a80f-c885f3b25b8e" containerName="test-curl-7642-140658098841040" Jul 23 04:39:47.183854 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:47.183808 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7b477ad-cc12-4b02-a80f-c885f3b25b8e" containerName="test-curl-7642-140658098841040" Jul 23 04:39:47.183887 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:47.183867 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7b477ad-cc12-4b02-a80f-c885f3b25b8e" containerName="test-curl-7642-140658098841040" Jul 23 04:39:47.186700 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:47.186686 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7642-140658098842048" Jul 23 04:39:47.196142 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:47.196119 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7642-140658098842048"] Jul 23 04:39:47.269963 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:47.269936 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-764bk\" (UniqueName: \"kubernetes.io/projected/b9b63933-87ed-4caf-80b9-93593449c643-kube-api-access-764bk\") pod \"test-curl-7642-140658098842048\" (UID: \"b9b63933-87ed-4caf-80b9-93593449c643\") " pod="opendatahub/test-curl-7642-140658098842048" Jul 23 04:39:47.370967 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:47.370937 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-764bk\" (UniqueName: \"kubernetes.io/projected/b9b63933-87ed-4caf-80b9-93593449c643-kube-api-access-764bk\") pod \"test-curl-7642-140658098842048\" (UID: \"b9b63933-87ed-4caf-80b9-93593449c643\") " pod="opendatahub/test-curl-7642-140658098842048" Jul 23 04:39:47.381121 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:47.381095 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-764bk\" (UniqueName: \"kubernetes.io/projected/b9b63933-87ed-4caf-80b9-93593449c643-kube-api-access-764bk\") pod \"test-curl-7642-140658098842048\" (UID: \"b9b63933-87ed-4caf-80b9-93593449c643\") " pod="opendatahub/test-curl-7642-140658098842048" Jul 23 04:39:47.496576 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:47.496518 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7642-140658098842048" Jul 23 04:39:47.625305 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:47.625281 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7642-140658098842048"] Jul 23 04:39:47.626932 ip-10-0-140-6 kubenswrapper[2567]: W0723 04:39:47.626909 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9b63933_87ed_4caf_80b9_93593449c643.slice/crio-1d921d9c63e15253e062a95f3a9601d68f717e6cece317bb0d73de7346c8463c WatchSource:0}: Error finding container 1d921d9c63e15253e062a95f3a9601d68f717e6cece317bb0d73de7346c8463c: Status 404 returned error can't find the container with id 1d921d9c63e15253e062a95f3a9601d68f717e6cece317bb0d73de7346c8463c Jul 23 04:39:47.862162 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:47.862080 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7642-140658098842048" event={"ID":"b9b63933-87ed-4caf-80b9-93593449c643","Type":"ContainerStarted","Data":"1d921d9c63e15253e062a95f3a9601d68f717e6cece317bb0d73de7346c8463c"} Jul 23 04:39:47.863359 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:47.863344 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7642-140658098841040" Jul 23 04:39:47.863468 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:47.863394 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb1e447147d8817528cb652cefec482201c453a8937a7a1a9bf48bc6ec7e9059" Jul 23 04:39:47.873774 ip-10-0-140-6 kubenswrapper[2567]: E0723 04:39:47.873749 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7642-140658098841040\" is forbidden: User \"system:node:ip-10-0-140-6.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-6.ec2.internal' and this object" podUID="e7b477ad-cc12-4b02-a80f-c885f3b25b8e" pod="opendatahub/test-curl-7642-140658098841040" Jul 23 04:39:48.492743 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:48.492709 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7b477ad-cc12-4b02-a80f-c885f3b25b8e" path="/var/lib/kubelet/pods/e7b477ad-cc12-4b02-a80f-c885f3b25b8e/volumes" Jul 23 04:39:48.870024 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:48.869936 2567 generic.go:356] "Generic (PLEG): container finished" podID="b9b63933-87ed-4caf-80b9-93593449c643" containerID="2eb4de4a050712f69ec91a8241fbba9ccae6dc1448cdff0107be817de594d32d" exitCode=0 Jul 23 04:39:48.870163 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:48.870025 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7642-140658098842048" event={"ID":"b9b63933-87ed-4caf-80b9-93593449c643","Type":"ContainerDied","Data":"2eb4de4a050712f69ec91a8241fbba9ccae6dc1448cdff0107be817de594d32d"} Jul 23 04:39:50.009002 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:50.008977 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7642-140658098842048" Jul 23 04:39:50.032176 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:50.032148 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7642-140658098842048_b9b63933-87ed-4caf-80b9-93593449c643/test-curl-7642-140658098842048/0.log" Jul 23 04:39:50.059931 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:50.059897 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7642-140658098842048"] Jul 23 04:39:50.065877 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:50.065853 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7642-140658098842048"] Jul 23 04:39:50.095128 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:50.095110 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-764bk\" (UniqueName: \"kubernetes.io/projected/b9b63933-87ed-4caf-80b9-93593449c643-kube-api-access-764bk\") pod \"b9b63933-87ed-4caf-80b9-93593449c643\" (UID: \"b9b63933-87ed-4caf-80b9-93593449c643\") " Jul 23 04:39:50.097230 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:50.097204 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9b63933-87ed-4caf-80b9-93593449c643-kube-api-access-764bk" (OuterVolumeSpecName: "kube-api-access-764bk") pod "b9b63933-87ed-4caf-80b9-93593449c643" (UID: "b9b63933-87ed-4caf-80b9-93593449c643"). InnerVolumeSpecName "kube-api-access-764bk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 04:39:50.191662 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:50.191635 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7642-140658098842048"] Jul 23 04:39:50.192007 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:50.191994 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9b63933-87ed-4caf-80b9-93593449c643" containerName="test-curl-7642-140658098842048" Jul 23 04:39:50.192045 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:50.192009 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9b63933-87ed-4caf-80b9-93593449c643" containerName="test-curl-7642-140658098842048" Jul 23 04:39:50.192089 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:50.192079 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9b63933-87ed-4caf-80b9-93593449c643" containerName="test-curl-7642-140658098842048" Jul 23 04:39:50.194920 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:50.194905 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7642-140658098842048" Jul 23 04:39:50.196069 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:50.196049 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-764bk\" (UniqueName: \"kubernetes.io/projected/b9b63933-87ed-4caf-80b9-93593449c643-kube-api-access-764bk\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:39:50.201778 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:50.201759 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7642-140658098842048"] Jul 23 04:39:50.296769 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:50.296739 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l96hb\" (UniqueName: \"kubernetes.io/projected/08e4a6be-aff9-428e-aab3-7c259f349480-kube-api-access-l96hb\") pod \"test-curl-7642-140658098842048\" (UID: \"08e4a6be-aff9-428e-aab3-7c259f349480\") " pod="opendatahub/test-curl-7642-140658098842048" Jul 23 04:39:50.398058 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:50.398028 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l96hb\" (UniqueName: \"kubernetes.io/projected/08e4a6be-aff9-428e-aab3-7c259f349480-kube-api-access-l96hb\") pod \"test-curl-7642-140658098842048\" (UID: \"08e4a6be-aff9-428e-aab3-7c259f349480\") " pod="opendatahub/test-curl-7642-140658098842048" Jul 23 04:39:50.407874 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:50.407844 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l96hb\" (UniqueName: \"kubernetes.io/projected/08e4a6be-aff9-428e-aab3-7c259f349480-kube-api-access-l96hb\") pod \"test-curl-7642-140658098842048\" (UID: \"08e4a6be-aff9-428e-aab3-7c259f349480\") " pod="opendatahub/test-curl-7642-140658098842048" Jul 23 04:39:50.493302 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:50.493217 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9b63933-87ed-4caf-80b9-93593449c643" path="/var/lib/kubelet/pods/b9b63933-87ed-4caf-80b9-93593449c643/volumes" Jul 23 04:39:50.505248 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:50.505229 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7642-140658098842048" Jul 23 04:39:50.638352 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:50.638327 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7642-140658098842048"] Jul 23 04:39:50.640145 ip-10-0-140-6 kubenswrapper[2567]: W0723 04:39:50.640121 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod08e4a6be_aff9_428e_aab3_7c259f349480.slice/crio-f3a47bf4a1484fd0db550e72503f843f0aa64b04310e5da81f855a5fd8929102 WatchSource:0}: Error finding container f3a47bf4a1484fd0db550e72503f843f0aa64b04310e5da81f855a5fd8929102: Status 404 returned error can't find the container with id f3a47bf4a1484fd0db550e72503f843f0aa64b04310e5da81f855a5fd8929102 Jul 23 04:39:50.883145 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:50.883051 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7642-140658098842048" event={"ID":"08e4a6be-aff9-428e-aab3-7c259f349480","Type":"ContainerStarted","Data":"f3a47bf4a1484fd0db550e72503f843f0aa64b04310e5da81f855a5fd8929102"} Jul 23 04:39:50.884195 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:50.884176 2567 scope.go:117] "RemoveContainer" containerID="2eb4de4a050712f69ec91a8241fbba9ccae6dc1448cdff0107be817de594d32d" Jul 23 04:39:50.884315 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:50.884197 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7642-140658098842048" Jul 23 04:39:51.890439 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:51.890401 2567 generic.go:356] "Generic (PLEG): container finished" podID="08e4a6be-aff9-428e-aab3-7c259f349480" containerID="4a97af16a8546976a2f15a55f32beaa596531001a123aae30dc24608e49c1528" exitCode=0 Jul 23 04:39:51.890914 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:51.890494 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7642-140658098842048" event={"ID":"08e4a6be-aff9-428e-aab3-7c259f349480","Type":"ContainerDied","Data":"4a97af16a8546976a2f15a55f32beaa596531001a123aae30dc24608e49c1528"} Jul 23 04:39:53.027493 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:53.027470 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7642-140658098842048" Jul 23 04:39:53.033688 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:53.033671 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l96hb\" (UniqueName: \"kubernetes.io/projected/08e4a6be-aff9-428e-aab3-7c259f349480-kube-api-access-l96hb\") pod \"08e4a6be-aff9-428e-aab3-7c259f349480\" (UID: \"08e4a6be-aff9-428e-aab3-7c259f349480\") " Jul 23 04:39:53.035759 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:53.035734 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08e4a6be-aff9-428e-aab3-7c259f349480-kube-api-access-l96hb" (OuterVolumeSpecName: "kube-api-access-l96hb") pod "08e4a6be-aff9-428e-aab3-7c259f349480" (UID: "08e4a6be-aff9-428e-aab3-7c259f349480"). InnerVolumeSpecName "kube-api-access-l96hb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 04:39:53.049783 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:53.049761 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7642-140658098842048_08e4a6be-aff9-428e-aab3-7c259f349480/test-curl-7642-140658098842048/0.log" Jul 23 04:39:53.076155 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:53.076129 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7642-140658098842048"] Jul 23 04:39:53.081795 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:53.081774 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7642-140658098842048"] Jul 23 04:39:53.135175 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:53.135156 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l96hb\" (UniqueName: \"kubernetes.io/projected/08e4a6be-aff9-428e-aab3-7c259f349480-kube-api-access-l96hb\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:39:53.270860 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:53.270825 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5fa859-z9mhc"] Jul 23 04:39:53.271153 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:53.271140 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08e4a6be-aff9-428e-aab3-7c259f349480" containerName="test-curl-7642-140658098842048" Jul 23 04:39:53.271197 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:53.271154 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="08e4a6be-aff9-428e-aab3-7c259f349480" containerName="test-curl-7642-140658098842048" Jul 23 04:39:53.271229 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:53.271218 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="08e4a6be-aff9-428e-aab3-7c259f349480" containerName="test-curl-7642-140658098842048" Jul 23 04:39:53.274022 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:53.274008 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5fa859-z9mhc" Jul 23 04:39:53.277389 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:53.277365 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dpftb\"" Jul 23 04:39:53.277389 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:53.277367 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 04:39:53.283242 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:53.283220 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5fa859-z9mhc"] Jul 23 04:39:53.336873 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:53.336841 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14ec5578-3d12-42c0-bef7-437a5f2af5b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-5fa859-z9mhc\" (UID: \"14ec5578-3d12-42c0-bef7-437a5f2af5b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5fa859-z9mhc" Jul 23 04:39:53.437530 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:53.437498 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14ec5578-3d12-42c0-bef7-437a5f2af5b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-5fa859-z9mhc\" (UID: \"14ec5578-3d12-42c0-bef7-437a5f2af5b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5fa859-z9mhc" Jul 23 04:39:53.438073 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:53.438054 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14ec5578-3d12-42c0-bef7-437a5f2af5b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-5fa859-z9mhc\" (UID: \"14ec5578-3d12-42c0-bef7-437a5f2af5b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5fa859-z9mhc" Jul 23 04:39:53.585601 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:53.585518 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5fa859-z9mhc" Jul 23 04:39:53.722364 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:53.722340 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5fa859-z9mhc"] Jul 23 04:39:53.724663 ip-10-0-140-6 kubenswrapper[2567]: W0723 04:39:53.724635 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14ec5578_3d12_42c0_bef7_437a5f2af5b9.slice/crio-58157263675a55766827b4caa0c4f4a2fff0b18252662b6314afa04b65697e53 WatchSource:0}: Error finding container 58157263675a55766827b4caa0c4f4a2fff0b18252662b6314afa04b65697e53: Status 404 returned error can't find the container with id 58157263675a55766827b4caa0c4f4a2fff0b18252662b6314afa04b65697e53 Jul 23 04:39:53.903235 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:53.903138 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5fa859-z9mhc" event={"ID":"14ec5578-3d12-42c0-bef7-437a5f2af5b9","Type":"ContainerStarted","Data":"58157263675a55766827b4caa0c4f4a2fff0b18252662b6314afa04b65697e53"} Jul 23 04:39:53.904449 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:53.904412 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3a47bf4a1484fd0db550e72503f843f0aa64b04310e5da81f855a5fd8929102" Jul 23 04:39:53.904561 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:53.904479 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7642-140658098842048" Jul 23 04:39:53.915130 ip-10-0-140-6 kubenswrapper[2567]: E0723 04:39:53.915102 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7642-140658098842048\" is forbidden: User \"system:node:ip-10-0-140-6.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-6.ec2.internal' and this object" podUID="08e4a6be-aff9-428e-aab3-7c259f349480" pod="opendatahub/test-curl-7642-140658098842048" Jul 23 04:39:54.492236 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:54.492207 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08e4a6be-aff9-428e-aab3-7c259f349480" path="/var/lib/kubelet/pods/08e4a6be-aff9-428e-aab3-7c259f349480/volumes" Jul 23 04:39:54.910767 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:54.910679 2567 generic.go:356] "Generic (PLEG): container finished" podID="14ec5578-3d12-42c0-bef7-437a5f2af5b9" containerID="919db2bfa25ba4129041f1e3debb18ff245dd204576fd529ab36116c163a23ca" exitCode=0 Jul 23 04:39:54.910767 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:54.910720 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5fa859-z9mhc" event={"ID":"14ec5578-3d12-42c0-bef7-437a5f2af5b9","Type":"ContainerDied","Data":"919db2bfa25ba4129041f1e3debb18ff245dd204576fd529ab36116c163a23ca"} Jul 23 04:39:56.048917 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:56.048896 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5fa859-z9mhc" Jul 23 04:39:56.057784 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:56.057763 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14ec5578-3d12-42c0-bef7-437a5f2af5b9-maas-api-service-ca\") pod \"14ec5578-3d12-42c0-bef7-437a5f2af5b9\" (UID: \"14ec5578-3d12-42c0-bef7-437a5f2af5b9\") " Jul 23 04:39:56.058071 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:56.058048 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14ec5578-3d12-42c0-bef7-437a5f2af5b9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "14ec5578-3d12-42c0-bef7-437a5f2af5b9" (UID: "14ec5578-3d12-42c0-bef7-437a5f2af5b9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 04:39:56.158795 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:56.158767 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14ec5578-3d12-42c0-bef7-437a5f2af5b9-maas-api-service-ca\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:39:56.921226 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:56.921197 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5fa859-z9mhc" Jul 23 04:39:56.921371 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:56.921230 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5fa859-z9mhc" event={"ID":"14ec5578-3d12-42c0-bef7-437a5f2af5b9","Type":"ContainerDied","Data":"58157263675a55766827b4caa0c4f4a2fff0b18252662b6314afa04b65697e53"} Jul 23 04:39:56.921371 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:39:56.921279 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58157263675a55766827b4caa0c4f4a2fff0b18252662b6314afa04b65697e53" Jul 23 04:40:08.355439 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:08.355379 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5fa859-z9mhc"] Jul 23 04:40:08.361776 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:08.361752 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5fa859-z9mhc"] Jul 23 04:40:08.491955 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:08.491918 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14ec5578-3d12-42c0-bef7-437a5f2af5b9" path="/var/lib/kubelet/pods/14ec5578-3d12-42c0-bef7-437a5f2af5b9/volumes" Jul 23 04:40:15.629237 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:15.629205 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt"] Jul 23 04:40:15.629738 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:15.629403 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt" podUID="68024583-771a-4ae3-88cb-fd39b8b1de1e" containerName="istio-proxy" containerID="cri-o://47378560a0e016cd1a85cc2134cc7a2fcdd223c9a3a9baf6b5c4ba835a21fb79" gracePeriod=30 Jul 23 04:40:15.931813 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:15.931778 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ab39e0-dkxj8"] Jul 23 04:40:15.932102 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:15.932091 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14ec5578-3d12-42c0-bef7-437a5f2af5b9" containerName="revoke-keys" Jul 23 04:40:15.932149 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:15.932104 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="14ec5578-3d12-42c0-bef7-437a5f2af5b9" containerName="revoke-keys" Jul 23 04:40:15.932184 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:15.932170 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="14ec5578-3d12-42c0-bef7-437a5f2af5b9" containerName="revoke-keys" Jul 23 04:40:15.936827 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:15.936807 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ab39e0-dkxj8" Jul 23 04:40:15.940345 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:15.940325 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 04:40:15.940484 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:15.940379 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dpftb\"" Jul 23 04:40:15.947858 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:15.947834 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ab39e0-dkxj8"] Jul 23 04:40:16.024710 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:16.024679 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85aef6bc-d0b1-4000-83ef-61f2bc1b6a60-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-ab39e0-dkxj8\" (UID: \"85aef6bc-d0b1-4000-83ef-61f2bc1b6a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ab39e0-dkxj8" Jul 23 04:40:16.125903 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:16.125868 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85aef6bc-d0b1-4000-83ef-61f2bc1b6a60-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-ab39e0-dkxj8\" (UID: \"85aef6bc-d0b1-4000-83ef-61f2bc1b6a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ab39e0-dkxj8" Jul 23 04:40:16.126619 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:16.126596 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85aef6bc-d0b1-4000-83ef-61f2bc1b6a60-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-ab39e0-dkxj8\" (UID: \"85aef6bc-d0b1-4000-83ef-61f2bc1b6a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ab39e0-dkxj8" Jul 23 04:40:16.247574 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:16.247477 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ab39e0-dkxj8" Jul 23 04:40:16.387685 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:16.387662 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ab39e0-dkxj8"] Jul 23 04:40:16.388994 ip-10-0-140-6 kubenswrapper[2567]: W0723 04:40:16.388969 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85aef6bc_d0b1_4000_83ef_61f2bc1b6a60.slice/crio-7c7c57af325c8a5696ab506501ed4d91a250d108b865fc950a47ebab689f5813 WatchSource:0}: Error finding container 7c7c57af325c8a5696ab506501ed4d91a250d108b865fc950a47ebab689f5813: Status 404 returned error can't find the container with id 7c7c57af325c8a5696ab506501ed4d91a250d108b865fc950a47ebab689f5813 Jul 23 04:40:17.029711 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:17.029668 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ab39e0-dkxj8" event={"ID":"85aef6bc-d0b1-4000-83ef-61f2bc1b6a60","Type":"ContainerStarted","Data":"bf13f6064c6c4eada39ea335c4153e4faf06227ec39104ad619e35bc3f94efac"} Jul 23 04:40:17.030120 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:17.029739 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ab39e0-dkxj8" event={"ID":"85aef6bc-d0b1-4000-83ef-61f2bc1b6a60","Type":"ContainerStarted","Data":"7c7c57af325c8a5696ab506501ed4d91a250d108b865fc950a47ebab689f5813"} Jul 23 04:40:17.049962 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:17.049911 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ab39e0-dkxj8" podStartSLOduration=2.049890464 podStartE2EDuration="2.049890464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 04:40:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 04:40:17.048719664 +0000 UTC m=+2696.954023956" watchObservedRunningTime="2026-07-23 04:40:17.049890464 +0000 UTC m=+2696.955194758" Jul 23 04:40:18.043018 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:18.042980 2567 generic.go:356] "Generic (PLEG): container finished" podID="85aef6bc-d0b1-4000-83ef-61f2bc1b6a60" containerID="bf13f6064c6c4eada39ea335c4153e4faf06227ec39104ad619e35bc3f94efac" exitCode=0 Jul 23 04:40:18.043018 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:18.043023 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ab39e0-dkxj8" event={"ID":"85aef6bc-d0b1-4000-83ef-61f2bc1b6a60","Type":"ContainerDied","Data":"bf13f6064c6c4eada39ea335c4153e4faf06227ec39104ad619e35bc3f94efac"} Jul 23 04:40:19.178085 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:19.178063 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ab39e0-dkxj8" Jul 23 04:40:19.252962 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:19.252934 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85aef6bc-d0b1-4000-83ef-61f2bc1b6a60-maas-api-service-ca\") pod \"85aef6bc-d0b1-4000-83ef-61f2bc1b6a60\" (UID: \"85aef6bc-d0b1-4000-83ef-61f2bc1b6a60\") " Jul 23 04:40:19.253272 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:19.253248 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85aef6bc-d0b1-4000-83ef-61f2bc1b6a60-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "85aef6bc-d0b1-4000-83ef-61f2bc1b6a60" (UID: "85aef6bc-d0b1-4000-83ef-61f2bc1b6a60"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 04:40:19.354401 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:19.354330 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85aef6bc-d0b1-4000-83ef-61f2bc1b6a60-maas-api-service-ca\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:40:20.054717 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:20.054682 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ab39e0-dkxj8" Jul 23 04:40:20.054889 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:20.054719 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ab39e0-dkxj8" event={"ID":"85aef6bc-d0b1-4000-83ef-61f2bc1b6a60","Type":"ContainerDied","Data":"7c7c57af325c8a5696ab506501ed4d91a250d108b865fc950a47ebab689f5813"} Jul 23 04:40:20.054889 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:20.054766 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c7c57af325c8a5696ab506501ed4d91a250d108b865fc950a47ebab689f5813" Jul 23 04:40:20.682184 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:20.682161 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wzgn9_dbd53593-e5e7-473a-b71c-8fbf9272913f/console-operator/2.log" Jul 23 04:40:20.690750 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:20.690731 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wzgn9_dbd53593-e5e7-473a-b71c-8fbf9272913f/console-operator/2.log" Jul 23 04:40:20.896263 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:20.896211 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt" Jul 23 04:40:20.947989 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:20.947962 2567 scope.go:117] "RemoveContainer" containerID="8ef71f9705cb3b02869350f319d20cfbecb53c66a02d295d5fb62f71a37c8a6c" Jul 23 04:40:20.956703 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:20.956686 2567 scope.go:117] "RemoveContainer" containerID="47378560a0e016cd1a85cc2134cc7a2fcdd223c9a3a9baf6b5c4ba835a21fb79" Jul 23 04:40:20.967877 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:20.967857 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68024583-771a-4ae3-88cb-fd39b8b1de1e-istiod-ca-cert\") pod \"68024583-771a-4ae3-88cb-fd39b8b1de1e\" (UID: \"68024583-771a-4ae3-88cb-fd39b8b1de1e\") " Jul 23 04:40:20.967980 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:20.967892 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68024583-771a-4ae3-88cb-fd39b8b1de1e-credential-socket\") pod \"68024583-771a-4ae3-88cb-fd39b8b1de1e\" (UID: \"68024583-771a-4ae3-88cb-fd39b8b1de1e\") " Jul 23 04:40:20.967980 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:20.967949 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6d798\" (UniqueName: \"kubernetes.io/projected/68024583-771a-4ae3-88cb-fd39b8b1de1e-kube-api-access-6d798\") pod \"68024583-771a-4ae3-88cb-fd39b8b1de1e\" (UID: \"68024583-771a-4ae3-88cb-fd39b8b1de1e\") " Jul 23 04:40:20.968089 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:20.967995 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68024583-771a-4ae3-88cb-fd39b8b1de1e-istio-podinfo\") pod \"68024583-771a-4ae3-88cb-fd39b8b1de1e\" (UID: \"68024583-771a-4ae3-88cb-fd39b8b1de1e\") " Jul 23 04:40:20.968089 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:20.968059 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68024583-771a-4ae3-88cb-fd39b8b1de1e-workload-certs\") pod \"68024583-771a-4ae3-88cb-fd39b8b1de1e\" (UID: \"68024583-771a-4ae3-88cb-fd39b8b1de1e\") " Jul 23 04:40:20.968195 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:20.968086 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68024583-771a-4ae3-88cb-fd39b8b1de1e-workload-socket\") pod \"68024583-771a-4ae3-88cb-fd39b8b1de1e\" (UID: \"68024583-771a-4ae3-88cb-fd39b8b1de1e\") " Jul 23 04:40:20.968195 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:20.968118 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68024583-771a-4ae3-88cb-fd39b8b1de1e-istio-token\") pod \"68024583-771a-4ae3-88cb-fd39b8b1de1e\" (UID: \"68024583-771a-4ae3-88cb-fd39b8b1de1e\") " Jul 23 04:40:20.968195 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:20.968189 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68024583-771a-4ae3-88cb-fd39b8b1de1e-istio-data\") pod \"68024583-771a-4ae3-88cb-fd39b8b1de1e\" (UID: \"68024583-771a-4ae3-88cb-fd39b8b1de1e\") " Jul 23 04:40:20.968335 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:20.968212 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68024583-771a-4ae3-88cb-fd39b8b1de1e-istio-envoy\") pod \"68024583-771a-4ae3-88cb-fd39b8b1de1e\" (UID: \"68024583-771a-4ae3-88cb-fd39b8b1de1e\") " Jul 23 04:40:20.968335 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:20.968224 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68024583-771a-4ae3-88cb-fd39b8b1de1e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "68024583-771a-4ae3-88cb-fd39b8b1de1e" (UID: "68024583-771a-4ae3-88cb-fd39b8b1de1e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 04:40:20.968585 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:20.968359 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68024583-771a-4ae3-88cb-fd39b8b1de1e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "68024583-771a-4ae3-88cb-fd39b8b1de1e" (UID: "68024583-771a-4ae3-88cb-fd39b8b1de1e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 04:40:20.968585 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:20.968483 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68024583-771a-4ae3-88cb-fd39b8b1de1e-istiod-ca-cert\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:40:20.968585 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:20.968500 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68024583-771a-4ae3-88cb-fd39b8b1de1e-credential-socket\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:40:20.968781 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:20.968594 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68024583-771a-4ae3-88cb-fd39b8b1de1e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "68024583-771a-4ae3-88cb-fd39b8b1de1e" (UID: "68024583-771a-4ae3-88cb-fd39b8b1de1e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 04:40:20.968939 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:20.968905 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68024583-771a-4ae3-88cb-fd39b8b1de1e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "68024583-771a-4ae3-88cb-fd39b8b1de1e" (UID: "68024583-771a-4ae3-88cb-fd39b8b1de1e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 04:40:20.969387 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:20.969278 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68024583-771a-4ae3-88cb-fd39b8b1de1e-istio-data" (OuterVolumeSpecName: "istio-data") pod "68024583-771a-4ae3-88cb-fd39b8b1de1e" (UID: "68024583-771a-4ae3-88cb-fd39b8b1de1e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 04:40:20.970545 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:20.970513 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68024583-771a-4ae3-88cb-fd39b8b1de1e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "68024583-771a-4ae3-88cb-fd39b8b1de1e" (UID: "68024583-771a-4ae3-88cb-fd39b8b1de1e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 04:40:20.970545 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:20.970526 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68024583-771a-4ae3-88cb-fd39b8b1de1e-istio-token" (OuterVolumeSpecName: "istio-token") pod "68024583-771a-4ae3-88cb-fd39b8b1de1e" (UID: "68024583-771a-4ae3-88cb-fd39b8b1de1e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 04:40:20.970705 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:20.970541 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68024583-771a-4ae3-88cb-fd39b8b1de1e-kube-api-access-6d798" (OuterVolumeSpecName: "kube-api-access-6d798") pod "68024583-771a-4ae3-88cb-fd39b8b1de1e" (UID: "68024583-771a-4ae3-88cb-fd39b8b1de1e"). InnerVolumeSpecName "kube-api-access-6d798". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 04:40:20.970705 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:20.970601 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68024583-771a-4ae3-88cb-fd39b8b1de1e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "68024583-771a-4ae3-88cb-fd39b8b1de1e" (UID: "68024583-771a-4ae3-88cb-fd39b8b1de1e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 04:40:21.064830 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:21.061052 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt" Jul 23 04:40:21.065668 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:21.065633 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt" event={"ID":"68024583-771a-4ae3-88cb-fd39b8b1de1e","Type":"ContainerDied","Data":"47378560a0e016cd1a85cc2134cc7a2fcdd223c9a3a9baf6b5c4ba835a21fb79"} Jul 23 04:40:21.065810 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:21.065777 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt" event={"ID":"68024583-771a-4ae3-88cb-fd39b8b1de1e","Type":"ContainerDied","Data":"26fd4fb9ca30f561142ad3e3c33bf66f40fed2d574323b9ab306e759f229cc80"} Jul 23 04:40:21.069093 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:21.069070 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68024583-771a-4ae3-88cb-fd39b8b1de1e-istio-data\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:40:21.069183 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:21.069100 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68024583-771a-4ae3-88cb-fd39b8b1de1e-istio-envoy\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:40:21.069183 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:21.069114 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d798\" (UniqueName: \"kubernetes.io/projected/68024583-771a-4ae3-88cb-fd39b8b1de1e-kube-api-access-6d798\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:40:21.069183 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:21.069123 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68024583-771a-4ae3-88cb-fd39b8b1de1e-istio-podinfo\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:40:21.069183 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:21.069132 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68024583-771a-4ae3-88cb-fd39b8b1de1e-workload-certs\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:40:21.069183 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:21.069139 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68024583-771a-4ae3-88cb-fd39b8b1de1e-workload-socket\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:40:21.069183 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:21.069147 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68024583-771a-4ae3-88cb-fd39b8b1de1e-istio-token\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:40:21.104506 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:21.104473 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt"] Jul 23 04:40:21.109587 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:21.109565 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-5fa859-openshift-default-6fbc9fcff9-fbvzt"] Jul 23 04:40:22.492622 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:22.492590 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68024583-771a-4ae3-88cb-fd39b8b1de1e" path="/var/lib/kubelet/pods/68024583-771a-4ae3-88cb-fd39b8b1de1e/volumes" Jul 23 04:40:31.014765 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:31.014733 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ab39e0-dkxj8"] Jul 23 04:40:31.016849 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:31.016829 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ab39e0-dkxj8"] Jul 23 04:40:32.492168 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:32.492129 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85aef6bc-d0b1-4000-83ef-61f2bc1b6a60" path="/var/lib/kubelet/pods/85aef6bc-d0b1-4000-83ef-61f2bc1b6a60/volumes" Jul 23 04:40:38.170157 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:38.170124 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7"] Jul 23 04:40:38.170610 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:38.170335 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7" podUID="87b4f602-0ad5-4a97-885a-7a7cf730f500" containerName="istio-proxy" containerID="cri-o://7d9abdba97191e3d8c1dab57797cc7a3780fd4837bb806d15ef6dfccd049e5e8" gracePeriod=30 Jul 23 04:40:38.827827 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:38.827797 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-9f544c994-gctq6_fa78f912-a721-4e62-91b4-0101d53b3f1c/authorino/0.log" Jul 23 04:40:43.423012 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:43.422989 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7" Jul 23 04:40:43.556836 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:43.556750 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87b4f602-0ad5-4a97-885a-7a7cf730f500-credential-socket\") pod \"87b4f602-0ad5-4a97-885a-7a7cf730f500\" (UID: \"87b4f602-0ad5-4a97-885a-7a7cf730f500\") " Jul 23 04:40:43.556836 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:43.556800 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4nw7m\" (UniqueName: \"kubernetes.io/projected/87b4f602-0ad5-4a97-885a-7a7cf730f500-kube-api-access-4nw7m\") pod \"87b4f602-0ad5-4a97-885a-7a7cf730f500\" (UID: \"87b4f602-0ad5-4a97-885a-7a7cf730f500\") " Jul 23 04:40:43.556836 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:43.556819 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87b4f602-0ad5-4a97-885a-7a7cf730f500-istio-envoy\") pod \"87b4f602-0ad5-4a97-885a-7a7cf730f500\" (UID: \"87b4f602-0ad5-4a97-885a-7a7cf730f500\") " Jul 23 04:40:43.556836 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:43.556841 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87b4f602-0ad5-4a97-885a-7a7cf730f500-istiod-ca-cert\") pod \"87b4f602-0ad5-4a97-885a-7a7cf730f500\" (UID: \"87b4f602-0ad5-4a97-885a-7a7cf730f500\") " Jul 23 04:40:43.557154 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:43.556878 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87b4f602-0ad5-4a97-885a-7a7cf730f500-istio-data\") pod \"87b4f602-0ad5-4a97-885a-7a7cf730f500\" (UID: \"87b4f602-0ad5-4a97-885a-7a7cf730f500\") " Jul 23 04:40:43.557154 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:43.556931 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87b4f602-0ad5-4a97-885a-7a7cf730f500-workload-socket\") pod \"87b4f602-0ad5-4a97-885a-7a7cf730f500\" (UID: \"87b4f602-0ad5-4a97-885a-7a7cf730f500\") " Jul 23 04:40:43.557154 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:43.556964 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87b4f602-0ad5-4a97-885a-7a7cf730f500-istio-token\") pod \"87b4f602-0ad5-4a97-885a-7a7cf730f500\" (UID: \"87b4f602-0ad5-4a97-885a-7a7cf730f500\") " Jul 23 04:40:43.557154 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:43.557005 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87b4f602-0ad5-4a97-885a-7a7cf730f500-workload-certs\") pod \"87b4f602-0ad5-4a97-885a-7a7cf730f500\" (UID: \"87b4f602-0ad5-4a97-885a-7a7cf730f500\") " Jul 23 04:40:43.557154 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:43.557034 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87b4f602-0ad5-4a97-885a-7a7cf730f500-istio-podinfo\") pod \"87b4f602-0ad5-4a97-885a-7a7cf730f500\" (UID: \"87b4f602-0ad5-4a97-885a-7a7cf730f500\") " Jul 23 04:40:43.557406 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:43.557122 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87b4f602-0ad5-4a97-885a-7a7cf730f500-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "87b4f602-0ad5-4a97-885a-7a7cf730f500" (UID: "87b4f602-0ad5-4a97-885a-7a7cf730f500"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 04:40:43.557406 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:43.557211 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87b4f602-0ad5-4a97-885a-7a7cf730f500-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "87b4f602-0ad5-4a97-885a-7a7cf730f500" (UID: "87b4f602-0ad5-4a97-885a-7a7cf730f500"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 04:40:43.557406 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:43.557305 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87b4f602-0ad5-4a97-885a-7a7cf730f500-workload-socket\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:40:43.557406 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:43.557324 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87b4f602-0ad5-4a97-885a-7a7cf730f500-credential-socket\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:40:43.557643 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:43.557411 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87b4f602-0ad5-4a97-885a-7a7cf730f500-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "87b4f602-0ad5-4a97-885a-7a7cf730f500" (UID: "87b4f602-0ad5-4a97-885a-7a7cf730f500"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 04:40:43.557770 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:43.557706 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87b4f602-0ad5-4a97-885a-7a7cf730f500-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "87b4f602-0ad5-4a97-885a-7a7cf730f500" (UID: "87b4f602-0ad5-4a97-885a-7a7cf730f500"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 04:40:43.557770 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:43.557729 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87b4f602-0ad5-4a97-885a-7a7cf730f500-istio-data" (OuterVolumeSpecName: "istio-data") pod "87b4f602-0ad5-4a97-885a-7a7cf730f500" (UID: "87b4f602-0ad5-4a97-885a-7a7cf730f500"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 04:40:43.559139 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:43.559114 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87b4f602-0ad5-4a97-885a-7a7cf730f500-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "87b4f602-0ad5-4a97-885a-7a7cf730f500" (UID: "87b4f602-0ad5-4a97-885a-7a7cf730f500"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 04:40:43.559139 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:43.559125 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87b4f602-0ad5-4a97-885a-7a7cf730f500-kube-api-access-4nw7m" (OuterVolumeSpecName: "kube-api-access-4nw7m") pod "87b4f602-0ad5-4a97-885a-7a7cf730f500" (UID: "87b4f602-0ad5-4a97-885a-7a7cf730f500"). InnerVolumeSpecName "kube-api-access-4nw7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 04:40:43.559288 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:43.559234 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87b4f602-0ad5-4a97-885a-7a7cf730f500-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "87b4f602-0ad5-4a97-885a-7a7cf730f500" (UID: "87b4f602-0ad5-4a97-885a-7a7cf730f500"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 04:40:43.559588 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:43.559567 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87b4f602-0ad5-4a97-885a-7a7cf730f500-istio-token" (OuterVolumeSpecName: "istio-token") pod "87b4f602-0ad5-4a97-885a-7a7cf730f500" (UID: "87b4f602-0ad5-4a97-885a-7a7cf730f500"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 04:40:43.658812 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:43.658683 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87b4f602-0ad5-4a97-885a-7a7cf730f500-istio-token\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:40:43.658812 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:43.658709 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87b4f602-0ad5-4a97-885a-7a7cf730f500-workload-certs\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:40:43.658812 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:43.658723 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87b4f602-0ad5-4a97-885a-7a7cf730f500-istio-podinfo\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:40:43.658812 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:43.658736 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nw7m\" (UniqueName: \"kubernetes.io/projected/87b4f602-0ad5-4a97-885a-7a7cf730f500-kube-api-access-4nw7m\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:40:43.658812 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:43.658749 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87b4f602-0ad5-4a97-885a-7a7cf730f500-istio-envoy\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:40:43.658812 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:43.658764 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87b4f602-0ad5-4a97-885a-7a7cf730f500-istiod-ca-cert\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:40:43.658812 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:43.658776 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87b4f602-0ad5-4a97-885a-7a7cf730f500-istio-data\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Jul 23 04:40:44.182047 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:44.182010 2567 generic.go:356] "Generic (PLEG): container finished" podID="87b4f602-0ad5-4a97-885a-7a7cf730f500" containerID="7d9abdba97191e3d8c1dab57797cc7a3780fd4837bb806d15ef6dfccd049e5e8" exitCode=0 Jul 23 04:40:44.182231 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:44.182063 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7" Jul 23 04:40:44.182231 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:44.182104 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7" event={"ID":"87b4f602-0ad5-4a97-885a-7a7cf730f500","Type":"ContainerDied","Data":"7d9abdba97191e3d8c1dab57797cc7a3780fd4837bb806d15ef6dfccd049e5e8"} Jul 23 04:40:44.182231 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:44.182160 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7" event={"ID":"87b4f602-0ad5-4a97-885a-7a7cf730f500","Type":"ContainerDied","Data":"9c8802e9f9bf7bb74712f3a6809c8932a80ce72a5750aa7c7ec557779ac799ce"} Jul 23 04:40:44.182231 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:44.182183 2567 scope.go:117] "RemoveContainer" containerID="7d9abdba97191e3d8c1dab57797cc7a3780fd4837bb806d15ef6dfccd049e5e8" Jul 23 04:40:44.191711 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:44.191693 2567 scope.go:117] "RemoveContainer" containerID="1d9483ddf0628fffdabaaccd7eafe82ff709632a501a38e3e16516dc201f6a16" Jul 23 04:40:44.196276 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:44.196258 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-rzrj5_d60f90ba-ef41-4ac7-80d6-d03eae7b361a/manager/0.log" Jul 23 04:40:44.200606 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:44.200587 2567 scope.go:117] "RemoveContainer" containerID="7d9abdba97191e3d8c1dab57797cc7a3780fd4837bb806d15ef6dfccd049e5e8" Jul 23 04:40:44.201230 ip-10-0-140-6 kubenswrapper[2567]: E0723 04:40:44.201042 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d9abdba97191e3d8c1dab57797cc7a3780fd4837bb806d15ef6dfccd049e5e8\": container with ID starting with 7d9abdba97191e3d8c1dab57797cc7a3780fd4837bb806d15ef6dfccd049e5e8 not found: ID does not exist" containerID="7d9abdba97191e3d8c1dab57797cc7a3780fd4837bb806d15ef6dfccd049e5e8" Jul 23 04:40:44.201230 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:44.201079 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d9abdba97191e3d8c1dab57797cc7a3780fd4837bb806d15ef6dfccd049e5e8"} err="failed to get container status \"7d9abdba97191e3d8c1dab57797cc7a3780fd4837bb806d15ef6dfccd049e5e8\": rpc error: code = NotFound desc = could not find container \"7d9abdba97191e3d8c1dab57797cc7a3780fd4837bb806d15ef6dfccd049e5e8\": container with ID starting with 7d9abdba97191e3d8c1dab57797cc7a3780fd4837bb806d15ef6dfccd049e5e8 not found: ID does not exist" Jul 23 04:40:44.201230 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:44.201101 2567 scope.go:117] "RemoveContainer" containerID="1d9483ddf0628fffdabaaccd7eafe82ff709632a501a38e3e16516dc201f6a16" Jul 23 04:40:44.201414 ip-10-0-140-6 kubenswrapper[2567]: E0723 04:40:44.201367 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d9483ddf0628fffdabaaccd7eafe82ff709632a501a38e3e16516dc201f6a16\": container with ID starting with 1d9483ddf0628fffdabaaccd7eafe82ff709632a501a38e3e16516dc201f6a16 not found: ID does not exist" containerID="1d9483ddf0628fffdabaaccd7eafe82ff709632a501a38e3e16516dc201f6a16" Jul 23 04:40:44.201414 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:44.201387 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d9483ddf0628fffdabaaccd7eafe82ff709632a501a38e3e16516dc201f6a16"} err="failed to get container status \"1d9483ddf0628fffdabaaccd7eafe82ff709632a501a38e3e16516dc201f6a16\": rpc error: code = NotFound desc = could not find container \"1d9483ddf0628fffdabaaccd7eafe82ff709632a501a38e3e16516dc201f6a16\": container with ID starting with 1d9483ddf0628fffdabaaccd7eafe82ff709632a501a38e3e16516dc201f6a16 not found: ID does not exist" Jul 23 04:40:44.203351 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:44.203332 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7"] Jul 23 04:40:44.207204 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:44.207185 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-ab39e0-openshift-default-9d6d575c9-zmpn7"] Jul 23 04:40:44.330354 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:44.330324 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-mwvqq_8fdae660-c7ad-4c99-9c82-f2881f07b883/manager/0.log" Jul 23 04:40:44.494054 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:44.493974 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87b4f602-0ad5-4a97-885a-7a7cf730f500" path="/var/lib/kubelet/pods/87b4f602-0ad5-4a97-885a-7a7cf730f500/volumes" Jul 23 04:40:45.091557 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:45.091533 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5775946f86-jh6jx_03f43fe7-a1ff-4b1c-847a-0c3eff57597f/copy-manifests/0.log" Jul 23 04:40:45.098699 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:45.098680 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5775946f86-jh6jx_03f43fe7-a1ff-4b1c-847a-0c3eff57597f/manager/0.log" Jul 23 04:40:45.571932 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:45.571905 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29746335-mgprt_7327303b-30a9-4589-9b5a-b5f65072eabd/cleanup/0.log" Jul 23 04:40:45.702735 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:45.702710 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29746350-vkldq_908620ef-a3f9-4dee-abec-3a074d51ba5b/cleanup/0.log" Jul 23 04:40:46.629666 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:46.629639 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sqh2r_12c78b78-909c-46b3-8c5c-b69ee1437002/extract/0.log" Jul 23 04:40:46.641898 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:46.641876 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sqh2r_12c78b78-909c-46b3-8c5c-b69ee1437002/util/0.log" Jul 23 04:40:46.649430 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:46.649413 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sqh2r_12c78b78-909c-46b3-8c5c-b69ee1437002/pull/0.log" Jul 23 04:40:46.761975 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:46.761953 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fq5lh_1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b/util/0.log" Jul 23 04:40:46.769022 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:46.769005 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fq5lh_1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b/pull/0.log" Jul 23 04:40:46.778956 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:46.778937 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fq5lh_1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b/extract/0.log" Jul 23 04:40:46.902193 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:46.902115 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vb9vz_7d344803-fa33-4428-b5e6-2accaa3a9637/extract/0.log" Jul 23 04:40:46.909351 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:46.909322 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vb9vz_7d344803-fa33-4428-b5e6-2accaa3a9637/util/0.log" Jul 23 04:40:46.916895 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:46.916877 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vb9vz_7d344803-fa33-4428-b5e6-2accaa3a9637/pull/0.log" Jul 23 04:40:47.030022 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:47.029998 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r7lhw_06ed2906-f99f-4fb8-9359-f6258a7f44af/util/0.log" Jul 23 04:40:47.037044 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:47.037024 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r7lhw_06ed2906-f99f-4fb8-9359-f6258a7f44af/pull/0.log" Jul 23 04:40:47.043807 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:47.043785 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r7lhw_06ed2906-f99f-4fb8-9359-f6258a7f44af/extract/0.log" Jul 23 04:40:47.166617 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:47.166550 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-9f544c994-gctq6_fa78f912-a721-4e62-91b4-0101d53b3f1c/authorino/0.log" Jul 23 04:40:47.292660 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:47.292628 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-56h6j_d9ee86b7-6d1a-4b01-80d8-6054cbc019dc/manager/0.log" Jul 23 04:40:48.026668 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:48.026641 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-6qglq_040429aa-732f-498d-bee1-3ee87e763646/manager/0.log" Jul 23 04:40:48.399464 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:48.399362 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4mc5m7_bb5a035e-cabb-4796-9094-30073eef0c21/istio-proxy/0.log" Jul 23 04:40:49.231543 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:49.231514 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-76fccd85c4-bhbr2_0a513261-862c-43e7-bf4a-2011100221f5/router/0.log" Jul 23 04:40:55.428676 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:55.428639 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-czc5s/must-gather-dppg6"] Jul 23 04:40:55.429065 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:55.428969 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68024583-771a-4ae3-88cb-fd39b8b1de1e" containerName="istio-proxy" Jul 23 04:40:55.429065 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:55.428985 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="68024583-771a-4ae3-88cb-fd39b8b1de1e" containerName="istio-proxy" Jul 23 04:40:55.429065 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:55.428998 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85aef6bc-d0b1-4000-83ef-61f2bc1b6a60" containerName="revoke-keys" Jul 23 04:40:55.429065 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:55.429005 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="85aef6bc-d0b1-4000-83ef-61f2bc1b6a60" containerName="revoke-keys" Jul 23 04:40:55.429065 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:55.429040 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87b4f602-0ad5-4a97-885a-7a7cf730f500" containerName="istio-proxy" Jul 23 04:40:55.429065 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:55.429051 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="87b4f602-0ad5-4a97-885a-7a7cf730f500" containerName="istio-proxy" Jul 23 04:40:55.429065 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:55.429063 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87b4f602-0ad5-4a97-885a-7a7cf730f500" containerName="istio-proxy" Jul 23 04:40:55.429065 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:55.429068 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="87b4f602-0ad5-4a97-885a-7a7cf730f500" containerName="istio-proxy" Jul 23 04:40:55.429467 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:55.429122 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="87b4f602-0ad5-4a97-885a-7a7cf730f500" containerName="istio-proxy" Jul 23 04:40:55.429467 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:55.429130 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="68024583-771a-4ae3-88cb-fd39b8b1de1e" containerName="istio-proxy" Jul 23 04:40:55.429467 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:55.429136 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="87b4f602-0ad5-4a97-885a-7a7cf730f500" containerName="istio-proxy" Jul 23 04:40:55.429467 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:55.429143 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="85aef6bc-d0b1-4000-83ef-61f2bc1b6a60" containerName="revoke-keys" Jul 23 04:40:55.429467 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:55.429197 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68024583-771a-4ae3-88cb-fd39b8b1de1e" containerName="istio-proxy" Jul 23 04:40:55.429467 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:55.429202 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="68024583-771a-4ae3-88cb-fd39b8b1de1e" containerName="istio-proxy" Jul 23 04:40:55.429467 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:55.429251 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="68024583-771a-4ae3-88cb-fd39b8b1de1e" containerName="istio-proxy" Jul 23 04:40:55.433685 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:55.433663 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-czc5s/must-gather-dppg6" Jul 23 04:40:55.436407 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:55.436386 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-czc5s\"/\"default-dockercfg-qqcf2\"" Jul 23 04:40:55.437548 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:55.437526 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-czc5s\"/\"openshift-service-ca.crt\"" Jul 23 04:40:55.437664 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:55.437645 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-czc5s\"/\"kube-root-ca.crt\"" Jul 23 04:40:55.441028 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:55.441001 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/95fd0036-3180-4f73-b34e-fe885e5981cf-must-gather-output\") pod \"must-gather-dppg6\" (UID: \"95fd0036-3180-4f73-b34e-fe885e5981cf\") " pod="openshift-must-gather-czc5s/must-gather-dppg6" Jul 23 04:40:55.441196 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:55.441179 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h5j8\" (UniqueName: \"kubernetes.io/projected/95fd0036-3180-4f73-b34e-fe885e5981cf-kube-api-access-2h5j8\") pod \"must-gather-dppg6\" (UID: \"95fd0036-3180-4f73-b34e-fe885e5981cf\") " pod="openshift-must-gather-czc5s/must-gather-dppg6" Jul 23 04:40:55.444878 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:55.444857 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-czc5s/must-gather-dppg6"] Jul 23 04:40:55.542140 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:55.542113 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h5j8\" (UniqueName: \"kubernetes.io/projected/95fd0036-3180-4f73-b34e-fe885e5981cf-kube-api-access-2h5j8\") pod \"must-gather-dppg6\" (UID: \"95fd0036-3180-4f73-b34e-fe885e5981cf\") " pod="openshift-must-gather-czc5s/must-gather-dppg6" Jul 23 04:40:55.542293 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:55.542154 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/95fd0036-3180-4f73-b34e-fe885e5981cf-must-gather-output\") pod \"must-gather-dppg6\" (UID: \"95fd0036-3180-4f73-b34e-fe885e5981cf\") " pod="openshift-must-gather-czc5s/must-gather-dppg6" Jul 23 04:40:55.542519 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:55.542499 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/95fd0036-3180-4f73-b34e-fe885e5981cf-must-gather-output\") pod \"must-gather-dppg6\" (UID: \"95fd0036-3180-4f73-b34e-fe885e5981cf\") " pod="openshift-must-gather-czc5s/must-gather-dppg6" Jul 23 04:40:55.550755 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:55.550733 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h5j8\" (UniqueName: \"kubernetes.io/projected/95fd0036-3180-4f73-b34e-fe885e5981cf-kube-api-access-2h5j8\") pod \"must-gather-dppg6\" (UID: \"95fd0036-3180-4f73-b34e-fe885e5981cf\") " pod="openshift-must-gather-czc5s/must-gather-dppg6" Jul 23 04:40:55.743514 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:55.743417 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-czc5s/must-gather-dppg6" Jul 23 04:40:55.874968 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:55.874946 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-czc5s/must-gather-dppg6"] Jul 23 04:40:55.877150 ip-10-0-140-6 kubenswrapper[2567]: W0723 04:40:55.877121 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod95fd0036_3180_4f73_b34e_fe885e5981cf.slice/crio-0d2f0f5c30c142934ce4d94fdeaf4222e5fe1df8148d908c1286b6008d327b12 WatchSource:0}: Error finding container 0d2f0f5c30c142934ce4d94fdeaf4222e5fe1df8148d908c1286b6008d327b12: Status 404 returned error can't find the container with id 0d2f0f5c30c142934ce4d94fdeaf4222e5fe1df8148d908c1286b6008d327b12 Jul 23 04:40:56.247809 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:56.247769 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-czc5s/must-gather-dppg6" event={"ID":"95fd0036-3180-4f73-b34e-fe885e5981cf","Type":"ContainerStarted","Data":"0d2f0f5c30c142934ce4d94fdeaf4222e5fe1df8148d908c1286b6008d327b12"} Jul 23 04:40:57.258086 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:57.258038 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-czc5s/must-gather-dppg6" event={"ID":"95fd0036-3180-4f73-b34e-fe885e5981cf","Type":"ContainerStarted","Data":"7bbeea8ccc2bcb1e2b0074f47bb36da29a989d9aa69559248f36ca4dc8f33846"} Jul 23 04:40:57.258086 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:57.258091 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-czc5s/must-gather-dppg6" event={"ID":"95fd0036-3180-4f73-b34e-fe885e5981cf","Type":"ContainerStarted","Data":"0ac2ee652c92c67fa15d5fd85a109c50d4952650a54b499436ad7caa4280b885"} Jul 23 04:40:57.282969 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:57.282907 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-czc5s/must-gather-dppg6" podStartSLOduration=1.425937801 podStartE2EDuration="2.282884125s" totalImagesPullingTime="856.946324ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 04:40:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 04:40:57.277228298 +0000 UTC m=+2737.182532591" watchObservedRunningTime="2026-07-23 04:40:57.282884125 +0000 UTC m=+2737.188188419" Jul 23 04:40:58.343128 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:58.343100 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-bjprg_ec74b46f-3165-4772-9090-b8adbf536767/global-pull-secret-syncer/0.log" Jul 23 04:40:58.529792 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:58.529762 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-kt7n9_46a75c21-dc90-40a4-b933-cc1e92dac8c7/konnectivity-agent/0.log" Jul 23 04:40:58.645656 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:40:58.645563 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-6.ec2.internal_b2183c9004d8f7a985e4b7edb883b58a/haproxy/0.log" Jul 23 04:41:02.900823 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:02.900784 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sqh2r_12c78b78-909c-46b3-8c5c-b69ee1437002/extract/0.log" Jul 23 04:41:02.925955 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:02.925902 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sqh2r_12c78b78-909c-46b3-8c5c-b69ee1437002/util/0.log" Jul 23 04:41:02.949178 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:02.949150 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759sqh2r_12c78b78-909c-46b3-8c5c-b69ee1437002/pull/0.log" Jul 23 04:41:02.986824 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:02.986790 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fq5lh_1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b/extract/0.log" Jul 23 04:41:03.012208 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:03.012142 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fq5lh_1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b/util/0.log" Jul 23 04:41:03.040945 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:03.040872 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0fq5lh_1799d3f5-a3e3-4fb5-ab9e-8be8b4bfff2b/pull/0.log" Jul 23 04:41:03.070984 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:03.070908 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vb9vz_7d344803-fa33-4428-b5e6-2accaa3a9637/extract/0.log" Jul 23 04:41:03.097290 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:03.097259 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vb9vz_7d344803-fa33-4428-b5e6-2accaa3a9637/util/0.log" Jul 23 04:41:03.120156 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:03.120101 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vb9vz_7d344803-fa33-4428-b5e6-2accaa3a9637/pull/0.log" Jul 23 04:41:03.152035 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:03.151875 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r7lhw_06ed2906-f99f-4fb8-9359-f6258a7f44af/extract/0.log" Jul 23 04:41:03.181959 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:03.181935 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r7lhw_06ed2906-f99f-4fb8-9359-f6258a7f44af/util/0.log" Jul 23 04:41:03.207309 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:03.207259 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1r7lhw_06ed2906-f99f-4fb8-9359-f6258a7f44af/pull/0.log" Jul 23 04:41:03.706451 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:03.704929 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-9f544c994-gctq6_fa78f912-a721-4e62-91b4-0101d53b3f1c/authorino/0.log" Jul 23 04:41:03.755973 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:03.755937 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-56h6j_d9ee86b7-6d1a-4b01-80d8-6054cbc019dc/manager/0.log" Jul 23 04:41:04.152458 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:04.152317 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-6qglq_040429aa-732f-498d-bee1-3ee87e763646/manager/0.log" Jul 23 04:41:06.614531 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:06.614500 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xsjwx_f39f698b-feb8-429d-942e-bc0de10e2860/node-exporter/0.log" Jul 23 04:41:06.649716 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:06.649688 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xsjwx_f39f698b-feb8-429d-942e-bc0de10e2860/kube-rbac-proxy/0.log" Jul 23 04:41:06.684687 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:06.684654 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xsjwx_f39f698b-feb8-429d-942e-bc0de10e2860/init-textfile/0.log" Jul 23 04:41:08.483205 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:08.483169 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-czc5s/perf-node-gather-daemonset-8g72x"] Jul 23 04:41:08.490817 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:08.490791 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-czc5s/perf-node-gather-daemonset-8g72x" Jul 23 04:41:08.500840 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:08.500811 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-czc5s/perf-node-gather-daemonset-8g72x"] Jul 23 04:41:08.514509 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:08.514456 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-7bc7489cb5-h6ksc_b6d9f954-7eb4-4d0d-9961-ad00ed214cd3/networking-console-plugin/0.log" Jul 23 04:41:08.598643 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:08.598599 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/56d1969e-abb8-43ab-9b90-263b7f3b0c39-proc\") pod \"perf-node-gather-daemonset-8g72x\" (UID: \"56d1969e-abb8-43ab-9b90-263b7f3b0c39\") " pod="openshift-must-gather-czc5s/perf-node-gather-daemonset-8g72x" Jul 23 04:41:08.598842 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:08.598734 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/56d1969e-abb8-43ab-9b90-263b7f3b0c39-podres\") pod \"perf-node-gather-daemonset-8g72x\" (UID: \"56d1969e-abb8-43ab-9b90-263b7f3b0c39\") " pod="openshift-must-gather-czc5s/perf-node-gather-daemonset-8g72x" Jul 23 04:41:08.598842 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:08.598776 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/56d1969e-abb8-43ab-9b90-263b7f3b0c39-lib-modules\") pod \"perf-node-gather-daemonset-8g72x\" (UID: \"56d1969e-abb8-43ab-9b90-263b7f3b0c39\") " pod="openshift-must-gather-czc5s/perf-node-gather-daemonset-8g72x" Jul 23 04:41:08.598842 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:08.598805 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxpsw\" (UniqueName: \"kubernetes.io/projected/56d1969e-abb8-43ab-9b90-263b7f3b0c39-kube-api-access-pxpsw\") pod \"perf-node-gather-daemonset-8g72x\" (UID: \"56d1969e-abb8-43ab-9b90-263b7f3b0c39\") " pod="openshift-must-gather-czc5s/perf-node-gather-daemonset-8g72x" Jul 23 04:41:08.599003 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:08.598844 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/56d1969e-abb8-43ab-9b90-263b7f3b0c39-sys\") pod \"perf-node-gather-daemonset-8g72x\" (UID: \"56d1969e-abb8-43ab-9b90-263b7f3b0c39\") " pod="openshift-must-gather-czc5s/perf-node-gather-daemonset-8g72x" Jul 23 04:41:08.699781 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:08.699748 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/56d1969e-abb8-43ab-9b90-263b7f3b0c39-podres\") pod \"perf-node-gather-daemonset-8g72x\" (UID: \"56d1969e-abb8-43ab-9b90-263b7f3b0c39\") " pod="openshift-must-gather-czc5s/perf-node-gather-daemonset-8g72x" Jul 23 04:41:08.699781 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:08.699783 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/56d1969e-abb8-43ab-9b90-263b7f3b0c39-lib-modules\") pod \"perf-node-gather-daemonset-8g72x\" (UID: \"56d1969e-abb8-43ab-9b90-263b7f3b0c39\") " pod="openshift-must-gather-czc5s/perf-node-gather-daemonset-8g72x" Jul 23 04:41:08.700032 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:08.699810 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxpsw\" (UniqueName: \"kubernetes.io/projected/56d1969e-abb8-43ab-9b90-263b7f3b0c39-kube-api-access-pxpsw\") pod \"perf-node-gather-daemonset-8g72x\" (UID: \"56d1969e-abb8-43ab-9b90-263b7f3b0c39\") " pod="openshift-must-gather-czc5s/perf-node-gather-daemonset-8g72x" Jul 23 04:41:08.700032 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:08.699911 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/56d1969e-abb8-43ab-9b90-263b7f3b0c39-podres\") pod \"perf-node-gather-daemonset-8g72x\" (UID: \"56d1969e-abb8-43ab-9b90-263b7f3b0c39\") " pod="openshift-must-gather-czc5s/perf-node-gather-daemonset-8g72x" Jul 23 04:41:08.700032 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:08.699948 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/56d1969e-abb8-43ab-9b90-263b7f3b0c39-sys\") pod \"perf-node-gather-daemonset-8g72x\" (UID: \"56d1969e-abb8-43ab-9b90-263b7f3b0c39\") " pod="openshift-must-gather-czc5s/perf-node-gather-daemonset-8g72x" Jul 23 04:41:08.700032 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:08.700002 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/56d1969e-abb8-43ab-9b90-263b7f3b0c39-sys\") pod \"perf-node-gather-daemonset-8g72x\" (UID: \"56d1969e-abb8-43ab-9b90-263b7f3b0c39\") " pod="openshift-must-gather-czc5s/perf-node-gather-daemonset-8g72x" Jul 23 04:41:08.700032 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:08.699957 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/56d1969e-abb8-43ab-9b90-263b7f3b0c39-lib-modules\") pod \"perf-node-gather-daemonset-8g72x\" (UID: \"56d1969e-abb8-43ab-9b90-263b7f3b0c39\") " pod="openshift-must-gather-czc5s/perf-node-gather-daemonset-8g72x" Jul 23 04:41:08.700253 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:08.700030 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/56d1969e-abb8-43ab-9b90-263b7f3b0c39-proc\") pod \"perf-node-gather-daemonset-8g72x\" (UID: \"56d1969e-abb8-43ab-9b90-263b7f3b0c39\") " pod="openshift-must-gather-czc5s/perf-node-gather-daemonset-8g72x" Jul 23 04:41:08.700253 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:08.700080 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/56d1969e-abb8-43ab-9b90-263b7f3b0c39-proc\") pod \"perf-node-gather-daemonset-8g72x\" (UID: \"56d1969e-abb8-43ab-9b90-263b7f3b0c39\") " pod="openshift-must-gather-czc5s/perf-node-gather-daemonset-8g72x" Jul 23 04:41:08.708481 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:08.708456 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxpsw\" (UniqueName: \"kubernetes.io/projected/56d1969e-abb8-43ab-9b90-263b7f3b0c39-kube-api-access-pxpsw\") pod \"perf-node-gather-daemonset-8g72x\" (UID: \"56d1969e-abb8-43ab-9b90-263b7f3b0c39\") " pod="openshift-must-gather-czc5s/perf-node-gather-daemonset-8g72x" Jul 23 04:41:08.807628 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:08.807539 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-czc5s/perf-node-gather-daemonset-8g72x" Jul 23 04:41:08.995313 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:08.995233 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-czc5s/perf-node-gather-daemonset-8g72x"] Jul 23 04:41:09.007724 ip-10-0-140-6 kubenswrapper[2567]: W0723 04:41:08.998396 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod56d1969e_abb8_43ab_9b90_263b7f3b0c39.slice/crio-ead96f7af2d6c3575758827588d894eec528bb827b9173ff99f2e7eb194bab5c WatchSource:0}: Error finding container ead96f7af2d6c3575758827588d894eec528bb827b9173ff99f2e7eb194bab5c: Status 404 returned error can't find the container with id ead96f7af2d6c3575758827588d894eec528bb827b9173ff99f2e7eb194bab5c Jul 23 04:41:09.175925 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:09.175893 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wzgn9_dbd53593-e5e7-473a-b71c-8fbf9272913f/console-operator/2.log" Jul 23 04:41:09.179848 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:09.179826 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wzgn9_dbd53593-e5e7-473a-b71c-8fbf9272913f/console-operator/3.log" Jul 23 04:41:09.352760 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:09.352677 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-czc5s/perf-node-gather-daemonset-8g72x" event={"ID":"56d1969e-abb8-43ab-9b90-263b7f3b0c39","Type":"ContainerStarted","Data":"3139c7c0b1ff88a765b02082b74d8bbd4513a0fcbb73f845b9083bc8923e8aa0"} Jul 23 04:41:09.352760 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:09.352733 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-czc5s/perf-node-gather-daemonset-8g72x" event={"ID":"56d1969e-abb8-43ab-9b90-263b7f3b0c39","Type":"ContainerStarted","Data":"ead96f7af2d6c3575758827588d894eec528bb827b9173ff99f2e7eb194bab5c"} Jul 23 04:41:09.352760 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:09.352760 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-czc5s/perf-node-gather-daemonset-8g72x" Jul 23 04:41:09.371803 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:09.371756 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-czc5s/perf-node-gather-daemonset-8g72x" podStartSLOduration=1.371740774 podStartE2EDuration="1.371740774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 04:41:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 04:41:09.36985876 +0000 UTC m=+2749.275163054" watchObservedRunningTime="2026-07-23 04:41:09.371740774 +0000 UTC m=+2749.277045065" Jul 23 04:41:10.266642 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:10.266618 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78cf4c6f6-qwlg8_b8b0f82e-c3c2-4bee-80c2-8ed759b47910/volume-data-source-validator/0.log" Jul 23 04:41:11.446945 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:11.446918 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w64s6_2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d/dns/0.log" Jul 23 04:41:11.470887 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:11.470863 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w64s6_2e1f5194-79cd-4bdb-9afc-133dbeb7cf5d/kube-rbac-proxy/0.log" Jul 23 04:41:11.525978 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:11.525950 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gpl6h_6b6f66fe-a3d8-4ef6-95ee-9016ff91606f/dns-node-resolver/0.log" Jul 23 04:41:12.095173 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:12.095105 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-fb4fdbc47-ppvss_84aef285-74a3-433d-bcdd-6a22df1e44d9/registry/0.log" Jul 23 04:41:12.117301 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:12.117260 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kw46l_787ce18c-a72b-426f-a9f0-51f2c6f7c9bb/node-ca/0.log" Jul 23 04:41:13.068753 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:13.068695 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4mc5m7_bb5a035e-cabb-4796-9094-30073eef0c21/istio-proxy/0.log" Jul 23 04:41:13.387354 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:13.387258 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-76fccd85c4-bhbr2_0a513261-862c-43e7-bf4a-2011100221f5/router/0.log" Jul 23 04:41:14.080462 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:14.080411 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nt27s_cc9ae7fe-6710-4847-bacc-01096f89cd3c/serve-healthcheck-canary/0.log" Jul 23 04:41:14.773712 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:14.773685 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rhkfn_8462a1d3-1d08-4b40-9790-9a60d9ff231e/kube-rbac-proxy/0.log" Jul 23 04:41:14.794123 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:14.794091 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rhkfn_8462a1d3-1d08-4b40-9790-9a60d9ff231e/exporter/0.log" Jul 23 04:41:14.816105 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:14.816079 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rhkfn_8462a1d3-1d08-4b40-9790-9a60d9ff231e/extractor/0.log" Jul 23 04:41:15.377567 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:15.377537 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-czc5s/perf-node-gather-daemonset-8g72x" Jul 23 04:41:17.230860 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:17.230834 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-rzrj5_d60f90ba-ef41-4ac7-80d6-d03eae7b361a/manager/0.log" Jul 23 04:41:17.469140 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:17.469110 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-mwvqq_8fdae660-c7ad-4c99-9c82-f2881f07b883/manager/0.log" Jul 23 04:41:17.915322 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:17.915256 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5775946f86-jh6jx_03f43fe7-a1ff-4b1c-847a-0c3eff57597f/manager/0.log" Jul 23 04:41:17.962376 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:17.962351 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5775946f86-jh6jx_03f43fe7-a1ff-4b1c-847a-0c3eff57597f/copy-manifests/0.log" Jul 23 04:41:19.269608 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:19.269532 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6776597644-gzpxr_a2d1aef1-4986-452b-b52d-7e35369e1a01/manager/0.log" Jul 23 04:41:19.279403 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:19.279373 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4mc5m7_bb5a035e-cabb-4796-9094-30073eef0c21/istio-proxy/0.log" Jul 23 04:41:19.321250 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:19.321201 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-ljbh7_e7e16755-e2e6-42b6-9ed1-e63706e73cd3/openshift-lws-operator/0.log" Jul 23 04:41:19.524872 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:19.524802 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-76fccd85c4-bhbr2_0a513261-862c-43e7-bf4a-2011100221f5/router/0.log" Jul 23 04:41:20.717566 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:20.717538 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4mc5m7_bb5a035e-cabb-4796-9094-30073eef0c21/istio-proxy/0.log" Jul 23 04:41:20.993373 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:20.993302 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-76fccd85c4-bhbr2_0a513261-862c-43e7-bf4a-2011100221f5/router/0.log" Jul 23 04:41:24.593867 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:24.593830 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-2nm67_83d126c6-3769-45ac-8b1e-8e04b8f775b3/kube-storage-version-migrator-operator/1.log" Jul 23 04:41:24.595261 ip-10-0-140-6 kubenswrapper[2567]: I0723 04:41:24.595156 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-2nm67_83d126c6-3769-45ac-8b1e-8e04b8f775b3/kube-storage-version-migrator-operator/0.log"