Jul 20 02:45:03.311040 ip-10-0-142-139 systemd[1]: Starting Kubernetes Kubelet... Jul 20 02:45:05.176889 ip-10-0-142-139 kubenswrapper[2558]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 20 02:45:05.176889 ip-10-0-142-139 kubenswrapper[2558]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 20 02:45:05.176889 ip-10-0-142-139 kubenswrapper[2558]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 20 02:45:05.176889 ip-10-0-142-139 kubenswrapper[2558]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 20 02:45:05.176889 ip-10-0-142-139 kubenswrapper[2558]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 20 02:45:05.176889 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.138295 2558 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 20 02:45:05.176889 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143377 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 02:45:05.176889 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143392 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 02:45:05.176889 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143396 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 02:45:05.176889 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143399 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 02:45:05.176889 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143402 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 02:45:05.176889 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143406 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 02:45:05.176889 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143409 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 02:45:05.176889 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143412 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 02:45:05.178545 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143415 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 02:45:05.178545 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143418 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 02:45:05.178545 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143426 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 02:45:05.178545 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143430 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 02:45:05.178545 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143433 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 02:45:05.178545 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143437 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 02:45:05.178545 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143442 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 02:45:05.178545 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143445 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 02:45:05.178545 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143448 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 02:45:05.178545 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143451 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 02:45:05.178545 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143454 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 02:45:05.178545 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143457 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 02:45:05.178545 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143459 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 02:45:05.178545 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143463 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 02:45:05.178545 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143466 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 02:45:05.178545 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143469 2558 feature_gate.go:328] unrecognized feature gate: Example Jul 20 02:45:05.178545 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143471 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 02:45:05.178545 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143474 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 02:45:05.178545 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143485 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 02:45:05.179166 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143488 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 02:45:05.179166 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143491 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 02:45:05.179166 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143494 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 02:45:05.179166 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143498 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 02:45:05.179166 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143503 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 02:45:05.179166 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143507 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 02:45:05.179166 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143511 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 02:45:05.179166 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143515 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 02:45:05.179166 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143519 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 02:45:05.179166 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143523 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 02:45:05.179166 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143527 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 02:45:05.179166 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143532 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 02:45:05.179166 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143536 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 02:45:05.179166 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143539 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 02:45:05.179166 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143543 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 02:45:05.179166 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143547 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 02:45:05.179166 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143552 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 02:45:05.179166 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143555 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 02:45:05.179166 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143559 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 02:45:05.179166 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143564 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 02:45:05.180734 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143567 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 02:45:05.180734 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143571 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 02:45:05.180734 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143575 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 02:45:05.180734 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143579 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 02:45:05.180734 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143582 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 02:45:05.180734 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143586 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 02:45:05.180734 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143590 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 02:45:05.180734 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143593 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 02:45:05.180734 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143599 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 02:45:05.180734 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143602 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 02:45:05.180734 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143606 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 02:45:05.180734 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143612 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 02:45:05.180734 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143617 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 02:45:05.180734 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143622 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 02:45:05.180734 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143626 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 02:45:05.180734 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143629 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 02:45:05.180734 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143633 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 02:45:05.180734 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143637 2558 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 02:45:05.180734 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143641 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 02:45:05.180734 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143645 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 02:45:05.181398 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143649 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 02:45:05.181398 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143652 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 02:45:05.181398 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143657 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 02:45:05.181398 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143661 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 02:45:05.181398 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143665 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 02:45:05.181398 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143668 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 02:45:05.181398 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143672 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 02:45:05.181398 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143676 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 02:45:05.181398 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143680 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 02:45:05.181398 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143683 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 02:45:05.181398 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143687 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 02:45:05.181398 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143692 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 02:45:05.181398 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143697 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 02:45:05.181398 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143700 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 02:45:05.181398 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143704 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 02:45:05.181398 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143710 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 02:45:05.181398 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.143714 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 02:45:05.181398 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.144280 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 02:45:05.181398 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.144287 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 02:45:05.181398 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.144292 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 02:45:05.181924 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.144296 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 02:45:05.181924 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.144300 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 02:45:05.181924 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.144304 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 02:45:05.181924 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.144308 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 02:45:05.181924 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.144312 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 02:45:05.181924 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.144316 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 02:45:05.181924 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.144320 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 02:45:05.181924 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.144324 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 02:45:05.181924 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.144328 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 02:45:05.181924 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.144332 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 02:45:05.181924 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.144336 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 02:45:05.181924 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.144341 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 02:45:05.181924 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146142 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 02:45:05.181924 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146151 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 02:45:05.181924 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146158 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 02:45:05.181924 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146165 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 02:45:05.181924 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146169 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 02:45:05.181924 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146173 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 02:45:05.181924 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146179 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 02:45:05.181924 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146182 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 02:45:05.182494 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146186 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 02:45:05.182494 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146190 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 02:45:05.182494 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146194 2558 feature_gate.go:328] unrecognized feature gate: Example Jul 20 02:45:05.182494 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146197 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 02:45:05.182494 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146201 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 02:45:05.182494 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146205 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 02:45:05.182494 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146209 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 02:45:05.182494 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146212 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 02:45:05.182494 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146216 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 02:45:05.182494 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146220 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 02:45:05.182494 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146223 2558 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 02:45:05.182494 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146227 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 02:45:05.182494 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146231 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 02:45:05.182494 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146235 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 02:45:05.182494 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146239 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 02:45:05.182494 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146243 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 02:45:05.182494 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146246 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 02:45:05.182494 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146250 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 02:45:05.182494 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146254 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 02:45:05.182494 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146258 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 02:45:05.183269 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146262 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 02:45:05.183269 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146266 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 02:45:05.183269 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146270 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 02:45:05.183269 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146275 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 02:45:05.183269 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146278 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 02:45:05.183269 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146283 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 02:45:05.183269 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146287 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 02:45:05.183269 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146291 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 02:45:05.183269 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146295 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 02:45:05.183269 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146298 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 02:45:05.183269 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146302 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 02:45:05.183269 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146306 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 02:45:05.183269 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146310 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 02:45:05.183269 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146313 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 02:45:05.183269 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146317 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 02:45:05.183269 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146321 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 02:45:05.183269 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146325 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 02:45:05.183269 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146328 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 02:45:05.183269 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146332 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 02:45:05.183870 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146336 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 02:45:05.183870 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146339 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 02:45:05.183870 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146343 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 02:45:05.183870 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146347 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 02:45:05.183870 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146350 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 02:45:05.183870 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146354 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 02:45:05.183870 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146358 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 02:45:05.183870 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146365 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 02:45:05.183870 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146369 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 02:45:05.183870 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146373 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 02:45:05.183870 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146377 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 02:45:05.183870 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146380 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 02:45:05.183870 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146384 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 02:45:05.183870 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146388 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 02:45:05.183870 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146391 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 02:45:05.183870 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146395 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 02:45:05.183870 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146398 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 02:45:05.183870 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146402 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 02:45:05.183870 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146406 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 02:45:05.183870 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146409 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 02:45:05.184495 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146413 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 02:45:05.184495 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146418 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 02:45:05.184495 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146424 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 02:45:05.184495 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146428 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 02:45:05.184495 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146433 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 02:45:05.184495 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146437 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 02:45:05.184495 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146440 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 02:45:05.184495 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146443 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 02:45:05.184495 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146447 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 02:45:05.184495 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146450 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 02:45:05.184495 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146454 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 02:45:05.184495 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146457 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 02:45:05.184495 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146461 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 02:45:05.184495 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146464 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 02:45:05.184495 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146468 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 02:45:05.184495 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146473 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 02:45:05.184495 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146477 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 02:45:05.184495 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146481 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 02:45:05.184495 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146484 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 02:45:05.184495 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146489 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 02:45:05.185116 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146493 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 02:45:05.185116 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146498 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 02:45:05.185116 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146502 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 02:45:05.185116 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146507 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 02:45:05.185116 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146511 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 02:45:05.185116 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146515 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 02:45:05.185116 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146519 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 02:45:05.185116 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146523 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 02:45:05.185116 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146527 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 02:45:05.185116 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146531 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 02:45:05.185116 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146536 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 02:45:05.185116 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.146540 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 02:45:05.185116 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147089 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 02:45:05.185116 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147114 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 02:45:05.185116 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147119 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 02:45:05.185116 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147124 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 02:45:05.185116 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147128 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 02:45:05.185116 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147131 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 02:45:05.185116 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147135 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 02:45:05.185848 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147139 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 02:45:05.185848 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147143 2558 feature_gate.go:328] unrecognized feature gate: Example Jul 20 02:45:05.185848 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147147 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 02:45:05.185848 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147150 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 02:45:05.185848 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147154 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 02:45:05.185848 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147158 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 02:45:05.185848 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147162 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 02:45:05.185848 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147166 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 02:45:05.185848 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147169 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 02:45:05.185848 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147175 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 02:45:05.185848 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147180 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 02:45:05.185848 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147184 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 02:45:05.185848 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147190 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 02:45:05.185848 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147194 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 02:45:05.185848 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147197 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 02:45:05.185848 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147201 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 02:45:05.185848 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147204 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 02:45:05.185848 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147208 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 02:45:05.185848 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147211 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 02:45:05.185848 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147215 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 02:45:05.186437 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147219 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 02:45:05.186437 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147223 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 02:45:05.186437 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147226 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 02:45:05.186437 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147230 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 02:45:05.186437 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147234 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 02:45:05.186437 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147237 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 02:45:05.186437 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147241 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 02:45:05.186437 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147245 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 02:45:05.186437 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147249 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 02:45:05.186437 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147253 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 02:45:05.186437 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147256 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 02:45:05.186437 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147260 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 02:45:05.186437 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147263 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 02:45:05.186437 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147267 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 02:45:05.186437 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147270 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 02:45:05.186437 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147274 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 02:45:05.186437 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147278 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 02:45:05.186437 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147282 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 02:45:05.186437 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147285 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 02:45:05.187005 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147289 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 02:45:05.187005 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147292 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 02:45:05.187005 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147296 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 02:45:05.187005 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147299 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 02:45:05.187005 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147303 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 02:45:05.187005 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147309 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 02:45:05.187005 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147312 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 02:45:05.187005 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147316 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 02:45:05.187005 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147320 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 02:45:05.187005 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147324 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 02:45:05.187005 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147330 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 02:45:05.187005 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147334 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 02:45:05.187005 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147337 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 02:45:05.187005 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147341 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 02:45:05.187005 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147347 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 02:45:05.187005 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147351 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 02:45:05.187005 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147355 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 02:45:05.187005 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147359 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 02:45:05.187005 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147363 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 02:45:05.187690 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147367 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 02:45:05.187690 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147371 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 02:45:05.187690 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147376 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 02:45:05.187690 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147385 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 02:45:05.187690 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147389 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 02:45:05.187690 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147392 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 02:45:05.187690 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147396 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 02:45:05.187690 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147399 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 02:45:05.187690 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147403 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 02:45:05.187690 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147407 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 02:45:05.187690 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147411 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 02:45:05.187690 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147415 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 02:45:05.187690 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147419 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 02:45:05.187690 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147422 2558 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 02:45:05.187690 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147425 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 02:45:05.187690 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147432 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 02:45:05.187690 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147436 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 02:45:05.187690 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147440 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 02:45:05.187690 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147446 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 02:45:05.188412 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147450 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 02:45:05.188412 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147454 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 02:45:05.188412 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147458 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 02:45:05.188412 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147462 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 02:45:05.188412 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147466 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 02:45:05.188412 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147470 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 02:45:05.188412 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147474 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 02:45:05.188412 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147484 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 02:45:05.188412 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147488 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 02:45:05.188412 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147492 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 02:45:05.188412 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147496 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 02:45:05.188412 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147499 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 02:45:05.188412 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147503 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 02:45:05.188412 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147507 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 02:45:05.188412 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.147511 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 02:45:05.188412 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147597 2558 flags.go:64] FLAG: --address="0.0.0.0" Jul 20 02:45:05.188412 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147606 2558 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 20 02:45:05.188412 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147615 2558 flags.go:64] FLAG: --anonymous-auth="true" Jul 20 02:45:05.188412 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147621 2558 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 20 02:45:05.188412 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147626 2558 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 20 02:45:05.188412 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147631 2558 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 20 02:45:05.188995 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147637 2558 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 20 02:45:05.188995 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147643 2558 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 20 02:45:05.188995 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147648 2558 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 20 02:45:05.188995 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147653 2558 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 20 02:45:05.188995 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147658 2558 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 20 02:45:05.188995 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147662 2558 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 20 02:45:05.188995 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147667 2558 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 20 02:45:05.188995 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147673 2558 flags.go:64] FLAG: --cgroup-root="" Jul 20 02:45:05.188995 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147678 2558 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 20 02:45:05.188995 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147682 2558 flags.go:64] FLAG: --client-ca-file="" Jul 20 02:45:05.188995 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147688 2558 flags.go:64] FLAG: --cloud-provider="external" Jul 20 02:45:05.188995 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147692 2558 flags.go:64] FLAG: --cluster-dns="[]" Jul 20 02:45:05.188995 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147698 2558 flags.go:64] FLAG: --cluster-domain="" Jul 20 02:45:05.188995 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147703 2558 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 20 02:45:05.188995 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147707 2558 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 20 02:45:05.188995 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147713 2558 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 20 02:45:05.188995 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147717 2558 flags.go:64] FLAG: --container-log-max-files="5" Jul 20 02:45:05.188995 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147723 2558 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 20 02:45:05.188995 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147727 2558 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 20 02:45:05.188995 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147732 2558 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 20 02:45:05.188995 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147737 2558 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 20 02:45:05.188995 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147742 2558 flags.go:64] FLAG: --contention-profiling="false" Jul 20 02:45:05.188995 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147746 2558 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 20 02:45:05.188995 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147750 2558 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 20 02:45:05.189660 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147755 2558 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 20 02:45:05.189660 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147759 2558 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 20 02:45:05.189660 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147766 2558 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 20 02:45:05.189660 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147771 2558 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 20 02:45:05.189660 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147775 2558 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 20 02:45:05.189660 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147780 2558 flags.go:64] FLAG: --enable-load-reader="false" Jul 20 02:45:05.189660 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147784 2558 flags.go:64] FLAG: --enable-server="true" Jul 20 02:45:05.189660 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147789 2558 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 20 02:45:05.189660 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147795 2558 flags.go:64] FLAG: --event-burst="100" Jul 20 02:45:05.189660 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147799 2558 flags.go:64] FLAG: --event-qps="50" Jul 20 02:45:05.189660 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147804 2558 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 20 02:45:05.189660 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147808 2558 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 20 02:45:05.189660 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147813 2558 flags.go:64] FLAG: --eviction-hard="" Jul 20 02:45:05.189660 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147818 2558 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 20 02:45:05.189660 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147822 2558 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 20 02:45:05.189660 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147829 2558 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 20 02:45:05.189660 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147833 2558 flags.go:64] FLAG: --eviction-soft="" Jul 20 02:45:05.189660 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147837 2558 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 20 02:45:05.189660 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147842 2558 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 20 02:45:05.189660 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147847 2558 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 20 02:45:05.189660 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147851 2558 flags.go:64] FLAG: --experimental-mounter-path="" Jul 20 02:45:05.189660 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147856 2558 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 20 02:45:05.189660 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147860 2558 flags.go:64] FLAG: --fail-swap-on="true" Jul 20 02:45:05.189660 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147864 2558 flags.go:64] FLAG: --feature-gates="" Jul 20 02:45:05.189660 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147870 2558 flags.go:64] FLAG: --file-check-frequency="20s" Jul 20 02:45:05.190385 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147874 2558 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 20 02:45:05.190385 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147879 2558 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 20 02:45:05.190385 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147883 2558 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 20 02:45:05.190385 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147888 2558 flags.go:64] FLAG: --healthz-port="10248" Jul 20 02:45:05.190385 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147892 2558 flags.go:64] FLAG: --help="false" Jul 20 02:45:05.190385 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147896 2558 flags.go:64] FLAG: --hostname-override="ip-10-0-142-139.ec2.internal" Jul 20 02:45:05.190385 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147901 2558 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 20 02:45:05.190385 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147905 2558 flags.go:64] FLAG: --http-check-frequency="20s" Jul 20 02:45:05.190385 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147910 2558 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 20 02:45:05.190385 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147914 2558 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 20 02:45:05.190385 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147919 2558 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 20 02:45:05.190385 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147923 2558 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 20 02:45:05.190385 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147928 2558 flags.go:64] FLAG: --image-service-endpoint="" Jul 20 02:45:05.190385 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147932 2558 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 20 02:45:05.190385 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147936 2558 flags.go:64] FLAG: --kube-api-burst="100" Jul 20 02:45:05.190385 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147941 2558 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 20 02:45:05.190385 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147946 2558 flags.go:64] FLAG: --kube-api-qps="50" Jul 20 02:45:05.190385 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147950 2558 flags.go:64] FLAG: --kube-reserved="" Jul 20 02:45:05.190385 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147955 2558 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 20 02:45:05.190385 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147959 2558 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 20 02:45:05.190385 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147963 2558 flags.go:64] FLAG: --kubelet-cgroups="" Jul 20 02:45:05.190385 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147968 2558 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 20 02:45:05.190385 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147973 2558 flags.go:64] FLAG: --lock-file="" Jul 20 02:45:05.190385 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147977 2558 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 20 02:45:05.191264 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147982 2558 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 20 02:45:05.191264 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147987 2558 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 20 02:45:05.191264 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.147996 2558 flags.go:64] FLAG: --log-json-split-stream="false" Jul 20 02:45:05.191264 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148000 2558 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 20 02:45:05.191264 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148004 2558 flags.go:64] FLAG: --log-text-split-stream="false" Jul 20 02:45:05.191264 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148008 2558 flags.go:64] FLAG: --logging-format="text" Jul 20 02:45:05.191264 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148013 2558 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 20 02:45:05.191264 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148018 2558 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 20 02:45:05.191264 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148022 2558 flags.go:64] FLAG: --manifest-url="" Jul 20 02:45:05.191264 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148026 2558 flags.go:64] FLAG: --manifest-url-header="" Jul 20 02:45:05.191264 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148033 2558 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 20 02:45:05.191264 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148037 2558 flags.go:64] FLAG: --max-open-files="1000000" Jul 20 02:45:05.191264 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148043 2558 flags.go:64] FLAG: --max-pods="110" Jul 20 02:45:05.191264 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148047 2558 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 20 02:45:05.191264 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148051 2558 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 20 02:45:05.191264 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148056 2558 flags.go:64] FLAG: --memory-manager-policy="None" Jul 20 02:45:05.191264 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148060 2558 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 20 02:45:05.191264 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148064 2558 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 20 02:45:05.191264 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148069 2558 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 20 02:45:05.191264 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148073 2558 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 20 02:45:05.191264 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148083 2558 flags.go:64] FLAG: --node-status-max-images="50" Jul 20 02:45:05.191264 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148088 2558 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 20 02:45:05.191264 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148108 2558 flags.go:64] FLAG: --oom-score-adj="-999" Jul 20 02:45:05.191264 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148113 2558 flags.go:64] FLAG: --pod-cidr="" Jul 20 02:45:05.192445 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148117 2558 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 20 02:45:05.192445 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148125 2558 flags.go:64] FLAG: --pod-manifest-path="" Jul 20 02:45:05.192445 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148130 2558 flags.go:64] FLAG: --pod-max-pids="-1" Jul 20 02:45:05.192445 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148134 2558 flags.go:64] FLAG: --pods-per-core="0" Jul 20 02:45:05.192445 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148139 2558 flags.go:64] FLAG: --port="10250" Jul 20 02:45:05.192445 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148143 2558 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 20 02:45:05.192445 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148149 2558 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ff6bd278045c232e" Jul 20 02:45:05.192445 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148154 2558 flags.go:64] FLAG: --qos-reserved="" Jul 20 02:45:05.192445 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148158 2558 flags.go:64] FLAG: --read-only-port="10255" Jul 20 02:45:05.192445 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148163 2558 flags.go:64] FLAG: --register-node="true" Jul 20 02:45:05.192445 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148168 2558 flags.go:64] FLAG: --register-with-taints="" Jul 20 02:45:05.192445 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148173 2558 flags.go:64] FLAG: --registry-burst="10" Jul 20 02:45:05.192445 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148177 2558 flags.go:64] FLAG: --registry-qps="5" Jul 20 02:45:05.192445 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148182 2558 flags.go:64] FLAG: --reserved-cpus="" Jul 20 02:45:05.192445 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148186 2558 flags.go:64] FLAG: --reserved-memory="" Jul 20 02:45:05.192445 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148192 2558 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 20 02:45:05.192445 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148196 2558 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 20 02:45:05.192445 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148201 2558 flags.go:64] FLAG: --rotate-certificates="false" Jul 20 02:45:05.192445 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148205 2558 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 20 02:45:05.192445 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148209 2558 flags.go:64] FLAG: --runonce="false" Jul 20 02:45:05.192445 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148213 2558 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 20 02:45:05.192445 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148217 2558 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 20 02:45:05.192445 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148222 2558 flags.go:64] FLAG: --seccomp-default="false" Jul 20 02:45:05.192445 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148226 2558 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 20 02:45:05.192445 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148230 2558 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 20 02:45:05.193482 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148235 2558 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 20 02:45:05.193482 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148239 2558 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 20 02:45:05.193482 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148244 2558 flags.go:64] FLAG: --storage-driver-password="root" Jul 20 02:45:05.193482 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148248 2558 flags.go:64] FLAG: --storage-driver-secure="false" Jul 20 02:45:05.193482 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148253 2558 flags.go:64] FLAG: --storage-driver-table="stats" Jul 20 02:45:05.193482 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148257 2558 flags.go:64] FLAG: --storage-driver-user="root" Jul 20 02:45:05.193482 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148262 2558 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 20 02:45:05.193482 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148267 2558 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 20 02:45:05.193482 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148272 2558 flags.go:64] FLAG: --system-cgroups="" Jul 20 02:45:05.193482 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148275 2558 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 20 02:45:05.193482 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148283 2558 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 20 02:45:05.193482 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148288 2558 flags.go:64] FLAG: --tls-cert-file="" Jul 20 02:45:05.193482 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148292 2558 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 20 02:45:05.193482 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148299 2558 flags.go:64] FLAG: --tls-min-version="" Jul 20 02:45:05.193482 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148304 2558 flags.go:64] FLAG: --tls-private-key-file="" Jul 20 02:45:05.193482 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148308 2558 flags.go:64] FLAG: --topology-manager-policy="none" Jul 20 02:45:05.193482 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148312 2558 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 20 02:45:05.193482 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148318 2558 flags.go:64] FLAG: --topology-manager-scope="container" Jul 20 02:45:05.193482 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148322 2558 flags.go:64] FLAG: --v="2" Jul 20 02:45:05.193482 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148328 2558 flags.go:64] FLAG: --version="false" Jul 20 02:45:05.193482 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148334 2558 flags.go:64] FLAG: --vmodule="" Jul 20 02:45:05.193482 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148340 2558 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 20 02:45:05.193482 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.148344 2558 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 20 02:45:05.193482 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.148915 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 02:45:05.193482 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.148920 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 02:45:05.194287 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.148925 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 02:45:05.194287 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.148929 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 02:45:05.194287 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.148934 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 02:45:05.194287 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.148938 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 02:45:05.194287 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.148942 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 02:45:05.194287 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.148945 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 02:45:05.194287 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.148949 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 02:45:05.194287 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.148953 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 02:45:05.194287 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.148957 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 02:45:05.194287 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.148961 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 02:45:05.194287 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.148965 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 02:45:05.194287 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.148969 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 02:45:05.194287 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.148972 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 02:45:05.194287 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.148976 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 02:45:05.194287 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.148981 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 02:45:05.194287 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.148985 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 02:45:05.194287 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.148989 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 02:45:05.194287 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.148992 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 02:45:05.194287 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.148996 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 02:45:05.194287 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149000 2558 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 02:45:05.194888 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149005 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 02:45:05.194888 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149011 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 02:45:05.194888 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149016 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 02:45:05.194888 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149020 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 02:45:05.194888 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149026 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 02:45:05.194888 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149030 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 02:45:05.194888 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149034 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 02:45:05.194888 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149038 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 02:45:05.194888 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149042 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 02:45:05.194888 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149045 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 02:45:05.194888 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149049 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 02:45:05.194888 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149053 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 02:45:05.194888 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149057 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 02:45:05.194888 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149060 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 02:45:05.194888 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149064 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 02:45:05.194888 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149068 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 02:45:05.194888 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149071 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 02:45:05.194888 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149075 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 02:45:05.194888 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149079 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 02:45:05.195486 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149083 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 02:45:05.195486 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149087 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 02:45:05.195486 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149091 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 02:45:05.195486 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149108 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 02:45:05.195486 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149112 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 02:45:05.195486 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149116 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 02:45:05.195486 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149120 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 02:45:05.195486 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149124 2558 feature_gate.go:328] unrecognized feature gate: Example Jul 20 02:45:05.195486 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149129 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 02:45:05.195486 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149133 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 02:45:05.195486 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149136 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 02:45:05.195486 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149140 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 02:45:05.195486 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149144 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 02:45:05.195486 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149149 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 02:45:05.195486 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149153 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 02:45:05.195486 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149157 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 02:45:05.195486 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149161 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 02:45:05.195486 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149166 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 02:45:05.195486 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149170 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 02:45:05.195486 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149174 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 02:45:05.196440 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149177 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 02:45:05.196440 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149181 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 02:45:05.196440 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149185 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 02:45:05.196440 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149189 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 02:45:05.196440 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149193 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 02:45:05.196440 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149197 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 02:45:05.196440 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149201 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 02:45:05.196440 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149204 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 02:45:05.196440 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149208 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 02:45:05.196440 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149212 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 02:45:05.196440 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149218 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 02:45:05.196440 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149222 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 02:45:05.196440 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149227 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 02:45:05.196440 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149231 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 02:45:05.196440 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149234 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 02:45:05.196440 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149238 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 02:45:05.196440 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149242 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 02:45:05.196440 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149246 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 02:45:05.196440 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149249 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 02:45:05.196440 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149253 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 02:45:05.365897 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149257 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 02:45:05.365897 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149261 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 02:45:05.365897 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149265 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 02:45:05.365897 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149268 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 02:45:05.365897 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149272 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 02:45:05.365897 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149278 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 02:45:05.365897 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149281 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 02:45:05.365897 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149285 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 02:45:05.365897 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149289 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 02:45:05.365897 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149293 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 02:45:05.365897 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149297 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 02:45:05.365897 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149301 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 02:45:05.365897 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149304 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 02:45:05.365897 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149308 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 02:45:05.365897 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149312 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 02:45:05.365897 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149317 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 02:45:05.365897 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149323 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 02:45:05.365897 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.149327 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 02:45:05.229713 ip-10-0-142-139 systemd[1]: Started Kubernetes Kubelet. Jul 20 02:45:05.389642 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.149334 2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 20 02:45:05.389642 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.158487 2558 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 20 02:45:05.389642 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.158504 2558 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 20 02:45:05.389642 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.159440 2558 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 20 02:45:05.389642 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.159448 2558 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 20 02:45:05.389642 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159523 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 02:45:05.389642 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159530 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 02:45:05.389642 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159535 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 02:45:05.389642 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159539 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 02:45:05.389642 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159543 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 02:45:05.389642 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159547 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 02:45:05.389642 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159552 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 02:45:05.389642 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159555 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 02:45:05.389642 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159560 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 02:45:05.391522 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159563 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 02:45:05.391522 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159567 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 02:45:05.391522 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159570 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 02:45:05.391522 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159575 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 02:45:05.391522 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159579 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 02:45:05.391522 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159586 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 02:45:05.391522 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159589 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 02:45:05.391522 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159593 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 02:45:05.391522 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159597 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 02:45:05.391522 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159601 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 02:45:05.391522 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159607 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 02:45:05.391522 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159611 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 02:45:05.391522 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159615 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 02:45:05.391522 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159620 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 02:45:05.391522 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159624 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 02:45:05.391522 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159628 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 02:45:05.391522 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159631 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 02:45:05.391522 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159635 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 02:45:05.391522 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159639 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 02:45:05.391522 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159643 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 02:45:05.392340 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159649 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 02:45:05.392340 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159653 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 02:45:05.392340 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159658 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 02:45:05.392340 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159663 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 02:45:05.392340 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159667 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 02:45:05.392340 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159671 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 02:45:05.392340 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159675 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 02:45:05.392340 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159680 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 02:45:05.392340 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159684 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 02:45:05.392340 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159687 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 02:45:05.392340 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159691 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 02:45:05.392340 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159695 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 02:45:05.392340 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159699 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 02:45:05.392340 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159702 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 02:45:05.392340 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159706 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 02:45:05.392340 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159710 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 02:45:05.392340 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159713 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 02:45:05.392340 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159719 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 02:45:05.392340 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159723 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 02:45:05.392340 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159726 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 02:45:05.393188 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159730 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 02:45:05.393188 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159734 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 02:45:05.393188 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159740 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 02:45:05.393188 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159744 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 02:45:05.393188 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159748 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 02:45:05.393188 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159752 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 02:45:05.393188 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159755 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 02:45:05.393188 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159759 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 02:45:05.393188 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159763 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 02:45:05.393188 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159767 2558 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 02:45:05.393188 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159771 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 02:45:05.393188 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159775 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 02:45:05.393188 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159778 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 02:45:05.393188 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159782 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 02:45:05.393188 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159787 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 02:45:05.393188 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159790 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 02:45:05.393188 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159794 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 02:45:05.393188 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159799 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 02:45:05.393188 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159802 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 02:45:05.393986 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159807 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 02:45:05.393986 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159811 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 02:45:05.393986 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159815 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 02:45:05.393986 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159818 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 02:45:05.393986 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159822 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 02:45:05.393986 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159828 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 02:45:05.393986 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159833 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 02:45:05.393986 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159837 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 02:45:05.393986 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159842 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 02:45:05.393986 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159846 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 02:45:05.393986 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159851 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 02:45:05.393986 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159855 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 02:45:05.393986 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159860 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 02:45:05.393986 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159863 2558 feature_gate.go:328] unrecognized feature gate: Example Jul 20 02:45:05.393986 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159868 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 02:45:05.393986 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159873 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 02:45:05.393986 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159877 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 02:45:05.393986 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159881 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 02:45:05.393986 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159884 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 02:45:05.393986 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159888 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 02:45:05.394859 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159892 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 02:45:05.394859 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159896 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 02:45:05.394859 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159900 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 02:45:05.394859 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159904 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 02:45:05.394859 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159907 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 02:45:05.394859 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159911 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 02:45:05.394859 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159915 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 02:45:05.394859 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159920 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 02:45:05.394859 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159926 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 02:45:05.394859 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159930 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 02:45:05.394859 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.159934 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 02:45:05.394859 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.159941 2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 20 02:45:05.394859 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160159 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 02:45:05.394859 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160168 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 02:45:05.394859 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160172 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 02:45:05.395690 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160176 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 02:45:05.395690 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160181 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 02:45:05.395690 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160185 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 02:45:05.395690 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160188 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 02:45:05.395690 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160192 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 02:45:05.395690 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160196 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 02:45:05.395690 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160200 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 02:45:05.395690 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160206 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 02:45:05.395690 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160209 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 02:45:05.395690 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160214 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 02:45:05.395690 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160218 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 02:45:05.395690 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160221 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 02:45:05.395690 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160225 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 02:45:05.395690 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160231 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 02:45:05.395690 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160235 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 02:45:05.395690 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160239 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 02:45:05.395690 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160242 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 02:45:05.395690 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160247 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 02:45:05.395690 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160250 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 02:45:05.395690 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160254 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 02:45:05.396507 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160257 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 02:45:05.396507 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160261 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 02:45:05.396507 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160265 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 02:45:05.396507 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160269 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 02:45:05.396507 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160272 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 02:45:05.396507 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160276 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 02:45:05.396507 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160279 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 02:45:05.396507 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160283 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 02:45:05.396507 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160287 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 02:45:05.396507 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160290 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 02:45:05.396507 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160296 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 02:45:05.396507 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160301 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 02:45:05.396507 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160305 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 02:45:05.396507 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160309 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 02:45:05.396507 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160313 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 02:45:05.396507 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160317 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 02:45:05.396507 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160321 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 02:45:05.396507 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160325 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 02:45:05.396507 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160329 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 02:45:05.396507 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160334 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 02:45:05.397366 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160337 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 02:45:05.397366 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160342 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 02:45:05.397366 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160346 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 02:45:05.397366 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160350 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 02:45:05.397366 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160354 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 02:45:05.397366 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160359 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 02:45:05.397366 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160362 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 02:45:05.397366 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160366 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 02:45:05.397366 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160370 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 02:45:05.397366 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160373 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 02:45:05.397366 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160377 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 02:45:05.397366 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160381 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 02:45:05.397366 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160384 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 02:45:05.397366 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160388 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 02:45:05.397366 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160391 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 02:45:05.397366 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160395 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 02:45:05.397366 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160398 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 02:45:05.397366 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160402 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 02:45:05.397366 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160407 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 02:45:05.397366 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160410 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 02:45:05.397995 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160414 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 02:45:05.397995 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160417 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 02:45:05.397995 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160421 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 02:45:05.397995 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160425 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 02:45:05.397995 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160430 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 02:45:05.397995 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160435 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 02:45:05.397995 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160439 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 02:45:05.397995 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160443 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 02:45:05.397995 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160447 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 02:45:05.397995 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160451 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 02:45:05.397995 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160456 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 02:45:05.397995 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160461 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 02:45:05.397995 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160464 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 02:45:05.397995 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160469 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 02:45:05.397995 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160473 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 02:45:05.397995 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160478 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 02:45:05.397995 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160482 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 02:45:05.397995 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160487 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 02:45:05.398675 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160491 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 02:45:05.398675 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160495 2558 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 02:45:05.398675 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160499 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 02:45:05.398675 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160503 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 02:45:05.398675 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160506 2558 feature_gate.go:328] unrecognized feature gate: Example Jul 20 02:45:05.398675 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160510 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 02:45:05.398675 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160513 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 02:45:05.398675 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160517 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 02:45:05.398675 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160521 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 02:45:05.398675 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160524 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 02:45:05.398675 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160528 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 02:45:05.398675 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160531 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 02:45:05.398675 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160535 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 02:45:05.398675 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160538 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 02:45:05.398675 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160542 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 02:45:05.398675 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160546 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 02:45:05.398675 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160549 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 02:45:05.398675 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:05.160553 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 02:45:05.399278 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.160559 2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 20 02:45:05.399278 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.160738 2558 server.go:962] "Client rotation is on, will bootstrap in background" Jul 20 02:45:05.399278 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.163963 2558 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 20 02:45:05.399278 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.165369 2558 server.go:1019] "Starting client certificate rotation" Jul 20 02:45:05.399278 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.165475 2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 20 02:45:05.399278 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.165514 2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 20 02:45:05.399278 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.167853 2558 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 20 02:45:05.399278 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.170817 2558 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 20 02:45:05.399278 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.180233 2558 log.go:25] "Validated CRI v1 runtime API" Jul 20 02:45:05.399278 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.182792 2558 log.go:25] "Validated CRI v1 image API" Jul 20 02:45:05.399278 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.185804 2558 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 20 02:45:05.399278 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.188856 2558 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a5c1df70-b972-4873-bacc-cab041b0e71a:/dev/nvme0n1p4 be0afcc4-4958-4d5f-8c14-511adaa86e3b:/dev/nvme0n1p3] Jul 20 02:45:05.399683 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.188875 2558 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 20 02:45:05.399787 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.199930 2558 manager.go:216] Machine: {Timestamp:2026-07-20 02:45:05.19838607 +0000 UTC m=+0.397635207 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103360 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec272d44ac110d9e00b91c862ce3d58d SystemUUID:ec272d44-ac11-0d9e-00b9-1c862ce3d58d BootID:7de32506-d976-46fe-b3ec-b883ff2d039c Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:5a:00:e7:5c:37 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:5a:00:e7:5c:37 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:da:a2:21:ee:46:21 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 20 02:45:05.399787 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.200082 2558 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 20 02:45:05.399787 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.200139 2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 20 02:45:05.399787 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.200291 2558 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 20 02:45:05.399787 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.203272 2558 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 20 02:45:05.399787 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.203296 2558 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-139.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 20 02:45:05.399787 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.204056 2558 topology_manager.go:138] "Creating topology manager with none policy" Jul 20 02:45:05.400083 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.204065 2558 container_manager_linux.go:309] "Creating device plugin manager" Jul 20 02:45:05.400083 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.204073 2558 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 20 02:45:05.400083 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.204732 2558 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 20 02:45:05.400083 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.205801 2558 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 20 02:45:05.400083 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.213077 2558 state_mem.go:36] "Initialized new in-memory state store" Jul 20 02:45:05.400083 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.213234 2558 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 20 02:45:05.400083 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.213269 2558 kubelet.go:486] "Attempting to sync node with API server" Jul 20 02:45:05.400083 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.213280 2558 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 20 02:45:05.400083 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.213292 2558 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 20 02:45:05.400083 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.213301 2558 kubelet.go:398] "Adding apiserver pod source" Jul 20 02:45:05.400083 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.213310 2558 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 20 02:45:05.400083 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.213692 2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 20 02:45:05.400083 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.213712 2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 20 02:45:05.400083 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.215942 2558 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 20 02:45:05.400083 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.216651 2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mdvx9" Jul 20 02:45:05.400083 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.217786 2558 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 20 02:45:05.400083 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.217825 2558 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 20 02:45:05.400083 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.219351 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 20 02:45:05.400083 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.219366 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 20 02:45:05.400083 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.219373 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 20 02:45:05.400083 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.219381 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 20 02:45:05.400083 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.219393 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 20 02:45:05.400083 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.219398 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 20 02:45:05.400083 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.219406 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 20 02:45:05.400083 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.219412 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 20 02:45:05.400083 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.219419 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 20 02:45:05.400083 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.219424 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 20 02:45:05.400083 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.219440 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 20 02:45:05.400083 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.219446 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 20 02:45:05.400083 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.220244 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 20 02:45:05.400083 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.220252 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 20 02:45:05.400083 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.223536 2558 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-139.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 20 02:45:05.400083 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:05.223660 2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 20 02:45:05.400083 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.224746 2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mdvx9" Jul 20 02:45:05.400083 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:05.228307 2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-139.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 20 02:45:05.400083 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.228834 2558 server.go:1268] "Started kubelet" Jul 20 02:45:05.400083 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.228964 2558 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 20 02:45:05.400083 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.229015 2558 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 20 02:45:05.400083 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.228975 2558 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 20 02:45:05.400083 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.231390 2558 server.go:310] "Adding debug handlers to kubelet server" Jul 20 02:45:05.401206 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:05.232633 2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-139.ec2.internal.18c3df65b156ff28 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-139.ec2.internal,UID:ip-10-0-142-139.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-139.ec2.internal,},FirstTimestamp:2026-07-20 02:45:05.228807976 +0000 UTC m=+0.428057114,LastTimestamp:2026-07-20 02:45:05.228807976 +0000 UTC m=+0.428057114,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-139.ec2.internal,}" Jul 20 02:45:05.401206 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.235953 2558 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 20 02:45:05.401206 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:05.237691 2558 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 20 02:45:05.401206 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.245929 2558 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 20 02:45:05.401206 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.245949 2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 20 02:45:05.401206 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:05.246132 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-139.ec2.internal\" not found" Jul 20 02:45:05.401206 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.246474 2558 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 20 02:45:05.401206 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.246480 2558 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 20 02:45:05.401206 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.246494 2558 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 20 02:45:05.401206 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.246653 2558 reconstruct.go:97] "Volume reconstruction finished" Jul 20 02:45:05.401206 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.246663 2558 reconciler.go:29] "Reconciler: start to sync state" Jul 20 02:45:05.401206 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.246810 2558 factory.go:55] Registering systemd factory Jul 20 02:45:05.401206 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.246825 2558 factory.go:223] Registration of the systemd container factory successfully Jul 20 02:45:05.401206 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.247036 2558 factory.go:153] Registering CRI-O factory Jul 20 02:45:05.401206 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.247047 2558 factory.go:223] Registration of the crio container factory successfully Jul 20 02:45:05.401206 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.247116 2558 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 20 02:45:05.401206 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.247137 2558 factory.go:103] Registering Raw factory Jul 20 02:45:05.401206 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.247151 2558 manager.go:1196] Started watching for new ooms in manager Jul 20 02:45:05.401206 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.247534 2558 manager.go:349] Starting recovery of all containers Jul 20 02:45:05.401206 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.257310 2558 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 02:45:05.401206 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.260450 2558 manager.go:354] Recovery completed Jul 20 02:45:05.401206 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:05.261271 2558 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-142-139.ec2.internal\" not found" node="ip-10-0-142-139.ec2.internal" Jul 20 02:45:05.401206 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.264797 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 02:45:05.401206 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.267160 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasSufficientMemory" Jul 20 02:45:05.401206 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.267211 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 02:45:05.401206 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.267232 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasSufficientPID" Jul 20 02:45:05.401206 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.267875 2558 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 20 02:45:05.401206 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.267890 2558 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 20 02:45:05.401206 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.267910 2558 state_mem.go:36] "Initialized new in-memory state store" Jul 20 02:45:05.401206 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.268381 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 02:45:05.401206 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.269255 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasSufficientMemory" Jul 20 02:45:05.401206 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.269287 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 02:45:05.401206 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.269301 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasSufficientPID" Jul 20 02:45:05.401206 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.270113 2558 policy_none.go:49] "None policy: Start" Jul 20 02:45:05.401206 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.270127 2558 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 20 02:45:05.402313 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.270140 2558 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 20 02:45:05.402313 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.272231 2558 policy_none.go:47] "Start" Jul 20 02:45:05.402313 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.317696 2558 manager.go:337] "Starting Device Plugin manager" Jul 20 02:45:05.402313 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:05.317873 2558 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 20 02:45:05.402313 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.317886 2558 server.go:87] "Starting device plugin registration server" Jul 20 02:45:05.402313 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.318149 2558 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 20 02:45:05.402313 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.318160 2558 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 20 02:45:05.402313 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.318259 2558 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 20 02:45:05.402313 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.318362 2558 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 20 02:45:05.402313 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.318371 2558 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 20 02:45:05.402313 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:05.318934 2558 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 20 02:45:05.402313 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:05.318999 2558 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-139.ec2.internal\" not found" Jul 20 02:45:05.517704 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.416539 2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 20 02:45:05.517704 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.417810 2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 20 02:45:05.517704 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.417856 2558 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 20 02:45:05.517704 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.417884 2558 kubelet.go:2448] "Starting kubelet main sync loop" Jul 20 02:45:05.517704 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:05.417926 2558 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 20 02:45:05.517704 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.418255 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 02:45:05.517704 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.419768 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasSufficientMemory" Jul 20 02:45:05.517704 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.419797 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 02:45:05.517704 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.419810 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasSufficientPID" Jul 20 02:45:05.517704 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.419832 2558 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-139.ec2.internal" Jul 20 02:45:05.517704 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.420430 2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 02:45:05.517704 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.427677 2558 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-139.ec2.internal" Jul 20 02:45:05.517704 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:05.427700 2558 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-139.ec2.internal\": node \"ip-10-0-142-139.ec2.internal\" not found" Jul 20 02:45:05.517704 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:05.450549 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-139.ec2.internal\" not found" Jul 20 02:45:05.518860 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.518738 2558 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-139.ec2.internal"] Jul 20 02:45:05.518860 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.518812 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 02:45:05.640253 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.520450 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasSufficientMemory" Jul 20 02:45:05.640253 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.520481 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 02:45:05.640253 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.520494 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasSufficientPID" Jul 20 02:45:05.640253 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.522665 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 02:45:05.640253 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.522816 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal" Jul 20 02:45:05.640253 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.522841 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 02:45:05.640253 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.523427 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasSufficientMemory" Jul 20 02:45:05.640253 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.523450 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 02:45:05.640253 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.523460 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasSufficientPID" Jul 20 02:45:05.640253 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.523427 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasSufficientMemory" Jul 20 02:45:05.640253 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.523525 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 02:45:05.640253 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.523544 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasSufficientPID" Jul 20 02:45:05.640253 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.525593 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-139.ec2.internal" Jul 20 02:45:05.640253 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.525618 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 02:45:05.640253 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.526303 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasSufficientMemory" Jul 20 02:45:05.640253 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.526334 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 02:45:05.640253 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.526349 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeHasSufficientPID" Jul 20 02:45:05.640253 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.548020 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/39560757f7588014bf49cd290ec77fb5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal\" (UID: \"39560757f7588014bf49cd290ec77fb5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal" Jul 20 02:45:05.640253 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.548042 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39560757f7588014bf49cd290ec77fb5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal\" (UID: \"39560757f7588014bf49cd290ec77fb5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal" Jul 20 02:45:05.640253 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.548061 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ceae9fddccd84171bbba0a3e9c0148c8-config\") pod \"kube-apiserver-proxy-ip-10-0-142-139.ec2.internal\" (UID: \"ceae9fddccd84171bbba0a3e9c0148c8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-139.ec2.internal" Jul 20 02:45:05.640253 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:05.551124 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-139.ec2.internal\" not found" Jul 20 02:45:05.640253 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:05.557880 2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-139.ec2.internal\" not found" node="ip-10-0-142-139.ec2.internal" Jul 20 02:45:05.640253 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:05.562602 2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-139.ec2.internal\" not found" node="ip-10-0-142-139.ec2.internal" Jul 20 02:45:05.734424 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.648225 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/39560757f7588014bf49cd290ec77fb5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal\" (UID: \"39560757f7588014bf49cd290ec77fb5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal" Jul 20 02:45:05.734424 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.648253 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39560757f7588014bf49cd290ec77fb5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal\" (UID: \"39560757f7588014bf49cd290ec77fb5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal" Jul 20 02:45:05.734424 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.648269 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ceae9fddccd84171bbba0a3e9c0148c8-config\") pod \"kube-apiserver-proxy-ip-10-0-142-139.ec2.internal\" (UID: \"ceae9fddccd84171bbba0a3e9c0148c8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-139.ec2.internal" Jul 20 02:45:05.734424 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.648316 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39560757f7588014bf49cd290ec77fb5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal\" (UID: \"39560757f7588014bf49cd290ec77fb5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal" Jul 20 02:45:05.734424 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.648320 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/39560757f7588014bf49cd290ec77fb5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal\" (UID: \"39560757f7588014bf49cd290ec77fb5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal" Jul 20 02:45:05.734424 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.648359 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ceae9fddccd84171bbba0a3e9c0148c8-config\") pod \"kube-apiserver-proxy-ip-10-0-142-139.ec2.internal\" (UID: \"ceae9fddccd84171bbba0a3e9c0148c8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-139.ec2.internal" Jul 20 02:45:05.734424 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:05.651325 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-139.ec2.internal\" not found" Jul 20 02:45:05.800073 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:05.752129 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-139.ec2.internal\" not found" Jul 20 02:45:05.904502 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:05.852615 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-139.ec2.internal\" not found" Jul 20 02:45:05.904502 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.860806 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal" Jul 20 02:45:05.904502 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:05.864583 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-139.ec2.internal" Jul 20 02:45:05.953362 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:05.953327 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-139.ec2.internal\" not found" Jul 20 02:45:06.053891 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:06.053815 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-139.ec2.internal\" not found" Jul 20 02:45:06.154338 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:06.154304 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-139.ec2.internal\" not found" Jul 20 02:45:06.164467 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:06.164446 2558 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 20 02:45:06.164600 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:06.164584 2558 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 20 02:45:06.164633 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:06.164613 2558 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 20 02:45:06.229593 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:06.229547 2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-19 02:40:05 +0000 UTC" deadline="2028-03-15 14:47:16.745104631 +0000 UTC" Jul 20 02:45:06.229593 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:06.229587 2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14508h2m10.515519662s" Jul 20 02:45:06.255196 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:06.255169 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-139.ec2.internal\" not found" Jul 20 02:45:06.355850 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:06.355820 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-139.ec2.internal\" not found" Jul 20 02:45:06.456486 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:06.456454 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-139.ec2.internal\" not found" Jul 20 02:45:06.457465 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:06.457415 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceae9fddccd84171bbba0a3e9c0148c8.slice/crio-2f112a23dcb66f6fbc7e0dd54edf27ed922994abe0211795d922d3aee20dd943 WatchSource:0}: Error finding container 2f112a23dcb66f6fbc7e0dd54edf27ed922994abe0211795d922d3aee20dd943: Status 404 returned error can't find the container with id 2f112a23dcb66f6fbc7e0dd54edf27ed922994abe0211795d922d3aee20dd943 Jul 20 02:45:06.457940 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:06.457924 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39560757f7588014bf49cd290ec77fb5.slice/crio-fbdfa133811ee41db83e3cf3a8c8cdeda25412f19129415882eeb04d0725fa2b WatchSource:0}: Error finding container fbdfa133811ee41db83e3cf3a8c8cdeda25412f19129415882eeb04d0725fa2b: Status 404 returned error can't find the container with id fbdfa133811ee41db83e3cf3a8c8cdeda25412f19129415882eeb04d0725fa2b Jul 20 02:45:06.462598 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:06.462582 2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 20 02:45:06.556959 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:06.556921 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-139.ec2.internal\" not found" Jul 20 02:45:06.600950 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:06.600876 2558 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 02:45:06.647155 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:06.647125 2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal" Jul 20 02:45:06.657558 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:06.657530 2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 20 02:45:06.659702 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:06.659684 2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-139.ec2.internal" Jul 20 02:45:06.667735 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:06.667718 2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 20 02:45:06.755195 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:06.755166 2558 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 02:45:07.214203 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.214165 2558 apiserver.go:52] "Watching apiserver" Jul 20 02:45:07.223044 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.223016 2558 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 20 02:45:07.223502 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.223455 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-d5bdr","openshift-multus/network-metrics-daemon-9x7hv","openshift-network-diagnostics/network-check-target-5cxtl","openshift-network-operator/iptables-alerter-4kbnx","kube-system/global-pull-secret-syncer-t9spk","kube-system/konnectivity-agent-mvvbt","kube-system/kube-apiserver-proxy-ip-10-0-142-139.ec2.internal","openshift-dns/node-resolver-lvkqp","openshift-image-registry/node-ca-9qxnl","openshift-multus/multus-5jp8t","openshift-ovn-kubernetes/ovnkube-node-n4b7k","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd68h","openshift-cluster-node-tuning-operator/tuned-pw2tj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal"] Jul 20 02:45:07.226161 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.226138 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5jp8t" Jul 20 02:45:07.228430 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.228407 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pw2tj" Jul 20 02:45:07.229209 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.229189 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 20 02:45:07.229300 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.229245 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 20 02:45:07.229365 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.229191 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 20 02:45:07.229435 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.229414 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 20 02:45:07.229542 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.229526 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-2bvlw\"" Jul 20 02:45:07.230475 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.230455 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cxtl" Jul 20 02:45:07.231017 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:07.230526 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5cxtl" podUID="26701b30-ec4b-4dec-80df-b483d7edb77d" Jul 20 02:45:07.232234 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.232208 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 20 02:45:07.232555 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.232531 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-99tmw\"" Jul 20 02:45:07.232643 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.232619 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 20 02:45:07.232802 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.232784 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4kbnx" Jul 20 02:45:07.234784 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.234765 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t9spk" Jul 20 02:45:07.234914 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:07.234828 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t9spk" podUID="6c7d134c-4e38-481c-8a81-e03434badf7d" Jul 20 02:45:07.235202 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.235170 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 20 02:45:07.235389 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.235351 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 20 02:45:07.235493 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.235464 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 20 02:45:07.235647 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.235628 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zb46q\"" Jul 20 02:45:07.237445 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.237048 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mvvbt" Jul 20 02:45:07.239421 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.239402 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qvv76\"" Jul 20 02:45:07.239633 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.239615 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 20 02:45:07.239772 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.239761 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 20 02:45:07.239932 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.239919 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9qxnl" Jul 20 02:45:07.241839 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.241529 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lvkqp" Jul 20 02:45:07.242614 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.242300 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 20 02:45:07.242614 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.242476 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 20 02:45:07.242614 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.242479 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qzrcz\"" Jul 20 02:45:07.242614 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.242547 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 20 02:45:07.244315 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.243915 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 20 02:45:07.244315 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.244157 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8hgtf\"" Jul 20 02:45:07.244456 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.244413 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 20 02:45:07.245758 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.245538 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9x7hv" Jul 20 02:45:07.245758 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:07.245603 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9x7hv" podUID="d70f5103-7dc9-4279-9781-38e395da435b" Jul 20 02:45:07.247109 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.247073 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d5bdr" Jul 20 02:45:07.247501 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.247470 2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 20 02:45:07.249007 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.248883 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.251389 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.251353 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 20 02:45:07.253185 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.252085 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd68h" Jul 20 02:45:07.257158 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.253639 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qzlwp\"" Jul 20 02:45:07.257158 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.253873 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 20 02:45:07.257158 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.254145 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 20 02:45:07.257158 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.254549 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vpwgg\"" Jul 20 02:45:07.257158 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.255600 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 20 02:45:07.257158 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.255797 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 20 02:45:07.257158 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.256217 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 20 02:45:07.257158 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.256436 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 20 02:45:07.257158 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.256641 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 20 02:45:07.257158 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.256894 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 20 02:45:07.257158 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.257139 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ttgmd\"" Jul 20 02:45:07.257653 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.257354 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 20 02:45:07.257653 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.257571 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 20 02:45:07.259634 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.259380 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1b8f7019-0b80-42f2-be36-db424e632410-os-release\") pod \"multus-5jp8t\" (UID: \"1b8f7019-0b80-42f2-be36-db424e632410\") " pod="openshift-multus/multus-5jp8t" Jul 20 02:45:07.259634 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.259415 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt4vl\" (UniqueName: \"kubernetes.io/projected/02007d87-46f6-4589-ac19-d1a89cf5465c-kube-api-access-jt4vl\") pod \"iptables-alerter-4kbnx\" (UID: \"02007d87-46f6-4589-ac19-d1a89cf5465c\") " pod="openshift-network-operator/iptables-alerter-4kbnx" Jul 20 02:45:07.259634 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.259439 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6b7f\" (UniqueName: \"kubernetes.io/projected/504cb1f0-841a-49f7-add8-2d4742a1cb68-kube-api-access-w6b7f\") pod \"node-ca-9qxnl\" (UID: \"504cb1f0-841a-49f7-add8-2d4742a1cb68\") " pod="openshift-image-registry/node-ca-9qxnl" Jul 20 02:45:07.259634 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.259474 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90740648-1f62-4e9d-a8af-67716403522e-etc-kubernetes\") pod \"tuned-pw2tj\" (UID: \"90740648-1f62-4e9d-a8af-67716403522e\") " pod="openshift-cluster-node-tuning-operator/tuned-pw2tj" Jul 20 02:45:07.259634 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.259507 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cb6c0d96-0866-493d-9e19-7f63dff241ad-host-cni-bin\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.259634 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.259528 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6c7d134c-4e38-481c-8a81-e03434badf7d-original-pull-secret\") pod \"global-pull-secret-syncer-t9spk\" (UID: \"6c7d134c-4e38-481c-8a81-e03434badf7d\") " pod="kube-system/global-pull-secret-syncer-t9spk" Jul 20 02:45:07.259634 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.259549 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/504cb1f0-841a-49f7-add8-2d4742a1cb68-serviceca\") pod \"node-ca-9qxnl\" (UID: \"504cb1f0-841a-49f7-add8-2d4742a1cb68\") " pod="openshift-image-registry/node-ca-9qxnl" Jul 20 02:45:07.259634 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.259591 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e8d00f9d-e63a-44a7-8a2c-268dc6787c75-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d5bdr\" (UID: \"e8d00f9d-e63a-44a7-8a2c-268dc6787c75\") " pod="openshift-multus/multus-additional-cni-plugins-d5bdr" Jul 20 02:45:07.259634 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.259622 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1b8f7019-0b80-42f2-be36-db424e632410-multus-daemon-config\") pod \"multus-5jp8t\" (UID: \"1b8f7019-0b80-42f2-be36-db424e632410\") " pod="openshift-multus/multus-5jp8t" Jul 20 02:45:07.260066 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.259644 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cb6c0d96-0866-493d-9e19-7f63dff241ad-systemd-units\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.260066 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.259665 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cb6c0d96-0866-493d-9e19-7f63dff241ad-ovnkube-config\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.260066 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.259699 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b8f7019-0b80-42f2-be36-db424e632410-host-var-lib-kubelet\") pod \"multus-5jp8t\" (UID: \"1b8f7019-0b80-42f2-be36-db424e632410\") " pod="openshift-multus/multus-5jp8t" Jul 20 02:45:07.260066 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.259726 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b8f7019-0b80-42f2-be36-db424e632410-system-cni-dir\") pod \"multus-5jp8t\" (UID: \"1b8f7019-0b80-42f2-be36-db424e632410\") " pod="openshift-multus/multus-5jp8t" Jul 20 02:45:07.260066 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.259749 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b8f7019-0b80-42f2-be36-db424e632410-host-var-lib-cni-bin\") pod \"multus-5jp8t\" (UID: \"1b8f7019-0b80-42f2-be36-db424e632410\") " pod="openshift-multus/multus-5jp8t" Jul 20 02:45:07.260066 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.259793 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/90740648-1f62-4e9d-a8af-67716403522e-etc-tuned\") pod \"tuned-pw2tj\" (UID: \"90740648-1f62-4e9d-a8af-67716403522e\") " pod="openshift-cluster-node-tuning-operator/tuned-pw2tj" Jul 20 02:45:07.260066 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.259823 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/02007d87-46f6-4589-ac19-d1a89cf5465c-iptables-alerter-script\") pod \"iptables-alerter-4kbnx\" (UID: \"02007d87-46f6-4589-ac19-d1a89cf5465c\") " pod="openshift-network-operator/iptables-alerter-4kbnx" Jul 20 02:45:07.260066 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.259849 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e8d00f9d-e63a-44a7-8a2c-268dc6787c75-cni-binary-copy\") pod \"multus-additional-cni-plugins-d5bdr\" (UID: \"e8d00f9d-e63a-44a7-8a2c-268dc6787c75\") " pod="openshift-multus/multus-additional-cni-plugins-d5bdr" Jul 20 02:45:07.260066 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.259864 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/90740648-1f62-4e9d-a8af-67716403522e-etc-sysctl-d\") pod \"tuned-pw2tj\" (UID: \"90740648-1f62-4e9d-a8af-67716403522e\") " pod="openshift-cluster-node-tuning-operator/tuned-pw2tj" Jul 20 02:45:07.260066 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.259879 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb6c0d96-0866-493d-9e19-7f63dff241ad-host-slash\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.260066 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.259894 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb6c0d96-0866-493d-9e19-7f63dff241ad-host-run-ovn-kubernetes\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.260066 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.259933 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b8f7019-0b80-42f2-be36-db424e632410-etc-kubernetes\") pod \"multus-5jp8t\" (UID: \"1b8f7019-0b80-42f2-be36-db424e632410\") " pod="openshift-multus/multus-5jp8t" Jul 20 02:45:07.260066 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.259951 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cb6c0d96-0866-493d-9e19-7f63dff241ad-run-systemd\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.260066 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.259965 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfk5c\" (UniqueName: \"kubernetes.io/projected/e8d00f9d-e63a-44a7-8a2c-268dc6787c75-kube-api-access-wfk5c\") pod \"multus-additional-cni-plugins-d5bdr\" (UID: \"e8d00f9d-e63a-44a7-8a2c-268dc6787c75\") " pod="openshift-multus/multus-additional-cni-plugins-d5bdr" Jul 20 02:45:07.260731 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.260000 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1b8f7019-0b80-42f2-be36-db424e632410-host-run-k8s-cni-cncf-io\") pod \"multus-5jp8t\" (UID: \"1b8f7019-0b80-42f2-be36-db424e632410\") " pod="openshift-multus/multus-5jp8t" Jul 20 02:45:07.260731 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.260026 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fht2c\" (UniqueName: \"kubernetes.io/projected/1b8f7019-0b80-42f2-be36-db424e632410-kube-api-access-fht2c\") pod \"multus-5jp8t\" (UID: \"1b8f7019-0b80-42f2-be36-db424e632410\") " pod="openshift-multus/multus-5jp8t" Jul 20 02:45:07.260731 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.260046 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1b8f7019-0b80-42f2-be36-db424e632410-multus-conf-dir\") pod \"multus-5jp8t\" (UID: \"1b8f7019-0b80-42f2-be36-db424e632410\") " pod="openshift-multus/multus-5jp8t" Jul 20 02:45:07.260731 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.260084 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/90740648-1f62-4e9d-a8af-67716403522e-etc-modprobe-d\") pod \"tuned-pw2tj\" (UID: \"90740648-1f62-4e9d-a8af-67716403522e\") " pod="openshift-cluster-node-tuning-operator/tuned-pw2tj" Jul 20 02:45:07.260731 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.260130 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c361af25-375d-4bff-bdb9-53f137cea0f6-tmp-dir\") pod \"node-resolver-lvkqp\" (UID: \"c361af25-375d-4bff-bdb9-53f137cea0f6\") " pod="openshift-dns/node-resolver-lvkqp" Jul 20 02:45:07.260731 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.260162 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cb6c0d96-0866-493d-9e19-7f63dff241ad-run-ovn\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.260731 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.260189 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cb6c0d96-0866-493d-9e19-7f63dff241ad-ovn-node-metrics-cert\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.260731 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.260215 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2be66185-38c5-43ac-ba3b-766093577e64-konnectivity-ca\") pod \"konnectivity-agent-mvvbt\" (UID: \"2be66185-38c5-43ac-ba3b-766093577e64\") " pod="kube-system/konnectivity-agent-mvvbt" Jul 20 02:45:07.260731 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.260238 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d70f5103-7dc9-4279-9781-38e395da435b-metrics-certs\") pod \"network-metrics-daemon-9x7hv\" (UID: \"d70f5103-7dc9-4279-9781-38e395da435b\") " pod="openshift-multus/network-metrics-daemon-9x7hv" Jul 20 02:45:07.260731 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.260260 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e8d00f9d-e63a-44a7-8a2c-268dc6787c75-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d5bdr\" (UID: \"e8d00f9d-e63a-44a7-8a2c-268dc6787c75\") " pod="openshift-multus/multus-additional-cni-plugins-d5bdr" Jul 20 02:45:07.260731 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.260283 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1b8f7019-0b80-42f2-be36-db424e632410-multus-socket-dir-parent\") pod \"multus-5jp8t\" (UID: \"1b8f7019-0b80-42f2-be36-db424e632410\") " pod="openshift-multus/multus-5jp8t" Jul 20 02:45:07.260731 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.260305 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/90740648-1f62-4e9d-a8af-67716403522e-etc-sysconfig\") pod \"tuned-pw2tj\" (UID: \"90740648-1f62-4e9d-a8af-67716403522e\") " pod="openshift-cluster-node-tuning-operator/tuned-pw2tj" Jul 20 02:45:07.260731 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.260327 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/90740648-1f62-4e9d-a8af-67716403522e-lib-modules\") pod \"tuned-pw2tj\" (UID: \"90740648-1f62-4e9d-a8af-67716403522e\") " pod="openshift-cluster-node-tuning-operator/tuned-pw2tj" Jul 20 02:45:07.260731 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.260349 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/90740648-1f62-4e9d-a8af-67716403522e-var-lib-kubelet\") pod \"tuned-pw2tj\" (UID: \"90740648-1f62-4e9d-a8af-67716403522e\") " pod="openshift-cluster-node-tuning-operator/tuned-pw2tj" Jul 20 02:45:07.261452 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.260372 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cb6c0d96-0866-493d-9e19-7f63dff241ad-node-log\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.261452 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.260395 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v964w\" (UniqueName: \"kubernetes.io/projected/d70f5103-7dc9-4279-9781-38e395da435b-kube-api-access-v964w\") pod \"network-metrics-daemon-9x7hv\" (UID: \"d70f5103-7dc9-4279-9781-38e395da435b\") " pod="openshift-multus/network-metrics-daemon-9x7hv" Jul 20 02:45:07.261452 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.260421 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e8d00f9d-e63a-44a7-8a2c-268dc6787c75-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d5bdr\" (UID: \"e8d00f9d-e63a-44a7-8a2c-268dc6787c75\") " pod="openshift-multus/multus-additional-cni-plugins-d5bdr" Jul 20 02:45:07.261452 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.260464 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/90740648-1f62-4e9d-a8af-67716403522e-etc-systemd\") pod \"tuned-pw2tj\" (UID: \"90740648-1f62-4e9d-a8af-67716403522e\") " pod="openshift-cluster-node-tuning-operator/tuned-pw2tj" Jul 20 02:45:07.261452 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.260487 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb6c0d96-0866-493d-9e19-7f63dff241ad-var-lib-openvswitch\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.261452 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.260509 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cb6c0d96-0866-493d-9e19-7f63dff241ad-log-socket\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.261452 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.260553 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2be66185-38c5-43ac-ba3b-766093577e64-agent-certs\") pod \"konnectivity-agent-mvvbt\" (UID: \"2be66185-38c5-43ac-ba3b-766093577e64\") " pod="kube-system/konnectivity-agent-mvvbt" Jul 20 02:45:07.261452 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.260576 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8d00f9d-e63a-44a7-8a2c-268dc6787c75-system-cni-dir\") pod \"multus-additional-cni-plugins-d5bdr\" (UID: \"e8d00f9d-e63a-44a7-8a2c-268dc6787c75\") " pod="openshift-multus/multus-additional-cni-plugins-d5bdr" Jul 20 02:45:07.261452 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.260592 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b8f7019-0b80-42f2-be36-db424e632410-multus-cni-dir\") pod \"multus-5jp8t\" (UID: \"1b8f7019-0b80-42f2-be36-db424e632410\") " pod="openshift-multus/multus-5jp8t" Jul 20 02:45:07.261452 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.260612 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb6c0d96-0866-493d-9e19-7f63dff241ad-host-kubelet\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.261452 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.260647 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb6c0d96-0866-493d-9e19-7f63dff241ad-run-openvswitch\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.261452 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.260675 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rsdj\" (UniqueName: \"kubernetes.io/projected/26701b30-ec4b-4dec-80df-b483d7edb77d-kube-api-access-4rsdj\") pod \"network-check-target-5cxtl\" (UID: \"26701b30-ec4b-4dec-80df-b483d7edb77d\") " pod="openshift-network-diagnostics/network-check-target-5cxtl" Jul 20 02:45:07.261452 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.260698 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90740648-1f62-4e9d-a8af-67716403522e-tmp\") pod \"tuned-pw2tj\" (UID: \"90740648-1f62-4e9d-a8af-67716403522e\") " pod="openshift-cluster-node-tuning-operator/tuned-pw2tj" Jul 20 02:45:07.262028 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.260727 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnj2l\" (UniqueName: \"kubernetes.io/projected/90740648-1f62-4e9d-a8af-67716403522e-kube-api-access-bnj2l\") pod \"tuned-pw2tj\" (UID: \"90740648-1f62-4e9d-a8af-67716403522e\") " pod="openshift-cluster-node-tuning-operator/tuned-pw2tj" Jul 20 02:45:07.262028 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.260747 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c361af25-375d-4bff-bdb9-53f137cea0f6-hosts-file\") pod \"node-resolver-lvkqp\" (UID: \"c361af25-375d-4bff-bdb9-53f137cea0f6\") " pod="openshift-dns/node-resolver-lvkqp" Jul 20 02:45:07.262028 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.260777 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/504cb1f0-841a-49f7-add8-2d4742a1cb68-host\") pod \"node-ca-9qxnl\" (UID: \"504cb1f0-841a-49f7-add8-2d4742a1cb68\") " pod="openshift-image-registry/node-ca-9qxnl" Jul 20 02:45:07.262028 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.260797 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e8d00f9d-e63a-44a7-8a2c-268dc6787c75-cnibin\") pod \"multus-additional-cni-plugins-d5bdr\" (UID: \"e8d00f9d-e63a-44a7-8a2c-268dc6787c75\") " pod="openshift-multus/multus-additional-cni-plugins-d5bdr" Jul 20 02:45:07.262028 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.260811 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1b8f7019-0b80-42f2-be36-db424e632410-host-run-multus-certs\") pod \"multus-5jp8t\" (UID: \"1b8f7019-0b80-42f2-be36-db424e632410\") " pod="openshift-multus/multus-5jp8t" Jul 20 02:45:07.262028 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.260877 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/90740648-1f62-4e9d-a8af-67716403522e-etc-sysctl-conf\") pod \"tuned-pw2tj\" (UID: \"90740648-1f62-4e9d-a8af-67716403522e\") " pod="openshift-cluster-node-tuning-operator/tuned-pw2tj" Jul 20 02:45:07.262028 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.260895 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/90740648-1f62-4e9d-a8af-67716403522e-run\") pod \"tuned-pw2tj\" (UID: \"90740648-1f62-4e9d-a8af-67716403522e\") " pod="openshift-cluster-node-tuning-operator/tuned-pw2tj" Jul 20 02:45:07.262028 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.260911 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cb6c0d96-0866-493d-9e19-7f63dff241ad-host-run-netns\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.262028 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.260941 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzxq4\" (UniqueName: \"kubernetes.io/projected/cb6c0d96-0866-493d-9e19-7f63dff241ad-kube-api-access-hzxq4\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.262028 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.261016 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1b8f7019-0b80-42f2-be36-db424e632410-cnibin\") pod \"multus-5jp8t\" (UID: \"1b8f7019-0b80-42f2-be36-db424e632410\") " pod="openshift-multus/multus-5jp8t" Jul 20 02:45:07.262028 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.261036 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b8f7019-0b80-42f2-be36-db424e632410-host-run-netns\") pod \"multus-5jp8t\" (UID: \"1b8f7019-0b80-42f2-be36-db424e632410\") " pod="openshift-multus/multus-5jp8t" Jul 20 02:45:07.262028 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.261057 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/90740648-1f62-4e9d-a8af-67716403522e-host\") pod \"tuned-pw2tj\" (UID: \"90740648-1f62-4e9d-a8af-67716403522e\") " pod="openshift-cluster-node-tuning-operator/tuned-pw2tj" Jul 20 02:45:07.262028 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.261090 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/02007d87-46f6-4589-ac19-d1a89cf5465c-host-slash\") pod \"iptables-alerter-4kbnx\" (UID: \"02007d87-46f6-4589-ac19-d1a89cf5465c\") " pod="openshift-network-operator/iptables-alerter-4kbnx" Jul 20 02:45:07.262028 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.261126 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cb6c0d96-0866-493d-9e19-7f63dff241ad-env-overrides\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.262747 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.261147 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/90740648-1f62-4e9d-a8af-67716403522e-sys\") pod \"tuned-pw2tj\" (UID: \"90740648-1f62-4e9d-a8af-67716403522e\") " pod="openshift-cluster-node-tuning-operator/tuned-pw2tj" Jul 20 02:45:07.262747 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.261169 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56h97\" (UniqueName: \"kubernetes.io/projected/c361af25-375d-4bff-bdb9-53f137cea0f6-kube-api-access-56h97\") pod \"node-resolver-lvkqp\" (UID: \"c361af25-375d-4bff-bdb9-53f137cea0f6\") " pod="openshift-dns/node-resolver-lvkqp" Jul 20 02:45:07.262747 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.261191 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb6c0d96-0866-493d-9e19-7f63dff241ad-etc-openvswitch\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.262747 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.261215 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb6c0d96-0866-493d-9e19-7f63dff241ad-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.262747 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.261261 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cb6c0d96-0866-493d-9e19-7f63dff241ad-ovnkube-script-lib\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.262747 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.261283 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6c7d134c-4e38-481c-8a81-e03434badf7d-kubelet-config\") pod \"global-pull-secret-syncer-t9spk\" (UID: \"6c7d134c-4e38-481c-8a81-e03434badf7d\") " pod="kube-system/global-pull-secret-syncer-t9spk" Jul 20 02:45:07.262747 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.261304 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6c7d134c-4e38-481c-8a81-e03434badf7d-dbus\") pod \"global-pull-secret-syncer-t9spk\" (UID: \"6c7d134c-4e38-481c-8a81-e03434badf7d\") " pod="kube-system/global-pull-secret-syncer-t9spk" Jul 20 02:45:07.262747 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.261325 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1b8f7019-0b80-42f2-be36-db424e632410-host-var-lib-cni-multus\") pod \"multus-5jp8t\" (UID: \"1b8f7019-0b80-42f2-be36-db424e632410\") " pod="openshift-multus/multus-5jp8t" Jul 20 02:45:07.262747 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.261562 2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 20 02:45:07.262747 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.261701 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1b8f7019-0b80-42f2-be36-db424e632410-cni-binary-copy\") pod \"multus-5jp8t\" (UID: \"1b8f7019-0b80-42f2-be36-db424e632410\") " pod="openshift-multus/multus-5jp8t" Jul 20 02:45:07.262747 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.261764 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1b8f7019-0b80-42f2-be36-db424e632410-hostroot\") pod \"multus-5jp8t\" (UID: \"1b8f7019-0b80-42f2-be36-db424e632410\") " pod="openshift-multus/multus-5jp8t" Jul 20 02:45:07.262747 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.261785 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cb6c0d96-0866-493d-9e19-7f63dff241ad-host-cni-netd\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.262747 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.261848 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e8d00f9d-e63a-44a7-8a2c-268dc6787c75-os-release\") pod \"multus-additional-cni-plugins-d5bdr\" (UID: \"e8d00f9d-e63a-44a7-8a2c-268dc6787c75\") " pod="openshift-multus/multus-additional-cni-plugins-d5bdr" Jul 20 02:45:07.279666 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.279606 2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-f22bk" Jul 20 02:45:07.287273 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.287247 2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-f22bk" Jul 20 02:45:07.319359 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.319333 2558 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 02:45:07.347796 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.347766 2558 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 20 02:45:07.364081 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.362983 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/90740648-1f62-4e9d-a8af-67716403522e-var-lib-kubelet\") pod \"tuned-pw2tj\" (UID: \"90740648-1f62-4e9d-a8af-67716403522e\") " pod="openshift-cluster-node-tuning-operator/tuned-pw2tj" Jul 20 02:45:07.364081 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.363021 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cb6c0d96-0866-493d-9e19-7f63dff241ad-node-log\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.364081 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.363050 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v964w\" (UniqueName: \"kubernetes.io/projected/d70f5103-7dc9-4279-9781-38e395da435b-kube-api-access-v964w\") pod \"network-metrics-daemon-9x7hv\" (UID: \"d70f5103-7dc9-4279-9781-38e395da435b\") " pod="openshift-multus/network-metrics-daemon-9x7hv" Jul 20 02:45:07.364081 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.363061 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/90740648-1f62-4e9d-a8af-67716403522e-var-lib-kubelet\") pod \"tuned-pw2tj\" (UID: \"90740648-1f62-4e9d-a8af-67716403522e\") " pod="openshift-cluster-node-tuning-operator/tuned-pw2tj" Jul 20 02:45:07.364081 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.363074 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e8d00f9d-e63a-44a7-8a2c-268dc6787c75-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d5bdr\" (UID: \"e8d00f9d-e63a-44a7-8a2c-268dc6787c75\") " pod="openshift-multus/multus-additional-cni-plugins-d5bdr" Jul 20 02:45:07.364081 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.363129 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9ef0be8f-01c8-4454-8713-630d2ca25963-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hd68h\" (UID: \"9ef0be8f-01c8-4454-8713-630d2ca25963\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd68h" Jul 20 02:45:07.364081 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.363158 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/90740648-1f62-4e9d-a8af-67716403522e-etc-systemd\") pod \"tuned-pw2tj\" (UID: \"90740648-1f62-4e9d-a8af-67716403522e\") " pod="openshift-cluster-node-tuning-operator/tuned-pw2tj" Jul 20 02:45:07.364081 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.363189 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb6c0d96-0866-493d-9e19-7f63dff241ad-var-lib-openvswitch\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.364081 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.363216 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cb6c0d96-0866-493d-9e19-7f63dff241ad-log-socket\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.364081 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.363237 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2be66185-38c5-43ac-ba3b-766093577e64-agent-certs\") pod \"konnectivity-agent-mvvbt\" (UID: \"2be66185-38c5-43ac-ba3b-766093577e64\") " pod="kube-system/konnectivity-agent-mvvbt" Jul 20 02:45:07.364081 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.363258 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8d00f9d-e63a-44a7-8a2c-268dc6787c75-system-cni-dir\") pod \"multus-additional-cni-plugins-d5bdr\" (UID: \"e8d00f9d-e63a-44a7-8a2c-268dc6787c75\") " pod="openshift-multus/multus-additional-cni-plugins-d5bdr" Jul 20 02:45:07.364081 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.363281 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9ef0be8f-01c8-4454-8713-630d2ca25963-etc-selinux\") pod \"aws-ebs-csi-driver-node-hd68h\" (UID: \"9ef0be8f-01c8-4454-8713-630d2ca25963\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd68h" Jul 20 02:45:07.364081 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.363308 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9ef0be8f-01c8-4454-8713-630d2ca25963-sys-fs\") pod \"aws-ebs-csi-driver-node-hd68h\" (UID: \"9ef0be8f-01c8-4454-8713-630d2ca25963\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd68h" Jul 20 02:45:07.364081 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.363335 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b8f7019-0b80-42f2-be36-db424e632410-multus-cni-dir\") pod \"multus-5jp8t\" (UID: \"1b8f7019-0b80-42f2-be36-db424e632410\") " pod="openshift-multus/multus-5jp8t" Jul 20 02:45:07.364885 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.363343 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cb6c0d96-0866-493d-9e19-7f63dff241ad-node-log\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.364885 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.363383 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb6c0d96-0866-493d-9e19-7f63dff241ad-host-kubelet\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.364885 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.363406 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8d00f9d-e63a-44a7-8a2c-268dc6787c75-system-cni-dir\") pod \"multus-additional-cni-plugins-d5bdr\" (UID: \"e8d00f9d-e63a-44a7-8a2c-268dc6787c75\") " pod="openshift-multus/multus-additional-cni-plugins-d5bdr" Jul 20 02:45:07.364885 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.363436 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb6c0d96-0866-493d-9e19-7f63dff241ad-run-openvswitch\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.364885 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.363451 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb6c0d96-0866-493d-9e19-7f63dff241ad-var-lib-openvswitch\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.364885 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.363461 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rsdj\" (UniqueName: \"kubernetes.io/projected/26701b30-ec4b-4dec-80df-b483d7edb77d-kube-api-access-4rsdj\") pod \"network-check-target-5cxtl\" (UID: \"26701b30-ec4b-4dec-80df-b483d7edb77d\") " pod="openshift-network-diagnostics/network-check-target-5cxtl" Jul 20 02:45:07.364885 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.363493 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90740648-1f62-4e9d-a8af-67716403522e-tmp\") pod \"tuned-pw2tj\" (UID: \"90740648-1f62-4e9d-a8af-67716403522e\") " pod="openshift-cluster-node-tuning-operator/tuned-pw2tj" Jul 20 02:45:07.364885 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.363499 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cb6c0d96-0866-493d-9e19-7f63dff241ad-log-socket\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.364885 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.363529 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnj2l\" (UniqueName: \"kubernetes.io/projected/90740648-1f62-4e9d-a8af-67716403522e-kube-api-access-bnj2l\") pod \"tuned-pw2tj\" (UID: \"90740648-1f62-4e9d-a8af-67716403522e\") " pod="openshift-cluster-node-tuning-operator/tuned-pw2tj" Jul 20 02:45:07.364885 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.363556 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c361af25-375d-4bff-bdb9-53f137cea0f6-hosts-file\") pod \"node-resolver-lvkqp\" (UID: \"c361af25-375d-4bff-bdb9-53f137cea0f6\") " pod="openshift-dns/node-resolver-lvkqp" Jul 20 02:45:07.364885 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.363581 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/504cb1f0-841a-49f7-add8-2d4742a1cb68-host\") pod \"node-ca-9qxnl\" (UID: \"504cb1f0-841a-49f7-add8-2d4742a1cb68\") " pod="openshift-image-registry/node-ca-9qxnl" Jul 20 02:45:07.364885 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.363604 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e8d00f9d-e63a-44a7-8a2c-268dc6787c75-cnibin\") pod \"multus-additional-cni-plugins-d5bdr\" (UID: \"e8d00f9d-e63a-44a7-8a2c-268dc6787c75\") " pod="openshift-multus/multus-additional-cni-plugins-d5bdr" Jul 20 02:45:07.364885 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.363631 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1b8f7019-0b80-42f2-be36-db424e632410-host-run-multus-certs\") pod \"multus-5jp8t\" (UID: \"1b8f7019-0b80-42f2-be36-db424e632410\") " pod="openshift-multus/multus-5jp8t" Jul 20 02:45:07.364885 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.363656 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/90740648-1f62-4e9d-a8af-67716403522e-etc-sysctl-conf\") pod \"tuned-pw2tj\" (UID: \"90740648-1f62-4e9d-a8af-67716403522e\") " pod="openshift-cluster-node-tuning-operator/tuned-pw2tj" Jul 20 02:45:07.364885 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.363309 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/90740648-1f62-4e9d-a8af-67716403522e-etc-systemd\") pod \"tuned-pw2tj\" (UID: \"90740648-1f62-4e9d-a8af-67716403522e\") " pod="openshift-cluster-node-tuning-operator/tuned-pw2tj" Jul 20 02:45:07.365628 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.363732 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c361af25-375d-4bff-bdb9-53f137cea0f6-hosts-file\") pod \"node-resolver-lvkqp\" (UID: \"c361af25-375d-4bff-bdb9-53f137cea0f6\") " pod="openshift-dns/node-resolver-lvkqp" Jul 20 02:45:07.365628 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.363740 2558 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 20 02:45:07.365628 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.363708 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e8d00f9d-e63a-44a7-8a2c-268dc6787c75-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d5bdr\" (UID: \"e8d00f9d-e63a-44a7-8a2c-268dc6787c75\") " pod="openshift-multus/multus-additional-cni-plugins-d5bdr" Jul 20 02:45:07.365628 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.363769 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/504cb1f0-841a-49f7-add8-2d4742a1cb68-host\") pod \"node-ca-9qxnl\" (UID: \"504cb1f0-841a-49f7-add8-2d4742a1cb68\") " pod="openshift-image-registry/node-ca-9qxnl" Jul 20 02:45:07.365628 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.363802 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1b8f7019-0b80-42f2-be36-db424e632410-host-run-multus-certs\") pod \"multus-5jp8t\" (UID: \"1b8f7019-0b80-42f2-be36-db424e632410\") " pod="openshift-multus/multus-5jp8t" Jul 20 02:45:07.365628 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.363862 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e8d00f9d-e63a-44a7-8a2c-268dc6787c75-cnibin\") pod \"multus-additional-cni-plugins-d5bdr\" (UID: \"e8d00f9d-e63a-44a7-8a2c-268dc6787c75\") " pod="openshift-multus/multus-additional-cni-plugins-d5bdr" Jul 20 02:45:07.365628 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.363898 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/90740648-1f62-4e9d-a8af-67716403522e-etc-sysctl-conf\") pod \"tuned-pw2tj\" (UID: \"90740648-1f62-4e9d-a8af-67716403522e\") " pod="openshift-cluster-node-tuning-operator/tuned-pw2tj" Jul 20 02:45:07.365628 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.363946 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb6c0d96-0866-493d-9e19-7f63dff241ad-run-openvswitch\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.365628 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.363980 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb6c0d96-0866-493d-9e19-7f63dff241ad-host-kubelet\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.365628 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.364045 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/90740648-1f62-4e9d-a8af-67716403522e-run\") pod \"tuned-pw2tj\" (UID: \"90740648-1f62-4e9d-a8af-67716403522e\") " pod="openshift-cluster-node-tuning-operator/tuned-pw2tj" Jul 20 02:45:07.365628 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.364051 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b8f7019-0b80-42f2-be36-db424e632410-multus-cni-dir\") pod \"multus-5jp8t\" (UID: \"1b8f7019-0b80-42f2-be36-db424e632410\") " pod="openshift-multus/multus-5jp8t" Jul 20 02:45:07.365628 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.363682 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/90740648-1f62-4e9d-a8af-67716403522e-run\") pod \"tuned-pw2tj\" (UID: \"90740648-1f62-4e9d-a8af-67716403522e\") " pod="openshift-cluster-node-tuning-operator/tuned-pw2tj" Jul 20 02:45:07.365628 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.364084 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cb6c0d96-0866-493d-9e19-7f63dff241ad-host-run-netns\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.365628 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.364138 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cb6c0d96-0866-493d-9e19-7f63dff241ad-host-run-netns\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.365628 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.364172 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzxq4\" (UniqueName: \"kubernetes.io/projected/cb6c0d96-0866-493d-9e19-7f63dff241ad-kube-api-access-hzxq4\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.366361 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.364218 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1b8f7019-0b80-42f2-be36-db424e632410-cnibin\") pod \"multus-5jp8t\" (UID: \"1b8f7019-0b80-42f2-be36-db424e632410\") " pod="openshift-multus/multus-5jp8t" Jul 20 02:45:07.366361 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.364243 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b8f7019-0b80-42f2-be36-db424e632410-host-run-netns\") pod \"multus-5jp8t\" (UID: \"1b8f7019-0b80-42f2-be36-db424e632410\") " pod="openshift-multus/multus-5jp8t" Jul 20 02:45:07.366361 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.364263 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/90740648-1f62-4e9d-a8af-67716403522e-host\") pod \"tuned-pw2tj\" (UID: \"90740648-1f62-4e9d-a8af-67716403522e\") " pod="openshift-cluster-node-tuning-operator/tuned-pw2tj" Jul 20 02:45:07.366361 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.364284 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/02007d87-46f6-4589-ac19-d1a89cf5465c-host-slash\") pod \"iptables-alerter-4kbnx\" (UID: \"02007d87-46f6-4589-ac19-d1a89cf5465c\") " pod="openshift-network-operator/iptables-alerter-4kbnx" Jul 20 02:45:07.366361 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.364308 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cb6c0d96-0866-493d-9e19-7f63dff241ad-env-overrides\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.366361 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.364331 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/90740648-1f62-4e9d-a8af-67716403522e-sys\") pod \"tuned-pw2tj\" (UID: \"90740648-1f62-4e9d-a8af-67716403522e\") " pod="openshift-cluster-node-tuning-operator/tuned-pw2tj" Jul 20 02:45:07.366361 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.364334 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b8f7019-0b80-42f2-be36-db424e632410-host-run-netns\") pod \"multus-5jp8t\" (UID: \"1b8f7019-0b80-42f2-be36-db424e632410\") " pod="openshift-multus/multus-5jp8t" Jul 20 02:45:07.366361 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.364355 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56h97\" (UniqueName: \"kubernetes.io/projected/c361af25-375d-4bff-bdb9-53f137cea0f6-kube-api-access-56h97\") pod \"node-resolver-lvkqp\" (UID: \"c361af25-375d-4bff-bdb9-53f137cea0f6\") " pod="openshift-dns/node-resolver-lvkqp" Jul 20 02:45:07.366361 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.364380 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb6c0d96-0866-493d-9e19-7f63dff241ad-etc-openvswitch\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.366361 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.364409 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb6c0d96-0866-493d-9e19-7f63dff241ad-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.366361 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.364421 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/90740648-1f62-4e9d-a8af-67716403522e-host\") pod \"tuned-pw2tj\" (UID: \"90740648-1f62-4e9d-a8af-67716403522e\") " pod="openshift-cluster-node-tuning-operator/tuned-pw2tj" Jul 20 02:45:07.366361 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.364412 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/90740648-1f62-4e9d-a8af-67716403522e-sys\") pod \"tuned-pw2tj\" (UID: \"90740648-1f62-4e9d-a8af-67716403522e\") " pod="openshift-cluster-node-tuning-operator/tuned-pw2tj" Jul 20 02:45:07.366361 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.364382 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1b8f7019-0b80-42f2-be36-db424e632410-cnibin\") pod \"multus-5jp8t\" (UID: \"1b8f7019-0b80-42f2-be36-db424e632410\") " pod="openshift-multus/multus-5jp8t" Jul 20 02:45:07.366361 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.364380 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/02007d87-46f6-4589-ac19-d1a89cf5465c-host-slash\") pod \"iptables-alerter-4kbnx\" (UID: \"02007d87-46f6-4589-ac19-d1a89cf5465c\") " pod="openshift-network-operator/iptables-alerter-4kbnx" Jul 20 02:45:07.366361 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.364464 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb6c0d96-0866-493d-9e19-7f63dff241ad-etc-openvswitch\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.367051 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.364471 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb6c0d96-0866-493d-9e19-7f63dff241ad-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.367051 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.364495 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cb6c0d96-0866-493d-9e19-7f63dff241ad-ovnkube-script-lib\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.367051 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.364519 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6c7d134c-4e38-481c-8a81-e03434badf7d-kubelet-config\") pod \"global-pull-secret-syncer-t9spk\" (UID: \"6c7d134c-4e38-481c-8a81-e03434badf7d\") " pod="kube-system/global-pull-secret-syncer-t9spk" Jul 20 02:45:07.367051 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.364542 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6c7d134c-4e38-481c-8a81-e03434badf7d-dbus\") pod \"global-pull-secret-syncer-t9spk\" (UID: \"6c7d134c-4e38-481c-8a81-e03434badf7d\") " pod="kube-system/global-pull-secret-syncer-t9spk" Jul 20 02:45:07.367051 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.364591 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1b8f7019-0b80-42f2-be36-db424e632410-host-var-lib-cni-multus\") pod \"multus-5jp8t\" (UID: \"1b8f7019-0b80-42f2-be36-db424e632410\") " pod="openshift-multus/multus-5jp8t" Jul 20 02:45:07.367051 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.364615 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1b8f7019-0b80-42f2-be36-db424e632410-cni-binary-copy\") pod \"multus-5jp8t\" (UID: \"1b8f7019-0b80-42f2-be36-db424e632410\") " pod="openshift-multus/multus-5jp8t" Jul 20 02:45:07.367051 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.364640 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1b8f7019-0b80-42f2-be36-db424e632410-hostroot\") pod \"multus-5jp8t\" (UID: \"1b8f7019-0b80-42f2-be36-db424e632410\") " pod="openshift-multus/multus-5jp8t" Jul 20 02:45:07.367051 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.364664 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cb6c0d96-0866-493d-9e19-7f63dff241ad-host-cni-netd\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.367051 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.364687 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e8d00f9d-e63a-44a7-8a2c-268dc6787c75-os-release\") pod \"multus-additional-cni-plugins-d5bdr\" (UID: \"e8d00f9d-e63a-44a7-8a2c-268dc6787c75\") " pod="openshift-multus/multus-additional-cni-plugins-d5bdr" Jul 20 02:45:07.367051 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.364696 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6c7d134c-4e38-481c-8a81-e03434badf7d-dbus\") pod \"global-pull-secret-syncer-t9spk\" (UID: \"6c7d134c-4e38-481c-8a81-e03434badf7d\") " pod="kube-system/global-pull-secret-syncer-t9spk" Jul 20 02:45:07.367051 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.364714 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7zr9\" (UniqueName: \"kubernetes.io/projected/9ef0be8f-01c8-4454-8713-630d2ca25963-kube-api-access-k7zr9\") pod \"aws-ebs-csi-driver-node-hd68h\" (UID: \"9ef0be8f-01c8-4454-8713-630d2ca25963\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd68h" Jul 20 02:45:07.367051 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.364744 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1b8f7019-0b80-42f2-be36-db424e632410-os-release\") pod \"multus-5jp8t\" (UID: \"1b8f7019-0b80-42f2-be36-db424e632410\") " pod="openshift-multus/multus-5jp8t" Jul 20 02:45:07.367051 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.364771 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt4vl\" (UniqueName: \"kubernetes.io/projected/02007d87-46f6-4589-ac19-d1a89cf5465c-kube-api-access-jt4vl\") pod \"iptables-alerter-4kbnx\" (UID: \"02007d87-46f6-4589-ac19-d1a89cf5465c\") " pod="openshift-network-operator/iptables-alerter-4kbnx" Jul 20 02:45:07.367051 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.364800 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6b7f\" (UniqueName: \"kubernetes.io/projected/504cb1f0-841a-49f7-add8-2d4742a1cb68-kube-api-access-w6b7f\") pod \"node-ca-9qxnl\" (UID: \"504cb1f0-841a-49f7-add8-2d4742a1cb68\") " pod="openshift-image-registry/node-ca-9qxnl" Jul 20 02:45:07.367798 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.364813 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cb6c0d96-0866-493d-9e19-7f63dff241ad-env-overrides\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.367798 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.364823 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90740648-1f62-4e9d-a8af-67716403522e-etc-kubernetes\") pod \"tuned-pw2tj\" (UID: \"90740648-1f62-4e9d-a8af-67716403522e\") " pod="openshift-cluster-node-tuning-operator/tuned-pw2tj" Jul 20 02:45:07.367798 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.364848 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cb6c0d96-0866-493d-9e19-7f63dff241ad-host-cni-bin\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.367798 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.364873 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6c7d134c-4e38-481c-8a81-e03434badf7d-original-pull-secret\") pod \"global-pull-secret-syncer-t9spk\" (UID: \"6c7d134c-4e38-481c-8a81-e03434badf7d\") " pod="kube-system/global-pull-secret-syncer-t9spk" Jul 20 02:45:07.367798 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.364878 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6c7d134c-4e38-481c-8a81-e03434badf7d-kubelet-config\") pod \"global-pull-secret-syncer-t9spk\" (UID: \"6c7d134c-4e38-481c-8a81-e03434badf7d\") " pod="kube-system/global-pull-secret-syncer-t9spk" Jul 20 02:45:07.367798 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.364897 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/504cb1f0-841a-49f7-add8-2d4742a1cb68-serviceca\") pod \"node-ca-9qxnl\" (UID: \"504cb1f0-841a-49f7-add8-2d4742a1cb68\") " pod="openshift-image-registry/node-ca-9qxnl" Jul 20 02:45:07.367798 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.364916 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1b8f7019-0b80-42f2-be36-db424e632410-host-var-lib-cni-multus\") pod \"multus-5jp8t\" (UID: \"1b8f7019-0b80-42f2-be36-db424e632410\") " pod="openshift-multus/multus-5jp8t" Jul 20 02:45:07.367798 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.364920 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e8d00f9d-e63a-44a7-8a2c-268dc6787c75-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d5bdr\" (UID: \"e8d00f9d-e63a-44a7-8a2c-268dc6787c75\") " pod="openshift-multus/multus-additional-cni-plugins-d5bdr" Jul 20 02:45:07.367798 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.364944 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1b8f7019-0b80-42f2-be36-db424e632410-multus-daemon-config\") pod \"multus-5jp8t\" (UID: \"1b8f7019-0b80-42f2-be36-db424e632410\") " pod="openshift-multus/multus-5jp8t" Jul 20 02:45:07.367798 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.364968 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cb6c0d96-0866-493d-9e19-7f63dff241ad-systemd-units\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.367798 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.364991 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cb6c0d96-0866-493d-9e19-7f63dff241ad-ovnkube-config\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.367798 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.365014 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b8f7019-0b80-42f2-be36-db424e632410-host-var-lib-kubelet\") pod \"multus-5jp8t\" (UID: \"1b8f7019-0b80-42f2-be36-db424e632410\") " pod="openshift-multus/multus-5jp8t" Jul 20 02:45:07.367798 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.365076 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b8f7019-0b80-42f2-be36-db424e632410-system-cni-dir\") pod \"multus-5jp8t\" (UID: \"1b8f7019-0b80-42f2-be36-db424e632410\") " pod="openshift-multus/multus-5jp8t" Jul 20 02:45:07.367798 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.365114 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b8f7019-0b80-42f2-be36-db424e632410-host-var-lib-cni-bin\") pod \"multus-5jp8t\" (UID: \"1b8f7019-0b80-42f2-be36-db424e632410\") " pod="openshift-multus/multus-5jp8t" Jul 20 02:45:07.367798 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.365139 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/90740648-1f62-4e9d-a8af-67716403522e-etc-tuned\") pod \"tuned-pw2tj\" (UID: \"90740648-1f62-4e9d-a8af-67716403522e\") " pod="openshift-cluster-node-tuning-operator/tuned-pw2tj" Jul 20 02:45:07.368532 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.365164 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/02007d87-46f6-4589-ac19-d1a89cf5465c-iptables-alerter-script\") pod \"iptables-alerter-4kbnx\" (UID: \"02007d87-46f6-4589-ac19-d1a89cf5465c\") " pod="openshift-network-operator/iptables-alerter-4kbnx" Jul 20 02:45:07.368532 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.365189 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e8d00f9d-e63a-44a7-8a2c-268dc6787c75-cni-binary-copy\") pod \"multus-additional-cni-plugins-d5bdr\" (UID: \"e8d00f9d-e63a-44a7-8a2c-268dc6787c75\") " pod="openshift-multus/multus-additional-cni-plugins-d5bdr" Jul 20 02:45:07.368532 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.365199 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cb6c0d96-0866-493d-9e19-7f63dff241ad-ovnkube-script-lib\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.368532 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.365211 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/90740648-1f62-4e9d-a8af-67716403522e-etc-sysctl-d\") pod \"tuned-pw2tj\" (UID: \"90740648-1f62-4e9d-a8af-67716403522e\") " pod="openshift-cluster-node-tuning-operator/tuned-pw2tj" Jul 20 02:45:07.368532 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.365236 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb6c0d96-0866-493d-9e19-7f63dff241ad-host-slash\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.368532 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.365284 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb6c0d96-0866-493d-9e19-7f63dff241ad-host-slash\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.368532 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.365328 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cb6c0d96-0866-493d-9e19-7f63dff241ad-systemd-units\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.368532 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.365442 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1b8f7019-0b80-42f2-be36-db424e632410-cni-binary-copy\") pod \"multus-5jp8t\" (UID: \"1b8f7019-0b80-42f2-be36-db424e632410\") " pod="openshift-multus/multus-5jp8t" Jul 20 02:45:07.368532 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.365498 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1b8f7019-0b80-42f2-be36-db424e632410-hostroot\") pod \"multus-5jp8t\" (UID: \"1b8f7019-0b80-42f2-be36-db424e632410\") " pod="openshift-multus/multus-5jp8t" Jul 20 02:45:07.368532 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.365541 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cb6c0d96-0866-493d-9e19-7f63dff241ad-host-cni-netd\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.368532 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.365604 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e8d00f9d-e63a-44a7-8a2c-268dc6787c75-os-release\") pod \"multus-additional-cni-plugins-d5bdr\" (UID: \"e8d00f9d-e63a-44a7-8a2c-268dc6787c75\") " pod="openshift-multus/multus-additional-cni-plugins-d5bdr" Jul 20 02:45:07.368532 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.365644 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1b8f7019-0b80-42f2-be36-db424e632410-multus-daemon-config\") pod \"multus-5jp8t\" (UID: \"1b8f7019-0b80-42f2-be36-db424e632410\") " pod="openshift-multus/multus-5jp8t" Jul 20 02:45:07.368532 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.365658 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b8f7019-0b80-42f2-be36-db424e632410-system-cni-dir\") pod \"multus-5jp8t\" (UID: \"1b8f7019-0b80-42f2-be36-db424e632410\") " pod="openshift-multus/multus-5jp8t" Jul 20 02:45:07.368532 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.365685 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb6c0d96-0866-493d-9e19-7f63dff241ad-host-run-ovn-kubernetes\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.368532 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.365700 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b8f7019-0b80-42f2-be36-db424e632410-host-var-lib-kubelet\") pod \"multus-5jp8t\" (UID: \"1b8f7019-0b80-42f2-be36-db424e632410\") " pod="openshift-multus/multus-5jp8t" Jul 20 02:45:07.369286 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.365715 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9ef0be8f-01c8-4454-8713-630d2ca25963-socket-dir\") pod \"aws-ebs-csi-driver-node-hd68h\" (UID: \"9ef0be8f-01c8-4454-8713-630d2ca25963\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd68h" Jul 20 02:45:07.369286 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.365734 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b8f7019-0b80-42f2-be36-db424e632410-host-var-lib-cni-bin\") pod \"multus-5jp8t\" (UID: \"1b8f7019-0b80-42f2-be36-db424e632410\") " pod="openshift-multus/multus-5jp8t" Jul 20 02:45:07.369286 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.365741 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9ef0be8f-01c8-4454-8713-630d2ca25963-device-dir\") pod \"aws-ebs-csi-driver-node-hd68h\" (UID: \"9ef0be8f-01c8-4454-8713-630d2ca25963\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd68h" Jul 20 02:45:07.369286 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.365759 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cb6c0d96-0866-493d-9e19-7f63dff241ad-host-cni-bin\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.369286 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.365770 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b8f7019-0b80-42f2-be36-db424e632410-etc-kubernetes\") pod \"multus-5jp8t\" (UID: \"1b8f7019-0b80-42f2-be36-db424e632410\") " pod="openshift-multus/multus-5jp8t" Jul 20 02:45:07.369286 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.365796 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cb6c0d96-0866-493d-9e19-7f63dff241ad-run-systemd\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.369286 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.365806 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1b8f7019-0b80-42f2-be36-db424e632410-os-release\") pod \"multus-5jp8t\" (UID: \"1b8f7019-0b80-42f2-be36-db424e632410\") " pod="openshift-multus/multus-5jp8t" Jul 20 02:45:07.369286 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.365821 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfk5c\" (UniqueName: \"kubernetes.io/projected/e8d00f9d-e63a-44a7-8a2c-268dc6787c75-kube-api-access-wfk5c\") pod \"multus-additional-cni-plugins-d5bdr\" (UID: \"e8d00f9d-e63a-44a7-8a2c-268dc6787c75\") " pod="openshift-multus/multus-additional-cni-plugins-d5bdr" Jul 20 02:45:07.369286 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.365846 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9ef0be8f-01c8-4454-8713-630d2ca25963-registration-dir\") pod \"aws-ebs-csi-driver-node-hd68h\" (UID: \"9ef0be8f-01c8-4454-8713-630d2ca25963\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd68h" Jul 20 02:45:07.369286 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.365874 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1b8f7019-0b80-42f2-be36-db424e632410-host-run-k8s-cni-cncf-io\") pod \"multus-5jp8t\" (UID: \"1b8f7019-0b80-42f2-be36-db424e632410\") " pod="openshift-multus/multus-5jp8t" Jul 20 02:45:07.369286 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.365899 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fht2c\" (UniqueName: \"kubernetes.io/projected/1b8f7019-0b80-42f2-be36-db424e632410-kube-api-access-fht2c\") pod \"multus-5jp8t\" (UID: \"1b8f7019-0b80-42f2-be36-db424e632410\") " pod="openshift-multus/multus-5jp8t" Jul 20 02:45:07.369286 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.365924 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1b8f7019-0b80-42f2-be36-db424e632410-multus-conf-dir\") pod \"multus-5jp8t\" (UID: \"1b8f7019-0b80-42f2-be36-db424e632410\") " pod="openshift-multus/multus-5jp8t" Jul 20 02:45:07.369286 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.365948 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/90740648-1f62-4e9d-a8af-67716403522e-etc-modprobe-d\") pod \"tuned-pw2tj\" (UID: \"90740648-1f62-4e9d-a8af-67716403522e\") " pod="openshift-cluster-node-tuning-operator/tuned-pw2tj" Jul 20 02:45:07.369286 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.365971 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c361af25-375d-4bff-bdb9-53f137cea0f6-tmp-dir\") pod \"node-resolver-lvkqp\" (UID: \"c361af25-375d-4bff-bdb9-53f137cea0f6\") " pod="openshift-dns/node-resolver-lvkqp" Jul 20 02:45:07.369932 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.365995 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cb6c0d96-0866-493d-9e19-7f63dff241ad-run-ovn\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.369932 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.366020 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cb6c0d96-0866-493d-9e19-7f63dff241ad-ovn-node-metrics-cert\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.369932 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.366045 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2be66185-38c5-43ac-ba3b-766093577e64-konnectivity-ca\") pod \"konnectivity-agent-mvvbt\" (UID: \"2be66185-38c5-43ac-ba3b-766093577e64\") " pod="kube-system/konnectivity-agent-mvvbt" Jul 20 02:45:07.369932 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.366072 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d70f5103-7dc9-4279-9781-38e395da435b-metrics-certs\") pod \"network-metrics-daemon-9x7hv\" (UID: \"d70f5103-7dc9-4279-9781-38e395da435b\") " pod="openshift-multus/network-metrics-daemon-9x7hv" Jul 20 02:45:07.369932 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.366120 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e8d00f9d-e63a-44a7-8a2c-268dc6787c75-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d5bdr\" (UID: \"e8d00f9d-e63a-44a7-8a2c-268dc6787c75\") " pod="openshift-multus/multus-additional-cni-plugins-d5bdr" Jul 20 02:45:07.369932 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.366146 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1b8f7019-0b80-42f2-be36-db424e632410-multus-socket-dir-parent\") pod \"multus-5jp8t\" (UID: \"1b8f7019-0b80-42f2-be36-db424e632410\") " pod="openshift-multus/multus-5jp8t" Jul 20 02:45:07.369932 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.366174 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/90740648-1f62-4e9d-a8af-67716403522e-etc-sysconfig\") pod \"tuned-pw2tj\" (UID: \"90740648-1f62-4e9d-a8af-67716403522e\") " pod="openshift-cluster-node-tuning-operator/tuned-pw2tj" Jul 20 02:45:07.369932 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.366197 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/90740648-1f62-4e9d-a8af-67716403522e-lib-modules\") pod \"tuned-pw2tj\" (UID: \"90740648-1f62-4e9d-a8af-67716403522e\") " pod="openshift-cluster-node-tuning-operator/tuned-pw2tj" Jul 20 02:45:07.369932 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.366220 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90740648-1f62-4e9d-a8af-67716403522e-etc-kubernetes\") pod \"tuned-pw2tj\" (UID: \"90740648-1f62-4e9d-a8af-67716403522e\") " pod="openshift-cluster-node-tuning-operator/tuned-pw2tj" Jul 20 02:45:07.369932 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.366325 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/90740648-1f62-4e9d-a8af-67716403522e-lib-modules\") pod \"tuned-pw2tj\" (UID: \"90740648-1f62-4e9d-a8af-67716403522e\") " pod="openshift-cluster-node-tuning-operator/tuned-pw2tj" Jul 20 02:45:07.369932 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.366343 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cb6c0d96-0866-493d-9e19-7f63dff241ad-ovnkube-config\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.369932 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:07.366406 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 02:45:07.369932 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:07.366467 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c7d134c-4e38-481c-8a81-e03434badf7d-original-pull-secret podName:6c7d134c-4e38-481c-8a81-e03434badf7d nodeName:}" failed. No retries permitted until 2026-07-20 02:45:07.866447012 +0000 UTC m=+3.065696140 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6c7d134c-4e38-481c-8a81-e03434badf7d-original-pull-secret") pod "global-pull-secret-syncer-t9spk" (UID: "6c7d134c-4e38-481c-8a81-e03434badf7d") : object "kube-system"/"original-pull-secret" not registered Jul 20 02:45:07.369932 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.366464 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/90740648-1f62-4e9d-a8af-67716403522e-etc-sysctl-d\") pod \"tuned-pw2tj\" (UID: \"90740648-1f62-4e9d-a8af-67716403522e\") " pod="openshift-cluster-node-tuning-operator/tuned-pw2tj" Jul 20 02:45:07.370585 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.366700 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e8d00f9d-e63a-44a7-8a2c-268dc6787c75-cni-binary-copy\") pod \"multus-additional-cni-plugins-d5bdr\" (UID: \"e8d00f9d-e63a-44a7-8a2c-268dc6787c75\") " pod="openshift-multus/multus-additional-cni-plugins-d5bdr" Jul 20 02:45:07.370585 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.366763 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1b8f7019-0b80-42f2-be36-db424e632410-multus-conf-dir\") pod \"multus-5jp8t\" (UID: \"1b8f7019-0b80-42f2-be36-db424e632410\") " pod="openshift-multus/multus-5jp8t" Jul 20 02:45:07.370585 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.366889 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/504cb1f0-841a-49f7-add8-2d4742a1cb68-serviceca\") pod \"node-ca-9qxnl\" (UID: \"504cb1f0-841a-49f7-add8-2d4742a1cb68\") " pod="openshift-image-registry/node-ca-9qxnl" Jul 20 02:45:07.370585 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.366982 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/90740648-1f62-4e9d-a8af-67716403522e-etc-modprobe-d\") pod \"tuned-pw2tj\" (UID: \"90740648-1f62-4e9d-a8af-67716403522e\") " pod="openshift-cluster-node-tuning-operator/tuned-pw2tj" Jul 20 02:45:07.370585 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.367238 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e8d00f9d-e63a-44a7-8a2c-268dc6787c75-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d5bdr\" (UID: \"e8d00f9d-e63a-44a7-8a2c-268dc6787c75\") " pod="openshift-multus/multus-additional-cni-plugins-d5bdr" Jul 20 02:45:07.370585 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.367260 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c361af25-375d-4bff-bdb9-53f137cea0f6-tmp-dir\") pod \"node-resolver-lvkqp\" (UID: \"c361af25-375d-4bff-bdb9-53f137cea0f6\") " pod="openshift-dns/node-resolver-lvkqp" Jul 20 02:45:07.370585 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.367298 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb6c0d96-0866-493d-9e19-7f63dff241ad-host-run-ovn-kubernetes\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.370585 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.367329 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cb6c0d96-0866-493d-9e19-7f63dff241ad-run-ovn\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.370585 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.367352 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b8f7019-0b80-42f2-be36-db424e632410-etc-kubernetes\") pod \"multus-5jp8t\" (UID: \"1b8f7019-0b80-42f2-be36-db424e632410\") " pod="openshift-multus/multus-5jp8t" Jul 20 02:45:07.370585 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.367397 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cb6c0d96-0866-493d-9e19-7f63dff241ad-run-systemd\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.370585 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.367541 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/02007d87-46f6-4589-ac19-d1a89cf5465c-iptables-alerter-script\") pod \"iptables-alerter-4kbnx\" (UID: \"02007d87-46f6-4589-ac19-d1a89cf5465c\") " pod="openshift-network-operator/iptables-alerter-4kbnx" Jul 20 02:45:07.370585 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.367637 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1b8f7019-0b80-42f2-be36-db424e632410-host-run-k8s-cni-cncf-io\") pod \"multus-5jp8t\" (UID: \"1b8f7019-0b80-42f2-be36-db424e632410\") " pod="openshift-multus/multus-5jp8t" Jul 20 02:45:07.370585 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.367687 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e8d00f9d-e63a-44a7-8a2c-268dc6787c75-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d5bdr\" (UID: \"e8d00f9d-e63a-44a7-8a2c-268dc6787c75\") " pod="openshift-multus/multus-additional-cni-plugins-d5bdr" Jul 20 02:45:07.370585 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:07.367860 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 02:45:07.371332 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:07.367920 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d70f5103-7dc9-4279-9781-38e395da435b-metrics-certs podName:d70f5103-7dc9-4279-9781-38e395da435b nodeName:}" failed. No retries permitted until 2026-07-20 02:45:07.867904214 +0000 UTC m=+3.067153342 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d70f5103-7dc9-4279-9781-38e395da435b-metrics-certs") pod "network-metrics-daemon-9x7hv" (UID: "d70f5103-7dc9-4279-9781-38e395da435b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 02:45:07.371332 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.367970 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/90740648-1f62-4e9d-a8af-67716403522e-etc-sysconfig\") pod \"tuned-pw2tj\" (UID: \"90740648-1f62-4e9d-a8af-67716403522e\") " pod="openshift-cluster-node-tuning-operator/tuned-pw2tj" Jul 20 02:45:07.371332 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.368026 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1b8f7019-0b80-42f2-be36-db424e632410-multus-socket-dir-parent\") pod \"multus-5jp8t\" (UID: \"1b8f7019-0b80-42f2-be36-db424e632410\") " pod="openshift-multus/multus-5jp8t" Jul 20 02:45:07.371332 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.368198 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90740648-1f62-4e9d-a8af-67716403522e-tmp\") pod \"tuned-pw2tj\" (UID: \"90740648-1f62-4e9d-a8af-67716403522e\") " pod="openshift-cluster-node-tuning-operator/tuned-pw2tj" Jul 20 02:45:07.371332 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.368514 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2be66185-38c5-43ac-ba3b-766093577e64-agent-certs\") pod \"konnectivity-agent-mvvbt\" (UID: \"2be66185-38c5-43ac-ba3b-766093577e64\") " pod="kube-system/konnectivity-agent-mvvbt" Jul 20 02:45:07.371332 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.368571 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2be66185-38c5-43ac-ba3b-766093577e64-konnectivity-ca\") pod \"konnectivity-agent-mvvbt\" (UID: \"2be66185-38c5-43ac-ba3b-766093577e64\") " pod="kube-system/konnectivity-agent-mvvbt" Jul 20 02:45:07.371332 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.369573 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/90740648-1f62-4e9d-a8af-67716403522e-etc-tuned\") pod \"tuned-pw2tj\" (UID: \"90740648-1f62-4e9d-a8af-67716403522e\") " pod="openshift-cluster-node-tuning-operator/tuned-pw2tj" Jul 20 02:45:07.371332 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.369794 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cb6c0d96-0866-493d-9e19-7f63dff241ad-ovn-node-metrics-cert\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.374505 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.374482 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v964w\" (UniqueName: \"kubernetes.io/projected/d70f5103-7dc9-4279-9781-38e395da435b-kube-api-access-v964w\") pod \"network-metrics-daemon-9x7hv\" (UID: \"d70f5103-7dc9-4279-9781-38e395da435b\") " pod="openshift-multus/network-metrics-daemon-9x7hv" Jul 20 02:45:07.379591 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:07.379411 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 02:45:07.379591 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:07.379434 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 02:45:07.379591 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:07.379449 2558 projected.go:196] Error preparing data for projected volume kube-api-access-4rsdj for pod openshift-network-diagnostics/network-check-target-5cxtl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 02:45:07.379591 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:07.379513 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/26701b30-ec4b-4dec-80df-b483d7edb77d-kube-api-access-4rsdj podName:26701b30-ec4b-4dec-80df-b483d7edb77d nodeName:}" failed. No retries permitted until 2026-07-20 02:45:07.879496379 +0000 UTC m=+3.078745506 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4rsdj" (UniqueName: "kubernetes.io/projected/26701b30-ec4b-4dec-80df-b483d7edb77d-kube-api-access-4rsdj") pod "network-check-target-5cxtl" (UID: "26701b30-ec4b-4dec-80df-b483d7edb77d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 02:45:07.382297 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.382250 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzxq4\" (UniqueName: \"kubernetes.io/projected/cb6c0d96-0866-493d-9e19-7f63dff241ad-kube-api-access-hzxq4\") pod \"ovnkube-node-n4b7k\" (UID: \"cb6c0d96-0866-493d-9e19-7f63dff241ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.382606 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.382586 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt4vl\" (UniqueName: \"kubernetes.io/projected/02007d87-46f6-4589-ac19-d1a89cf5465c-kube-api-access-jt4vl\") pod \"iptables-alerter-4kbnx\" (UID: \"02007d87-46f6-4589-ac19-d1a89cf5465c\") " pod="openshift-network-operator/iptables-alerter-4kbnx" Jul 20 02:45:07.382692 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.382613 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnj2l\" (UniqueName: \"kubernetes.io/projected/90740648-1f62-4e9d-a8af-67716403522e-kube-api-access-bnj2l\") pod \"tuned-pw2tj\" (UID: \"90740648-1f62-4e9d-a8af-67716403522e\") " pod="openshift-cluster-node-tuning-operator/tuned-pw2tj" Jul 20 02:45:07.383937 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.383871 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56h97\" (UniqueName: \"kubernetes.io/projected/c361af25-375d-4bff-bdb9-53f137cea0f6-kube-api-access-56h97\") pod \"node-resolver-lvkqp\" (UID: \"c361af25-375d-4bff-bdb9-53f137cea0f6\") " pod="openshift-dns/node-resolver-lvkqp" Jul 20 02:45:07.384128 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.384077 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6b7f\" (UniqueName: \"kubernetes.io/projected/504cb1f0-841a-49f7-add8-2d4742a1cb68-kube-api-access-w6b7f\") pod \"node-ca-9qxnl\" (UID: \"504cb1f0-841a-49f7-add8-2d4742a1cb68\") " pod="openshift-image-registry/node-ca-9qxnl" Jul 20 02:45:07.388936 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.388913 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fht2c\" (UniqueName: \"kubernetes.io/projected/1b8f7019-0b80-42f2-be36-db424e632410-kube-api-access-fht2c\") pod \"multus-5jp8t\" (UID: \"1b8f7019-0b80-42f2-be36-db424e632410\") " pod="openshift-multus/multus-5jp8t" Jul 20 02:45:07.389422 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.389403 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfk5c\" (UniqueName: \"kubernetes.io/projected/e8d00f9d-e63a-44a7-8a2c-268dc6787c75-kube-api-access-wfk5c\") pod \"multus-additional-cni-plugins-d5bdr\" (UID: \"e8d00f9d-e63a-44a7-8a2c-268dc6787c75\") " pod="openshift-multus/multus-additional-cni-plugins-d5bdr" Jul 20 02:45:07.391230 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.391168 2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 02:45:07.423599 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.423523 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-139.ec2.internal" event={"ID":"ceae9fddccd84171bbba0a3e9c0148c8","Type":"ContainerStarted","Data":"2f112a23dcb66f6fbc7e0dd54edf27ed922994abe0211795d922d3aee20dd943"} Jul 20 02:45:07.425497 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.425465 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal" event={"ID":"39560757f7588014bf49cd290ec77fb5","Type":"ContainerStarted","Data":"fbdfa133811ee41db83e3cf3a8c8cdeda25412f19129415882eeb04d0725fa2b"} Jul 20 02:45:07.466998 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.466901 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7zr9\" (UniqueName: \"kubernetes.io/projected/9ef0be8f-01c8-4454-8713-630d2ca25963-kube-api-access-k7zr9\") pod \"aws-ebs-csi-driver-node-hd68h\" (UID: \"9ef0be8f-01c8-4454-8713-630d2ca25963\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd68h" Jul 20 02:45:07.466998 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.466981 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9ef0be8f-01c8-4454-8713-630d2ca25963-socket-dir\") pod \"aws-ebs-csi-driver-node-hd68h\" (UID: \"9ef0be8f-01c8-4454-8713-630d2ca25963\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd68h" Jul 20 02:45:07.467262 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.467007 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9ef0be8f-01c8-4454-8713-630d2ca25963-device-dir\") pod \"aws-ebs-csi-driver-node-hd68h\" (UID: \"9ef0be8f-01c8-4454-8713-630d2ca25963\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd68h" Jul 20 02:45:07.467262 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.467044 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9ef0be8f-01c8-4454-8713-630d2ca25963-registration-dir\") pod \"aws-ebs-csi-driver-node-hd68h\" (UID: \"9ef0be8f-01c8-4454-8713-630d2ca25963\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd68h" Jul 20 02:45:07.467262 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.467091 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9ef0be8f-01c8-4454-8713-630d2ca25963-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hd68h\" (UID: \"9ef0be8f-01c8-4454-8713-630d2ca25963\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd68h" Jul 20 02:45:07.467262 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.467131 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9ef0be8f-01c8-4454-8713-630d2ca25963-etc-selinux\") pod \"aws-ebs-csi-driver-node-hd68h\" (UID: \"9ef0be8f-01c8-4454-8713-630d2ca25963\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd68h" Jul 20 02:45:07.467262 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.467157 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9ef0be8f-01c8-4454-8713-630d2ca25963-sys-fs\") pod \"aws-ebs-csi-driver-node-hd68h\" (UID: \"9ef0be8f-01c8-4454-8713-630d2ca25963\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd68h" Jul 20 02:45:07.467262 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.467253 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9ef0be8f-01c8-4454-8713-630d2ca25963-sys-fs\") pod \"aws-ebs-csi-driver-node-hd68h\" (UID: \"9ef0be8f-01c8-4454-8713-630d2ca25963\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd68h" Jul 20 02:45:07.467704 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.467622 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9ef0be8f-01c8-4454-8713-630d2ca25963-socket-dir\") pod \"aws-ebs-csi-driver-node-hd68h\" (UID: \"9ef0be8f-01c8-4454-8713-630d2ca25963\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd68h" Jul 20 02:45:07.467704 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.467633 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9ef0be8f-01c8-4454-8713-630d2ca25963-device-dir\") pod \"aws-ebs-csi-driver-node-hd68h\" (UID: \"9ef0be8f-01c8-4454-8713-630d2ca25963\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd68h" Jul 20 02:45:07.467704 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.467647 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9ef0be8f-01c8-4454-8713-630d2ca25963-registration-dir\") pod \"aws-ebs-csi-driver-node-hd68h\" (UID: \"9ef0be8f-01c8-4454-8713-630d2ca25963\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd68h" Jul 20 02:45:07.467704 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.467679 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9ef0be8f-01c8-4454-8713-630d2ca25963-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hd68h\" (UID: \"9ef0be8f-01c8-4454-8713-630d2ca25963\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd68h" Jul 20 02:45:07.467974 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.467743 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9ef0be8f-01c8-4454-8713-630d2ca25963-etc-selinux\") pod \"aws-ebs-csi-driver-node-hd68h\" (UID: \"9ef0be8f-01c8-4454-8713-630d2ca25963\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd68h" Jul 20 02:45:07.485281 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.485254 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7zr9\" (UniqueName: \"kubernetes.io/projected/9ef0be8f-01c8-4454-8713-630d2ca25963-kube-api-access-k7zr9\") pod \"aws-ebs-csi-driver-node-hd68h\" (UID: \"9ef0be8f-01c8-4454-8713-630d2ca25963\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd68h" Jul 20 02:45:07.542043 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.542013 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5jp8t" Jul 20 02:45:07.559184 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.559149 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pw2tj" Jul 20 02:45:07.569037 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.569008 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4kbnx" Jul 20 02:45:07.576635 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.576613 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mvvbt" Jul 20 02:45:07.586273 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.586240 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9qxnl" Jul 20 02:45:07.595902 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.595880 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lvkqp" Jul 20 02:45:07.603579 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.603559 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d5bdr" Jul 20 02:45:07.611195 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.611177 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:07.617861 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.617844 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd68h" Jul 20 02:45:07.870121 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.870026 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6c7d134c-4e38-481c-8a81-e03434badf7d-original-pull-secret\") pod \"global-pull-secret-syncer-t9spk\" (UID: \"6c7d134c-4e38-481c-8a81-e03434badf7d\") " pod="kube-system/global-pull-secret-syncer-t9spk" Jul 20 02:45:07.870121 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.870083 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d70f5103-7dc9-4279-9781-38e395da435b-metrics-certs\") pod \"network-metrics-daemon-9x7hv\" (UID: \"d70f5103-7dc9-4279-9781-38e395da435b\") " pod="openshift-multus/network-metrics-daemon-9x7hv" Jul 20 02:45:07.870291 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:07.870217 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 02:45:07.870291 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:07.870284 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d70f5103-7dc9-4279-9781-38e395da435b-metrics-certs podName:d70f5103-7dc9-4279-9781-38e395da435b nodeName:}" failed. No retries permitted until 2026-07-20 02:45:08.870263657 +0000 UTC m=+4.069512783 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d70f5103-7dc9-4279-9781-38e395da435b-metrics-certs") pod "network-metrics-daemon-9x7hv" (UID: "d70f5103-7dc9-4279-9781-38e395da435b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 02:45:07.870396 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:07.870218 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 02:45:07.870396 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:07.870365 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c7d134c-4e38-481c-8a81-e03434badf7d-original-pull-secret podName:6c7d134c-4e38-481c-8a81-e03434badf7d nodeName:}" failed. No retries permitted until 2026-07-20 02:45:08.870347841 +0000 UTC m=+4.069596970 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6c7d134c-4e38-481c-8a81-e03434badf7d-original-pull-secret") pod "global-pull-secret-syncer-t9spk" (UID: "6c7d134c-4e38-481c-8a81-e03434badf7d") : object "kube-system"/"original-pull-secret" not registered Jul 20 02:45:07.970835 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:07.970795 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rsdj\" (UniqueName: \"kubernetes.io/projected/26701b30-ec4b-4dec-80df-b483d7edb77d-kube-api-access-4rsdj\") pod \"network-check-target-5cxtl\" (UID: \"26701b30-ec4b-4dec-80df-b483d7edb77d\") " pod="openshift-network-diagnostics/network-check-target-5cxtl" Jul 20 02:45:07.971014 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:07.970939 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 02:45:07.971014 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:07.970957 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 02:45:07.971014 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:07.970969 2558 projected.go:196] Error preparing data for projected volume kube-api-access-4rsdj for pod openshift-network-diagnostics/network-check-target-5cxtl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 02:45:07.971182 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:07.971035 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/26701b30-ec4b-4dec-80df-b483d7edb77d-kube-api-access-4rsdj podName:26701b30-ec4b-4dec-80df-b483d7edb77d nodeName:}" failed. No retries permitted until 2026-07-20 02:45:08.97101693 +0000 UTC m=+4.170266072 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4rsdj" (UniqueName: "kubernetes.io/projected/26701b30-ec4b-4dec-80df-b483d7edb77d-kube-api-access-4rsdj") pod "network-check-target-5cxtl" (UID: "26701b30-ec4b-4dec-80df-b483d7edb77d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 02:45:08.290809 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:08.290722 2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-19 02:40:07 +0000 UTC" deadline="2028-05-02 05:25:02.883352112 +0000 UTC" Jul 20 02:45:08.290809 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:08.290757 2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15650h39m54.592599868s" Jul 20 02:45:08.489691 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:08.489658 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2be66185_38c5_43ac_ba3b_766093577e64.slice/crio-46ab7afc9de3de40997f8cf7f6788903870e7289fee93a212ef6a3f528867dae WatchSource:0}: Error finding container 46ab7afc9de3de40997f8cf7f6788903870e7289fee93a212ef6a3f528867dae: Status 404 returned error can't find the container with id 46ab7afc9de3de40997f8cf7f6788903870e7289fee93a212ef6a3f528867dae Jul 20 02:45:08.491238 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:08.491211 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ef0be8f_01c8_4454_8713_630d2ca25963.slice/crio-f5631590911d3d50a351953889831ad3ef0c662340ef8d85ee1136533f62aaf1 WatchSource:0}: Error finding container f5631590911d3d50a351953889831ad3ef0c662340ef8d85ee1136533f62aaf1: Status 404 returned error can't find the container with id f5631590911d3d50a351953889831ad3ef0c662340ef8d85ee1136533f62aaf1 Jul 20 02:45:08.492480 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:08.492454 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b8f7019_0b80_42f2_be36_db424e632410.slice/crio-ead8bff7dc1bb40544510b7b83c2a42f181641769576ce17fadee70ebbdd0c10 WatchSource:0}: Error finding container ead8bff7dc1bb40544510b7b83c2a42f181641769576ce17fadee70ebbdd0c10: Status 404 returned error can't find the container with id ead8bff7dc1bb40544510b7b83c2a42f181641769576ce17fadee70ebbdd0c10 Jul 20 02:45:08.497991 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:08.497733 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90740648_1f62_4e9d_a8af_67716403522e.slice/crio-97517668f8d3c20314ec32d9be3452853ee024996b4172888393e47f287743ec WatchSource:0}: Error finding container 97517668f8d3c20314ec32d9be3452853ee024996b4172888393e47f287743ec: Status 404 returned error can't find the container with id 97517668f8d3c20314ec32d9be3452853ee024996b4172888393e47f287743ec Jul 20 02:45:08.503604 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:08.503581 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02007d87_46f6_4589_ac19_d1a89cf5465c.slice/crio-dad5c99234841bbfdf8ebb7bbb2762b7d614c1490b2ebcedb77e3618d96619f1 WatchSource:0}: Error finding container dad5c99234841bbfdf8ebb7bbb2762b7d614c1490b2ebcedb77e3618d96619f1: Status 404 returned error can't find the container with id dad5c99234841bbfdf8ebb7bbb2762b7d614c1490b2ebcedb77e3618d96619f1 Jul 20 02:45:08.504446 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:08.504420 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc361af25_375d_4bff_bdb9_53f137cea0f6.slice/crio-d2667d25d499accf61ba1b11f3d5e56a5b71cefb4955a9eba901bf5c17028ac4 WatchSource:0}: Error finding container d2667d25d499accf61ba1b11f3d5e56a5b71cefb4955a9eba901bf5c17028ac4: Status 404 returned error can't find the container with id d2667d25d499accf61ba1b11f3d5e56a5b71cefb4955a9eba901bf5c17028ac4 Jul 20 02:45:08.877396 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:08.877186 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6c7d134c-4e38-481c-8a81-e03434badf7d-original-pull-secret\") pod \"global-pull-secret-syncer-t9spk\" (UID: \"6c7d134c-4e38-481c-8a81-e03434badf7d\") " pod="kube-system/global-pull-secret-syncer-t9spk" Jul 20 02:45:08.877568 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:08.877433 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d70f5103-7dc9-4279-9781-38e395da435b-metrics-certs\") pod \"network-metrics-daemon-9x7hv\" (UID: \"d70f5103-7dc9-4279-9781-38e395da435b\") " pod="openshift-multus/network-metrics-daemon-9x7hv" Jul 20 02:45:08.877568 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:08.877332 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 02:45:08.877676 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:08.877602 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c7d134c-4e38-481c-8a81-e03434badf7d-original-pull-secret podName:6c7d134c-4e38-481c-8a81-e03434badf7d nodeName:}" failed. No retries permitted until 2026-07-20 02:45:10.877582693 +0000 UTC m=+6.076831841 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6c7d134c-4e38-481c-8a81-e03434badf7d-original-pull-secret") pod "global-pull-secret-syncer-t9spk" (UID: "6c7d134c-4e38-481c-8a81-e03434badf7d") : object "kube-system"/"original-pull-secret" not registered Jul 20 02:45:08.877845 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:08.877541 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 02:45:08.877845 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:08.877813 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d70f5103-7dc9-4279-9781-38e395da435b-metrics-certs podName:d70f5103-7dc9-4279-9781-38e395da435b nodeName:}" failed. No retries permitted until 2026-07-20 02:45:10.877793517 +0000 UTC m=+6.077042659 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d70f5103-7dc9-4279-9781-38e395da435b-metrics-certs") pod "network-metrics-daemon-9x7hv" (UID: "d70f5103-7dc9-4279-9781-38e395da435b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 02:45:08.978159 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:08.978119 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rsdj\" (UniqueName: \"kubernetes.io/projected/26701b30-ec4b-4dec-80df-b483d7edb77d-kube-api-access-4rsdj\") pod \"network-check-target-5cxtl\" (UID: \"26701b30-ec4b-4dec-80df-b483d7edb77d\") " pod="openshift-network-diagnostics/network-check-target-5cxtl" Jul 20 02:45:08.978347 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:08.978287 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 02:45:08.978347 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:08.978308 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 02:45:08.978347 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:08.978319 2558 projected.go:196] Error preparing data for projected volume kube-api-access-4rsdj for pod openshift-network-diagnostics/network-check-target-5cxtl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 02:45:08.978518 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:08.978384 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/26701b30-ec4b-4dec-80df-b483d7edb77d-kube-api-access-4rsdj podName:26701b30-ec4b-4dec-80df-b483d7edb77d nodeName:}" failed. No retries permitted until 2026-07-20 02:45:10.978365785 +0000 UTC m=+6.177614929 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4rsdj" (UniqueName: "kubernetes.io/projected/26701b30-ec4b-4dec-80df-b483d7edb77d-kube-api-access-4rsdj") pod "network-check-target-5cxtl" (UID: "26701b30-ec4b-4dec-80df-b483d7edb77d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 02:45:09.291668 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:09.291622 2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-19 02:40:07 +0000 UTC" deadline="2028-02-15 10:50:50.535621741 +0000 UTC" Jul 20 02:45:09.291668 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:09.291666 2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13808h5m41.243960187s" Jul 20 02:45:09.422518 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:09.421525 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cxtl" Jul 20 02:45:09.422518 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:09.421648 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5cxtl" podUID="26701b30-ec4b-4dec-80df-b483d7edb77d" Jul 20 02:45:09.422518 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:09.422107 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t9spk" Jul 20 02:45:09.422518 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:09.422211 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t9spk" podUID="6c7d134c-4e38-481c-8a81-e03434badf7d" Jul 20 02:45:09.422518 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:09.422279 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9x7hv" Jul 20 02:45:09.422518 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:09.422355 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9x7hv" podUID="d70f5103-7dc9-4279-9781-38e395da435b" Jul 20 02:45:09.438207 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:09.437263 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-139.ec2.internal" event={"ID":"ceae9fddccd84171bbba0a3e9c0148c8","Type":"ContainerStarted","Data":"416206a18bc2b62aa0b513064d61b5476cca75bcbd53cc316c8287c20293c0dd"} Jul 20 02:45:09.465062 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:09.464913 2558 generic.go:356] "Generic (PLEG): container finished" podID="39560757f7588014bf49cd290ec77fb5" containerID="47df3eea01339c87814d7c01340f3258c0cfcbe6e61a55fb845332efc9378faf" exitCode=0 Jul 20 02:45:09.465062 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:09.464991 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal" event={"ID":"39560757f7588014bf49cd290ec77fb5","Type":"ContainerDied","Data":"47df3eea01339c87814d7c01340f3258c0cfcbe6e61a55fb845332efc9378faf"} Jul 20 02:45:09.477798 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:09.477765 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d5bdr" event={"ID":"e8d00f9d-e63a-44a7-8a2c-268dc6787c75","Type":"ContainerStarted","Data":"17412277b625df83061a73574f41808f5e21f7de8a15f443fab64ed1e4b38981"} Jul 20 02:45:09.498856 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:09.498818 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pw2tj" event={"ID":"90740648-1f62-4e9d-a8af-67716403522e","Type":"ContainerStarted","Data":"97517668f8d3c20314ec32d9be3452853ee024996b4172888393e47f287743ec"} Jul 20 02:45:09.508397 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:09.508320 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5jp8t" event={"ID":"1b8f7019-0b80-42f2-be36-db424e632410","Type":"ContainerStarted","Data":"ead8bff7dc1bb40544510b7b83c2a42f181641769576ce17fadee70ebbdd0c10"} Jul 20 02:45:09.512240 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:09.511218 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-139.ec2.internal" podStartSLOduration=3.511200707 podStartE2EDuration="3.511200707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 02:45:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:45:09.453611435 +0000 UTC m=+4.652860578" watchObservedRunningTime="2026-07-20 02:45:09.511200707 +0000 UTC m=+4.710449854" Jul 20 02:45:09.512387 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:09.512313 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lvkqp" event={"ID":"c361af25-375d-4bff-bdb9-53f137cea0f6","Type":"ContainerStarted","Data":"d2667d25d499accf61ba1b11f3d5e56a5b71cefb4955a9eba901bf5c17028ac4"} Jul 20 02:45:09.515420 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:09.515394 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" event={"ID":"cb6c0d96-0866-493d-9e19-7f63dff241ad","Type":"ContainerStarted","Data":"fe422b28c6d82049c57e02ca068a3c7653e61154e2b303ae9c5d073a2f221405"} Jul 20 02:45:09.517329 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:09.517305 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4kbnx" event={"ID":"02007d87-46f6-4589-ac19-d1a89cf5465c","Type":"ContainerStarted","Data":"dad5c99234841bbfdf8ebb7bbb2762b7d614c1490b2ebcedb77e3618d96619f1"} Jul 20 02:45:09.521449 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:09.521428 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9qxnl" event={"ID":"504cb1f0-841a-49f7-add8-2d4742a1cb68","Type":"ContainerStarted","Data":"a720f1d80feca01771b6c19258899c96383f5e43eba6900aa8abb581c2c9be9c"} Jul 20 02:45:09.524031 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:09.524010 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd68h" event={"ID":"9ef0be8f-01c8-4454-8713-630d2ca25963","Type":"ContainerStarted","Data":"f5631590911d3d50a351953889831ad3ef0c662340ef8d85ee1136533f62aaf1"} Jul 20 02:45:09.526685 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:09.526665 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mvvbt" event={"ID":"2be66185-38c5-43ac-ba3b-766093577e64","Type":"ContainerStarted","Data":"46ab7afc9de3de40997f8cf7f6788903870e7289fee93a212ef6a3f528867dae"} Jul 20 02:45:10.544213 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:10.543460 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal" event={"ID":"39560757f7588014bf49cd290ec77fb5","Type":"ContainerStarted","Data":"a48029e6272dbc46c17b900a85522084f9e7cbf1abad034a7ac264f9c222f667"} Jul 20 02:45:10.908200 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:10.908154 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6c7d134c-4e38-481c-8a81-e03434badf7d-original-pull-secret\") pod \"global-pull-secret-syncer-t9spk\" (UID: \"6c7d134c-4e38-481c-8a81-e03434badf7d\") " pod="kube-system/global-pull-secret-syncer-t9spk" Jul 20 02:45:10.908408 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:10.908229 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d70f5103-7dc9-4279-9781-38e395da435b-metrics-certs\") pod \"network-metrics-daemon-9x7hv\" (UID: \"d70f5103-7dc9-4279-9781-38e395da435b\") " pod="openshift-multus/network-metrics-daemon-9x7hv" Jul 20 02:45:10.908408 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:10.908304 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 02:45:10.908408 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:10.908387 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c7d134c-4e38-481c-8a81-e03434badf7d-original-pull-secret podName:6c7d134c-4e38-481c-8a81-e03434badf7d nodeName:}" failed. No retries permitted until 2026-07-20 02:45:14.908365244 +0000 UTC m=+10.107614381 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6c7d134c-4e38-481c-8a81-e03434badf7d-original-pull-secret") pod "global-pull-secret-syncer-t9spk" (UID: "6c7d134c-4e38-481c-8a81-e03434badf7d") : object "kube-system"/"original-pull-secret" not registered Jul 20 02:45:10.908585 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:10.908414 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 02:45:10.908585 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:10.908478 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d70f5103-7dc9-4279-9781-38e395da435b-metrics-certs podName:d70f5103-7dc9-4279-9781-38e395da435b nodeName:}" failed. No retries permitted until 2026-07-20 02:45:14.908460287 +0000 UTC m=+10.107709425 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d70f5103-7dc9-4279-9781-38e395da435b-metrics-certs") pod "network-metrics-daemon-9x7hv" (UID: "d70f5103-7dc9-4279-9781-38e395da435b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 02:45:11.008888 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:11.008835 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rsdj\" (UniqueName: \"kubernetes.io/projected/26701b30-ec4b-4dec-80df-b483d7edb77d-kube-api-access-4rsdj\") pod \"network-check-target-5cxtl\" (UID: \"26701b30-ec4b-4dec-80df-b483d7edb77d\") " pod="openshift-network-diagnostics/network-check-target-5cxtl" Jul 20 02:45:11.009115 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:11.009074 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 02:45:11.009115 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:11.009116 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 02:45:11.009253 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:11.009130 2558 projected.go:196] Error preparing data for projected volume kube-api-access-4rsdj for pod openshift-network-diagnostics/network-check-target-5cxtl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 02:45:11.009253 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:11.009188 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/26701b30-ec4b-4dec-80df-b483d7edb77d-kube-api-access-4rsdj podName:26701b30-ec4b-4dec-80df-b483d7edb77d nodeName:}" failed. No retries permitted until 2026-07-20 02:45:15.009170087 +0000 UTC m=+10.208419214 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4rsdj" (UniqueName: "kubernetes.io/projected/26701b30-ec4b-4dec-80df-b483d7edb77d-kube-api-access-4rsdj") pod "network-check-target-5cxtl" (UID: "26701b30-ec4b-4dec-80df-b483d7edb77d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 02:45:11.421348 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:11.420518 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9x7hv" Jul 20 02:45:11.421348 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:11.420644 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9x7hv" podUID="d70f5103-7dc9-4279-9781-38e395da435b" Jul 20 02:45:11.421348 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:11.420733 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cxtl" Jul 20 02:45:11.421348 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:11.420800 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5cxtl" podUID="26701b30-ec4b-4dec-80df-b483d7edb77d" Jul 20 02:45:11.421348 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:11.420856 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t9spk" Jul 20 02:45:11.421348 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:11.420923 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t9spk" podUID="6c7d134c-4e38-481c-8a81-e03434badf7d" Jul 20 02:45:13.419654 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:13.419220 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cxtl" Jul 20 02:45:13.419654 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:13.419360 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5cxtl" podUID="26701b30-ec4b-4dec-80df-b483d7edb77d" Jul 20 02:45:13.419654 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:13.419219 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9x7hv" Jul 20 02:45:13.419654 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:13.419402 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t9spk" Jul 20 02:45:13.419654 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:13.419472 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9x7hv" podUID="d70f5103-7dc9-4279-9781-38e395da435b" Jul 20 02:45:13.419654 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:13.419565 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t9spk" podUID="6c7d134c-4e38-481c-8a81-e03434badf7d" Jul 20 02:45:14.942603 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:14.942568 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d70f5103-7dc9-4279-9781-38e395da435b-metrics-certs\") pod \"network-metrics-daemon-9x7hv\" (UID: \"d70f5103-7dc9-4279-9781-38e395da435b\") " pod="openshift-multus/network-metrics-daemon-9x7hv" Jul 20 02:45:14.943084 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:14.942630 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6c7d134c-4e38-481c-8a81-e03434badf7d-original-pull-secret\") pod \"global-pull-secret-syncer-t9spk\" (UID: \"6c7d134c-4e38-481c-8a81-e03434badf7d\") " pod="kube-system/global-pull-secret-syncer-t9spk" Jul 20 02:45:14.943084 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:14.942723 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 02:45:14.943084 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:14.942777 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c7d134c-4e38-481c-8a81-e03434badf7d-original-pull-secret podName:6c7d134c-4e38-481c-8a81-e03434badf7d nodeName:}" failed. No retries permitted until 2026-07-20 02:45:22.942757907 +0000 UTC m=+18.142007049 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6c7d134c-4e38-481c-8a81-e03434badf7d-original-pull-secret") pod "global-pull-secret-syncer-t9spk" (UID: "6c7d134c-4e38-481c-8a81-e03434badf7d") : object "kube-system"/"original-pull-secret" not registered Jul 20 02:45:14.943278 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:14.943208 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 02:45:14.943310 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:14.943280 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d70f5103-7dc9-4279-9781-38e395da435b-metrics-certs podName:d70f5103-7dc9-4279-9781-38e395da435b nodeName:}" failed. No retries permitted until 2026-07-20 02:45:22.943262536 +0000 UTC m=+18.142511667 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d70f5103-7dc9-4279-9781-38e395da435b-metrics-certs") pod "network-metrics-daemon-9x7hv" (UID: "d70f5103-7dc9-4279-9781-38e395da435b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 02:45:15.043366 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:15.043324 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rsdj\" (UniqueName: \"kubernetes.io/projected/26701b30-ec4b-4dec-80df-b483d7edb77d-kube-api-access-4rsdj\") pod \"network-check-target-5cxtl\" (UID: \"26701b30-ec4b-4dec-80df-b483d7edb77d\") " pod="openshift-network-diagnostics/network-check-target-5cxtl" Jul 20 02:45:15.043570 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:15.043504 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 02:45:15.043570 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:15.043526 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 02:45:15.043570 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:15.043539 2558 projected.go:196] Error preparing data for projected volume kube-api-access-4rsdj for pod openshift-network-diagnostics/network-check-target-5cxtl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 02:45:15.043727 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:15.043598 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/26701b30-ec4b-4dec-80df-b483d7edb77d-kube-api-access-4rsdj podName:26701b30-ec4b-4dec-80df-b483d7edb77d nodeName:}" failed. No retries permitted until 2026-07-20 02:45:23.043578191 +0000 UTC m=+18.242827336 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4rsdj" (UniqueName: "kubernetes.io/projected/26701b30-ec4b-4dec-80df-b483d7edb77d-kube-api-access-4rsdj") pod "network-check-target-5cxtl" (UID: "26701b30-ec4b-4dec-80df-b483d7edb77d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 02:45:15.419758 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:15.419720 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cxtl" Jul 20 02:45:15.419946 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:15.419828 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5cxtl" podUID="26701b30-ec4b-4dec-80df-b483d7edb77d" Jul 20 02:45:15.420290 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:15.420267 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t9spk" Jul 20 02:45:15.420696 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:15.420439 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t9spk" podUID="6c7d134c-4e38-481c-8a81-e03434badf7d" Jul 20 02:45:15.420696 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:15.420521 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9x7hv" Jul 20 02:45:15.420696 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:15.420611 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9x7hv" podUID="d70f5103-7dc9-4279-9781-38e395da435b" Jul 20 02:45:17.418500 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:17.418461 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9x7hv" Jul 20 02:45:17.418952 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:17.418461 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t9spk" Jul 20 02:45:17.418952 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:17.418609 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9x7hv" podUID="d70f5103-7dc9-4279-9781-38e395da435b" Jul 20 02:45:17.418952 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:17.418700 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t9spk" podUID="6c7d134c-4e38-481c-8a81-e03434badf7d" Jul 20 02:45:17.418952 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:17.418740 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cxtl" Jul 20 02:45:17.418952 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:17.418806 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5cxtl" podUID="26701b30-ec4b-4dec-80df-b483d7edb77d" Jul 20 02:45:19.418346 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:19.418309 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cxtl" Jul 20 02:45:19.418761 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:19.418319 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t9spk" Jul 20 02:45:19.418761 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:19.418441 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5cxtl" podUID="26701b30-ec4b-4dec-80df-b483d7edb77d" Jul 20 02:45:19.418761 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:19.418470 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9x7hv" Jul 20 02:45:19.418761 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:19.418572 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t9spk" podUID="6c7d134c-4e38-481c-8a81-e03434badf7d" Jul 20 02:45:19.418761 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:19.418662 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9x7hv" podUID="d70f5103-7dc9-4279-9781-38e395da435b" Jul 20 02:45:21.418753 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:21.418718 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9x7hv" Jul 20 02:45:21.419222 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:21.418720 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cxtl" Jul 20 02:45:21.419222 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:21.418862 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9x7hv" podUID="d70f5103-7dc9-4279-9781-38e395da435b" Jul 20 02:45:21.419222 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:21.418930 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5cxtl" podUID="26701b30-ec4b-4dec-80df-b483d7edb77d" Jul 20 02:45:21.419222 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:21.419001 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t9spk" Jul 20 02:45:21.419222 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:21.419078 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t9spk" podUID="6c7d134c-4e38-481c-8a81-e03434badf7d" Jul 20 02:45:23.002110 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:23.002061 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d70f5103-7dc9-4279-9781-38e395da435b-metrics-certs\") pod \"network-metrics-daemon-9x7hv\" (UID: \"d70f5103-7dc9-4279-9781-38e395da435b\") " pod="openshift-multus/network-metrics-daemon-9x7hv" Jul 20 02:45:23.002570 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:23.002164 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6c7d134c-4e38-481c-8a81-e03434badf7d-original-pull-secret\") pod \"global-pull-secret-syncer-t9spk\" (UID: \"6c7d134c-4e38-481c-8a81-e03434badf7d\") " pod="kube-system/global-pull-secret-syncer-t9spk" Jul 20 02:45:23.002570 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:23.002207 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 02:45:23.002570 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:23.002286 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d70f5103-7dc9-4279-9781-38e395da435b-metrics-certs podName:d70f5103-7dc9-4279-9781-38e395da435b nodeName:}" failed. No retries permitted until 2026-07-20 02:45:39.002261087 +0000 UTC m=+34.201510226 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d70f5103-7dc9-4279-9781-38e395da435b-metrics-certs") pod "network-metrics-daemon-9x7hv" (UID: "d70f5103-7dc9-4279-9781-38e395da435b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 02:45:23.002570 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:23.002312 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 02:45:23.002570 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:23.002367 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c7d134c-4e38-481c-8a81-e03434badf7d-original-pull-secret podName:6c7d134c-4e38-481c-8a81-e03434badf7d nodeName:}" failed. No retries permitted until 2026-07-20 02:45:39.002349946 +0000 UTC m=+34.201599084 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6c7d134c-4e38-481c-8a81-e03434badf7d-original-pull-secret") pod "global-pull-secret-syncer-t9spk" (UID: "6c7d134c-4e38-481c-8a81-e03434badf7d") : object "kube-system"/"original-pull-secret" not registered Jul 20 02:45:23.102881 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:23.102842 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rsdj\" (UniqueName: \"kubernetes.io/projected/26701b30-ec4b-4dec-80df-b483d7edb77d-kube-api-access-4rsdj\") pod \"network-check-target-5cxtl\" (UID: \"26701b30-ec4b-4dec-80df-b483d7edb77d\") " pod="openshift-network-diagnostics/network-check-target-5cxtl" Jul 20 02:45:23.103053 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:23.102994 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 02:45:23.103053 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:23.103016 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 02:45:23.103053 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:23.103027 2558 projected.go:196] Error preparing data for projected volume kube-api-access-4rsdj for pod openshift-network-diagnostics/network-check-target-5cxtl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 02:45:23.103299 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:23.103115 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/26701b30-ec4b-4dec-80df-b483d7edb77d-kube-api-access-4rsdj podName:26701b30-ec4b-4dec-80df-b483d7edb77d nodeName:}" failed. No retries permitted until 2026-07-20 02:45:39.103077074 +0000 UTC m=+34.302326219 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4rsdj" (UniqueName: "kubernetes.io/projected/26701b30-ec4b-4dec-80df-b483d7edb77d-kube-api-access-4rsdj") pod "network-check-target-5cxtl" (UID: "26701b30-ec4b-4dec-80df-b483d7edb77d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 02:45:23.418424 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:23.418386 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t9spk" Jul 20 02:45:23.418604 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:23.418386 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cxtl" Jul 20 02:45:23.418604 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:23.418524 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t9spk" podUID="6c7d134c-4e38-481c-8a81-e03434badf7d" Jul 20 02:45:23.418719 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:23.418386 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9x7hv" Jul 20 02:45:23.418719 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:23.418606 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5cxtl" podUID="26701b30-ec4b-4dec-80df-b483d7edb77d" Jul 20 02:45:23.418819 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:23.418724 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9x7hv" podUID="d70f5103-7dc9-4279-9781-38e395da435b" Jul 20 02:45:25.419587 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:25.419507 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cxtl" Jul 20 02:45:25.420085 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:25.419602 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t9spk" Jul 20 02:45:25.420085 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:25.419635 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9x7hv" Jul 20 02:45:25.420085 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:25.419732 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9x7hv" podUID="d70f5103-7dc9-4279-9781-38e395da435b" Jul 20 02:45:25.420085 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:25.419737 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5cxtl" podUID="26701b30-ec4b-4dec-80df-b483d7edb77d" Jul 20 02:45:25.420085 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:25.419842 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t9spk" podUID="6c7d134c-4e38-481c-8a81-e03434badf7d" Jul 20 02:45:27.418581 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:27.418540 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cxtl" Jul 20 02:45:27.418989 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:27.418551 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9x7hv" Jul 20 02:45:27.418989 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:27.418672 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5cxtl" podUID="26701b30-ec4b-4dec-80df-b483d7edb77d" Jul 20 02:45:27.418989 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:27.418561 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t9spk" Jul 20 02:45:27.418989 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:27.418750 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9x7hv" podUID="d70f5103-7dc9-4279-9781-38e395da435b" Jul 20 02:45:27.418989 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:27.418886 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t9spk" podUID="6c7d134c-4e38-481c-8a81-e03434badf7d" Jul 20 02:45:27.584994 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:27.584287 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pw2tj" event={"ID":"90740648-1f62-4e9d-a8af-67716403522e","Type":"ContainerStarted","Data":"ea909cf562a43206aff039a95af0d96983c2395c2df63bb558c9cfbaea993b17"} Jul 20 02:45:27.601139 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:27.600290 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-139.ec2.internal" podStartSLOduration=21.600272033 podStartE2EDuration="21.600272033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 02:45:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:45:10.559282906 +0000 UTC m=+5.758532053" watchObservedRunningTime="2026-07-20 02:45:27.600272033 +0000 UTC m=+22.799521182" Jul 20 02:45:28.588393 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:28.588143 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lvkqp" event={"ID":"c361af25-375d-4bff-bdb9-53f137cea0f6","Type":"ContainerStarted","Data":"26128383ec63be121b6fa303eca0526896e12256c865fd494c2ca0bd89039a34"} Jul 20 02:45:28.591789 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:28.591754 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" event={"ID":"cb6c0d96-0866-493d-9e19-7f63dff241ad","Type":"ContainerStarted","Data":"49d82ec6b3c04739709d196cee7fd52778b10abec8b97eb5c4f954cab0564479"} Jul 20 02:45:28.591938 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:28.591837 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" event={"ID":"cb6c0d96-0866-493d-9e19-7f63dff241ad","Type":"ContainerStarted","Data":"6fa5697d93fd0fa05ee58e5b93bd4b747221188a56ab73823b284132cb6a9c85"} Jul 20 02:45:28.591938 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:28.591868 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" event={"ID":"cb6c0d96-0866-493d-9e19-7f63dff241ad","Type":"ContainerStarted","Data":"edbf53964b0b9da0f594f0bc916ffd3cbaa6af5319f7e568edabf7abd6c8dba9"} Jul 20 02:45:28.591938 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:28.591895 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" event={"ID":"cb6c0d96-0866-493d-9e19-7f63dff241ad","Type":"ContainerStarted","Data":"c3916a4be6259262ec7820ecd1170c1e7f54d8b973faaa04d73092cc3746c769"} Jul 20 02:45:28.591938 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:28.591933 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" event={"ID":"cb6c0d96-0866-493d-9e19-7f63dff241ad","Type":"ContainerStarted","Data":"2fdc1126bb7e50b0acaef30b20ac069349083f56aaf4ffb365e15422ac7a01cb"} Jul 20 02:45:28.592246 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:28.591969 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" event={"ID":"cb6c0d96-0866-493d-9e19-7f63dff241ad","Type":"ContainerStarted","Data":"4bfb91504ae22ac880abefe784f5414e664f8820a8c85e6f36a3c7309c5a1e51"} Jul 20 02:45:28.593504 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:28.593469 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9qxnl" event={"ID":"504cb1f0-841a-49f7-add8-2d4742a1cb68","Type":"ContainerStarted","Data":"23e2e728e83d437776eb3b79970c6523a33eeea976f23e9011894eb8965dac7f"} Jul 20 02:45:28.594929 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:28.594907 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd68h" event={"ID":"9ef0be8f-01c8-4454-8713-630d2ca25963","Type":"ContainerStarted","Data":"d92612b6d174b146f057132c5e5ee511f4dc8a9986d9da4aeeea3ddbf462d3d1"} Jul 20 02:45:28.596286 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:28.596257 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mvvbt" event={"ID":"2be66185-38c5-43ac-ba3b-766093577e64","Type":"ContainerStarted","Data":"3770f9e33eeaa52546d776206860a8846a57a6cb7433520f5f1404f01ff47781"} Jul 20 02:45:28.597665 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:28.597645 2558 generic.go:356] "Generic (PLEG): container finished" podID="e8d00f9d-e63a-44a7-8a2c-268dc6787c75" containerID="584974d0d1071b612e9613646a601b9254ef1f6f4d8fdc4d7fba80898bf85ee9" exitCode=0 Jul 20 02:45:28.597766 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:28.597708 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d5bdr" event={"ID":"e8d00f9d-e63a-44a7-8a2c-268dc6787c75","Type":"ContainerDied","Data":"584974d0d1071b612e9613646a601b9254ef1f6f4d8fdc4d7fba80898bf85ee9"} Jul 20 02:45:28.600553 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:28.600528 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5jp8t" event={"ID":"1b8f7019-0b80-42f2-be36-db424e632410","Type":"ContainerStarted","Data":"d88a45da6bad6db83662fffc885d9e0c5537c0c3205cff06c261694c3265d6b2"} Jul 20 02:45:28.601652 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:28.601613 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-pw2tj" podStartSLOduration=4.67676484 podStartE2EDuration="23.601602161s" totalImagesPullingTime="18.924837321s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 02:45:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:45:27.600954823 +0000 UTC m=+22.800203972" watchObservedRunningTime="2026-07-20 02:45:28.601602161 +0000 UTC m=+23.800851308" Jul 20 02:45:28.601989 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:28.601970 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-lvkqp" podStartSLOduration=4.681641988 podStartE2EDuration="23.601964886s" totalImagesPullingTime="18.920322898s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 02:45:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:45:28.601852058 +0000 UTC m=+23.801101205" watchObservedRunningTime="2026-07-20 02:45:28.601964886 +0000 UTC m=+23.801214024" Jul 20 02:45:28.618918 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:28.618879 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-5jp8t" podStartSLOduration=4.637278438 podStartE2EDuration="23.618866222s" totalImagesPullingTime="18.981587784s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 02:45:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:45:28.618484693 +0000 UTC m=+23.817733843" watchObservedRunningTime="2026-07-20 02:45:28.618866222 +0000 UTC m=+23.818115368" Jul 20 02:45:28.631463 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:28.631423 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mvvbt" podStartSLOduration=4.695089908 podStartE2EDuration="23.631411953s" totalImagesPullingTime="18.936322045s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:45:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:45:28.631409487 +0000 UTC m=+23.830658634" watchObservedRunningTime="2026-07-20 02:45:28.631411953 +0000 UTC m=+23.830661099" Jul 20 02:45:28.644908 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:28.644870 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9qxnl" podStartSLOduration=4.7402428709999995 podStartE2EDuration="23.644863015s" totalImagesPullingTime="18.904620144s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 02:45:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:45:28.64459353 +0000 UTC m=+23.843842677" watchObservedRunningTime="2026-07-20 02:45:28.644863015 +0000 UTC m=+23.844112161" Jul 20 02:45:29.065649 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:29.065623 2558 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 20 02:45:29.332014 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:29.331853 2558 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-20T02:45:29.065640361Z","UUID":"7f5d2b5e-02f4-406e-8a5a-fa4637f3a819","Handler":null,"Name":"","Endpoint":""} Jul 20 02:45:29.333505 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:29.333476 2558 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 20 02:45:29.333635 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:29.333523 2558 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 20 02:45:29.418883 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:29.418853 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t9spk" Jul 20 02:45:29.419059 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:29.418853 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cxtl" Jul 20 02:45:29.419059 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:29.418980 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t9spk" podUID="6c7d134c-4e38-481c-8a81-e03434badf7d" Jul 20 02:45:29.419059 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:29.418853 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9x7hv" Jul 20 02:45:29.419253 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:29.419059 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5cxtl" podUID="26701b30-ec4b-4dec-80df-b483d7edb77d" Jul 20 02:45:29.419253 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:29.419191 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9x7hv" podUID="d70f5103-7dc9-4279-9781-38e395da435b" Jul 20 02:45:29.604395 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:29.604308 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4kbnx" event={"ID":"02007d87-46f6-4589-ac19-d1a89cf5465c","Type":"ContainerStarted","Data":"6d2b9abc3fc843e0c807e652479c7346cb0c7f62817165ef636d0619f13d135b"} Jul 20 02:45:29.606867 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:29.606826 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd68h" event={"ID":"9ef0be8f-01c8-4454-8713-630d2ca25963","Type":"ContainerStarted","Data":"faf7f9488bebe9fa2c01dbf3531fff1570b192fd15fa337720be71aaf947661f"} Jul 20 02:45:30.612764 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:30.612675 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" event={"ID":"cb6c0d96-0866-493d-9e19-7f63dff241ad","Type":"ContainerStarted","Data":"32558935667cde2c072c915cb40193df4ce732f1061d250fe8dc48884bd25c2a"} Jul 20 02:45:30.614918 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:30.614890 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd68h" event={"ID":"9ef0be8f-01c8-4454-8713-630d2ca25963","Type":"ContainerStarted","Data":"383120356f13bde0a77af72557b109404edf053dc0724ab4bebf7f5398f8023f"} Jul 20 02:45:30.632896 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:30.632854 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hd68h" podStartSLOduration=4.14938694 podStartE2EDuration="25.632840147s" totalImagesPullingTime="21.483453207s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 02:45:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:45:30.632596387 +0000 UTC m=+25.831845548" watchObservedRunningTime="2026-07-20 02:45:30.632840147 +0000 UTC m=+25.832089293" Jul 20 02:45:30.633192 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:30.633169 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4kbnx" podStartSLOduration=6.7317497490000004 podStartE2EDuration="25.633163193s" totalImagesPullingTime="18.901413444s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 02:45:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:45:29.618870654 +0000 UTC m=+24.818119802" watchObservedRunningTime="2026-07-20 02:45:30.633163193 +0000 UTC m=+25.832412339" Jul 20 02:45:31.418789 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:31.418754 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t9spk" Jul 20 02:45:31.418952 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:31.418762 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9x7hv" Jul 20 02:45:31.418952 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:31.418871 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t9spk" podUID="6c7d134c-4e38-481c-8a81-e03434badf7d" Jul 20 02:45:31.418952 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:31.418890 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cxtl" Jul 20 02:45:31.419123 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:31.418973 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9x7hv" podUID="d70f5103-7dc9-4279-9781-38e395da435b" Jul 20 02:45:31.419123 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:31.419056 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5cxtl" podUID="26701b30-ec4b-4dec-80df-b483d7edb77d" Jul 20 02:45:31.574056 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:31.574020 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mvvbt" Jul 20 02:45:31.574660 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:31.574637 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mvvbt" Jul 20 02:45:31.617221 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:31.617190 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mvvbt" Jul 20 02:45:31.617663 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:31.617629 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mvvbt" Jul 20 02:45:33.422272 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:33.421952 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9x7hv" Jul 20 02:45:33.422988 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:33.422000 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t9spk" Jul 20 02:45:33.422988 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:33.422353 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9x7hv" podUID="d70f5103-7dc9-4279-9781-38e395da435b" Jul 20 02:45:33.422988 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:33.422436 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t9spk" podUID="6c7d134c-4e38-481c-8a81-e03434badf7d" Jul 20 02:45:33.422988 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:33.422000 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cxtl" Jul 20 02:45:33.422988 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:33.422517 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5cxtl" podUID="26701b30-ec4b-4dec-80df-b483d7edb77d" Jul 20 02:45:33.622425 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:33.622394 2558 generic.go:356] "Generic (PLEG): container finished" podID="e8d00f9d-e63a-44a7-8a2c-268dc6787c75" containerID="ca5accdb1d69ac679597b05bded91d1633a4944f7fd3f624249ca468f90c05df" exitCode=0 Jul 20 02:45:33.622587 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:33.622479 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d5bdr" event={"ID":"e8d00f9d-e63a-44a7-8a2c-268dc6787c75","Type":"ContainerDied","Data":"ca5accdb1d69ac679597b05bded91d1633a4944f7fd3f624249ca468f90c05df"} Jul 20 02:45:33.626366 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:33.626340 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" event={"ID":"cb6c0d96-0866-493d-9e19-7f63dff241ad","Type":"ContainerStarted","Data":"f9b0048071ece4836c50727dbe6379f78666167e236600326546eef5e80a9213"} Jul 20 02:45:33.626561 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:33.626543 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:33.626648 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:33.626604 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:33.641901 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:33.641880 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:33.670153 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:33.670113 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" podStartSLOduration=9.683980462 podStartE2EDuration="28.670083744s" totalImagesPullingTime="18.986103282s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 02:45:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:45:33.668511882 +0000 UTC m=+28.867761029" watchObservedRunningTime="2026-07-20 02:45:33.670083744 +0000 UTC m=+28.869332891" Jul 20 02:45:34.629528 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:34.629499 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:34.644184 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:34.644161 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:45:34.975110 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:34.975017 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-t9spk"] Jul 20 02:45:34.975255 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:34.975185 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t9spk" Jul 20 02:45:34.975317 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:34.975267 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t9spk" podUID="6c7d134c-4e38-481c-8a81-e03434badf7d" Jul 20 02:45:34.977783 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:34.977757 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5cxtl"] Jul 20 02:45:34.977917 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:34.977868 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cxtl" Jul 20 02:45:34.977979 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:34.977963 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5cxtl" podUID="26701b30-ec4b-4dec-80df-b483d7edb77d" Jul 20 02:45:34.985966 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:34.985945 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9x7hv"] Jul 20 02:45:34.986056 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:34.986037 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9x7hv" Jul 20 02:45:34.986141 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:34.986126 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9x7hv" podUID="d70f5103-7dc9-4279-9781-38e395da435b" Jul 20 02:45:35.633360 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:35.633327 2558 generic.go:356] "Generic (PLEG): container finished" podID="e8d00f9d-e63a-44a7-8a2c-268dc6787c75" containerID="c96ebb886d9a4d0cc361f8e88c7c9dd886c195e9be708be9a13bbae03c8fb729" exitCode=0 Jul 20 02:45:35.633851 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:35.633406 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d5bdr" event={"ID":"e8d00f9d-e63a-44a7-8a2c-268dc6787c75","Type":"ContainerDied","Data":"c96ebb886d9a4d0cc361f8e88c7c9dd886c195e9be708be9a13bbae03c8fb729"} Jul 20 02:45:36.418435 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:36.418402 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9x7hv" Jul 20 02:45:36.418610 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:36.418402 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cxtl" Jul 20 02:45:36.418610 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:36.418542 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9x7hv" podUID="d70f5103-7dc9-4279-9781-38e395da435b" Jul 20 02:45:36.418610 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:36.418418 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t9spk" Jul 20 02:45:36.418762 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:36.418635 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5cxtl" podUID="26701b30-ec4b-4dec-80df-b483d7edb77d" Jul 20 02:45:36.418762 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:36.418668 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t9spk" podUID="6c7d134c-4e38-481c-8a81-e03434badf7d" Jul 20 02:45:37.639801 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:37.639615 2558 generic.go:356] "Generic (PLEG): container finished" podID="e8d00f9d-e63a-44a7-8a2c-268dc6787c75" containerID="041dce567d8fac21a30735c2b11ed9252c61299277af2d38fc2483c58c865095" exitCode=0 Jul 20 02:45:37.640212 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:37.639685 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d5bdr" event={"ID":"e8d00f9d-e63a-44a7-8a2c-268dc6787c75","Type":"ContainerDied","Data":"041dce567d8fac21a30735c2b11ed9252c61299277af2d38fc2483c58c865095"} Jul 20 02:45:38.418435 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:38.418395 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9x7hv" Jul 20 02:45:38.418435 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:38.418416 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cxtl" Jul 20 02:45:38.418673 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:38.418507 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t9spk" Jul 20 02:45:38.418673 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:38.418520 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5cxtl" podUID="26701b30-ec4b-4dec-80df-b483d7edb77d" Jul 20 02:45:38.418673 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:38.418580 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t9spk" podUID="6c7d134c-4e38-481c-8a81-e03434badf7d" Jul 20 02:45:38.418823 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:38.418673 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9x7hv" podUID="d70f5103-7dc9-4279-9781-38e395da435b" Jul 20 02:45:39.018880 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:39.018849 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6c7d134c-4e38-481c-8a81-e03434badf7d-original-pull-secret\") pod \"global-pull-secret-syncer-t9spk\" (UID: \"6c7d134c-4e38-481c-8a81-e03434badf7d\") " pod="kube-system/global-pull-secret-syncer-t9spk" Jul 20 02:45:39.019352 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:39.018910 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d70f5103-7dc9-4279-9781-38e395da435b-metrics-certs\") pod \"network-metrics-daemon-9x7hv\" (UID: \"d70f5103-7dc9-4279-9781-38e395da435b\") " pod="openshift-multus/network-metrics-daemon-9x7hv" Jul 20 02:45:39.019352 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:39.018986 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 02:45:39.019352 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:39.019047 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 02:45:39.019352 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:39.019062 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c7d134c-4e38-481c-8a81-e03434badf7d-original-pull-secret podName:6c7d134c-4e38-481c-8a81-e03434badf7d nodeName:}" failed. No retries permitted until 2026-07-20 02:46:11.019041624 +0000 UTC m=+66.218290749 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6c7d134c-4e38-481c-8a81-e03434badf7d-original-pull-secret") pod "global-pull-secret-syncer-t9spk" (UID: "6c7d134c-4e38-481c-8a81-e03434badf7d") : object "kube-system"/"original-pull-secret" not registered Jul 20 02:45:39.019352 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:39.019115 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d70f5103-7dc9-4279-9781-38e395da435b-metrics-certs podName:d70f5103-7dc9-4279-9781-38e395da435b nodeName:}" failed. No retries permitted until 2026-07-20 02:46:11.019077788 +0000 UTC m=+66.218326917 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d70f5103-7dc9-4279-9781-38e395da435b-metrics-certs") pod "network-metrics-daemon-9x7hv" (UID: "d70f5103-7dc9-4279-9781-38e395da435b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 02:45:39.119896 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:39.119859 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rsdj\" (UniqueName: \"kubernetes.io/projected/26701b30-ec4b-4dec-80df-b483d7edb77d-kube-api-access-4rsdj\") pod \"network-check-target-5cxtl\" (UID: \"26701b30-ec4b-4dec-80df-b483d7edb77d\") " pod="openshift-network-diagnostics/network-check-target-5cxtl" Jul 20 02:45:39.120068 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:39.120032 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 02:45:39.120068 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:39.120060 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 02:45:39.120173 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:39.120073 2558 projected.go:196] Error preparing data for projected volume kube-api-access-4rsdj for pod openshift-network-diagnostics/network-check-target-5cxtl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 02:45:39.120173 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:39.120154 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/26701b30-ec4b-4dec-80df-b483d7edb77d-kube-api-access-4rsdj podName:26701b30-ec4b-4dec-80df-b483d7edb77d nodeName:}" failed. No retries permitted until 2026-07-20 02:46:11.120135941 +0000 UTC m=+66.319385147 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-4rsdj" (UniqueName: "kubernetes.io/projected/26701b30-ec4b-4dec-80df-b483d7edb77d-kube-api-access-4rsdj") pod "network-check-target-5cxtl" (UID: "26701b30-ec4b-4dec-80df-b483d7edb77d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 02:45:39.121392 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:39.121369 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-139.ec2.internal" event="NodeReady" Jul 20 02:45:39.121537 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:39.121498 2558 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 20 02:45:39.160817 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:39.160789 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rkzf8"] Jul 20 02:45:39.164172 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:39.164154 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-d5lxm"] Jul 20 02:45:39.164287 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:39.164269 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rkzf8" Jul 20 02:45:39.166943 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:39.166920 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 20 02:45:39.166943 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:39.166939 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 20 02:45:39.167312 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:39.167297 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nz45m\"" Jul 20 02:45:39.167940 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:39.167921 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-d5lxm" Jul 20 02:45:39.170494 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:39.170308 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-mhmn8\"" Jul 20 02:45:39.170494 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:39.170368 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 20 02:45:39.170494 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:39.170425 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 20 02:45:39.170494 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:39.170458 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 20 02:45:39.176596 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:39.176381 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rkzf8"] Jul 20 02:45:39.178900 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:39.177197 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-d5lxm"] Jul 20 02:45:39.321436 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:39.321397 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f32d6cda-7b64-4aa8-90ee-21c1e3043ed3-tmp-dir\") pod \"dns-default-rkzf8\" (UID: \"f32d6cda-7b64-4aa8-90ee-21c1e3043ed3\") " pod="openshift-dns/dns-default-rkzf8" Jul 20 02:45:39.321436 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:39.321440 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f534591e-00ab-4312-baa1-18298304041f-cert\") pod \"ingress-canary-d5lxm\" (UID: \"f534591e-00ab-4312-baa1-18298304041f\") " pod="openshift-ingress-canary/ingress-canary-d5lxm" Jul 20 02:45:39.321685 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:39.321486 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-664gc\" (UniqueName: \"kubernetes.io/projected/f534591e-00ab-4312-baa1-18298304041f-kube-api-access-664gc\") pod \"ingress-canary-d5lxm\" (UID: \"f534591e-00ab-4312-baa1-18298304041f\") " pod="openshift-ingress-canary/ingress-canary-d5lxm" Jul 20 02:45:39.321685 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:39.321515 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2vjj\" (UniqueName: \"kubernetes.io/projected/f32d6cda-7b64-4aa8-90ee-21c1e3043ed3-kube-api-access-c2vjj\") pod \"dns-default-rkzf8\" (UID: \"f32d6cda-7b64-4aa8-90ee-21c1e3043ed3\") " pod="openshift-dns/dns-default-rkzf8" Jul 20 02:45:39.321685 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:39.321557 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f32d6cda-7b64-4aa8-90ee-21c1e3043ed3-config-volume\") pod \"dns-default-rkzf8\" (UID: \"f32d6cda-7b64-4aa8-90ee-21c1e3043ed3\") " pod="openshift-dns/dns-default-rkzf8" Jul 20 02:45:39.321685 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:39.321621 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f32d6cda-7b64-4aa8-90ee-21c1e3043ed3-metrics-tls\") pod \"dns-default-rkzf8\" (UID: \"f32d6cda-7b64-4aa8-90ee-21c1e3043ed3\") " pod="openshift-dns/dns-default-rkzf8" Jul 20 02:45:39.422228 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:39.422194 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-664gc\" (UniqueName: \"kubernetes.io/projected/f534591e-00ab-4312-baa1-18298304041f-kube-api-access-664gc\") pod \"ingress-canary-d5lxm\" (UID: \"f534591e-00ab-4312-baa1-18298304041f\") " pod="openshift-ingress-canary/ingress-canary-d5lxm" Jul 20 02:45:39.422424 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:39.422240 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2vjj\" (UniqueName: \"kubernetes.io/projected/f32d6cda-7b64-4aa8-90ee-21c1e3043ed3-kube-api-access-c2vjj\") pod \"dns-default-rkzf8\" (UID: \"f32d6cda-7b64-4aa8-90ee-21c1e3043ed3\") " pod="openshift-dns/dns-default-rkzf8" Jul 20 02:45:39.422424 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:39.422295 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f32d6cda-7b64-4aa8-90ee-21c1e3043ed3-config-volume\") pod \"dns-default-rkzf8\" (UID: \"f32d6cda-7b64-4aa8-90ee-21c1e3043ed3\") " pod="openshift-dns/dns-default-rkzf8" Jul 20 02:45:39.422424 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:39.422346 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f32d6cda-7b64-4aa8-90ee-21c1e3043ed3-metrics-tls\") pod \"dns-default-rkzf8\" (UID: \"f32d6cda-7b64-4aa8-90ee-21c1e3043ed3\") " pod="openshift-dns/dns-default-rkzf8" Jul 20 02:45:39.422598 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:39.422442 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 02:45:39.422598 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:39.422497 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f32d6cda-7b64-4aa8-90ee-21c1e3043ed3-metrics-tls podName:f32d6cda-7b64-4aa8-90ee-21c1e3043ed3 nodeName:}" failed. No retries permitted until 2026-07-20 02:45:39.922479176 +0000 UTC m=+35.121728302 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f32d6cda-7b64-4aa8-90ee-21c1e3043ed3-metrics-tls") pod "dns-default-rkzf8" (UID: "f32d6cda-7b64-4aa8-90ee-21c1e3043ed3") : secret "dns-default-metrics-tls" not found Jul 20 02:45:39.422598 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:39.422514 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f32d6cda-7b64-4aa8-90ee-21c1e3043ed3-tmp-dir\") pod \"dns-default-rkzf8\" (UID: \"f32d6cda-7b64-4aa8-90ee-21c1e3043ed3\") " pod="openshift-dns/dns-default-rkzf8" Jul 20 02:45:39.422598 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:39.422537 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f534591e-00ab-4312-baa1-18298304041f-cert\") pod \"ingress-canary-d5lxm\" (UID: \"f534591e-00ab-4312-baa1-18298304041f\") " pod="openshift-ingress-canary/ingress-canary-d5lxm" Jul 20 02:45:39.422807 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:39.422787 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f32d6cda-7b64-4aa8-90ee-21c1e3043ed3-tmp-dir\") pod \"dns-default-rkzf8\" (UID: \"f32d6cda-7b64-4aa8-90ee-21c1e3043ed3\") " pod="openshift-dns/dns-default-rkzf8" Jul 20 02:45:39.422921 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:39.422897 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f32d6cda-7b64-4aa8-90ee-21c1e3043ed3-config-volume\") pod \"dns-default-rkzf8\" (UID: \"f32d6cda-7b64-4aa8-90ee-21c1e3043ed3\") " pod="openshift-dns/dns-default-rkzf8" Jul 20 02:45:39.427136 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:39.427114 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f534591e-00ab-4312-baa1-18298304041f-cert\") pod \"ingress-canary-d5lxm\" (UID: \"f534591e-00ab-4312-baa1-18298304041f\") " pod="openshift-ingress-canary/ingress-canary-d5lxm" Jul 20 02:45:39.431215 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:39.431188 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-664gc\" (UniqueName: \"kubernetes.io/projected/f534591e-00ab-4312-baa1-18298304041f-kube-api-access-664gc\") pod \"ingress-canary-d5lxm\" (UID: \"f534591e-00ab-4312-baa1-18298304041f\") " pod="openshift-ingress-canary/ingress-canary-d5lxm" Jul 20 02:45:39.431344 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:39.431195 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2vjj\" (UniqueName: \"kubernetes.io/projected/f32d6cda-7b64-4aa8-90ee-21c1e3043ed3-kube-api-access-c2vjj\") pod \"dns-default-rkzf8\" (UID: \"f32d6cda-7b64-4aa8-90ee-21c1e3043ed3\") " pod="openshift-dns/dns-default-rkzf8" Jul 20 02:45:39.487935 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:39.487899 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-d5lxm" Jul 20 02:45:39.638455 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:39.638420 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-d5lxm"] Jul 20 02:45:39.642727 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:45:39.642696 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf534591e_00ab_4312_baa1_18298304041f.slice/crio-7658d12f6e24b830b10acb8dcb230c51ce62743d773d4bc8536d090d4202cbea WatchSource:0}: Error finding container 7658d12f6e24b830b10acb8dcb230c51ce62743d773d4bc8536d090d4202cbea: Status 404 returned error can't find the container with id 7658d12f6e24b830b10acb8dcb230c51ce62743d773d4bc8536d090d4202cbea Jul 20 02:45:39.927213 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:39.927003 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f32d6cda-7b64-4aa8-90ee-21c1e3043ed3-metrics-tls\") pod \"dns-default-rkzf8\" (UID: \"f32d6cda-7b64-4aa8-90ee-21c1e3043ed3\") " pod="openshift-dns/dns-default-rkzf8" Jul 20 02:45:39.927354 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:39.927167 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 02:45:39.927354 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:39.927279 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f32d6cda-7b64-4aa8-90ee-21c1e3043ed3-metrics-tls podName:f32d6cda-7b64-4aa8-90ee-21c1e3043ed3 nodeName:}" failed. No retries permitted until 2026-07-20 02:45:40.927261515 +0000 UTC m=+36.126510643 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f32d6cda-7b64-4aa8-90ee-21c1e3043ed3-metrics-tls") pod "dns-default-rkzf8" (UID: "f32d6cda-7b64-4aa8-90ee-21c1e3043ed3") : secret "dns-default-metrics-tls" not found Jul 20 02:45:40.418775 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:40.418736 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t9spk" Jul 20 02:45:40.419654 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:40.418736 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cxtl" Jul 20 02:45:40.419654 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:40.418736 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9x7hv" Jul 20 02:45:40.421865 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:40.421842 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 20 02:45:40.423783 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:40.423195 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 20 02:45:40.423783 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:40.423233 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 20 02:45:40.423783 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:40.423240 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jphph\"" Jul 20 02:45:40.423783 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:40.423484 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 20 02:45:40.423783 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:40.423536 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8vd4r\"" Jul 20 02:45:40.648908 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:40.648869 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-d5lxm" event={"ID":"f534591e-00ab-4312-baa1-18298304041f","Type":"ContainerStarted","Data":"7658d12f6e24b830b10acb8dcb230c51ce62743d773d4bc8536d090d4202cbea"} Jul 20 02:45:40.935260 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:40.935226 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f32d6cda-7b64-4aa8-90ee-21c1e3043ed3-metrics-tls\") pod \"dns-default-rkzf8\" (UID: \"f32d6cda-7b64-4aa8-90ee-21c1e3043ed3\") " pod="openshift-dns/dns-default-rkzf8" Jul 20 02:45:40.935438 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:40.935356 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 02:45:40.935438 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:40.935430 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f32d6cda-7b64-4aa8-90ee-21c1e3043ed3-metrics-tls podName:f32d6cda-7b64-4aa8-90ee-21c1e3043ed3 nodeName:}" failed. No retries permitted until 2026-07-20 02:45:42.935409232 +0000 UTC m=+38.134658360 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f32d6cda-7b64-4aa8-90ee-21c1e3043ed3-metrics-tls") pod "dns-default-rkzf8" (UID: "f32d6cda-7b64-4aa8-90ee-21c1e3043ed3") : secret "dns-default-metrics-tls" not found Jul 20 02:45:42.951907 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:42.951869 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f32d6cda-7b64-4aa8-90ee-21c1e3043ed3-metrics-tls\") pod \"dns-default-rkzf8\" (UID: \"f32d6cda-7b64-4aa8-90ee-21c1e3043ed3\") " pod="openshift-dns/dns-default-rkzf8" Jul 20 02:45:42.952347 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:42.952035 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 02:45:42.952347 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:42.952124 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f32d6cda-7b64-4aa8-90ee-21c1e3043ed3-metrics-tls podName:f32d6cda-7b64-4aa8-90ee-21c1e3043ed3 nodeName:}" failed. No retries permitted until 2026-07-20 02:45:46.952086501 +0000 UTC m=+42.151335625 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f32d6cda-7b64-4aa8-90ee-21c1e3043ed3-metrics-tls") pod "dns-default-rkzf8" (UID: "f32d6cda-7b64-4aa8-90ee-21c1e3043ed3") : secret "dns-default-metrics-tls" not found Jul 20 02:45:43.657447 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:43.657412 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-d5lxm" event={"ID":"f534591e-00ab-4312-baa1-18298304041f","Type":"ContainerStarted","Data":"74fbbb34879376fc3bf053629415c33f3bed33b3f31e90eee2460d94b9216797"} Jul 20 02:45:43.660186 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:43.660160 2558 generic.go:356] "Generic (PLEG): container finished" podID="e8d00f9d-e63a-44a7-8a2c-268dc6787c75" containerID="403f4578923922eb86f907e1e7dac6492f00ff6701ab3f938af6620312f253c5" exitCode=0 Jul 20 02:45:43.660314 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:43.660196 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d5bdr" event={"ID":"e8d00f9d-e63a-44a7-8a2c-268dc6787c75","Type":"ContainerDied","Data":"403f4578923922eb86f907e1e7dac6492f00ff6701ab3f938af6620312f253c5"} Jul 20 02:45:43.672558 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:43.672521 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-d5lxm" podStartSLOduration=1.170270575 podStartE2EDuration="4.672509276s" totalImagesPullingTime="3.502238701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:45:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:45:43.672044951 +0000 UTC m=+38.871294099" watchObservedRunningTime="2026-07-20 02:45:43.672509276 +0000 UTC m=+38.871758422" Jul 20 02:45:44.665808 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:44.665777 2558 generic.go:356] "Generic (PLEG): container finished" podID="e8d00f9d-e63a-44a7-8a2c-268dc6787c75" containerID="d3c0a318d4a954ee51b39c5862b0e1440fc1fff573621da4e7559723af880430" exitCode=0 Jul 20 02:45:44.666241 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:44.665854 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d5bdr" event={"ID":"e8d00f9d-e63a-44a7-8a2c-268dc6787c75","Type":"ContainerDied","Data":"d3c0a318d4a954ee51b39c5862b0e1440fc1fff573621da4e7559723af880430"} Jul 20 02:45:45.671620 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:45.671438 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d5bdr" event={"ID":"e8d00f9d-e63a-44a7-8a2c-268dc6787c75","Type":"ContainerStarted","Data":"21595e90dd4309d8b86fa516c970f70ad57cd5f3210930d84ed23c09320f0a2b"} Jul 20 02:45:45.693554 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:45.693505 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-d5bdr" podStartSLOduration=9.83168958 podStartE2EDuration="40.693491305s" totalImagesPullingTime="30.861801725s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 02:45:05 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:45:45.692261737 +0000 UTC m=+40.891510881" watchObservedRunningTime="2026-07-20 02:45:45.693491305 +0000 UTC m=+40.892740451" Jul 20 02:45:46.980511 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:46.980466 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f32d6cda-7b64-4aa8-90ee-21c1e3043ed3-metrics-tls\") pod \"dns-default-rkzf8\" (UID: \"f32d6cda-7b64-4aa8-90ee-21c1e3043ed3\") " pod="openshift-dns/dns-default-rkzf8" Jul 20 02:45:46.980892 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:46.980582 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 02:45:46.980892 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:46.980636 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f32d6cda-7b64-4aa8-90ee-21c1e3043ed3-metrics-tls podName:f32d6cda-7b64-4aa8-90ee-21c1e3043ed3 nodeName:}" failed. No retries permitted until 2026-07-20 02:45:54.980622857 +0000 UTC m=+50.179871982 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f32d6cda-7b64-4aa8-90ee-21c1e3043ed3-metrics-tls") pod "dns-default-rkzf8" (UID: "f32d6cda-7b64-4aa8-90ee-21c1e3043ed3") : secret "dns-default-metrics-tls" not found Jul 20 02:45:55.033714 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:45:55.033672 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f32d6cda-7b64-4aa8-90ee-21c1e3043ed3-metrics-tls\") pod \"dns-default-rkzf8\" (UID: \"f32d6cda-7b64-4aa8-90ee-21c1e3043ed3\") " pod="openshift-dns/dns-default-rkzf8" Jul 20 02:45:55.034191 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:55.033808 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 02:45:55.034191 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:45:55.033871 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f32d6cda-7b64-4aa8-90ee-21c1e3043ed3-metrics-tls podName:f32d6cda-7b64-4aa8-90ee-21c1e3043ed3 nodeName:}" failed. No retries permitted until 2026-07-20 02:46:11.033854442 +0000 UTC m=+66.233103567 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f32d6cda-7b64-4aa8-90ee-21c1e3043ed3-metrics-tls") pod "dns-default-rkzf8" (UID: "f32d6cda-7b64-4aa8-90ee-21c1e3043ed3") : secret "dns-default-metrics-tls" not found Jul 20 02:46:06.645886 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:06.645856 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n4b7k" Jul 20 02:46:11.043148 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:11.043084 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f32d6cda-7b64-4aa8-90ee-21c1e3043ed3-metrics-tls\") pod \"dns-default-rkzf8\" (UID: \"f32d6cda-7b64-4aa8-90ee-21c1e3043ed3\") " pod="openshift-dns/dns-default-rkzf8" Jul 20 02:46:11.043148 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:11.043152 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6c7d134c-4e38-481c-8a81-e03434badf7d-original-pull-secret\") pod \"global-pull-secret-syncer-t9spk\" (UID: \"6c7d134c-4e38-481c-8a81-e03434badf7d\") " pod="kube-system/global-pull-secret-syncer-t9spk" Jul 20 02:46:11.043550 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:11.043181 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d70f5103-7dc9-4279-9781-38e395da435b-metrics-certs\") pod \"network-metrics-daemon-9x7hv\" (UID: \"d70f5103-7dc9-4279-9781-38e395da435b\") " pod="openshift-multus/network-metrics-daemon-9x7hv" Jul 20 02:46:11.043550 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:46:11.043252 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 02:46:11.043550 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:46:11.043330 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f32d6cda-7b64-4aa8-90ee-21c1e3043ed3-metrics-tls podName:f32d6cda-7b64-4aa8-90ee-21c1e3043ed3 nodeName:}" failed. No retries permitted until 2026-07-20 02:46:43.043313505 +0000 UTC m=+98.242562630 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f32d6cda-7b64-4aa8-90ee-21c1e3043ed3-metrics-tls") pod "dns-default-rkzf8" (UID: "f32d6cda-7b64-4aa8-90ee-21c1e3043ed3") : secret "dns-default-metrics-tls" not found Jul 20 02:46:11.046176 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:11.046159 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 20 02:46:11.046256 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:11.046167 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 20 02:46:11.054171 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:46:11.054153 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 20 02:46:11.054243 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:46:11.054227 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d70f5103-7dc9-4279-9781-38e395da435b-metrics-certs podName:d70f5103-7dc9-4279-9781-38e395da435b nodeName:}" failed. No retries permitted until 2026-07-20 02:47:15.054207275 +0000 UTC m=+130.253456400 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d70f5103-7dc9-4279-9781-38e395da435b-metrics-certs") pod "network-metrics-daemon-9x7hv" (UID: "d70f5103-7dc9-4279-9781-38e395da435b") : secret "metrics-daemon-secret" not found Jul 20 02:46:11.058057 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:11.058021 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6c7d134c-4e38-481c-8a81-e03434badf7d-original-pull-secret\") pod \"global-pull-secret-syncer-t9spk\" (UID: \"6c7d134c-4e38-481c-8a81-e03434badf7d\") " pod="kube-system/global-pull-secret-syncer-t9spk" Jul 20 02:46:11.144489 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:11.144455 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rsdj\" (UniqueName: \"kubernetes.io/projected/26701b30-ec4b-4dec-80df-b483d7edb77d-kube-api-access-4rsdj\") pod \"network-check-target-5cxtl\" (UID: \"26701b30-ec4b-4dec-80df-b483d7edb77d\") " pod="openshift-network-diagnostics/network-check-target-5cxtl" Jul 20 02:46:11.147129 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:11.147089 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 20 02:46:11.157430 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:11.157410 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 20 02:46:11.167679 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:11.167652 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rsdj\" (UniqueName: \"kubernetes.io/projected/26701b30-ec4b-4dec-80df-b483d7edb77d-kube-api-access-4rsdj\") pod \"network-check-target-5cxtl\" (UID: \"26701b30-ec4b-4dec-80df-b483d7edb77d\") " pod="openshift-network-diagnostics/network-check-target-5cxtl" Jul 20 02:46:11.332257 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:11.332163 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t9spk" Jul 20 02:46:11.347959 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:11.347936 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jphph\"" Jul 20 02:46:11.356154 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:11.356129 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5cxtl" Jul 20 02:46:11.467700 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:11.467660 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-t9spk"] Jul 20 02:46:11.483485 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:11.483458 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5cxtl"] Jul 20 02:46:11.486086 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:46:11.486061 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26701b30_ec4b_4dec_80df_b483d7edb77d.slice/crio-1e61b97cd55f3e7e26b93819839d40842a9c104b404b4e31fb1687c721262ee8 WatchSource:0}: Error finding container 1e61b97cd55f3e7e26b93819839d40842a9c104b404b4e31fb1687c721262ee8: Status 404 returned error can't find the container with id 1e61b97cd55f3e7e26b93819839d40842a9c104b404b4e31fb1687c721262ee8 Jul 20 02:46:11.722091 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:11.722058 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6484fddbb8-bw7l9"] Jul 20 02:46:11.754120 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:11.754067 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6484fddbb8-bw7l9"] Jul 20 02:46:11.754286 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:11.754253 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5cxtl" event={"ID":"26701b30-ec4b-4dec-80df-b483d7edb77d","Type":"ContainerStarted","Data":"1e61b97cd55f3e7e26b93819839d40842a9c104b404b4e31fb1687c721262ee8"} Jul 20 02:46:11.754342 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:11.754295 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-t9spk" event={"ID":"6c7d134c-4e38-481c-8a81-e03434badf7d","Type":"ContainerStarted","Data":"eb8a5ec7febf859516532701a7ad4aeb18c55447c3f27035d8c468404c1a0bf1"} Jul 20 02:46:11.754381 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:11.754352 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6484fddbb8-bw7l9" Jul 20 02:46:11.757007 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:11.756977 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-ppzkm\"" Jul 20 02:46:11.757153 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:11.757020 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 20 02:46:11.757153 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:11.757024 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 20 02:46:11.758266 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:11.758245 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 20 02:46:11.758380 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:11.758251 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 20 02:46:11.761388 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:11.761370 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-f5b4c9dd9-wx8vr"] Jul 20 02:46:11.775928 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:11.775907 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-f5b4c9dd9-wx8vr"] Jul 20 02:46:11.776009 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:11.775941 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f5b4c9dd9-wx8vr" Jul 20 02:46:11.778852 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:11.778829 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 20 02:46:11.849068 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:11.849036 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb110dff-4ae0-43b2-8b14-2b558b3320dd-tmp\") pod \"klusterlet-addon-workmgr-f5b4c9dd9-wx8vr\" (UID: \"bb110dff-4ae0-43b2-8b14-2b558b3320dd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f5b4c9dd9-wx8vr" Jul 20 02:46:11.849068 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:11.849067 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/bb110dff-4ae0-43b2-8b14-2b558b3320dd-klusterlet-config\") pod \"klusterlet-addon-workmgr-f5b4c9dd9-wx8vr\" (UID: \"bb110dff-4ae0-43b2-8b14-2b558b3320dd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f5b4c9dd9-wx8vr" Jul 20 02:46:11.849283 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:11.849109 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ef3fb814-aa53-4829-ad88-dd070dabcb80-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6484fddbb8-bw7l9\" (UID: \"ef3fb814-aa53-4829-ad88-dd070dabcb80\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6484fddbb8-bw7l9" Jul 20 02:46:11.849283 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:11.849152 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd7jx\" (UniqueName: \"kubernetes.io/projected/ef3fb814-aa53-4829-ad88-dd070dabcb80-kube-api-access-nd7jx\") pod \"managed-serviceaccount-addon-agent-6484fddbb8-bw7l9\" (UID: \"ef3fb814-aa53-4829-ad88-dd070dabcb80\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6484fddbb8-bw7l9" Jul 20 02:46:11.849283 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:11.849214 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh7ww\" (UniqueName: \"kubernetes.io/projected/bb110dff-4ae0-43b2-8b14-2b558b3320dd-kube-api-access-kh7ww\") pod \"klusterlet-addon-workmgr-f5b4c9dd9-wx8vr\" (UID: \"bb110dff-4ae0-43b2-8b14-2b558b3320dd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f5b4c9dd9-wx8vr" Jul 20 02:46:11.949477 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:11.949445 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh7ww\" (UniqueName: \"kubernetes.io/projected/bb110dff-4ae0-43b2-8b14-2b558b3320dd-kube-api-access-kh7ww\") pod \"klusterlet-addon-workmgr-f5b4c9dd9-wx8vr\" (UID: \"bb110dff-4ae0-43b2-8b14-2b558b3320dd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f5b4c9dd9-wx8vr" Jul 20 02:46:11.949640 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:11.949512 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb110dff-4ae0-43b2-8b14-2b558b3320dd-tmp\") pod \"klusterlet-addon-workmgr-f5b4c9dd9-wx8vr\" (UID: \"bb110dff-4ae0-43b2-8b14-2b558b3320dd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f5b4c9dd9-wx8vr" Jul 20 02:46:11.949640 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:11.949537 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/bb110dff-4ae0-43b2-8b14-2b558b3320dd-klusterlet-config\") pod \"klusterlet-addon-workmgr-f5b4c9dd9-wx8vr\" (UID: \"bb110dff-4ae0-43b2-8b14-2b558b3320dd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f5b4c9dd9-wx8vr" Jul 20 02:46:11.949640 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:11.949555 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ef3fb814-aa53-4829-ad88-dd070dabcb80-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6484fddbb8-bw7l9\" (UID: \"ef3fb814-aa53-4829-ad88-dd070dabcb80\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6484fddbb8-bw7l9" Jul 20 02:46:11.949640 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:11.949572 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd7jx\" (UniqueName: \"kubernetes.io/projected/ef3fb814-aa53-4829-ad88-dd070dabcb80-kube-api-access-nd7jx\") pod \"managed-serviceaccount-addon-agent-6484fddbb8-bw7l9\" (UID: \"ef3fb814-aa53-4829-ad88-dd070dabcb80\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6484fddbb8-bw7l9" Jul 20 02:46:11.949996 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:11.949969 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb110dff-4ae0-43b2-8b14-2b558b3320dd-tmp\") pod \"klusterlet-addon-workmgr-f5b4c9dd9-wx8vr\" (UID: \"bb110dff-4ae0-43b2-8b14-2b558b3320dd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f5b4c9dd9-wx8vr" Jul 20 02:46:11.952357 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:11.952332 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/bb110dff-4ae0-43b2-8b14-2b558b3320dd-klusterlet-config\") pod \"klusterlet-addon-workmgr-f5b4c9dd9-wx8vr\" (UID: \"bb110dff-4ae0-43b2-8b14-2b558b3320dd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f5b4c9dd9-wx8vr" Jul 20 02:46:11.952455 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:11.952337 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ef3fb814-aa53-4829-ad88-dd070dabcb80-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6484fddbb8-bw7l9\" (UID: \"ef3fb814-aa53-4829-ad88-dd070dabcb80\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6484fddbb8-bw7l9" Jul 20 02:46:11.957084 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:11.957056 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh7ww\" (UniqueName: \"kubernetes.io/projected/bb110dff-4ae0-43b2-8b14-2b558b3320dd-kube-api-access-kh7ww\") pod \"klusterlet-addon-workmgr-f5b4c9dd9-wx8vr\" (UID: \"bb110dff-4ae0-43b2-8b14-2b558b3320dd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f5b4c9dd9-wx8vr" Jul 20 02:46:11.957209 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:11.957171 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd7jx\" (UniqueName: \"kubernetes.io/projected/ef3fb814-aa53-4829-ad88-dd070dabcb80-kube-api-access-nd7jx\") pod \"managed-serviceaccount-addon-agent-6484fddbb8-bw7l9\" (UID: \"ef3fb814-aa53-4829-ad88-dd070dabcb80\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6484fddbb8-bw7l9" Jul 20 02:46:12.072375 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:12.072286 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6484fddbb8-bw7l9" Jul 20 02:46:12.105976 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:12.105946 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f5b4c9dd9-wx8vr" Jul 20 02:46:12.242056 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:12.241999 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6484fddbb8-bw7l9"] Jul 20 02:46:12.248596 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:46:12.248562 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef3fb814_aa53_4829_ad88_dd070dabcb80.slice/crio-516356c098b7004ecf2ec453f9edc81f41c770784fbdf2b7ee248fd2bd0630a7 WatchSource:0}: Error finding container 516356c098b7004ecf2ec453f9edc81f41c770784fbdf2b7ee248fd2bd0630a7: Status 404 returned error can't find the container with id 516356c098b7004ecf2ec453f9edc81f41c770784fbdf2b7ee248fd2bd0630a7 Jul 20 02:46:12.269086 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:12.268951 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-f5b4c9dd9-wx8vr"] Jul 20 02:46:12.272412 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:46:12.272377 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb110dff_4ae0_43b2_8b14_2b558b3320dd.slice/crio-1fae3f3141c9503cc8ed68857bad95f5efbb3eefac27dd263543227a1bea90a5 WatchSource:0}: Error finding container 1fae3f3141c9503cc8ed68857bad95f5efbb3eefac27dd263543227a1bea90a5: Status 404 returned error can't find the container with id 1fae3f3141c9503cc8ed68857bad95f5efbb3eefac27dd263543227a1bea90a5 Jul 20 02:46:12.736805 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:12.736741 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6484fddbb8-bw7l9" event={"ID":"ef3fb814-aa53-4829-ad88-dd070dabcb80","Type":"ContainerStarted","Data":"516356c098b7004ecf2ec453f9edc81f41c770784fbdf2b7ee248fd2bd0630a7"} Jul 20 02:46:12.737989 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:12.737958 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f5b4c9dd9-wx8vr" event={"ID":"bb110dff-4ae0-43b2-8b14-2b558b3320dd","Type":"ContainerStarted","Data":"1fae3f3141c9503cc8ed68857bad95f5efbb3eefac27dd263543227a1bea90a5"} Jul 20 02:46:19.758016 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:19.757976 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-t9spk" event={"ID":"6c7d134c-4e38-481c-8a81-e03434badf7d","Type":"ContainerStarted","Data":"70ae0bec1d50879234f3d71a811fd25574d0f4af0300f0070ea74a7b09c474b1"} Jul 20 02:46:19.759455 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:19.759426 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5cxtl" event={"ID":"26701b30-ec4b-4dec-80df-b483d7edb77d","Type":"ContainerStarted","Data":"3b628215a8fb61937a2608d376d35367080d201ab8eee77956eb581ee0146c0b"} Jul 20 02:46:19.759596 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:19.759497 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5cxtl" Jul 20 02:46:19.760845 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:19.760825 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6484fddbb8-bw7l9" event={"ID":"ef3fb814-aa53-4829-ad88-dd070dabcb80","Type":"ContainerStarted","Data":"79f5e5674e21f21f191044451f9fa4668fe52a1508cc187e846989337d5a18ff"} Jul 20 02:46:19.762086 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:19.762062 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f5b4c9dd9-wx8vr" event={"ID":"bb110dff-4ae0-43b2-8b14-2b558b3320dd","Type":"ContainerStarted","Data":"df29666c19c580c97c1cb5cd579aa0720a092c52239e190837c37713b8d3df65"} Jul 20 02:46:19.762268 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:19.762257 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f5b4c9dd9-wx8vr" Jul 20 02:46:19.763773 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:19.763755 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f5b4c9dd9-wx8vr" Jul 20 02:46:19.772835 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:19.772799 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-t9spk" podStartSLOduration=67.368804791 podStartE2EDuration="1m14.772786036s" totalImagesPullingTime="7.403981245s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 02:45:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:46:19.772327875 +0000 UTC m=+74.971577021" watchObservedRunningTime="2026-07-20 02:46:19.772786036 +0000 UTC m=+74.972035177" Jul 20 02:46:19.785830 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:19.785794 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6484fddbb8-bw7l9" podStartSLOduration=2.160978287 podStartE2EDuration="8.785781422s" totalImagesPullingTime="6.624803135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:46:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:46:19.785298943 +0000 UTC m=+74.984548089" watchObservedRunningTime="2026-07-20 02:46:19.785781422 +0000 UTC m=+74.985030566" Jul 20 02:46:19.798835 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:19.798795 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5cxtl" podStartSLOduration=67.410169428 podStartE2EDuration="1m14.798784744s" totalImagesPullingTime="7.388615316s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:45:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:46:19.797506415 +0000 UTC m=+74.996755573" watchObservedRunningTime="2026-07-20 02:46:19.798784744 +0000 UTC m=+74.998033869" Jul 20 02:46:19.815137 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:19.815088 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f5b4c9dd9-wx8vr" podStartSLOduration=2.135320377 podStartE2EDuration="8.815076317s" totalImagesPullingTime="6.67975594s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:46:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:46:19.814498709 +0000 UTC m=+75.013747857" watchObservedRunningTime="2026-07-20 02:46:19.815076317 +0000 UTC m=+75.014325464" Jul 20 02:46:43.066248 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:43.066110 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f32d6cda-7b64-4aa8-90ee-21c1e3043ed3-metrics-tls\") pod \"dns-default-rkzf8\" (UID: \"f32d6cda-7b64-4aa8-90ee-21c1e3043ed3\") " pod="openshift-dns/dns-default-rkzf8" Jul 20 02:46:43.066711 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:46:43.066275 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 02:46:43.066711 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:46:43.066344 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f32d6cda-7b64-4aa8-90ee-21c1e3043ed3-metrics-tls podName:f32d6cda-7b64-4aa8-90ee-21c1e3043ed3 nodeName:}" failed. No retries permitted until 2026-07-20 02:47:47.066328137 +0000 UTC m=+162.265577261 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f32d6cda-7b64-4aa8-90ee-21c1e3043ed3-metrics-tls") pod "dns-default-rkzf8" (UID: "f32d6cda-7b64-4aa8-90ee-21c1e3043ed3") : secret "dns-default-metrics-tls" not found Jul 20 02:46:50.766650 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:46:50.766614 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5cxtl" Jul 20 02:47:15.089500 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:47:15.089457 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d70f5103-7dc9-4279-9781-38e395da435b-metrics-certs\") pod \"network-metrics-daemon-9x7hv\" (UID: \"d70f5103-7dc9-4279-9781-38e395da435b\") " pod="openshift-multus/network-metrics-daemon-9x7hv" Jul 20 02:47:15.089972 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:47:15.089579 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 20 02:47:15.089972 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:47:15.089633 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d70f5103-7dc9-4279-9781-38e395da435b-metrics-certs podName:d70f5103-7dc9-4279-9781-38e395da435b nodeName:}" failed. No retries permitted until 2026-07-20 02:49:17.089618329 +0000 UTC m=+252.288867454 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d70f5103-7dc9-4279-9781-38e395da435b-metrics-certs") pod "network-metrics-daemon-9x7hv" (UID: "d70f5103-7dc9-4279-9781-38e395da435b") : secret "metrics-daemon-secret" not found Jul 20 02:47:42.182672 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:47:42.182618 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-rkzf8" podUID="f32d6cda-7b64-4aa8-90ee-21c1e3043ed3" Jul 20 02:47:42.963500 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:47:42.963466 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rkzf8" Jul 20 02:47:43.440283 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:47:43.440238 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-9x7hv" podUID="d70f5103-7dc9-4279-9781-38e395da435b" Jul 20 02:47:47.102879 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:47:47.102819 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f32d6cda-7b64-4aa8-90ee-21c1e3043ed3-metrics-tls\") pod \"dns-default-rkzf8\" (UID: \"f32d6cda-7b64-4aa8-90ee-21c1e3043ed3\") " pod="openshift-dns/dns-default-rkzf8" Jul 20 02:47:47.103327 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:47:47.102981 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 02:47:47.103327 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:47:47.103047 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f32d6cda-7b64-4aa8-90ee-21c1e3043ed3-metrics-tls podName:f32d6cda-7b64-4aa8-90ee-21c1e3043ed3 nodeName:}" failed. No retries permitted until 2026-07-20 02:49:49.103031298 +0000 UTC m=+284.302280423 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f32d6cda-7b64-4aa8-90ee-21c1e3043ed3-metrics-tls") pod "dns-default-rkzf8" (UID: "f32d6cda-7b64-4aa8-90ee-21c1e3043ed3") : secret "dns-default-metrics-tls" not found Jul 20 02:47:57.418372 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:47:57.418281 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9x7hv" Jul 20 02:48:19.053565 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:48:19.053533 2558 generic.go:356] "Generic (PLEG): container finished" podID="ef3fb814-aa53-4829-ad88-dd070dabcb80" containerID="79f5e5674e21f21f191044451f9fa4668fe52a1508cc187e846989337d5a18ff" exitCode=255 Jul 20 02:48:19.054021 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:48:19.053575 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6484fddbb8-bw7l9" event={"ID":"ef3fb814-aa53-4829-ad88-dd070dabcb80","Type":"ContainerDied","Data":"79f5e5674e21f21f191044451f9fa4668fe52a1508cc187e846989337d5a18ff"} Jul 20 02:48:19.054021 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:48:19.053928 2558 scope.go:117] "RemoveContainer" containerID="79f5e5674e21f21f191044451f9fa4668fe52a1508cc187e846989337d5a18ff" Jul 20 02:48:19.762814 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:48:19.762752 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f5b4c9dd9-wx8vr" podUID="bb110dff-4ae0-43b2-8b14-2b558b3320dd" containerName="acm-agent" probeResult="failure" output="Get \"http://10.133.0.9:8000/readyz\": dial tcp 10.133.0.9:8000: connect: connection refused" Jul 20 02:48:20.058986 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:48:20.058886 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6484fddbb8-bw7l9" event={"ID":"ef3fb814-aa53-4829-ad88-dd070dabcb80","Type":"ContainerStarted","Data":"fcf852783f200b12457c63a0ea1e6a00bd140660c86fd5e3329119d02fd26b6f"} Jul 20 02:48:20.060242 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:48:20.060220 2558 generic.go:356] "Generic (PLEG): container finished" podID="bb110dff-4ae0-43b2-8b14-2b558b3320dd" containerID="df29666c19c580c97c1cb5cd579aa0720a092c52239e190837c37713b8d3df65" exitCode=1 Jul 20 02:48:20.060336 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:48:20.060267 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f5b4c9dd9-wx8vr" event={"ID":"bb110dff-4ae0-43b2-8b14-2b558b3320dd","Type":"ContainerDied","Data":"df29666c19c580c97c1cb5cd579aa0720a092c52239e190837c37713b8d3df65"} Jul 20 02:48:20.060517 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:48:20.060504 2558 scope.go:117] "RemoveContainer" containerID="df29666c19c580c97c1cb5cd579aa0720a092c52239e190837c37713b8d3df65" Jul 20 02:48:21.065446 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:48:21.065409 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f5b4c9dd9-wx8vr" event={"ID":"bb110dff-4ae0-43b2-8b14-2b558b3320dd","Type":"ContainerStarted","Data":"59747bd5487c6a67a4731f21954792ae242f913fbcd6c4f6fc32f4efd3442f6a"} Jul 20 02:48:21.065893 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:48:21.065773 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f5b4c9dd9-wx8vr" Jul 20 02:48:21.066349 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:48:21.066334 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f5b4c9dd9-wx8vr" Jul 20 02:49:10.927747 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:10.927708 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-8vxh2"] Jul 20 02:49:10.931653 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:10.931485 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-8vxh2" Jul 20 02:49:10.934227 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:10.934204 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-g2skg\"" Jul 20 02:49:10.944787 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:10.944763 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-8vxh2"] Jul 20 02:49:11.019006 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:11.018976 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-rx856"] Jul 20 02:49:11.021707 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:11.021691 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-rx856" Jul 20 02:49:11.024371 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:11.024352 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 20 02:49:11.024497 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:11.024413 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 20 02:49:11.024497 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:11.024459 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 20 02:49:11.024612 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:11.024497 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-scjvq\"" Jul 20 02:49:11.024701 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:11.024683 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 20 02:49:11.028802 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:11.028786 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-84f4d8756b-ltpj5"] Jul 20 02:49:11.031566 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:11.031551 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84f4d8756b-ltpj5" Jul 20 02:49:11.035349 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:11.035332 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 20 02:49:11.035441 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:11.035334 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dwwjm\"" Jul 20 02:49:11.035510 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:11.035496 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 20 02:49:11.035895 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:11.035863 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-rx856"] Jul 20 02:49:11.036858 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:11.036837 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 20 02:49:11.041305 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:11.041288 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 20 02:49:11.047964 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:11.047945 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84f4d8756b-ltpj5"] Jul 20 02:49:11.062391 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:11.062364 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4thkj\" (UniqueName: \"kubernetes.io/projected/0dd69072-56fa-4f7c-bed7-ac8e3d654203-kube-api-access-4thkj\") pod \"network-check-source-7bc5b6bf85-8vxh2\" (UID: \"0dd69072-56fa-4f7c-bed7-ac8e3d654203\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-8vxh2" Jul 20 02:49:11.163609 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:11.163575 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/463f86e5-8599-4846-875b-82cdad3c331e-config\") pod \"service-ca-operator-5bcbc94c78-rx856\" (UID: \"463f86e5-8599-4846-875b-82cdad3c331e\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-rx856" Jul 20 02:49:11.163609 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:11.163608 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9zvk\" (UniqueName: \"kubernetes.io/projected/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-kube-api-access-d9zvk\") pod \"image-registry-84f4d8756b-ltpj5\" (UID: \"537fa96e-a0c2-48ff-877b-5aca6c1e40f7\") " pod="openshift-image-registry/image-registry-84f4d8756b-ltpj5" Jul 20 02:49:11.163798 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:11.163631 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4thkj\" (UniqueName: \"kubernetes.io/projected/0dd69072-56fa-4f7c-bed7-ac8e3d654203-kube-api-access-4thkj\") pod \"network-check-source-7bc5b6bf85-8vxh2\" (UID: \"0dd69072-56fa-4f7c-bed7-ac8e3d654203\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-8vxh2" Jul 20 02:49:11.163798 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:11.163649 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-installation-pull-secrets\") pod \"image-registry-84f4d8756b-ltpj5\" (UID: \"537fa96e-a0c2-48ff-877b-5aca6c1e40f7\") " pod="openshift-image-registry/image-registry-84f4d8756b-ltpj5" Jul 20 02:49:11.163798 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:11.163692 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-bound-sa-token\") pod \"image-registry-84f4d8756b-ltpj5\" (UID: \"537fa96e-a0c2-48ff-877b-5aca6c1e40f7\") " pod="openshift-image-registry/image-registry-84f4d8756b-ltpj5" Jul 20 02:49:11.163798 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:11.163720 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/463f86e5-8599-4846-875b-82cdad3c331e-serving-cert\") pod \"service-ca-operator-5bcbc94c78-rx856\" (UID: \"463f86e5-8599-4846-875b-82cdad3c331e\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-rx856" Jul 20 02:49:11.163798 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:11.163743 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-registry-certificates\") pod \"image-registry-84f4d8756b-ltpj5\" (UID: \"537fa96e-a0c2-48ff-877b-5aca6c1e40f7\") " pod="openshift-image-registry/image-registry-84f4d8756b-ltpj5" Jul 20 02:49:11.163798 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:11.163794 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-image-registry-private-configuration\") pod \"image-registry-84f4d8756b-ltpj5\" (UID: \"537fa96e-a0c2-48ff-877b-5aca6c1e40f7\") " pod="openshift-image-registry/image-registry-84f4d8756b-ltpj5" Jul 20 02:49:11.163971 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:11.163819 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-ca-trust-extracted\") pod \"image-registry-84f4d8756b-ltpj5\" (UID: \"537fa96e-a0c2-48ff-877b-5aca6c1e40f7\") " pod="openshift-image-registry/image-registry-84f4d8756b-ltpj5" Jul 20 02:49:11.163971 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:11.163844 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt7np\" (UniqueName: \"kubernetes.io/projected/463f86e5-8599-4846-875b-82cdad3c331e-kube-api-access-qt7np\") pod \"service-ca-operator-5bcbc94c78-rx856\" (UID: \"463f86e5-8599-4846-875b-82cdad3c331e\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-rx856" Jul 20 02:49:11.163971 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:11.163876 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-trusted-ca\") pod \"image-registry-84f4d8756b-ltpj5\" (UID: \"537fa96e-a0c2-48ff-877b-5aca6c1e40f7\") " pod="openshift-image-registry/image-registry-84f4d8756b-ltpj5" Jul 20 02:49:11.163971 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:11.163923 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-registry-tls\") pod \"image-registry-84f4d8756b-ltpj5\" (UID: \"537fa96e-a0c2-48ff-877b-5aca6c1e40f7\") " pod="openshift-image-registry/image-registry-84f4d8756b-ltpj5" Jul 20 02:49:11.171492 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:11.171470 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4thkj\" (UniqueName: \"kubernetes.io/projected/0dd69072-56fa-4f7c-bed7-ac8e3d654203-kube-api-access-4thkj\") pod \"network-check-source-7bc5b6bf85-8vxh2\" (UID: \"0dd69072-56fa-4f7c-bed7-ac8e3d654203\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-8vxh2" Jul 20 02:49:11.240362 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:11.240288 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-8vxh2" Jul 20 02:49:11.264243 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:11.264214 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-image-registry-private-configuration\") pod \"image-registry-84f4d8756b-ltpj5\" (UID: \"537fa96e-a0c2-48ff-877b-5aca6c1e40f7\") " pod="openshift-image-registry/image-registry-84f4d8756b-ltpj5" Jul 20 02:49:11.264386 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:11.264249 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-ca-trust-extracted\") pod \"image-registry-84f4d8756b-ltpj5\" (UID: \"537fa96e-a0c2-48ff-877b-5aca6c1e40f7\") " pod="openshift-image-registry/image-registry-84f4d8756b-ltpj5" Jul 20 02:49:11.264386 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:11.264276 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt7np\" (UniqueName: \"kubernetes.io/projected/463f86e5-8599-4846-875b-82cdad3c331e-kube-api-access-qt7np\") pod \"service-ca-operator-5bcbc94c78-rx856\" (UID: \"463f86e5-8599-4846-875b-82cdad3c331e\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-rx856" Jul 20 02:49:11.264386 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:11.264301 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-trusted-ca\") pod \"image-registry-84f4d8756b-ltpj5\" (UID: \"537fa96e-a0c2-48ff-877b-5aca6c1e40f7\") " pod="openshift-image-registry/image-registry-84f4d8756b-ltpj5" Jul 20 02:49:11.264386 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:11.264352 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-registry-tls\") pod \"image-registry-84f4d8756b-ltpj5\" (UID: \"537fa96e-a0c2-48ff-877b-5aca6c1e40f7\") " pod="openshift-image-registry/image-registry-84f4d8756b-ltpj5" Jul 20 02:49:11.264598 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:11.264377 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/463f86e5-8599-4846-875b-82cdad3c331e-config\") pod \"service-ca-operator-5bcbc94c78-rx856\" (UID: \"463f86e5-8599-4846-875b-82cdad3c331e\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-rx856" Jul 20 02:49:11.264598 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:11.264410 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9zvk\" (UniqueName: \"kubernetes.io/projected/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-kube-api-access-d9zvk\") pod \"image-registry-84f4d8756b-ltpj5\" (UID: \"537fa96e-a0c2-48ff-877b-5aca6c1e40f7\") " pod="openshift-image-registry/image-registry-84f4d8756b-ltpj5" Jul 20 02:49:11.264598 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:11.264441 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-installation-pull-secrets\") pod \"image-registry-84f4d8756b-ltpj5\" (UID: \"537fa96e-a0c2-48ff-877b-5aca6c1e40f7\") " pod="openshift-image-registry/image-registry-84f4d8756b-ltpj5" Jul 20 02:49:11.264598 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:49:11.264465 2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 20 02:49:11.264598 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:11.264479 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-bound-sa-token\") pod \"image-registry-84f4d8756b-ltpj5\" (UID: \"537fa96e-a0c2-48ff-877b-5aca6c1e40f7\") " pod="openshift-image-registry/image-registry-84f4d8756b-ltpj5" Jul 20 02:49:11.264598 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:49:11.264487 2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84f4d8756b-ltpj5: secret "image-registry-tls" not found Jul 20 02:49:11.264598 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:11.264514 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/463f86e5-8599-4846-875b-82cdad3c331e-serving-cert\") pod \"service-ca-operator-5bcbc94c78-rx856\" (UID: \"463f86e5-8599-4846-875b-82cdad3c331e\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-rx856" Jul 20 02:49:11.264598 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:11.264537 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-registry-certificates\") pod \"image-registry-84f4d8756b-ltpj5\" (UID: \"537fa96e-a0c2-48ff-877b-5aca6c1e40f7\") " pod="openshift-image-registry/image-registry-84f4d8756b-ltpj5" Jul 20 02:49:11.264598 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:49:11.264589 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-registry-tls podName:537fa96e-a0c2-48ff-877b-5aca6c1e40f7 nodeName:}" failed. No retries permitted until 2026-07-20 02:49:11.76455119 +0000 UTC m=+246.963800336 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-registry-tls") pod "image-registry-84f4d8756b-ltpj5" (UID: "537fa96e-a0c2-48ff-877b-5aca6c1e40f7") : secret "image-registry-tls" not found Jul 20 02:49:11.265021 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:11.264673 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-ca-trust-extracted\") pod \"image-registry-84f4d8756b-ltpj5\" (UID: \"537fa96e-a0c2-48ff-877b-5aca6c1e40f7\") " pod="openshift-image-registry/image-registry-84f4d8756b-ltpj5" Jul 20 02:49:11.265180 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:11.265026 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-registry-certificates\") pod \"image-registry-84f4d8756b-ltpj5\" (UID: \"537fa96e-a0c2-48ff-877b-5aca6c1e40f7\") " pod="openshift-image-registry/image-registry-84f4d8756b-ltpj5" Jul 20 02:49:11.265351 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:11.265326 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/463f86e5-8599-4846-875b-82cdad3c331e-config\") pod \"service-ca-operator-5bcbc94c78-rx856\" (UID: \"463f86e5-8599-4846-875b-82cdad3c331e\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-rx856" Jul 20 02:49:11.265540 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:11.265519 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-trusted-ca\") pod \"image-registry-84f4d8756b-ltpj5\" (UID: \"537fa96e-a0c2-48ff-877b-5aca6c1e40f7\") " pod="openshift-image-registry/image-registry-84f4d8756b-ltpj5" Jul 20 02:49:11.267205 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:11.267180 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/463f86e5-8599-4846-875b-82cdad3c331e-serving-cert\") pod \"service-ca-operator-5bcbc94c78-rx856\" (UID: \"463f86e5-8599-4846-875b-82cdad3c331e\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-rx856" Jul 20 02:49:11.267318 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:11.267242 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-installation-pull-secrets\") pod \"image-registry-84f4d8756b-ltpj5\" (UID: \"537fa96e-a0c2-48ff-877b-5aca6c1e40f7\") " pod="openshift-image-registry/image-registry-84f4d8756b-ltpj5" Jul 20 02:49:11.267374 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:11.267319 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-image-registry-private-configuration\") pod \"image-registry-84f4d8756b-ltpj5\" (UID: \"537fa96e-a0c2-48ff-877b-5aca6c1e40f7\") " pod="openshift-image-registry/image-registry-84f4d8756b-ltpj5" Jul 20 02:49:11.272595 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:11.272568 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt7np\" (UniqueName: \"kubernetes.io/projected/463f86e5-8599-4846-875b-82cdad3c331e-kube-api-access-qt7np\") pod \"service-ca-operator-5bcbc94c78-rx856\" (UID: \"463f86e5-8599-4846-875b-82cdad3c331e\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-rx856" Jul 20 02:49:11.272688 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:11.272668 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-bound-sa-token\") pod \"image-registry-84f4d8756b-ltpj5\" (UID: \"537fa96e-a0c2-48ff-877b-5aca6c1e40f7\") " pod="openshift-image-registry/image-registry-84f4d8756b-ltpj5" Jul 20 02:49:11.272729 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:11.272695 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9zvk\" (UniqueName: \"kubernetes.io/projected/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-kube-api-access-d9zvk\") pod \"image-registry-84f4d8756b-ltpj5\" (UID: \"537fa96e-a0c2-48ff-877b-5aca6c1e40f7\") " pod="openshift-image-registry/image-registry-84f4d8756b-ltpj5" Jul 20 02:49:11.330390 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:11.330360 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-rx856" Jul 20 02:49:11.365258 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:11.365205 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-8vxh2"] Jul 20 02:49:11.368708 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:49:11.368685 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dd69072_56fa_4f7c_bed7_ac8e3d654203.slice/crio-39f3477551055c1a097cbd1e54fb497f7114730e546677a707962101382078ee WatchSource:0}: Error finding container 39f3477551055c1a097cbd1e54fb497f7114730e546677a707962101382078ee: Status 404 returned error can't find the container with id 39f3477551055c1a097cbd1e54fb497f7114730e546677a707962101382078ee Jul 20 02:49:11.453915 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:11.453885 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-rx856"] Jul 20 02:49:11.458378 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:49:11.458344 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod463f86e5_8599_4846_875b_82cdad3c331e.slice/crio-12c360a61f0004130613891fea9a5b27ddd5b3c0907216a7d95962cd57404c3f WatchSource:0}: Error finding container 12c360a61f0004130613891fea9a5b27ddd5b3c0907216a7d95962cd57404c3f: Status 404 returned error can't find the container with id 12c360a61f0004130613891fea9a5b27ddd5b3c0907216a7d95962cd57404c3f Jul 20 02:49:11.767767 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:11.767724 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-registry-tls\") pod \"image-registry-84f4d8756b-ltpj5\" (UID: \"537fa96e-a0c2-48ff-877b-5aca6c1e40f7\") " pod="openshift-image-registry/image-registry-84f4d8756b-ltpj5" Jul 20 02:49:11.767972 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:49:11.767847 2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 20 02:49:11.767972 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:49:11.767859 2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84f4d8756b-ltpj5: secret "image-registry-tls" not found Jul 20 02:49:11.767972 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:49:11.767905 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-registry-tls podName:537fa96e-a0c2-48ff-877b-5aca6c1e40f7 nodeName:}" failed. No retries permitted until 2026-07-20 02:49:12.767891888 +0000 UTC m=+247.967141013 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-registry-tls") pod "image-registry-84f4d8756b-ltpj5" (UID: "537fa96e-a0c2-48ff-877b-5aca6c1e40f7") : secret "image-registry-tls" not found Jul 20 02:49:12.209404 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:12.209369 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-rx856" event={"ID":"463f86e5-8599-4846-875b-82cdad3c331e","Type":"ContainerStarted","Data":"12c360a61f0004130613891fea9a5b27ddd5b3c0907216a7d95962cd57404c3f"} Jul 20 02:49:12.210745 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:12.210721 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-8vxh2" event={"ID":"0dd69072-56fa-4f7c-bed7-ac8e3d654203","Type":"ContainerStarted","Data":"c2ce97fd89ed63d0bda0a33e4dd2850cbd594b21810e1482c9b39e14bd93d1ce"} Jul 20 02:49:12.210861 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:12.210757 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-8vxh2" event={"ID":"0dd69072-56fa-4f7c-bed7-ac8e3d654203","Type":"ContainerStarted","Data":"39f3477551055c1a097cbd1e54fb497f7114730e546677a707962101382078ee"} Jul 20 02:49:12.224709 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:12.224665 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-8vxh2" podStartSLOduration=2.224652143 podStartE2EDuration="2.224652143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:49:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:49:12.224089124 +0000 UTC m=+247.423338283" watchObservedRunningTime="2026-07-20 02:49:12.224652143 +0000 UTC m=+247.423901351" Jul 20 02:49:12.776787 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:12.776733 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-registry-tls\") pod \"image-registry-84f4d8756b-ltpj5\" (UID: \"537fa96e-a0c2-48ff-877b-5aca6c1e40f7\") " pod="openshift-image-registry/image-registry-84f4d8756b-ltpj5" Jul 20 02:49:12.776968 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:49:12.776896 2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 20 02:49:12.776968 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:49:12.776916 2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84f4d8756b-ltpj5: secret "image-registry-tls" not found Jul 20 02:49:12.776968 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:49:12.776969 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-registry-tls podName:537fa96e-a0c2-48ff-877b-5aca6c1e40f7 nodeName:}" failed. No retries permitted until 2026-07-20 02:49:14.77695252 +0000 UTC m=+249.976201649 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-registry-tls") pod "image-registry-84f4d8756b-ltpj5" (UID: "537fa96e-a0c2-48ff-877b-5aca6c1e40f7") : secret "image-registry-tls" not found Jul 20 02:49:14.218571 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:14.218534 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-rx856" event={"ID":"463f86e5-8599-4846-875b-82cdad3c331e","Type":"ContainerStarted","Data":"2901dab0f894e4b936bcfbb85ba7d5d9031eb531a567265a31e57529c577129c"} Jul 20 02:49:14.234211 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:14.234164 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-rx856" podStartSLOduration=1.9048805949999998 podStartE2EDuration="4.23414985s" totalImagesPullingTime="2.329269255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:49:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:49:14.233522632 +0000 UTC m=+249.432771801" watchObservedRunningTime="2026-07-20 02:49:14.23414985 +0000 UTC m=+249.433398996" Jul 20 02:49:14.602123 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:14.602027 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-pc9x7"] Jul 20 02:49:14.605164 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:14.605148 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pc9x7" Jul 20 02:49:14.607741 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:14.607717 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 20 02:49:14.607935 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:14.607922 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 20 02:49:14.608030 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:14.608012 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kg8wf\"" Jul 20 02:49:14.608359 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:14.608342 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 20 02:49:14.608417 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:14.608397 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 20 02:49:14.616872 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:14.616849 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pc9x7"] Jul 20 02:49:14.694622 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:14.694587 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn4w6\" (UniqueName: \"kubernetes.io/projected/88ef3efd-9021-4686-9adc-936513135679-kube-api-access-jn4w6\") pod \"insights-runtime-extractor-pc9x7\" (UID: \"88ef3efd-9021-4686-9adc-936513135679\") " pod="openshift-insights/insights-runtime-extractor-pc9x7" Jul 20 02:49:14.694622 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:14.694624 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/88ef3efd-9021-4686-9adc-936513135679-data-volume\") pod \"insights-runtime-extractor-pc9x7\" (UID: \"88ef3efd-9021-4686-9adc-936513135679\") " pod="openshift-insights/insights-runtime-extractor-pc9x7" Jul 20 02:49:14.694820 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:14.694642 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/88ef3efd-9021-4686-9adc-936513135679-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pc9x7\" (UID: \"88ef3efd-9021-4686-9adc-936513135679\") " pod="openshift-insights/insights-runtime-extractor-pc9x7" Jul 20 02:49:14.694820 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:14.694660 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/88ef3efd-9021-4686-9adc-936513135679-crio-socket\") pod \"insights-runtime-extractor-pc9x7\" (UID: \"88ef3efd-9021-4686-9adc-936513135679\") " pod="openshift-insights/insights-runtime-extractor-pc9x7" Jul 20 02:49:14.694820 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:14.694800 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88ef3efd-9021-4686-9adc-936513135679-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pc9x7\" (UID: \"88ef3efd-9021-4686-9adc-936513135679\") " pod="openshift-insights/insights-runtime-extractor-pc9x7" Jul 20 02:49:14.795686 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:14.795645 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/88ef3efd-9021-4686-9adc-936513135679-data-volume\") pod \"insights-runtime-extractor-pc9x7\" (UID: \"88ef3efd-9021-4686-9adc-936513135679\") " pod="openshift-insights/insights-runtime-extractor-pc9x7" Jul 20 02:49:14.795686 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:14.795689 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/88ef3efd-9021-4686-9adc-936513135679-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pc9x7\" (UID: \"88ef3efd-9021-4686-9adc-936513135679\") " pod="openshift-insights/insights-runtime-extractor-pc9x7" Jul 20 02:49:14.795859 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:14.795747 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/88ef3efd-9021-4686-9adc-936513135679-crio-socket\") pod \"insights-runtime-extractor-pc9x7\" (UID: \"88ef3efd-9021-4686-9adc-936513135679\") " pod="openshift-insights/insights-runtime-extractor-pc9x7" Jul 20 02:49:14.795859 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:49:14.795767 2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 20 02:49:14.795859 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:49:14.795845 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/88ef3efd-9021-4686-9adc-936513135679-kube-rbac-proxy-cm podName:88ef3efd-9021-4686-9adc-936513135679 nodeName:}" failed. No retries permitted until 2026-07-20 02:49:15.295824136 +0000 UTC m=+250.495073264 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/88ef3efd-9021-4686-9adc-936513135679-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pc9x7" (UID: "88ef3efd-9021-4686-9adc-936513135679") : configmap "kube-rbac-proxy" not found Jul 20 02:49:14.795989 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:14.795863 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88ef3efd-9021-4686-9adc-936513135679-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pc9x7\" (UID: \"88ef3efd-9021-4686-9adc-936513135679\") " pod="openshift-insights/insights-runtime-extractor-pc9x7" Jul 20 02:49:14.795989 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:14.795870 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/88ef3efd-9021-4686-9adc-936513135679-crio-socket\") pod \"insights-runtime-extractor-pc9x7\" (UID: \"88ef3efd-9021-4686-9adc-936513135679\") " pod="openshift-insights/insights-runtime-extractor-pc9x7" Jul 20 02:49:14.795989 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:14.795904 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-registry-tls\") pod \"image-registry-84f4d8756b-ltpj5\" (UID: \"537fa96e-a0c2-48ff-877b-5aca6c1e40f7\") " pod="openshift-image-registry/image-registry-84f4d8756b-ltpj5" Jul 20 02:49:14.795989 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:14.795948 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn4w6\" (UniqueName: \"kubernetes.io/projected/88ef3efd-9021-4686-9adc-936513135679-kube-api-access-jn4w6\") pod \"insights-runtime-extractor-pc9x7\" (UID: \"88ef3efd-9021-4686-9adc-936513135679\") " pod="openshift-insights/insights-runtime-extractor-pc9x7" Jul 20 02:49:14.795989 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:49:14.795955 2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 20 02:49:14.795989 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:49:14.795988 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88ef3efd-9021-4686-9adc-936513135679-insights-runtime-extractor-tls podName:88ef3efd-9021-4686-9adc-936513135679 nodeName:}" failed. No retries permitted until 2026-07-20 02:49:15.295977826 +0000 UTC m=+250.495226954 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/88ef3efd-9021-4686-9adc-936513135679-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pc9x7" (UID: "88ef3efd-9021-4686-9adc-936513135679") : secret "insights-runtime-extractor-tls" not found Jul 20 02:49:14.796276 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:49:14.796041 2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 20 02:49:14.796276 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:49:14.796057 2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84f4d8756b-ltpj5: secret "image-registry-tls" not found Jul 20 02:49:14.796276 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:49:14.796123 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-registry-tls podName:537fa96e-a0c2-48ff-877b-5aca6c1e40f7 nodeName:}" failed. No retries permitted until 2026-07-20 02:49:18.796091328 +0000 UTC m=+253.995340467 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-registry-tls") pod "image-registry-84f4d8756b-ltpj5" (UID: "537fa96e-a0c2-48ff-877b-5aca6c1e40f7") : secret "image-registry-tls" not found Jul 20 02:49:14.796276 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:14.796164 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/88ef3efd-9021-4686-9adc-936513135679-data-volume\") pod \"insights-runtime-extractor-pc9x7\" (UID: \"88ef3efd-9021-4686-9adc-936513135679\") " pod="openshift-insights/insights-runtime-extractor-pc9x7" Jul 20 02:49:14.803167 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:14.803144 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn4w6\" (UniqueName: \"kubernetes.io/projected/88ef3efd-9021-4686-9adc-936513135679-kube-api-access-jn4w6\") pod \"insights-runtime-extractor-pc9x7\" (UID: \"88ef3efd-9021-4686-9adc-936513135679\") " pod="openshift-insights/insights-runtime-extractor-pc9x7" Jul 20 02:49:15.300738 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:15.300700 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/88ef3efd-9021-4686-9adc-936513135679-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pc9x7\" (UID: \"88ef3efd-9021-4686-9adc-936513135679\") " pod="openshift-insights/insights-runtime-extractor-pc9x7" Jul 20 02:49:15.301167 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:15.300807 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88ef3efd-9021-4686-9adc-936513135679-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pc9x7\" (UID: \"88ef3efd-9021-4686-9adc-936513135679\") " pod="openshift-insights/insights-runtime-extractor-pc9x7" Jul 20 02:49:15.301167 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:49:15.300874 2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 20 02:49:15.301167 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:49:15.300901 2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 20 02:49:15.301167 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:49:15.300951 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/88ef3efd-9021-4686-9adc-936513135679-kube-rbac-proxy-cm podName:88ef3efd-9021-4686-9adc-936513135679 nodeName:}" failed. No retries permitted until 2026-07-20 02:49:16.300935411 +0000 UTC m=+251.500184535 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/88ef3efd-9021-4686-9adc-936513135679-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pc9x7" (UID: "88ef3efd-9021-4686-9adc-936513135679") : configmap "kube-rbac-proxy" not found Jul 20 02:49:15.301167 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:49:15.300967 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88ef3efd-9021-4686-9adc-936513135679-insights-runtime-extractor-tls podName:88ef3efd-9021-4686-9adc-936513135679 nodeName:}" failed. No retries permitted until 2026-07-20 02:49:16.300961288 +0000 UTC m=+251.500210413 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/88ef3efd-9021-4686-9adc-936513135679-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pc9x7" (UID: "88ef3efd-9021-4686-9adc-936513135679") : secret "insights-runtime-extractor-tls" not found Jul 20 02:49:16.308411 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:16.308372 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/88ef3efd-9021-4686-9adc-936513135679-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pc9x7\" (UID: \"88ef3efd-9021-4686-9adc-936513135679\") " pod="openshift-insights/insights-runtime-extractor-pc9x7" Jul 20 02:49:16.308909 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:16.308463 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88ef3efd-9021-4686-9adc-936513135679-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pc9x7\" (UID: \"88ef3efd-9021-4686-9adc-936513135679\") " pod="openshift-insights/insights-runtime-extractor-pc9x7" Jul 20 02:49:16.308909 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:49:16.308498 2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 20 02:49:16.308909 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:49:16.308564 2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 20 02:49:16.308909 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:49:16.308580 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/88ef3efd-9021-4686-9adc-936513135679-kube-rbac-proxy-cm podName:88ef3efd-9021-4686-9adc-936513135679 nodeName:}" failed. No retries permitted until 2026-07-20 02:49:18.308560967 +0000 UTC m=+253.507810112 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/88ef3efd-9021-4686-9adc-936513135679-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pc9x7" (UID: "88ef3efd-9021-4686-9adc-936513135679") : configmap "kube-rbac-proxy" not found Jul 20 02:49:16.308909 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:49:16.308597 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88ef3efd-9021-4686-9adc-936513135679-insights-runtime-extractor-tls podName:88ef3efd-9021-4686-9adc-936513135679 nodeName:}" failed. No retries permitted until 2026-07-20 02:49:18.30858796 +0000 UTC m=+253.507837086 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/88ef3efd-9021-4686-9adc-936513135679-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pc9x7" (UID: "88ef3efd-9021-4686-9adc-936513135679") : secret "insights-runtime-extractor-tls" not found Jul 20 02:49:16.582788 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:16.582721 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lvkqp_c361af25-375d-4bff-bdb9-53f137cea0f6/dns-node-resolver/0.log" Jul 20 02:49:17.115314 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:17.115270 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d70f5103-7dc9-4279-9781-38e395da435b-metrics-certs\") pod \"network-metrics-daemon-9x7hv\" (UID: \"d70f5103-7dc9-4279-9781-38e395da435b\") " pod="openshift-multus/network-metrics-daemon-9x7hv" Jul 20 02:49:17.115483 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:49:17.115420 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 20 02:49:17.115483 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:49:17.115478 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d70f5103-7dc9-4279-9781-38e395da435b-metrics-certs podName:d70f5103-7dc9-4279-9781-38e395da435b nodeName:}" failed. No retries permitted until 2026-07-20 02:51:19.115462731 +0000 UTC m=+374.314711871 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d70f5103-7dc9-4279-9781-38e395da435b-metrics-certs") pod "network-metrics-daemon-9x7hv" (UID: "d70f5103-7dc9-4279-9781-38e395da435b") : secret "metrics-daemon-secret" not found Jul 20 02:49:17.383166 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:17.383138 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9qxnl_504cb1f0-841a-49f7-add8-2d4742a1cb68/node-ca/0.log" Jul 20 02:49:17.550752 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:17.550715 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-cwznc"] Jul 20 02:49:17.553771 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:17.553752 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-cwznc" Jul 20 02:49:17.556382 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:17.556365 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 20 02:49:17.556483 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:17.556418 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 20 02:49:17.557397 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:17.557380 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 20 02:49:17.557397 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:17.557391 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-pg6jh\"" Jul 20 02:49:17.557535 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:17.557416 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 20 02:49:17.560796 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:17.560776 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-cwznc"] Jul 20 02:49:17.719466 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:17.719376 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1ecfba3e-9f5c-41e9-9679-900a1995faf7-signing-key\") pod \"service-ca-58598695f7-cwznc\" (UID: \"1ecfba3e-9f5c-41e9-9679-900a1995faf7\") " pod="openshift-service-ca/service-ca-58598695f7-cwznc" Jul 20 02:49:17.719466 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:17.719448 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r4z2\" (UniqueName: \"kubernetes.io/projected/1ecfba3e-9f5c-41e9-9679-900a1995faf7-kube-api-access-9r4z2\") pod \"service-ca-58598695f7-cwznc\" (UID: \"1ecfba3e-9f5c-41e9-9679-900a1995faf7\") " pod="openshift-service-ca/service-ca-58598695f7-cwznc" Jul 20 02:49:17.719671 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:17.719492 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1ecfba3e-9f5c-41e9-9679-900a1995faf7-signing-cabundle\") pod \"service-ca-58598695f7-cwznc\" (UID: \"1ecfba3e-9f5c-41e9-9679-900a1995faf7\") " pod="openshift-service-ca/service-ca-58598695f7-cwznc" Jul 20 02:49:17.819879 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:17.819841 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r4z2\" (UniqueName: \"kubernetes.io/projected/1ecfba3e-9f5c-41e9-9679-900a1995faf7-kube-api-access-9r4z2\") pod \"service-ca-58598695f7-cwznc\" (UID: \"1ecfba3e-9f5c-41e9-9679-900a1995faf7\") " pod="openshift-service-ca/service-ca-58598695f7-cwznc" Jul 20 02:49:17.820035 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:17.819893 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1ecfba3e-9f5c-41e9-9679-900a1995faf7-signing-cabundle\") pod \"service-ca-58598695f7-cwznc\" (UID: \"1ecfba3e-9f5c-41e9-9679-900a1995faf7\") " pod="openshift-service-ca/service-ca-58598695f7-cwznc" Jul 20 02:49:17.820035 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:17.819973 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1ecfba3e-9f5c-41e9-9679-900a1995faf7-signing-key\") pod \"service-ca-58598695f7-cwznc\" (UID: \"1ecfba3e-9f5c-41e9-9679-900a1995faf7\") " pod="openshift-service-ca/service-ca-58598695f7-cwznc" Jul 20 02:49:17.820548 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:17.820527 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1ecfba3e-9f5c-41e9-9679-900a1995faf7-signing-cabundle\") pod \"service-ca-58598695f7-cwznc\" (UID: \"1ecfba3e-9f5c-41e9-9679-900a1995faf7\") " pod="openshift-service-ca/service-ca-58598695f7-cwznc" Jul 20 02:49:17.822352 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:17.822335 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1ecfba3e-9f5c-41e9-9679-900a1995faf7-signing-key\") pod \"service-ca-58598695f7-cwznc\" (UID: \"1ecfba3e-9f5c-41e9-9679-900a1995faf7\") " pod="openshift-service-ca/service-ca-58598695f7-cwznc" Jul 20 02:49:17.827549 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:17.827524 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r4z2\" (UniqueName: \"kubernetes.io/projected/1ecfba3e-9f5c-41e9-9679-900a1995faf7-kube-api-access-9r4z2\") pod \"service-ca-58598695f7-cwznc\" (UID: \"1ecfba3e-9f5c-41e9-9679-900a1995faf7\") " pod="openshift-service-ca/service-ca-58598695f7-cwznc" Jul 20 02:49:17.863335 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:17.863300 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-cwznc" Jul 20 02:49:17.985595 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:17.985528 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-cwznc"] Jul 20 02:49:17.988652 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:49:17.988625 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ecfba3e_9f5c_41e9_9679_900a1995faf7.slice/crio-8037e28d562ba6671ec1b2dde546c5f974980cf858bd2c2cb42da9ecaab06835 WatchSource:0}: Error finding container 8037e28d562ba6671ec1b2dde546c5f974980cf858bd2c2cb42da9ecaab06835: Status 404 returned error can't find the container with id 8037e28d562ba6671ec1b2dde546c5f974980cf858bd2c2cb42da9ecaab06835 Jul 20 02:49:18.183593 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:18.183563 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-d5lxm_f534591e-00ab-4312-baa1-18298304041f/serve-healthcheck-canary/0.log" Jul 20 02:49:18.231666 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:18.231631 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-cwznc" event={"ID":"1ecfba3e-9f5c-41e9-9679-900a1995faf7","Type":"ContainerStarted","Data":"120e3b0373c114d5962ea1d3fafe82bc19d4d554a91f3413c52cc25dca82b2a1"} Jul 20 02:49:18.231835 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:18.231678 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-cwznc" event={"ID":"1ecfba3e-9f5c-41e9-9679-900a1995faf7","Type":"ContainerStarted","Data":"8037e28d562ba6671ec1b2dde546c5f974980cf858bd2c2cb42da9ecaab06835"} Jul 20 02:49:18.249429 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:18.249331 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-cwznc" podStartSLOduration=1.249317463 podStartE2EDuration="1.249317463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:49:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:49:18.248961671 +0000 UTC m=+253.448210827" watchObservedRunningTime="2026-07-20 02:49:18.249317463 +0000 UTC m=+253.448566610" Jul 20 02:49:18.324047 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:18.324016 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/88ef3efd-9021-4686-9adc-936513135679-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pc9x7\" (UID: \"88ef3efd-9021-4686-9adc-936513135679\") " pod="openshift-insights/insights-runtime-extractor-pc9x7" Jul 20 02:49:18.324233 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:18.324110 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88ef3efd-9021-4686-9adc-936513135679-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pc9x7\" (UID: \"88ef3efd-9021-4686-9adc-936513135679\") " pod="openshift-insights/insights-runtime-extractor-pc9x7" Jul 20 02:49:18.324233 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:49:18.324178 2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 20 02:49:18.324347 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:49:18.324258 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/88ef3efd-9021-4686-9adc-936513135679-kube-rbac-proxy-cm podName:88ef3efd-9021-4686-9adc-936513135679 nodeName:}" failed. No retries permitted until 2026-07-20 02:49:22.32423738 +0000 UTC m=+257.523486519 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/88ef3efd-9021-4686-9adc-936513135679-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pc9x7" (UID: "88ef3efd-9021-4686-9adc-936513135679") : configmap "kube-rbac-proxy" not found Jul 20 02:49:18.324347 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:49:18.324278 2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 20 02:49:18.324435 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:49:18.324349 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88ef3efd-9021-4686-9adc-936513135679-insights-runtime-extractor-tls podName:88ef3efd-9021-4686-9adc-936513135679 nodeName:}" failed. No retries permitted until 2026-07-20 02:49:22.324326885 +0000 UTC m=+257.523576026 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/88ef3efd-9021-4686-9adc-936513135679-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pc9x7" (UID: "88ef3efd-9021-4686-9adc-936513135679") : secret "insights-runtime-extractor-tls" not found Jul 20 02:49:18.826607 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:18.826574 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-registry-tls\") pod \"image-registry-84f4d8756b-ltpj5\" (UID: \"537fa96e-a0c2-48ff-877b-5aca6c1e40f7\") " pod="openshift-image-registry/image-registry-84f4d8756b-ltpj5" Jul 20 02:49:18.827072 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:49:18.826713 2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 20 02:49:18.827072 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:49:18.826732 2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84f4d8756b-ltpj5: secret "image-registry-tls" not found Jul 20 02:49:18.827072 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:49:18.826792 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-registry-tls podName:537fa96e-a0c2-48ff-877b-5aca6c1e40f7 nodeName:}" failed. No retries permitted until 2026-07-20 02:49:26.826772456 +0000 UTC m=+262.026021582 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-registry-tls") pod "image-registry-84f4d8756b-ltpj5" (UID: "537fa96e-a0c2-48ff-877b-5aca6c1e40f7") : secret "image-registry-tls" not found Jul 20 02:49:22.354690 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:22.354648 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88ef3efd-9021-4686-9adc-936513135679-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pc9x7\" (UID: \"88ef3efd-9021-4686-9adc-936513135679\") " pod="openshift-insights/insights-runtime-extractor-pc9x7" Jul 20 02:49:22.355122 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:22.354715 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/88ef3efd-9021-4686-9adc-936513135679-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pc9x7\" (UID: \"88ef3efd-9021-4686-9adc-936513135679\") " pod="openshift-insights/insights-runtime-extractor-pc9x7" Jul 20 02:49:22.355122 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:49:22.354804 2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 20 02:49:22.355122 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:49:22.354807 2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 20 02:49:22.355122 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:49:22.354861 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/88ef3efd-9021-4686-9adc-936513135679-kube-rbac-proxy-cm podName:88ef3efd-9021-4686-9adc-936513135679 nodeName:}" failed. No retries permitted until 2026-07-20 02:49:30.354845764 +0000 UTC m=+265.554094890 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/88ef3efd-9021-4686-9adc-936513135679-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pc9x7" (UID: "88ef3efd-9021-4686-9adc-936513135679") : configmap "kube-rbac-proxy" not found Jul 20 02:49:22.355122 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:49:22.354874 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88ef3efd-9021-4686-9adc-936513135679-insights-runtime-extractor-tls podName:88ef3efd-9021-4686-9adc-936513135679 nodeName:}" failed. No retries permitted until 2026-07-20 02:49:30.354867827 +0000 UTC m=+265.554116952 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/88ef3efd-9021-4686-9adc-936513135679-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pc9x7" (UID: "88ef3efd-9021-4686-9adc-936513135679") : secret "insights-runtime-extractor-tls" not found Jul 20 02:49:26.890794 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:26.890711 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-registry-tls\") pod \"image-registry-84f4d8756b-ltpj5\" (UID: \"537fa96e-a0c2-48ff-877b-5aca6c1e40f7\") " pod="openshift-image-registry/image-registry-84f4d8756b-ltpj5" Jul 20 02:49:26.893139 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:26.893114 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-registry-tls\") pod \"image-registry-84f4d8756b-ltpj5\" (UID: \"537fa96e-a0c2-48ff-877b-5aca6c1e40f7\") " pod="openshift-image-registry/image-registry-84f4d8756b-ltpj5" Jul 20 02:49:26.940786 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:26.940756 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84f4d8756b-ltpj5" Jul 20 02:49:27.073542 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:27.073511 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84f4d8756b-ltpj5"] Jul 20 02:49:27.077312 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:49:27.077280 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod537fa96e_a0c2_48ff_877b_5aca6c1e40f7.slice/crio-cb94d45149af0e2e69367428f1379382162d7db55f822e9904c4ca8fd3d17ac8 WatchSource:0}: Error finding container cb94d45149af0e2e69367428f1379382162d7db55f822e9904c4ca8fd3d17ac8: Status 404 returned error can't find the container with id cb94d45149af0e2e69367428f1379382162d7db55f822e9904c4ca8fd3d17ac8 Jul 20 02:49:27.261601 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:27.261557 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84f4d8756b-ltpj5" event={"ID":"537fa96e-a0c2-48ff-877b-5aca6c1e40f7","Type":"ContainerStarted","Data":"058a8584f827a75ad94788f7e523fe2faeb9b39a1b8aeb8fc01292a7b36692d1"} Jul 20 02:49:27.261769 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:27.261617 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84f4d8756b-ltpj5" event={"ID":"537fa96e-a0c2-48ff-877b-5aca6c1e40f7","Type":"ContainerStarted","Data":"cb94d45149af0e2e69367428f1379382162d7db55f822e9904c4ca8fd3d17ac8"} Jul 20 02:49:27.261769 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:27.261694 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-84f4d8756b-ltpj5" Jul 20 02:49:27.280175 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:27.280125 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-84f4d8756b-ltpj5" podStartSLOduration=16.280090287 podStartE2EDuration="16.280090287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:49:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:49:27.279291804 +0000 UTC m=+262.478540942" watchObservedRunningTime="2026-07-20 02:49:27.280090287 +0000 UTC m=+262.479339458" Jul 20 02:49:30.418564 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:30.418512 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/88ef3efd-9021-4686-9adc-936513135679-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pc9x7\" (UID: \"88ef3efd-9021-4686-9adc-936513135679\") " pod="openshift-insights/insights-runtime-extractor-pc9x7" Jul 20 02:49:30.418949 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:30.418629 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88ef3efd-9021-4686-9adc-936513135679-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pc9x7\" (UID: \"88ef3efd-9021-4686-9adc-936513135679\") " pod="openshift-insights/insights-runtime-extractor-pc9x7" Jul 20 02:49:30.423261 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:30.419514 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/88ef3efd-9021-4686-9adc-936513135679-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pc9x7\" (UID: \"88ef3efd-9021-4686-9adc-936513135679\") " pod="openshift-insights/insights-runtime-extractor-pc9x7" Jul 20 02:49:30.423892 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:30.423870 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88ef3efd-9021-4686-9adc-936513135679-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pc9x7\" (UID: \"88ef3efd-9021-4686-9adc-936513135679\") " pod="openshift-insights/insights-runtime-extractor-pc9x7" Jul 20 02:49:30.515310 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:30.515275 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pc9x7" Jul 20 02:49:30.641564 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:30.641524 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pc9x7"] Jul 20 02:49:30.644440 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:49:30.644417 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88ef3efd_9021_4686_9adc_936513135679.slice/crio-7fae45f36d40bca2455524e7525005b1b0e3ab6dc63441d322c5beea5836d8df WatchSource:0}: Error finding container 7fae45f36d40bca2455524e7525005b1b0e3ab6dc63441d322c5beea5836d8df: Status 404 returned error can't find the container with id 7fae45f36d40bca2455524e7525005b1b0e3ab6dc63441d322c5beea5836d8df Jul 20 02:49:31.276278 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:31.276220 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pc9x7" event={"ID":"88ef3efd-9021-4686-9adc-936513135679","Type":"ContainerStarted","Data":"7fae45f36d40bca2455524e7525005b1b0e3ab6dc63441d322c5beea5836d8df"} Jul 20 02:49:36.018055 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:36.018017 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-84f4d8756b-ltpj5"] Jul 20 02:49:36.030318 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:36.029672 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-dwj5r"] Jul 20 02:49:36.033328 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:36.033223 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-dwj5r" Jul 20 02:49:36.036954 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:36.036874 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-48h2h\"" Jul 20 02:49:36.037297 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:36.037227 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 20 02:49:36.053176 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:36.053111 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-dwj5r"] Jul 20 02:49:36.056970 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:36.056938 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ff2fffa2-659c-4f26-85b0-7e644ce93348-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-dwj5r\" (UID: \"ff2fffa2-659c-4f26-85b0-7e644ce93348\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-dwj5r" Jul 20 02:49:36.157970 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:36.157821 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ff2fffa2-659c-4f26-85b0-7e644ce93348-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-dwj5r\" (UID: \"ff2fffa2-659c-4f26-85b0-7e644ce93348\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-dwj5r" Jul 20 02:49:36.161246 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:36.161213 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ff2fffa2-659c-4f26-85b0-7e644ce93348-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-dwj5r\" (UID: \"ff2fffa2-659c-4f26-85b0-7e644ce93348\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-dwj5r" Jul 20 02:49:36.294119 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:36.294024 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pc9x7" event={"ID":"88ef3efd-9021-4686-9adc-936513135679","Type":"ContainerStarted","Data":"e66984bee8d3312bfa95026844943daf9cecbb797ae320fefcd02ef5e03e7ecc"} Jul 20 02:49:36.345932 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:36.345893 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-dwj5r" Jul 20 02:49:36.496898 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:36.496679 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-dwj5r"] Jul 20 02:49:36.499856 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:49:36.499821 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff2fffa2_659c_4f26_85b0_7e644ce93348.slice/crio-d1889971b9aea665abb2477be9351ae1c67c9c735170ec5270dfeed86a0d6818 WatchSource:0}: Error finding container d1889971b9aea665abb2477be9351ae1c67c9c735170ec5270dfeed86a0d6818: Status 404 returned error can't find the container with id d1889971b9aea665abb2477be9351ae1c67c9c735170ec5270dfeed86a0d6818 Jul 20 02:49:37.299401 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:37.299361 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-dwj5r" event={"ID":"ff2fffa2-659c-4f26-85b0-7e644ce93348","Type":"ContainerStarted","Data":"d1889971b9aea665abb2477be9351ae1c67c9c735170ec5270dfeed86a0d6818"} Jul 20 02:49:39.308219 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:39.308179 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pc9x7" event={"ID":"88ef3efd-9021-4686-9adc-936513135679","Type":"ContainerStarted","Data":"41a2f881d26ae874c427af734388c058c2d27c3bb34cc6004e1b2348bb672b00"} Jul 20 02:49:39.309601 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:39.309572 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-dwj5r" event={"ID":"ff2fffa2-659c-4f26-85b0-7e644ce93348","Type":"ContainerStarted","Data":"af92d398794fbdc6c7cb0cda78e846556cd8593bbe2a34403df86205a3400df0"} Jul 20 02:49:39.309773 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:39.309755 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-dwj5r" Jul 20 02:49:39.314288 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:39.314269 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-dwj5r" Jul 20 02:49:39.323955 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:39.323884 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-dwj5r" podStartSLOduration=0.722208755 podStartE2EDuration="3.323870923s" totalImagesPullingTime="2.601662168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:49:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:49:39.323206222 +0000 UTC m=+274.522455368" watchObservedRunningTime="2026-07-20 02:49:39.323870923 +0000 UTC m=+274.523120069" Jul 20 02:49:43.324713 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:43.324677 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pc9x7" event={"ID":"88ef3efd-9021-4686-9adc-936513135679","Type":"ContainerStarted","Data":"677cec5f7491a7165225d9b9ae02f1a5a0f7360cc8982bae74aa1c6c8ef2b5e2"} Jul 20 02:49:43.342626 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:43.342581 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-pc9x7" podStartSLOduration=17.598119021 podStartE2EDuration="29.342566993s" totalImagesPullingTime="11.744447972s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 02:49:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:49:43.340918356 +0000 UTC m=+278.540167505" watchObservedRunningTime="2026-07-20 02:49:43.342566993 +0000 UTC m=+278.541816140" Jul 20 02:49:45.390922 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:45.390890 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-74bcf79799-8xc5s"] Jul 20 02:49:45.394054 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:45.394035 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74bcf79799-8xc5s" Jul 20 02:49:45.400324 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:45.400107 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 20 02:49:45.400324 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:45.400312 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 20 02:49:45.400450 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:45.400323 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 20 02:49:45.400450 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:45.400412 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 20 02:49:45.400655 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:45.400637 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 20 02:49:45.400722 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:45.400692 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-q5q4k\"" Jul 20 02:49:45.400778 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:45.400723 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 20 02:49:45.400778 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:45.400750 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 20 02:49:45.409179 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:45.409160 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74bcf79799-8xc5s"] Jul 20 02:49:45.427676 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:45.427647 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/95a990cb-fd40-4d72-8a18-fc3ae2930904-console-config\") pod \"console-74bcf79799-8xc5s\" (UID: \"95a990cb-fd40-4d72-8a18-fc3ae2930904\") " pod="openshift-console/console-74bcf79799-8xc5s" Jul 20 02:49:45.427776 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:45.427729 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/95a990cb-fd40-4d72-8a18-fc3ae2930904-console-serving-cert\") pod \"console-74bcf79799-8xc5s\" (UID: \"95a990cb-fd40-4d72-8a18-fc3ae2930904\") " pod="openshift-console/console-74bcf79799-8xc5s" Jul 20 02:49:45.427776 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:45.427768 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/95a990cb-fd40-4d72-8a18-fc3ae2930904-console-oauth-config\") pod \"console-74bcf79799-8xc5s\" (UID: \"95a990cb-fd40-4d72-8a18-fc3ae2930904\") " pod="openshift-console/console-74bcf79799-8xc5s" Jul 20 02:49:45.427845 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:45.427829 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zzcj\" (UniqueName: \"kubernetes.io/projected/95a990cb-fd40-4d72-8a18-fc3ae2930904-kube-api-access-5zzcj\") pod \"console-74bcf79799-8xc5s\" (UID: \"95a990cb-fd40-4d72-8a18-fc3ae2930904\") " pod="openshift-console/console-74bcf79799-8xc5s" Jul 20 02:49:45.427877 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:45.427866 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/95a990cb-fd40-4d72-8a18-fc3ae2930904-oauth-serving-cert\") pod \"console-74bcf79799-8xc5s\" (UID: \"95a990cb-fd40-4d72-8a18-fc3ae2930904\") " pod="openshift-console/console-74bcf79799-8xc5s" Jul 20 02:49:45.427935 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:45.427919 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/95a990cb-fd40-4d72-8a18-fc3ae2930904-service-ca\") pod \"console-74bcf79799-8xc5s\" (UID: \"95a990cb-fd40-4d72-8a18-fc3ae2930904\") " pod="openshift-console/console-74bcf79799-8xc5s" Jul 20 02:49:45.528494 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:45.528456 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/95a990cb-fd40-4d72-8a18-fc3ae2930904-console-oauth-config\") pod \"console-74bcf79799-8xc5s\" (UID: \"95a990cb-fd40-4d72-8a18-fc3ae2930904\") " pod="openshift-console/console-74bcf79799-8xc5s" Jul 20 02:49:45.528666 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:45.528517 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zzcj\" (UniqueName: \"kubernetes.io/projected/95a990cb-fd40-4d72-8a18-fc3ae2930904-kube-api-access-5zzcj\") pod \"console-74bcf79799-8xc5s\" (UID: \"95a990cb-fd40-4d72-8a18-fc3ae2930904\") " pod="openshift-console/console-74bcf79799-8xc5s" Jul 20 02:49:45.528666 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:45.528541 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/95a990cb-fd40-4d72-8a18-fc3ae2930904-oauth-serving-cert\") pod \"console-74bcf79799-8xc5s\" (UID: \"95a990cb-fd40-4d72-8a18-fc3ae2930904\") " pod="openshift-console/console-74bcf79799-8xc5s" Jul 20 02:49:45.528666 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:45.528580 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/95a990cb-fd40-4d72-8a18-fc3ae2930904-service-ca\") pod \"console-74bcf79799-8xc5s\" (UID: \"95a990cb-fd40-4d72-8a18-fc3ae2930904\") " pod="openshift-console/console-74bcf79799-8xc5s" Jul 20 02:49:45.528666 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:45.528609 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/95a990cb-fd40-4d72-8a18-fc3ae2930904-console-config\") pod \"console-74bcf79799-8xc5s\" (UID: \"95a990cb-fd40-4d72-8a18-fc3ae2930904\") " pod="openshift-console/console-74bcf79799-8xc5s" Jul 20 02:49:45.528666 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:45.528661 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/95a990cb-fd40-4d72-8a18-fc3ae2930904-console-serving-cert\") pod \"console-74bcf79799-8xc5s\" (UID: \"95a990cb-fd40-4d72-8a18-fc3ae2930904\") " pod="openshift-console/console-74bcf79799-8xc5s" Jul 20 02:49:45.529385 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:45.529351 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/95a990cb-fd40-4d72-8a18-fc3ae2930904-oauth-serving-cert\") pod \"console-74bcf79799-8xc5s\" (UID: \"95a990cb-fd40-4d72-8a18-fc3ae2930904\") " pod="openshift-console/console-74bcf79799-8xc5s" Jul 20 02:49:45.529503 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:45.529400 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/95a990cb-fd40-4d72-8a18-fc3ae2930904-service-ca\") pod \"console-74bcf79799-8xc5s\" (UID: \"95a990cb-fd40-4d72-8a18-fc3ae2930904\") " pod="openshift-console/console-74bcf79799-8xc5s" Jul 20 02:49:45.529503 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:45.529418 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/95a990cb-fd40-4d72-8a18-fc3ae2930904-console-config\") pod \"console-74bcf79799-8xc5s\" (UID: \"95a990cb-fd40-4d72-8a18-fc3ae2930904\") " pod="openshift-console/console-74bcf79799-8xc5s" Jul 20 02:49:45.530856 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:45.530826 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/95a990cb-fd40-4d72-8a18-fc3ae2930904-console-oauth-config\") pod \"console-74bcf79799-8xc5s\" (UID: \"95a990cb-fd40-4d72-8a18-fc3ae2930904\") " pod="openshift-console/console-74bcf79799-8xc5s" Jul 20 02:49:45.530991 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:45.530968 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/95a990cb-fd40-4d72-8a18-fc3ae2930904-console-serving-cert\") pod \"console-74bcf79799-8xc5s\" (UID: \"95a990cb-fd40-4d72-8a18-fc3ae2930904\") " pod="openshift-console/console-74bcf79799-8xc5s" Jul 20 02:49:45.536766 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:45.536741 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zzcj\" (UniqueName: \"kubernetes.io/projected/95a990cb-fd40-4d72-8a18-fc3ae2930904-kube-api-access-5zzcj\") pod \"console-74bcf79799-8xc5s\" (UID: \"95a990cb-fd40-4d72-8a18-fc3ae2930904\") " pod="openshift-console/console-74bcf79799-8xc5s" Jul 20 02:49:45.703198 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:45.703088 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74bcf79799-8xc5s" Jul 20 02:49:45.828869 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:45.828838 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74bcf79799-8xc5s"] Jul 20 02:49:45.831571 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:49:45.831543 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95a990cb_fd40_4d72_8a18_fc3ae2930904.slice/crio-ed4a97eaf6003d105c49e353c8551d244e9aeda855fc2433bbe61f939ec3372b WatchSource:0}: Error finding container ed4a97eaf6003d105c49e353c8551d244e9aeda855fc2433bbe61f939ec3372b: Status 404 returned error can't find the container with id ed4a97eaf6003d105c49e353c8551d244e9aeda855fc2433bbe61f939ec3372b Jul 20 02:49:45.964432 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:49:45.964343 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-rkzf8" podUID="f32d6cda-7b64-4aa8-90ee-21c1e3043ed3" Jul 20 02:49:46.024648 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:46.024611 2558 patch_prober.go:28] interesting pod/image-registry-84f4d8756b-ltpj5 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 20 02:49:46.024795 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:46.024662 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-84f4d8756b-ltpj5" podUID="537fa96e-a0c2-48ff-877b-5aca6c1e40f7" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 20 02:49:46.338812 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:46.338717 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74bcf79799-8xc5s" event={"ID":"95a990cb-fd40-4d72-8a18-fc3ae2930904","Type":"ContainerStarted","Data":"ed4a97eaf6003d105c49e353c8551d244e9aeda855fc2433bbe61f939ec3372b"} Jul 20 02:49:46.338812 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:46.338738 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rkzf8" Jul 20 02:49:47.221630 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:47.221590 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-nxwvq"] Jul 20 02:49:47.225353 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:47.225263 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nxwvq" Jul 20 02:49:47.228276 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:47.228257 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 20 02:49:47.228587 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:47.228569 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 20 02:49:47.229668 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:47.229473 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 20 02:49:47.229668 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:47.229499 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 20 02:49:47.229668 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:47.229539 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-z7hth\"" Jul 20 02:49:47.229869 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:47.229752 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 20 02:49:47.229928 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:47.229911 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 20 02:49:47.243396 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:47.243367 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/804f2ad0-31c6-4bab-a698-d1d2224d9e5f-metrics-client-ca\") pod \"node-exporter-nxwvq\" (UID: \"804f2ad0-31c6-4bab-a698-d1d2224d9e5f\") " pod="openshift-monitoring/node-exporter-nxwvq" Jul 20 02:49:47.243516 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:47.243417 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/804f2ad0-31c6-4bab-a698-d1d2224d9e5f-root\") pod \"node-exporter-nxwvq\" (UID: \"804f2ad0-31c6-4bab-a698-d1d2224d9e5f\") " pod="openshift-monitoring/node-exporter-nxwvq" Jul 20 02:49:47.243516 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:47.243443 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/804f2ad0-31c6-4bab-a698-d1d2224d9e5f-sys\") pod \"node-exporter-nxwvq\" (UID: \"804f2ad0-31c6-4bab-a698-d1d2224d9e5f\") " pod="openshift-monitoring/node-exporter-nxwvq" Jul 20 02:49:47.243516 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:47.243462 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lns5j\" (UniqueName: \"kubernetes.io/projected/804f2ad0-31c6-4bab-a698-d1d2224d9e5f-kube-api-access-lns5j\") pod \"node-exporter-nxwvq\" (UID: \"804f2ad0-31c6-4bab-a698-d1d2224d9e5f\") " pod="openshift-monitoring/node-exporter-nxwvq" Jul 20 02:49:47.243516 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:47.243477 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/804f2ad0-31c6-4bab-a698-d1d2224d9e5f-node-exporter-wtmp\") pod \"node-exporter-nxwvq\" (UID: \"804f2ad0-31c6-4bab-a698-d1d2224d9e5f\") " pod="openshift-monitoring/node-exporter-nxwvq" Jul 20 02:49:47.243516 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:47.243507 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/804f2ad0-31c6-4bab-a698-d1d2224d9e5f-node-exporter-accelerators-collector-config\") pod \"node-exporter-nxwvq\" (UID: \"804f2ad0-31c6-4bab-a698-d1d2224d9e5f\") " pod="openshift-monitoring/node-exporter-nxwvq" Jul 20 02:49:47.243762 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:47.243556 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/804f2ad0-31c6-4bab-a698-d1d2224d9e5f-node-exporter-textfile\") pod \"node-exporter-nxwvq\" (UID: \"804f2ad0-31c6-4bab-a698-d1d2224d9e5f\") " pod="openshift-monitoring/node-exporter-nxwvq" Jul 20 02:49:47.243762 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:47.243584 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/804f2ad0-31c6-4bab-a698-d1d2224d9e5f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nxwvq\" (UID: \"804f2ad0-31c6-4bab-a698-d1d2224d9e5f\") " pod="openshift-monitoring/node-exporter-nxwvq" Jul 20 02:49:47.243762 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:47.243610 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/804f2ad0-31c6-4bab-a698-d1d2224d9e5f-node-exporter-tls\") pod \"node-exporter-nxwvq\" (UID: \"804f2ad0-31c6-4bab-a698-d1d2224d9e5f\") " pod="openshift-monitoring/node-exporter-nxwvq" Jul 20 02:49:47.344570 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:47.344535 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/804f2ad0-31c6-4bab-a698-d1d2224d9e5f-node-exporter-textfile\") pod \"node-exporter-nxwvq\" (UID: \"804f2ad0-31c6-4bab-a698-d1d2224d9e5f\") " pod="openshift-monitoring/node-exporter-nxwvq" Jul 20 02:49:47.344762 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:47.344582 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/804f2ad0-31c6-4bab-a698-d1d2224d9e5f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nxwvq\" (UID: \"804f2ad0-31c6-4bab-a698-d1d2224d9e5f\") " pod="openshift-monitoring/node-exporter-nxwvq" Jul 20 02:49:47.344762 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:47.344634 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/804f2ad0-31c6-4bab-a698-d1d2224d9e5f-node-exporter-tls\") pod \"node-exporter-nxwvq\" (UID: \"804f2ad0-31c6-4bab-a698-d1d2224d9e5f\") " pod="openshift-monitoring/node-exporter-nxwvq" Jul 20 02:49:47.344762 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:47.344696 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/804f2ad0-31c6-4bab-a698-d1d2224d9e5f-metrics-client-ca\") pod \"node-exporter-nxwvq\" (UID: \"804f2ad0-31c6-4bab-a698-d1d2224d9e5f\") " pod="openshift-monitoring/node-exporter-nxwvq" Jul 20 02:49:47.344762 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:47.344736 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/804f2ad0-31c6-4bab-a698-d1d2224d9e5f-root\") pod \"node-exporter-nxwvq\" (UID: \"804f2ad0-31c6-4bab-a698-d1d2224d9e5f\") " pod="openshift-monitoring/node-exporter-nxwvq" Jul 20 02:49:47.344762 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:47.344762 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/804f2ad0-31c6-4bab-a698-d1d2224d9e5f-sys\") pod \"node-exporter-nxwvq\" (UID: \"804f2ad0-31c6-4bab-a698-d1d2224d9e5f\") " pod="openshift-monitoring/node-exporter-nxwvq" Jul 20 02:49:47.345022 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:47.344791 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lns5j\" (UniqueName: \"kubernetes.io/projected/804f2ad0-31c6-4bab-a698-d1d2224d9e5f-kube-api-access-lns5j\") pod \"node-exporter-nxwvq\" (UID: \"804f2ad0-31c6-4bab-a698-d1d2224d9e5f\") " pod="openshift-monitoring/node-exporter-nxwvq" Jul 20 02:49:47.345022 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:47.344813 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/804f2ad0-31c6-4bab-a698-d1d2224d9e5f-node-exporter-wtmp\") pod \"node-exporter-nxwvq\" (UID: \"804f2ad0-31c6-4bab-a698-d1d2224d9e5f\") " pod="openshift-monitoring/node-exporter-nxwvq" Jul 20 02:49:47.345022 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:47.344869 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/804f2ad0-31c6-4bab-a698-d1d2224d9e5f-node-exporter-accelerators-collector-config\") pod \"node-exporter-nxwvq\" (UID: \"804f2ad0-31c6-4bab-a698-d1d2224d9e5f\") " pod="openshift-monitoring/node-exporter-nxwvq" Jul 20 02:49:47.345022 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:47.344897 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/804f2ad0-31c6-4bab-a698-d1d2224d9e5f-root\") pod \"node-exporter-nxwvq\" (UID: \"804f2ad0-31c6-4bab-a698-d1d2224d9e5f\") " pod="openshift-monitoring/node-exporter-nxwvq" Jul 20 02:49:47.345022 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:47.344943 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/804f2ad0-31c6-4bab-a698-d1d2224d9e5f-node-exporter-textfile\") pod \"node-exporter-nxwvq\" (UID: \"804f2ad0-31c6-4bab-a698-d1d2224d9e5f\") " pod="openshift-monitoring/node-exporter-nxwvq" Jul 20 02:49:47.345272 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:47.345122 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/804f2ad0-31c6-4bab-a698-d1d2224d9e5f-node-exporter-wtmp\") pod \"node-exporter-nxwvq\" (UID: \"804f2ad0-31c6-4bab-a698-d1d2224d9e5f\") " pod="openshift-monitoring/node-exporter-nxwvq" Jul 20 02:49:47.345272 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:47.345165 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/804f2ad0-31c6-4bab-a698-d1d2224d9e5f-sys\") pod \"node-exporter-nxwvq\" (UID: \"804f2ad0-31c6-4bab-a698-d1d2224d9e5f\") " pod="openshift-monitoring/node-exporter-nxwvq" Jul 20 02:49:47.346458 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:47.345993 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/804f2ad0-31c6-4bab-a698-d1d2224d9e5f-metrics-client-ca\") pod \"node-exporter-nxwvq\" (UID: \"804f2ad0-31c6-4bab-a698-d1d2224d9e5f\") " pod="openshift-monitoring/node-exporter-nxwvq" Jul 20 02:49:47.346458 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:47.346426 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/804f2ad0-31c6-4bab-a698-d1d2224d9e5f-node-exporter-accelerators-collector-config\") pod \"node-exporter-nxwvq\" (UID: \"804f2ad0-31c6-4bab-a698-d1d2224d9e5f\") " pod="openshift-monitoring/node-exporter-nxwvq" Jul 20 02:49:47.348151 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:47.348128 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/804f2ad0-31c6-4bab-a698-d1d2224d9e5f-node-exporter-tls\") pod \"node-exporter-nxwvq\" (UID: \"804f2ad0-31c6-4bab-a698-d1d2224d9e5f\") " pod="openshift-monitoring/node-exporter-nxwvq" Jul 20 02:49:47.348228 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:47.348210 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/804f2ad0-31c6-4bab-a698-d1d2224d9e5f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nxwvq\" (UID: \"804f2ad0-31c6-4bab-a698-d1d2224d9e5f\") " pod="openshift-monitoring/node-exporter-nxwvq" Jul 20 02:49:47.353953 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:47.353929 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lns5j\" (UniqueName: \"kubernetes.io/projected/804f2ad0-31c6-4bab-a698-d1d2224d9e5f-kube-api-access-lns5j\") pod \"node-exporter-nxwvq\" (UID: \"804f2ad0-31c6-4bab-a698-d1d2224d9e5f\") " pod="openshift-monitoring/node-exporter-nxwvq" Jul 20 02:49:47.538419 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:47.538307 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nxwvq" Jul 20 02:49:47.552872 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:49:47.552837 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod804f2ad0_31c6_4bab_a698_d1d2224d9e5f.slice/crio-09ec41b91018348c00838c2df6269de0969e3c15fa027a6de1235c6d63cca628 WatchSource:0}: Error finding container 09ec41b91018348c00838c2df6269de0969e3c15fa027a6de1235c6d63cca628: Status 404 returned error can't find the container with id 09ec41b91018348c00838c2df6269de0969e3c15fa027a6de1235c6d63cca628 Jul 20 02:49:48.347752 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:48.347704 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nxwvq" event={"ID":"804f2ad0-31c6-4bab-a698-d1d2224d9e5f","Type":"ContainerStarted","Data":"09ec41b91018348c00838c2df6269de0969e3c15fa027a6de1235c6d63cca628"} Jul 20 02:49:49.161534 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:49.161503 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f32d6cda-7b64-4aa8-90ee-21c1e3043ed3-metrics-tls\") pod \"dns-default-rkzf8\" (UID: \"f32d6cda-7b64-4aa8-90ee-21c1e3043ed3\") " pod="openshift-dns/dns-default-rkzf8" Jul 20 02:49:49.164069 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:49.164046 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f32d6cda-7b64-4aa8-90ee-21c1e3043ed3-metrics-tls\") pod \"dns-default-rkzf8\" (UID: \"f32d6cda-7b64-4aa8-90ee-21c1e3043ed3\") " pod="openshift-dns/dns-default-rkzf8" Jul 20 02:49:49.342294 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:49.342259 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nz45m\"" Jul 20 02:49:49.350290 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:49.350266 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rkzf8" Jul 20 02:49:49.352486 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:49.352461 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74bcf79799-8xc5s" event={"ID":"95a990cb-fd40-4d72-8a18-fc3ae2930904","Type":"ContainerStarted","Data":"a3ffc9198d0e71707803c2072e43eaec397a78e974ec6bcd29ed9acf50911856"} Jul 20 02:49:49.371430 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:49.371381 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-74bcf79799-8xc5s" podStartSLOduration=1.394422836 podStartE2EDuration="4.371366193s" totalImagesPullingTime="2.976943357s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:49:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:49:49.370475001 +0000 UTC m=+284.569724148" watchObservedRunningTime="2026-07-20 02:49:49.371366193 +0000 UTC m=+284.570615340" Jul 20 02:49:49.505064 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:49.504961 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rkzf8"] Jul 20 02:49:49.507786 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:49:49.507748 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf32d6cda_7b64_4aa8_90ee_21c1e3043ed3.slice/crio-d10276d017bdb1ed64457f65e32ce955512b90b20cbe1edb0eed8ebcf3e7f375 WatchSource:0}: Error finding container d10276d017bdb1ed64457f65e32ce955512b90b20cbe1edb0eed8ebcf3e7f375: Status 404 returned error can't find the container with id d10276d017bdb1ed64457f65e32ce955512b90b20cbe1edb0eed8ebcf3e7f375 Jul 20 02:49:50.198405 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:50.198369 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6f8ffbf86f-6jjqd"] Jul 20 02:49:50.202176 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:50.202154 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6f8ffbf86f-6jjqd" Jul 20 02:49:50.205377 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:50.205289 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 20 02:49:50.205513 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:50.205448 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 20 02:49:50.205584 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:50.205561 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-gwnhh\"" Jul 20 02:49:50.205700 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:50.205661 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 20 02:49:50.205787 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:50.205770 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 20 02:49:50.205882 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:50.205866 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-1quh0hdm7c45e\"" Jul 20 02:49:50.206053 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:50.206037 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 20 02:49:50.215666 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:50.215640 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6f8ffbf86f-6jjqd"] Jul 20 02:49:50.271221 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:50.271182 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/657d8ade-0d0c-4aac-8302-491148a27723-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6f8ffbf86f-6jjqd\" (UID: \"657d8ade-0d0c-4aac-8302-491148a27723\") " pod="openshift-monitoring/thanos-querier-6f8ffbf86f-6jjqd" Jul 20 02:49:50.271221 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:50.271231 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/657d8ade-0d0c-4aac-8302-491148a27723-metrics-client-ca\") pod \"thanos-querier-6f8ffbf86f-6jjqd\" (UID: \"657d8ade-0d0c-4aac-8302-491148a27723\") " pod="openshift-monitoring/thanos-querier-6f8ffbf86f-6jjqd" Jul 20 02:49:50.271486 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:50.271276 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/657d8ade-0d0c-4aac-8302-491148a27723-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6f8ffbf86f-6jjqd\" (UID: \"657d8ade-0d0c-4aac-8302-491148a27723\") " pod="openshift-monitoring/thanos-querier-6f8ffbf86f-6jjqd" Jul 20 02:49:50.271486 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:50.271333 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/657d8ade-0d0c-4aac-8302-491148a27723-secret-thanos-querier-tls\") pod \"thanos-querier-6f8ffbf86f-6jjqd\" (UID: \"657d8ade-0d0c-4aac-8302-491148a27723\") " pod="openshift-monitoring/thanos-querier-6f8ffbf86f-6jjqd" Jul 20 02:49:50.271486 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:50.271373 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf2zg\" (UniqueName: \"kubernetes.io/projected/657d8ade-0d0c-4aac-8302-491148a27723-kube-api-access-vf2zg\") pod \"thanos-querier-6f8ffbf86f-6jjqd\" (UID: \"657d8ade-0d0c-4aac-8302-491148a27723\") " pod="openshift-monitoring/thanos-querier-6f8ffbf86f-6jjqd" Jul 20 02:49:50.271486 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:50.271426 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/657d8ade-0d0c-4aac-8302-491148a27723-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6f8ffbf86f-6jjqd\" (UID: \"657d8ade-0d0c-4aac-8302-491148a27723\") " pod="openshift-monitoring/thanos-querier-6f8ffbf86f-6jjqd" Jul 20 02:49:50.271486 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:50.271475 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/657d8ade-0d0c-4aac-8302-491148a27723-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6f8ffbf86f-6jjqd\" (UID: \"657d8ade-0d0c-4aac-8302-491148a27723\") " pod="openshift-monitoring/thanos-querier-6f8ffbf86f-6jjqd" Jul 20 02:49:50.271695 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:50.271503 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/657d8ade-0d0c-4aac-8302-491148a27723-secret-grpc-tls\") pod \"thanos-querier-6f8ffbf86f-6jjqd\" (UID: \"657d8ade-0d0c-4aac-8302-491148a27723\") " pod="openshift-monitoring/thanos-querier-6f8ffbf86f-6jjqd" Jul 20 02:49:50.358167 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:50.358121 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rkzf8" event={"ID":"f32d6cda-7b64-4aa8-90ee-21c1e3043ed3","Type":"ContainerStarted","Data":"d10276d017bdb1ed64457f65e32ce955512b90b20cbe1edb0eed8ebcf3e7f375"} Jul 20 02:49:50.360238 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:50.360209 2558 generic.go:356] "Generic (PLEG): container finished" podID="804f2ad0-31c6-4bab-a698-d1d2224d9e5f" containerID="dd039266d4af0f9771334060b78595f48a7cd1050402197d487f36b8e0b460ed" exitCode=0 Jul 20 02:49:50.360491 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:50.360458 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nxwvq" event={"ID":"804f2ad0-31c6-4bab-a698-d1d2224d9e5f","Type":"ContainerDied","Data":"dd039266d4af0f9771334060b78595f48a7cd1050402197d487f36b8e0b460ed"} Jul 20 02:49:50.372666 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:50.372624 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/657d8ade-0d0c-4aac-8302-491148a27723-secret-grpc-tls\") pod \"thanos-querier-6f8ffbf86f-6jjqd\" (UID: \"657d8ade-0d0c-4aac-8302-491148a27723\") " pod="openshift-monitoring/thanos-querier-6f8ffbf86f-6jjqd" Jul 20 02:49:50.372781 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:50.372697 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/657d8ade-0d0c-4aac-8302-491148a27723-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6f8ffbf86f-6jjqd\" (UID: \"657d8ade-0d0c-4aac-8302-491148a27723\") " pod="openshift-monitoring/thanos-querier-6f8ffbf86f-6jjqd" Jul 20 02:49:50.372781 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:50.372729 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/657d8ade-0d0c-4aac-8302-491148a27723-metrics-client-ca\") pod \"thanos-querier-6f8ffbf86f-6jjqd\" (UID: \"657d8ade-0d0c-4aac-8302-491148a27723\") " pod="openshift-monitoring/thanos-querier-6f8ffbf86f-6jjqd" Jul 20 02:49:50.373651 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:50.372943 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/657d8ade-0d0c-4aac-8302-491148a27723-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6f8ffbf86f-6jjqd\" (UID: \"657d8ade-0d0c-4aac-8302-491148a27723\") " pod="openshift-monitoring/thanos-querier-6f8ffbf86f-6jjqd" Jul 20 02:49:50.373651 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:50.372988 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/657d8ade-0d0c-4aac-8302-491148a27723-secret-thanos-querier-tls\") pod \"thanos-querier-6f8ffbf86f-6jjqd\" (UID: \"657d8ade-0d0c-4aac-8302-491148a27723\") " pod="openshift-monitoring/thanos-querier-6f8ffbf86f-6jjqd" Jul 20 02:49:50.373651 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:50.373016 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf2zg\" (UniqueName: \"kubernetes.io/projected/657d8ade-0d0c-4aac-8302-491148a27723-kube-api-access-vf2zg\") pod \"thanos-querier-6f8ffbf86f-6jjqd\" (UID: \"657d8ade-0d0c-4aac-8302-491148a27723\") " pod="openshift-monitoring/thanos-querier-6f8ffbf86f-6jjqd" Jul 20 02:49:50.373651 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:50.373066 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/657d8ade-0d0c-4aac-8302-491148a27723-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6f8ffbf86f-6jjqd\" (UID: \"657d8ade-0d0c-4aac-8302-491148a27723\") " pod="openshift-monitoring/thanos-querier-6f8ffbf86f-6jjqd" Jul 20 02:49:50.373651 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:50.373133 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/657d8ade-0d0c-4aac-8302-491148a27723-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6f8ffbf86f-6jjqd\" (UID: \"657d8ade-0d0c-4aac-8302-491148a27723\") " pod="openshift-monitoring/thanos-querier-6f8ffbf86f-6jjqd" Jul 20 02:49:50.373651 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:50.373569 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/657d8ade-0d0c-4aac-8302-491148a27723-metrics-client-ca\") pod \"thanos-querier-6f8ffbf86f-6jjqd\" (UID: \"657d8ade-0d0c-4aac-8302-491148a27723\") " pod="openshift-monitoring/thanos-querier-6f8ffbf86f-6jjqd" Jul 20 02:49:50.377965 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:50.377916 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/657d8ade-0d0c-4aac-8302-491148a27723-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6f8ffbf86f-6jjqd\" (UID: \"657d8ade-0d0c-4aac-8302-491148a27723\") " pod="openshift-monitoring/thanos-querier-6f8ffbf86f-6jjqd" Jul 20 02:49:50.378444 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:50.378421 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/657d8ade-0d0c-4aac-8302-491148a27723-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6f8ffbf86f-6jjqd\" (UID: \"657d8ade-0d0c-4aac-8302-491148a27723\") " pod="openshift-monitoring/thanos-querier-6f8ffbf86f-6jjqd" Jul 20 02:49:50.378775 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:50.378756 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/657d8ade-0d0c-4aac-8302-491148a27723-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6f8ffbf86f-6jjqd\" (UID: \"657d8ade-0d0c-4aac-8302-491148a27723\") " pod="openshift-monitoring/thanos-querier-6f8ffbf86f-6jjqd" Jul 20 02:49:50.379587 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:50.379545 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/657d8ade-0d0c-4aac-8302-491148a27723-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6f8ffbf86f-6jjqd\" (UID: \"657d8ade-0d0c-4aac-8302-491148a27723\") " pod="openshift-monitoring/thanos-querier-6f8ffbf86f-6jjqd" Jul 20 02:49:50.379680 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:50.379662 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/657d8ade-0d0c-4aac-8302-491148a27723-secret-grpc-tls\") pod \"thanos-querier-6f8ffbf86f-6jjqd\" (UID: \"657d8ade-0d0c-4aac-8302-491148a27723\") " pod="openshift-monitoring/thanos-querier-6f8ffbf86f-6jjqd" Jul 20 02:49:50.381297 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:50.381273 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/657d8ade-0d0c-4aac-8302-491148a27723-secret-thanos-querier-tls\") pod \"thanos-querier-6f8ffbf86f-6jjqd\" (UID: \"657d8ade-0d0c-4aac-8302-491148a27723\") " pod="openshift-monitoring/thanos-querier-6f8ffbf86f-6jjqd" Jul 20 02:49:50.387329 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:50.387309 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf2zg\" (UniqueName: \"kubernetes.io/projected/657d8ade-0d0c-4aac-8302-491148a27723-kube-api-access-vf2zg\") pod \"thanos-querier-6f8ffbf86f-6jjqd\" (UID: \"657d8ade-0d0c-4aac-8302-491148a27723\") " pod="openshift-monitoring/thanos-querier-6f8ffbf86f-6jjqd" Jul 20 02:49:50.515709 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:50.515549 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6f8ffbf86f-6jjqd" Jul 20 02:49:51.129459 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:51.129434 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6f8ffbf86f-6jjqd"] Jul 20 02:49:51.132169 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:49:51.132144 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod657d8ade_0d0c_4aac_8302_491148a27723.slice/crio-27731ae85194b1e5dde10869833ac858138ad2bfea55822c24d70b83060fafe6 WatchSource:0}: Error finding container 27731ae85194b1e5dde10869833ac858138ad2bfea55822c24d70b83060fafe6: Status 404 returned error can't find the container with id 27731ae85194b1e5dde10869833ac858138ad2bfea55822c24d70b83060fafe6 Jul 20 02:49:51.365976 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:51.365888 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rkzf8" event={"ID":"f32d6cda-7b64-4aa8-90ee-21c1e3043ed3","Type":"ContainerStarted","Data":"9427c14f9b1011e1d1ad8d5b74b79ee52b83a2769bca5bfb1a1fe51481662f90"} Jul 20 02:49:51.365976 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:51.365945 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rkzf8" event={"ID":"f32d6cda-7b64-4aa8-90ee-21c1e3043ed3","Type":"ContainerStarted","Data":"b3df7b0b3ef1d0166e3d3f0ec8bd2a48174c95077dcc9ce65476d81f53e6e1f7"} Jul 20 02:49:51.365976 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:51.365974 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rkzf8" Jul 20 02:49:51.367154 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:51.367132 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f8ffbf86f-6jjqd" event={"ID":"657d8ade-0d0c-4aac-8302-491148a27723","Type":"ContainerStarted","Data":"27731ae85194b1e5dde10869833ac858138ad2bfea55822c24d70b83060fafe6"} Jul 20 02:49:51.369146 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:51.369124 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nxwvq" event={"ID":"804f2ad0-31c6-4bab-a698-d1d2224d9e5f","Type":"ContainerStarted","Data":"bde080031c63b67e5ba36e217953e0f581513623b370eb558b7bc44ec76fd150"} Jul 20 02:49:51.369223 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:51.369163 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nxwvq" event={"ID":"804f2ad0-31c6-4bab-a698-d1d2224d9e5f","Type":"ContainerStarted","Data":"6bc07967d721fd91cb574d984525c855b5e27ec706ca85acce13e4c785ac68b0"} Jul 20 02:49:51.383689 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:51.383633 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-rkzf8" podStartSLOduration=250.852656085 podStartE2EDuration="4m12.383613697s" totalImagesPullingTime="1.530957612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:45:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:49:51.38260796 +0000 UTC m=+286.581857108" watchObservedRunningTime="2026-07-20 02:49:51.383613697 +0000 UTC m=+286.582862856" Jul 20 02:49:51.399690 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:51.399627 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-nxwvq" podStartSLOduration=2.507180366 podStartE2EDuration="4.399611864s" totalImagesPullingTime="1.826472716s" totalInitContainerRuntime="65.958782ms" isStatefulPod=true podCreationTimestamp="2026-07-20 02:49:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:49:51.39856759 +0000 UTC m=+286.597816750" watchObservedRunningTime="2026-07-20 02:49:51.399611864 +0000 UTC m=+286.598861010" Jul 20 02:49:53.417440 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.417411 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 20 02:49:53.422316 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.422295 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.425062 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.425040 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 20 02:49:53.425062 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.425055 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 20 02:49:53.425239 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.425130 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 20 02:49:53.425239 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.425155 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 20 02:49:53.425239 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.425134 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-c5zkf\"" Jul 20 02:49:53.425383 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.425249 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 20 02:49:53.425460 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.425440 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 20 02:49:53.426525 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.426485 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 20 02:49:53.426525 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.426496 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 20 02:49:53.426685 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.426507 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 20 02:49:53.427849 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.427556 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 20 02:49:53.427849 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.427566 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 20 02:49:53.427849 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.427620 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fok20bnkg872k\"" Jul 20 02:49:53.427849 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.427560 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 20 02:49:53.428417 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.428177 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 20 02:49:53.428417 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.428292 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 20 02:49:53.429398 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.429377 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 20 02:49:53.437366 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.437344 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 20 02:49:53.499993 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.499965 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/01b09630-9c37-419f-8ad8-e0b97aa6df82-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.500128 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.500007 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/01b09630-9c37-419f-8ad8-e0b97aa6df82-config-out\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.500128 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.500026 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/01b09630-9c37-419f-8ad8-e0b97aa6df82-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.500128 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.500091 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/01b09630-9c37-419f-8ad8-e0b97aa6df82-config\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.500245 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.500148 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/01b09630-9c37-419f-8ad8-e0b97aa6df82-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.500245 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.500172 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbz2s\" (UniqueName: \"kubernetes.io/projected/01b09630-9c37-419f-8ad8-e0b97aa6df82-kube-api-access-kbz2s\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.500245 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.500193 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/01b09630-9c37-419f-8ad8-e0b97aa6df82-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.500245 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.500225 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01b09630-9c37-419f-8ad8-e0b97aa6df82-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.500365 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.500285 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/01b09630-9c37-419f-8ad8-e0b97aa6df82-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.500365 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.500312 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/01b09630-9c37-419f-8ad8-e0b97aa6df82-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.500422 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.500369 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/01b09630-9c37-419f-8ad8-e0b97aa6df82-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.500422 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.500386 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01b09630-9c37-419f-8ad8-e0b97aa6df82-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.500422 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.500401 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01b09630-9c37-419f-8ad8-e0b97aa6df82-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.500422 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.500419 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01b09630-9c37-419f-8ad8-e0b97aa6df82-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.500533 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.500439 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/01b09630-9c37-419f-8ad8-e0b97aa6df82-web-config\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.500533 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.500459 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/01b09630-9c37-419f-8ad8-e0b97aa6df82-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.500533 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.500474 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/01b09630-9c37-419f-8ad8-e0b97aa6df82-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.500533 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.500517 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/01b09630-9c37-419f-8ad8-e0b97aa6df82-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.500649 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.500534 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/01b09630-9c37-419f-8ad8-e0b97aa6df82-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.500649 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.500555 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/01b09630-9c37-419f-8ad8-e0b97aa6df82-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.601860 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.601828 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/01b09630-9c37-419f-8ad8-e0b97aa6df82-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.602004 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.601888 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/01b09630-9c37-419f-8ad8-e0b97aa6df82-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.602004 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.601913 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01b09630-9c37-419f-8ad8-e0b97aa6df82-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.602004 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.601936 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01b09630-9c37-419f-8ad8-e0b97aa6df82-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.602004 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.601964 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01b09630-9c37-419f-8ad8-e0b97aa6df82-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.602004 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.601988 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/01b09630-9c37-419f-8ad8-e0b97aa6df82-web-config\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.602263 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.602015 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/01b09630-9c37-419f-8ad8-e0b97aa6df82-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.602263 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.602234 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/01b09630-9c37-419f-8ad8-e0b97aa6df82-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.602372 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.602299 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/01b09630-9c37-419f-8ad8-e0b97aa6df82-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.602372 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.602332 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/01b09630-9c37-419f-8ad8-e0b97aa6df82-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.602482 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.602376 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/01b09630-9c37-419f-8ad8-e0b97aa6df82-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.602482 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.602409 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/01b09630-9c37-419f-8ad8-e0b97aa6df82-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.602482 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.602454 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/01b09630-9c37-419f-8ad8-e0b97aa6df82-config-out\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.602632 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.602480 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/01b09630-9c37-419f-8ad8-e0b97aa6df82-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.602632 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.602514 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/01b09630-9c37-419f-8ad8-e0b97aa6df82-config\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.602632 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.602549 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/01b09630-9c37-419f-8ad8-e0b97aa6df82-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.602632 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.602572 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbz2s\" (UniqueName: \"kubernetes.io/projected/01b09630-9c37-419f-8ad8-e0b97aa6df82-kube-api-access-kbz2s\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.602632 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.602601 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/01b09630-9c37-419f-8ad8-e0b97aa6df82-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.602632 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.602623 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01b09630-9c37-419f-8ad8-e0b97aa6df82-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.602920 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.602667 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/01b09630-9c37-419f-8ad8-e0b97aa6df82-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.602920 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.602759 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01b09630-9c37-419f-8ad8-e0b97aa6df82-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.602920 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.602793 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/01b09630-9c37-419f-8ad8-e0b97aa6df82-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.602920 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.602823 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01b09630-9c37-419f-8ad8-e0b97aa6df82-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.603089 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.602946 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01b09630-9c37-419f-8ad8-e0b97aa6df82-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.604777 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.604753 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/01b09630-9c37-419f-8ad8-e0b97aa6df82-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.605634 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.605575 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/01b09630-9c37-419f-8ad8-e0b97aa6df82-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.605820 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.605795 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01b09630-9c37-419f-8ad8-e0b97aa6df82-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.606175 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.606143 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/01b09630-9c37-419f-8ad8-e0b97aa6df82-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.606327 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.606261 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/01b09630-9c37-419f-8ad8-e0b97aa6df82-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.606391 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.606332 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/01b09630-9c37-419f-8ad8-e0b97aa6df82-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.606671 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.606648 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/01b09630-9c37-419f-8ad8-e0b97aa6df82-web-config\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.607206 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.606916 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/01b09630-9c37-419f-8ad8-e0b97aa6df82-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.607206 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.607163 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/01b09630-9c37-419f-8ad8-e0b97aa6df82-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.607644 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.607622 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/01b09630-9c37-419f-8ad8-e0b97aa6df82-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.607762 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.607743 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/01b09630-9c37-419f-8ad8-e0b97aa6df82-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.608166 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.608140 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/01b09630-9c37-419f-8ad8-e0b97aa6df82-config\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.608249 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.608233 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/01b09630-9c37-419f-8ad8-e0b97aa6df82-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.608590 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.608573 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/01b09630-9c37-419f-8ad8-e0b97aa6df82-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.608670 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.608652 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/01b09630-9c37-419f-8ad8-e0b97aa6df82-config-out\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.610658 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.610637 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbz2s\" (UniqueName: \"kubernetes.io/projected/01b09630-9c37-419f-8ad8-e0b97aa6df82-kube-api-access-kbz2s\") pod \"prometheus-k8s-0\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.732657 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.732619 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:49:53.867486 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:53.867450 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 20 02:49:53.870776 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:49:53.870752 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01b09630_9c37_419f_8ad8_e0b97aa6df82.slice/crio-77e070f963eab3b96bd8125b995ec469331c23f5d6e1225d7714ab712d75579f WatchSource:0}: Error finding container 77e070f963eab3b96bd8125b995ec469331c23f5d6e1225d7714ab712d75579f: Status 404 returned error can't find the container with id 77e070f963eab3b96bd8125b995ec469331c23f5d6e1225d7714ab712d75579f Jul 20 02:49:54.382980 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:54.382940 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f8ffbf86f-6jjqd" event={"ID":"657d8ade-0d0c-4aac-8302-491148a27723","Type":"ContainerStarted","Data":"ab3b104d45085c4238dd2563aee682d25ede7d1b0e19c5c6c77fb37f1b354597"} Jul 20 02:49:54.383188 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:54.383039 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f8ffbf86f-6jjqd" event={"ID":"657d8ade-0d0c-4aac-8302-491148a27723","Type":"ContainerStarted","Data":"fb298ed984ee7beddbadf2a346f6e9577d63e753967cc54b9303b999d8115e31"} Jul 20 02:49:54.383188 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:54.383081 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f8ffbf86f-6jjqd" event={"ID":"657d8ade-0d0c-4aac-8302-491148a27723","Type":"ContainerStarted","Data":"b5105ff1e707a34d19701e97933371bdefe6d661f99abfe4e8f9aac7bcc4b512"} Jul 20 02:49:54.384317 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:54.384287 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01b09630-9c37-419f-8ad8-e0b97aa6df82","Type":"ContainerStarted","Data":"77e070f963eab3b96bd8125b995ec469331c23f5d6e1225d7714ab712d75579f"} Jul 20 02:49:55.390037 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:55.390004 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f8ffbf86f-6jjqd" event={"ID":"657d8ade-0d0c-4aac-8302-491148a27723","Type":"ContainerStarted","Data":"3b79c2e729e7b21f9868ef3456fdebdff8dfa4b88555aebf212f8d4b3cc8ad4e"} Jul 20 02:49:55.390469 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:55.390068 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f8ffbf86f-6jjqd" event={"ID":"657d8ade-0d0c-4aac-8302-491148a27723","Type":"ContainerStarted","Data":"69a3185d28a1fd6577a373c9f12546dd64f7646e5e5eb28648eec959c4a326c7"} Jul 20 02:49:55.390469 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:55.390110 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f8ffbf86f-6jjqd" event={"ID":"657d8ade-0d0c-4aac-8302-491148a27723","Type":"ContainerStarted","Data":"87e04864885d747f7d4bb680776e47161ca6b0dc84a5587e749c25f757145ad3"} Jul 20 02:49:55.390469 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:55.390259 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6f8ffbf86f-6jjqd" Jul 20 02:49:55.391555 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:55.391533 2558 generic.go:356] "Generic (PLEG): container finished" podID="01b09630-9c37-419f-8ad8-e0b97aa6df82" containerID="3394177568c88e299ac9ea0feab4432f10d6b7a2ebc05bed2a457ab6993f3b8e" exitCode=0 Jul 20 02:49:55.391639 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:55.391576 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01b09630-9c37-419f-8ad8-e0b97aa6df82","Type":"ContainerDied","Data":"3394177568c88e299ac9ea0feab4432f10d6b7a2ebc05bed2a457ab6993f3b8e"} Jul 20 02:49:55.412480 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:55.412430 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6f8ffbf86f-6jjqd" podStartSLOduration=2.284491159 podStartE2EDuration="5.412411367s" totalImagesPullingTime="3.127920208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:49:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:49:55.410316521 +0000 UTC m=+290.609565680" watchObservedRunningTime="2026-07-20 02:49:55.412411367 +0000 UTC m=+290.611660558" Jul 20 02:49:55.703935 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:55.703853 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74bcf79799-8xc5s" Jul 20 02:49:55.703935 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:55.703904 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-74bcf79799-8xc5s" Jul 20 02:49:55.705286 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:55.705261 2558 patch_prober.go:28] interesting pod/console-74bcf79799-8xc5s container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.16:8443/health\": dial tcp 10.133.0.16:8443: connect: connection refused" start-of-body= Jul 20 02:49:55.705402 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:55.705310 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-74bcf79799-8xc5s" podUID="95a990cb-fd40-4d72-8a18-fc3ae2930904" containerName="console" probeResult="failure" output="Get \"https://10.133.0.16:8443/health\": dial tcp 10.133.0.16:8443: connect: connection refused" Jul 20 02:49:55.746842 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:55.746812 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74bcf79799-8xc5s"] Jul 20 02:49:56.023503 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:56.023411 2558 patch_prober.go:28] interesting pod/image-registry-84f4d8756b-ltpj5 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 20 02:49:56.023503 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:56.023465 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-84f4d8756b-ltpj5" podUID="537fa96e-a0c2-48ff-877b-5aca6c1e40f7" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 20 02:49:58.406892 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:58.406865 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01b09630-9c37-419f-8ad8-e0b97aa6df82","Type":"ContainerStarted","Data":"454280bab7fcdcad8e564a92bac238ffdc6c89743cbe3660c537e50f70d4eea9"} Jul 20 02:49:59.414224 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:59.414184 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01b09630-9c37-419f-8ad8-e0b97aa6df82","Type":"ContainerStarted","Data":"cdc12e352a5377a18c6f5cdf4ccd7d3c07d7c80b8f1f409f7924d88106b9db86"} Jul 20 02:49:59.414602 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:59.414273 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01b09630-9c37-419f-8ad8-e0b97aa6df82","Type":"ContainerStarted","Data":"d3c3f2319689f0f448374b987914f00d7e950b93bb253eadc00efd75f1269bfa"} Jul 20 02:49:59.414602 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:59.414323 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01b09630-9c37-419f-8ad8-e0b97aa6df82","Type":"ContainerStarted","Data":"0ff20bc1a0992a582fa8ad26c0f2ef2c54a35f27c097825bba7d5c4c55e370c8"} Jul 20 02:49:59.414602 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:59.414348 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01b09630-9c37-419f-8ad8-e0b97aa6df82","Type":"ContainerStarted","Data":"259ed9af1282abb0563757e4b43308b21f71b13e55f3e5c80f3ad1a5e2154d6f"} Jul 20 02:49:59.414602 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:59.414371 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01b09630-9c37-419f-8ad8-e0b97aa6df82","Type":"ContainerStarted","Data":"931c8bb914b43643f2cbd7630c5c3ffaae8672cf2770c9a63615141ba5db773f"} Jul 20 02:49:59.445179 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:49:59.445131 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.339216991 podStartE2EDuration="6.445114397s" totalImagesPullingTime="4.089527331s" totalInitContainerRuntime="16.370075ms" isStatefulPod=false podCreationTimestamp="2026-07-20 02:49:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:49:59.44416042 +0000 UTC m=+294.643409568" watchObservedRunningTime="2026-07-20 02:49:59.445114397 +0000 UTC m=+294.644363554" Jul 20 02:50:00.419014 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:50:00.418977 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-9x7hv" podUID="d70f5103-7dc9-4279-9781-38e395da435b" Jul 20 02:50:01.048702 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:01.048638 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-84f4d8756b-ltpj5" podUID="537fa96e-a0c2-48ff-877b-5aca6c1e40f7" containerName="registry" containerID="cri-o://058a8584f827a75ad94788f7e523fe2faeb9b39a1b8aeb8fc01292a7b36692d1" gracePeriod=30 Jul 20 02:50:01.279803 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:01.279779 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84f4d8756b-ltpj5" Jul 20 02:50:01.376226 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:01.376203 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rkzf8" Jul 20 02:50:01.377697 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:01.377677 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-ca-trust-extracted\") pod \"537fa96e-a0c2-48ff-877b-5aca6c1e40f7\" (UID: \"537fa96e-a0c2-48ff-877b-5aca6c1e40f7\") " Jul 20 02:50:01.377788 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:01.377706 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-registry-tls\") pod \"537fa96e-a0c2-48ff-877b-5aca6c1e40f7\" (UID: \"537fa96e-a0c2-48ff-877b-5aca6c1e40f7\") " Jul 20 02:50:01.377788 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:01.377736 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-installation-pull-secrets\") pod \"537fa96e-a0c2-48ff-877b-5aca6c1e40f7\" (UID: \"537fa96e-a0c2-48ff-877b-5aca6c1e40f7\") " Jul 20 02:50:01.377788 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:01.377771 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-trusted-ca\") pod \"537fa96e-a0c2-48ff-877b-5aca6c1e40f7\" (UID: \"537fa96e-a0c2-48ff-877b-5aca6c1e40f7\") " Jul 20 02:50:01.377947 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:01.377825 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-registry-certificates\") pod \"537fa96e-a0c2-48ff-877b-5aca6c1e40f7\" (UID: \"537fa96e-a0c2-48ff-877b-5aca6c1e40f7\") " Jul 20 02:50:01.377947 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:01.377849 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-image-registry-private-configuration\") pod \"537fa96e-a0c2-48ff-877b-5aca6c1e40f7\" (UID: \"537fa96e-a0c2-48ff-877b-5aca6c1e40f7\") " Jul 20 02:50:01.377947 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:01.377877 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-bound-sa-token\") pod \"537fa96e-a0c2-48ff-877b-5aca6c1e40f7\" (UID: \"537fa96e-a0c2-48ff-877b-5aca6c1e40f7\") " Jul 20 02:50:01.377947 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:01.377936 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9zvk\" (UniqueName: \"kubernetes.io/projected/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-kube-api-access-d9zvk\") pod \"537fa96e-a0c2-48ff-877b-5aca6c1e40f7\" (UID: \"537fa96e-a0c2-48ff-877b-5aca6c1e40f7\") " Jul 20 02:50:01.378362 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:01.378299 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "537fa96e-a0c2-48ff-877b-5aca6c1e40f7" (UID: "537fa96e-a0c2-48ff-877b-5aca6c1e40f7"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 02:50:01.378362 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:01.378295 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "537fa96e-a0c2-48ff-877b-5aca6c1e40f7" (UID: "537fa96e-a0c2-48ff-877b-5aca6c1e40f7"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 02:50:01.380482 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:01.380453 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "537fa96e-a0c2-48ff-877b-5aca6c1e40f7" (UID: "537fa96e-a0c2-48ff-877b-5aca6c1e40f7"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 02:50:01.380482 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:01.380473 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "537fa96e-a0c2-48ff-877b-5aca6c1e40f7" (UID: "537fa96e-a0c2-48ff-877b-5aca6c1e40f7"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 02:50:01.380625 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:01.380463 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-kube-api-access-d9zvk" (OuterVolumeSpecName: "kube-api-access-d9zvk") pod "537fa96e-a0c2-48ff-877b-5aca6c1e40f7" (UID: "537fa96e-a0c2-48ff-877b-5aca6c1e40f7"). InnerVolumeSpecName "kube-api-access-d9zvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 02:50:01.380625 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:01.380489 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "537fa96e-a0c2-48ff-877b-5aca6c1e40f7" (UID: "537fa96e-a0c2-48ff-877b-5aca6c1e40f7"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 02:50:01.380625 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:01.380616 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "537fa96e-a0c2-48ff-877b-5aca6c1e40f7" (UID: "537fa96e-a0c2-48ff-877b-5aca6c1e40f7"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 02:50:01.386537 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:01.386510 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "537fa96e-a0c2-48ff-877b-5aca6c1e40f7" (UID: "537fa96e-a0c2-48ff-877b-5aca6c1e40f7"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 02:50:01.402256 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:01.402227 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6f8ffbf86f-6jjqd" Jul 20 02:50:01.422435 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:01.422400 2558 generic.go:356] "Generic (PLEG): container finished" podID="537fa96e-a0c2-48ff-877b-5aca6c1e40f7" containerID="058a8584f827a75ad94788f7e523fe2faeb9b39a1b8aeb8fc01292a7b36692d1" exitCode=0 Jul 20 02:50:01.422972 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:01.422770 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84f4d8756b-ltpj5" Jul 20 02:50:01.428355 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:01.428329 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84f4d8756b-ltpj5" event={"ID":"537fa96e-a0c2-48ff-877b-5aca6c1e40f7","Type":"ContainerDied","Data":"058a8584f827a75ad94788f7e523fe2faeb9b39a1b8aeb8fc01292a7b36692d1"} Jul 20 02:50:01.428475 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:01.428379 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84f4d8756b-ltpj5" event={"ID":"537fa96e-a0c2-48ff-877b-5aca6c1e40f7","Type":"ContainerDied","Data":"cb94d45149af0e2e69367428f1379382162d7db55f822e9904c4ca8fd3d17ac8"} Jul 20 02:50:01.428475 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:01.428407 2558 scope.go:117] "RemoveContainer" containerID="058a8584f827a75ad94788f7e523fe2faeb9b39a1b8aeb8fc01292a7b36692d1" Jul 20 02:50:01.439674 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:01.439659 2558 scope.go:117] "RemoveContainer" containerID="058a8584f827a75ad94788f7e523fe2faeb9b39a1b8aeb8fc01292a7b36692d1" Jul 20 02:50:01.439950 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:50:01.439924 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"058a8584f827a75ad94788f7e523fe2faeb9b39a1b8aeb8fc01292a7b36692d1\": container with ID starting with 058a8584f827a75ad94788f7e523fe2faeb9b39a1b8aeb8fc01292a7b36692d1 not found: ID does not exist" containerID="058a8584f827a75ad94788f7e523fe2faeb9b39a1b8aeb8fc01292a7b36692d1" Jul 20 02:50:01.440000 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:01.439958 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"058a8584f827a75ad94788f7e523fe2faeb9b39a1b8aeb8fc01292a7b36692d1"} err="failed to get container status \"058a8584f827a75ad94788f7e523fe2faeb9b39a1b8aeb8fc01292a7b36692d1\": rpc error: code = NotFound desc = could not find container \"058a8584f827a75ad94788f7e523fe2faeb9b39a1b8aeb8fc01292a7b36692d1\": container with ID starting with 058a8584f827a75ad94788f7e523fe2faeb9b39a1b8aeb8fc01292a7b36692d1 not found: ID does not exist" Jul 20 02:50:01.451824 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:01.451792 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-84f4d8756b-ltpj5"] Jul 20 02:50:01.454522 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:01.454499 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-84f4d8756b-ltpj5"] Jul 20 02:50:01.479401 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:01.479376 2558 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-trusted-ca\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 02:50:01.479490 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:01.479408 2558 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-registry-certificates\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 02:50:01.479490 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:01.479426 2558 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-image-registry-private-configuration\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 02:50:01.479490 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:01.479441 2558 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-bound-sa-token\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 02:50:01.479490 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:01.479457 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9zvk\" (UniqueName: \"kubernetes.io/projected/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-kube-api-access-d9zvk\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 02:50:01.479490 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:01.479472 2558 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-ca-trust-extracted\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 02:50:01.479490 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:01.479487 2558 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-registry-tls\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 02:50:01.479676 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:01.479502 2558 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/537fa96e-a0c2-48ff-877b-5aca6c1e40f7-installation-pull-secrets\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 02:50:03.423416 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:03.423378 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="537fa96e-a0c2-48ff-877b-5aca6c1e40f7" path="/var/lib/kubelet/pods/537fa96e-a0c2-48ff-877b-5aca6c1e40f7/volumes" Jul 20 02:50:03.732835 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:03.732756 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:50:05.271517 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:05.271486 2558 kubelet.go:1648] "Image garbage collection succeeded" Jul 20 02:50:14.419060 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:14.419024 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9x7hv" Jul 20 02:50:21.421518 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:21.421461 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-74bcf79799-8xc5s" podUID="95a990cb-fd40-4d72-8a18-fc3ae2930904" containerName="console" containerID="cri-o://a3ffc9198d0e71707803c2072e43eaec397a78e974ec6bcd29ed9acf50911856" gracePeriod=15 Jul 20 02:50:21.678223 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:21.678172 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74bcf79799-8xc5s" Jul 20 02:50:21.849417 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:21.849375 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/95a990cb-fd40-4d72-8a18-fc3ae2930904-console-oauth-config\") pod \"95a990cb-fd40-4d72-8a18-fc3ae2930904\" (UID: \"95a990cb-fd40-4d72-8a18-fc3ae2930904\") " Jul 20 02:50:21.849591 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:21.849426 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/95a990cb-fd40-4d72-8a18-fc3ae2930904-oauth-serving-cert\") pod \"95a990cb-fd40-4d72-8a18-fc3ae2930904\" (UID: \"95a990cb-fd40-4d72-8a18-fc3ae2930904\") " Jul 20 02:50:21.849591 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:21.849475 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/95a990cb-fd40-4d72-8a18-fc3ae2930904-console-config\") pod \"95a990cb-fd40-4d72-8a18-fc3ae2930904\" (UID: \"95a990cb-fd40-4d72-8a18-fc3ae2930904\") " Jul 20 02:50:21.849591 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:21.849535 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/95a990cb-fd40-4d72-8a18-fc3ae2930904-service-ca\") pod \"95a990cb-fd40-4d72-8a18-fc3ae2930904\" (UID: \"95a990cb-fd40-4d72-8a18-fc3ae2930904\") " Jul 20 02:50:21.849591 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:21.849576 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zzcj\" (UniqueName: \"kubernetes.io/projected/95a990cb-fd40-4d72-8a18-fc3ae2930904-kube-api-access-5zzcj\") pod \"95a990cb-fd40-4d72-8a18-fc3ae2930904\" (UID: \"95a990cb-fd40-4d72-8a18-fc3ae2930904\") " Jul 20 02:50:21.849809 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:21.849607 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/95a990cb-fd40-4d72-8a18-fc3ae2930904-console-serving-cert\") pod \"95a990cb-fd40-4d72-8a18-fc3ae2930904\" (UID: \"95a990cb-fd40-4d72-8a18-fc3ae2930904\") " Jul 20 02:50:21.849953 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:21.849923 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95a990cb-fd40-4d72-8a18-fc3ae2930904-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "95a990cb-fd40-4d72-8a18-fc3ae2930904" (UID: "95a990cb-fd40-4d72-8a18-fc3ae2930904"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 02:50:21.850019 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:21.849935 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95a990cb-fd40-4d72-8a18-fc3ae2930904-console-config" (OuterVolumeSpecName: "console-config") pod "95a990cb-fd40-4d72-8a18-fc3ae2930904" (UID: "95a990cb-fd40-4d72-8a18-fc3ae2930904"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 02:50:21.850061 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:21.850039 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95a990cb-fd40-4d72-8a18-fc3ae2930904-service-ca" (OuterVolumeSpecName: "service-ca") pod "95a990cb-fd40-4d72-8a18-fc3ae2930904" (UID: "95a990cb-fd40-4d72-8a18-fc3ae2930904"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 02:50:21.851824 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:21.851790 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95a990cb-fd40-4d72-8a18-fc3ae2930904-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "95a990cb-fd40-4d72-8a18-fc3ae2930904" (UID: "95a990cb-fd40-4d72-8a18-fc3ae2930904"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 02:50:21.851824 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:21.851802 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95a990cb-fd40-4d72-8a18-fc3ae2930904-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "95a990cb-fd40-4d72-8a18-fc3ae2930904" (UID: "95a990cb-fd40-4d72-8a18-fc3ae2930904"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 02:50:21.851996 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:21.851864 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95a990cb-fd40-4d72-8a18-fc3ae2930904-kube-api-access-5zzcj" (OuterVolumeSpecName: "kube-api-access-5zzcj") pod "95a990cb-fd40-4d72-8a18-fc3ae2930904" (UID: "95a990cb-fd40-4d72-8a18-fc3ae2930904"). InnerVolumeSpecName "kube-api-access-5zzcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 02:50:21.951205 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:21.951123 2558 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/95a990cb-fd40-4d72-8a18-fc3ae2930904-service-ca\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 02:50:21.951205 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:21.951150 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zzcj\" (UniqueName: \"kubernetes.io/projected/95a990cb-fd40-4d72-8a18-fc3ae2930904-kube-api-access-5zzcj\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 02:50:21.951205 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:21.951160 2558 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/95a990cb-fd40-4d72-8a18-fc3ae2930904-console-serving-cert\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 02:50:21.951205 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:21.951170 2558 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/95a990cb-fd40-4d72-8a18-fc3ae2930904-console-oauth-config\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 02:50:21.951205 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:21.951178 2558 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/95a990cb-fd40-4d72-8a18-fc3ae2930904-oauth-serving-cert\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 02:50:21.951205 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:21.951186 2558 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/95a990cb-fd40-4d72-8a18-fc3ae2930904-console-config\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 02:50:22.501695 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:22.501659 2558 generic.go:356] "Generic (PLEG): container finished" podID="95a990cb-fd40-4d72-8a18-fc3ae2930904" containerID="a3ffc9198d0e71707803c2072e43eaec397a78e974ec6bcd29ed9acf50911856" exitCode=0 Jul 20 02:50:22.501695 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:22.501699 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74bcf79799-8xc5s" Jul 20 02:50:22.502185 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:22.501747 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74bcf79799-8xc5s" event={"ID":"95a990cb-fd40-4d72-8a18-fc3ae2930904","Type":"ContainerDied","Data":"a3ffc9198d0e71707803c2072e43eaec397a78e974ec6bcd29ed9acf50911856"} Jul 20 02:50:22.502185 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:22.501797 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74bcf79799-8xc5s" event={"ID":"95a990cb-fd40-4d72-8a18-fc3ae2930904","Type":"ContainerDied","Data":"ed4a97eaf6003d105c49e353c8551d244e9aeda855fc2433bbe61f939ec3372b"} Jul 20 02:50:22.502185 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:22.501818 2558 scope.go:117] "RemoveContainer" containerID="a3ffc9198d0e71707803c2072e43eaec397a78e974ec6bcd29ed9acf50911856" Jul 20 02:50:22.510256 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:22.510234 2558 scope.go:117] "RemoveContainer" containerID="a3ffc9198d0e71707803c2072e43eaec397a78e974ec6bcd29ed9acf50911856" Jul 20 02:50:22.510490 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:50:22.510471 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3ffc9198d0e71707803c2072e43eaec397a78e974ec6bcd29ed9acf50911856\": container with ID starting with a3ffc9198d0e71707803c2072e43eaec397a78e974ec6bcd29ed9acf50911856 not found: ID does not exist" containerID="a3ffc9198d0e71707803c2072e43eaec397a78e974ec6bcd29ed9acf50911856" Jul 20 02:50:22.510535 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:22.510497 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3ffc9198d0e71707803c2072e43eaec397a78e974ec6bcd29ed9acf50911856"} err="failed to get container status \"a3ffc9198d0e71707803c2072e43eaec397a78e974ec6bcd29ed9acf50911856\": rpc error: code = NotFound desc = could not find container \"a3ffc9198d0e71707803c2072e43eaec397a78e974ec6bcd29ed9acf50911856\": container with ID starting with a3ffc9198d0e71707803c2072e43eaec397a78e974ec6bcd29ed9acf50911856 not found: ID does not exist" Jul 20 02:50:22.521444 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:22.521423 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74bcf79799-8xc5s"] Jul 20 02:50:22.525530 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:22.525505 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-74bcf79799-8xc5s"] Jul 20 02:50:23.422728 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:23.422693 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95a990cb-fd40-4d72-8a18-fc3ae2930904" path="/var/lib/kubelet/pods/95a990cb-fd40-4d72-8a18-fc3ae2930904/volumes" Jul 20 02:50:25.514438 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:25.514405 2558 generic.go:356] "Generic (PLEG): container finished" podID="463f86e5-8599-4846-875b-82cdad3c331e" containerID="2901dab0f894e4b936bcfbb85ba7d5d9031eb531a567265a31e57529c577129c" exitCode=0 Jul 20 02:50:25.514823 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:25.514460 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-rx856" event={"ID":"463f86e5-8599-4846-875b-82cdad3c331e","Type":"ContainerDied","Data":"2901dab0f894e4b936bcfbb85ba7d5d9031eb531a567265a31e57529c577129c"} Jul 20 02:50:25.514823 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:25.514741 2558 scope.go:117] "RemoveContainer" containerID="2901dab0f894e4b936bcfbb85ba7d5d9031eb531a567265a31e57529c577129c" Jul 20 02:50:26.521008 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:26.520974 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-rx856" event={"ID":"463f86e5-8599-4846-875b-82cdad3c331e","Type":"ContainerStarted","Data":"3ed22b31b476d1e5f14c93534ebfeb11c3e59b9f82aa2879857464069a00531c"} Jul 20 02:50:53.733565 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:53.733534 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:50:53.752431 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:53.752406 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:50:54.635370 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:50:54.635345 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:11.817272 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:11.817238 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 20 02:51:11.817661 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:11.817548 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="01b09630-9c37-419f-8ad8-e0b97aa6df82" containerName="prometheus" containerID="cri-o://454280bab7fcdcad8e564a92bac238ffdc6c89743cbe3660c537e50f70d4eea9" gracePeriod=600 Jul 20 02:51:11.817661 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:11.817569 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="01b09630-9c37-419f-8ad8-e0b97aa6df82" containerName="kube-rbac-proxy" containerID="cri-o://d3c3f2319689f0f448374b987914f00d7e950b93bb253eadc00efd75f1269bfa" gracePeriod=600 Jul 20 02:51:11.817661 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:11.817586 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="01b09630-9c37-419f-8ad8-e0b97aa6df82" containerName="thanos-sidecar" containerID="cri-o://259ed9af1282abb0563757e4b43308b21f71b13e55f3e5c80f3ad1a5e2154d6f" gracePeriod=600 Jul 20 02:51:11.817661 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:11.817609 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="01b09630-9c37-419f-8ad8-e0b97aa6df82" containerName="kube-rbac-proxy-web" containerID="cri-o://0ff20bc1a0992a582fa8ad26c0f2ef2c54a35f27c097825bba7d5c4c55e370c8" gracePeriod=600 Jul 20 02:51:11.817661 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:11.817629 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="01b09630-9c37-419f-8ad8-e0b97aa6df82" containerName="kube-rbac-proxy-thanos" containerID="cri-o://cdc12e352a5377a18c6f5cdf4ccd7d3c07d7c80b8f1f409f7924d88106b9db86" gracePeriod=600 Jul 20 02:51:11.817661 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:11.817633 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="01b09630-9c37-419f-8ad8-e0b97aa6df82" containerName="config-reloader" containerID="cri-o://931c8bb914b43643f2cbd7630c5c3ffaae8672cf2770c9a63615141ba5db773f" gracePeriod=600 Jul 20 02:51:12.068834 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.068774 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.161248 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.161215 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01b09630-9c37-419f-8ad8-e0b97aa6df82-prometheus-trusted-ca-bundle\") pod \"01b09630-9c37-419f-8ad8-e0b97aa6df82\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " Jul 20 02:51:12.161248 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.161249 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/01b09630-9c37-419f-8ad8-e0b97aa6df82-secret-metrics-client-certs\") pod \"01b09630-9c37-419f-8ad8-e0b97aa6df82\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " Jul 20 02:51:12.161488 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.161268 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/01b09630-9c37-419f-8ad8-e0b97aa6df82-tls-assets\") pod \"01b09630-9c37-419f-8ad8-e0b97aa6df82\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " Jul 20 02:51:12.161488 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.161287 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/01b09630-9c37-419f-8ad8-e0b97aa6df82-prometheus-k8s-db\") pod \"01b09630-9c37-419f-8ad8-e0b97aa6df82\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " Jul 20 02:51:12.161488 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.161320 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/01b09630-9c37-419f-8ad8-e0b97aa6df82-config\") pod \"01b09630-9c37-419f-8ad8-e0b97aa6df82\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " Jul 20 02:51:12.161488 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.161346 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/01b09630-9c37-419f-8ad8-e0b97aa6df82-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"01b09630-9c37-419f-8ad8-e0b97aa6df82\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " Jul 20 02:51:12.161488 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.161379 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/01b09630-9c37-419f-8ad8-e0b97aa6df82-secret-kube-rbac-proxy\") pod \"01b09630-9c37-419f-8ad8-e0b97aa6df82\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " Jul 20 02:51:12.161488 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.161406 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/01b09630-9c37-419f-8ad8-e0b97aa6df82-prometheus-k8s-rulefiles-0\") pod \"01b09630-9c37-419f-8ad8-e0b97aa6df82\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " Jul 20 02:51:12.161488 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.161441 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01b09630-9c37-419f-8ad8-e0b97aa6df82-configmap-serving-certs-ca-bundle\") pod \"01b09630-9c37-419f-8ad8-e0b97aa6df82\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " Jul 20 02:51:12.161488 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.161467 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/01b09630-9c37-419f-8ad8-e0b97aa6df82-prometheus-k8s-rulefiles-2\") pod \"01b09630-9c37-419f-8ad8-e0b97aa6df82\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " Jul 20 02:51:12.161488 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.161489 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/01b09630-9c37-419f-8ad8-e0b97aa6df82-thanos-prometheus-http-client-file\") pod \"01b09630-9c37-419f-8ad8-e0b97aa6df82\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " Jul 20 02:51:12.161903 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.161514 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/01b09630-9c37-419f-8ad8-e0b97aa6df82-secret-grpc-tls\") pod \"01b09630-9c37-419f-8ad8-e0b97aa6df82\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " Jul 20 02:51:12.161903 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.161549 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/01b09630-9c37-419f-8ad8-e0b97aa6df82-secret-prometheus-k8s-tls\") pod \"01b09630-9c37-419f-8ad8-e0b97aa6df82\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " Jul 20 02:51:12.161903 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.161584 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/01b09630-9c37-419f-8ad8-e0b97aa6df82-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"01b09630-9c37-419f-8ad8-e0b97aa6df82\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " Jul 20 02:51:12.161903 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.161612 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/01b09630-9c37-419f-8ad8-e0b97aa6df82-prometheus-k8s-rulefiles-1\") pod \"01b09630-9c37-419f-8ad8-e0b97aa6df82\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " Jul 20 02:51:12.161903 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.161642 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/01b09630-9c37-419f-8ad8-e0b97aa6df82-config-out\") pod \"01b09630-9c37-419f-8ad8-e0b97aa6df82\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " Jul 20 02:51:12.161903 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.161710 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbz2s\" (UniqueName: \"kubernetes.io/projected/01b09630-9c37-419f-8ad8-e0b97aa6df82-kube-api-access-kbz2s\") pod \"01b09630-9c37-419f-8ad8-e0b97aa6df82\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " Jul 20 02:51:12.161903 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.161722 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01b09630-9c37-419f-8ad8-e0b97aa6df82-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "01b09630-9c37-419f-8ad8-e0b97aa6df82" (UID: "01b09630-9c37-419f-8ad8-e0b97aa6df82"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 02:51:12.161903 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.161749 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01b09630-9c37-419f-8ad8-e0b97aa6df82-configmap-metrics-client-ca\") pod \"01b09630-9c37-419f-8ad8-e0b97aa6df82\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " Jul 20 02:51:12.161903 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.161777 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01b09630-9c37-419f-8ad8-e0b97aa6df82-configmap-kubelet-serving-ca-bundle\") pod \"01b09630-9c37-419f-8ad8-e0b97aa6df82\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " Jul 20 02:51:12.161903 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.161802 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/01b09630-9c37-419f-8ad8-e0b97aa6df82-web-config\") pod \"01b09630-9c37-419f-8ad8-e0b97aa6df82\" (UID: \"01b09630-9c37-419f-8ad8-e0b97aa6df82\") " Jul 20 02:51:12.162423 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.162043 2558 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01b09630-9c37-419f-8ad8-e0b97aa6df82-prometheus-trusted-ca-bundle\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 02:51:12.162423 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.162045 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01b09630-9c37-419f-8ad8-e0b97aa6df82-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "01b09630-9c37-419f-8ad8-e0b97aa6df82" (UID: "01b09630-9c37-419f-8ad8-e0b97aa6df82"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 02:51:12.163079 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.162848 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01b09630-9c37-419f-8ad8-e0b97aa6df82-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "01b09630-9c37-419f-8ad8-e0b97aa6df82" (UID: "01b09630-9c37-419f-8ad8-e0b97aa6df82"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 02:51:12.163079 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.162983 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01b09630-9c37-419f-8ad8-e0b97aa6df82-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "01b09630-9c37-419f-8ad8-e0b97aa6df82" (UID: "01b09630-9c37-419f-8ad8-e0b97aa6df82"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 02:51:12.163647 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.163617 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01b09630-9c37-419f-8ad8-e0b97aa6df82-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "01b09630-9c37-419f-8ad8-e0b97aa6df82" (UID: "01b09630-9c37-419f-8ad8-e0b97aa6df82"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 02:51:12.163825 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.163797 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01b09630-9c37-419f-8ad8-e0b97aa6df82-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "01b09630-9c37-419f-8ad8-e0b97aa6df82" (UID: "01b09630-9c37-419f-8ad8-e0b97aa6df82"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 02:51:12.163972 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.163948 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01b09630-9c37-419f-8ad8-e0b97aa6df82-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "01b09630-9c37-419f-8ad8-e0b97aa6df82" (UID: "01b09630-9c37-419f-8ad8-e0b97aa6df82"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 02:51:12.164390 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.164353 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01b09630-9c37-419f-8ad8-e0b97aa6df82-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "01b09630-9c37-419f-8ad8-e0b97aa6df82" (UID: "01b09630-9c37-419f-8ad8-e0b97aa6df82"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 02:51:12.164925 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.164363 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01b09630-9c37-419f-8ad8-e0b97aa6df82-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "01b09630-9c37-419f-8ad8-e0b97aa6df82" (UID: "01b09630-9c37-419f-8ad8-e0b97aa6df82"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 02:51:12.164925 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.164619 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01b09630-9c37-419f-8ad8-e0b97aa6df82-config" (OuterVolumeSpecName: "config") pod "01b09630-9c37-419f-8ad8-e0b97aa6df82" (UID: "01b09630-9c37-419f-8ad8-e0b97aa6df82"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 02:51:12.164925 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.164798 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01b09630-9c37-419f-8ad8-e0b97aa6df82-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "01b09630-9c37-419f-8ad8-e0b97aa6df82" (UID: "01b09630-9c37-419f-8ad8-e0b97aa6df82"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 02:51:12.165352 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.165322 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01b09630-9c37-419f-8ad8-e0b97aa6df82-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "01b09630-9c37-419f-8ad8-e0b97aa6df82" (UID: "01b09630-9c37-419f-8ad8-e0b97aa6df82"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 02:51:12.165982 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.165939 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01b09630-9c37-419f-8ad8-e0b97aa6df82-kube-api-access-kbz2s" (OuterVolumeSpecName: "kube-api-access-kbz2s") pod "01b09630-9c37-419f-8ad8-e0b97aa6df82" (UID: "01b09630-9c37-419f-8ad8-e0b97aa6df82"). InnerVolumeSpecName "kube-api-access-kbz2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 02:51:12.166085 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.166035 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01b09630-9c37-419f-8ad8-e0b97aa6df82-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "01b09630-9c37-419f-8ad8-e0b97aa6df82" (UID: "01b09630-9c37-419f-8ad8-e0b97aa6df82"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 02:51:12.166407 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.166378 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01b09630-9c37-419f-8ad8-e0b97aa6df82-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "01b09630-9c37-419f-8ad8-e0b97aa6df82" (UID: "01b09630-9c37-419f-8ad8-e0b97aa6df82"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 02:51:12.166853 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.166830 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01b09630-9c37-419f-8ad8-e0b97aa6df82-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "01b09630-9c37-419f-8ad8-e0b97aa6df82" (UID: "01b09630-9c37-419f-8ad8-e0b97aa6df82"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 02:51:12.166930 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.166900 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01b09630-9c37-419f-8ad8-e0b97aa6df82-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "01b09630-9c37-419f-8ad8-e0b97aa6df82" (UID: "01b09630-9c37-419f-8ad8-e0b97aa6df82"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 02:51:12.166990 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.166964 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01b09630-9c37-419f-8ad8-e0b97aa6df82-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "01b09630-9c37-419f-8ad8-e0b97aa6df82" (UID: "01b09630-9c37-419f-8ad8-e0b97aa6df82"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 02:51:12.166990 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.166967 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01b09630-9c37-419f-8ad8-e0b97aa6df82-config-out" (OuterVolumeSpecName: "config-out") pod "01b09630-9c37-419f-8ad8-e0b97aa6df82" (UID: "01b09630-9c37-419f-8ad8-e0b97aa6df82"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 02:51:12.174752 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.174732 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01b09630-9c37-419f-8ad8-e0b97aa6df82-web-config" (OuterVolumeSpecName: "web-config") pod "01b09630-9c37-419f-8ad8-e0b97aa6df82" (UID: "01b09630-9c37-419f-8ad8-e0b97aa6df82"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 02:51:12.262991 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.262947 2558 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/01b09630-9c37-419f-8ad8-e0b97aa6df82-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 02:51:12.262991 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.262987 2558 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/01b09630-9c37-419f-8ad8-e0b97aa6df82-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 02:51:12.262991 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.262998 2558 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/01b09630-9c37-419f-8ad8-e0b97aa6df82-config-out\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 02:51:12.262991 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.263007 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbz2s\" (UniqueName: \"kubernetes.io/projected/01b09630-9c37-419f-8ad8-e0b97aa6df82-kube-api-access-kbz2s\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 02:51:12.263260 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.263018 2558 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01b09630-9c37-419f-8ad8-e0b97aa6df82-configmap-metrics-client-ca\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 02:51:12.263260 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.263027 2558 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01b09630-9c37-419f-8ad8-e0b97aa6df82-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 02:51:12.263260 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.263037 2558 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/01b09630-9c37-419f-8ad8-e0b97aa6df82-web-config\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 02:51:12.263260 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.263045 2558 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/01b09630-9c37-419f-8ad8-e0b97aa6df82-secret-metrics-client-certs\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 02:51:12.263260 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.263054 2558 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/01b09630-9c37-419f-8ad8-e0b97aa6df82-tls-assets\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 02:51:12.263260 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.263061 2558 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/01b09630-9c37-419f-8ad8-e0b97aa6df82-prometheus-k8s-db\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 02:51:12.263260 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.263069 2558 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/01b09630-9c37-419f-8ad8-e0b97aa6df82-config\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 02:51:12.263260 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.263080 2558 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/01b09630-9c37-419f-8ad8-e0b97aa6df82-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 02:51:12.263260 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.263087 2558 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/01b09630-9c37-419f-8ad8-e0b97aa6df82-secret-kube-rbac-proxy\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 02:51:12.263260 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.263108 2558 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/01b09630-9c37-419f-8ad8-e0b97aa6df82-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 02:51:12.263260 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.263116 2558 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01b09630-9c37-419f-8ad8-e0b97aa6df82-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 02:51:12.263260 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.263124 2558 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/01b09630-9c37-419f-8ad8-e0b97aa6df82-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 02:51:12.263260 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.263132 2558 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/01b09630-9c37-419f-8ad8-e0b97aa6df82-thanos-prometheus-http-client-file\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 02:51:12.263260 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.263140 2558 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/01b09630-9c37-419f-8ad8-e0b97aa6df82-secret-grpc-tls\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 02:51:12.263260 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.263148 2558 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/01b09630-9c37-419f-8ad8-e0b97aa6df82-secret-prometheus-k8s-tls\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 02:51:12.684648 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.684613 2558 generic.go:356] "Generic (PLEG): container finished" podID="01b09630-9c37-419f-8ad8-e0b97aa6df82" containerID="cdc12e352a5377a18c6f5cdf4ccd7d3c07d7c80b8f1f409f7924d88106b9db86" exitCode=0 Jul 20 02:51:12.684648 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.684639 2558 generic.go:356] "Generic (PLEG): container finished" podID="01b09630-9c37-419f-8ad8-e0b97aa6df82" containerID="d3c3f2319689f0f448374b987914f00d7e950b93bb253eadc00efd75f1269bfa" exitCode=0 Jul 20 02:51:12.684648 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.684647 2558 generic.go:356] "Generic (PLEG): container finished" podID="01b09630-9c37-419f-8ad8-e0b97aa6df82" containerID="0ff20bc1a0992a582fa8ad26c0f2ef2c54a35f27c097825bba7d5c4c55e370c8" exitCode=0 Jul 20 02:51:12.684648 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.684652 2558 generic.go:356] "Generic (PLEG): container finished" podID="01b09630-9c37-419f-8ad8-e0b97aa6df82" containerID="259ed9af1282abb0563757e4b43308b21f71b13e55f3e5c80f3ad1a5e2154d6f" exitCode=0 Jul 20 02:51:12.684648 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.684658 2558 generic.go:356] "Generic (PLEG): container finished" podID="01b09630-9c37-419f-8ad8-e0b97aa6df82" containerID="931c8bb914b43643f2cbd7630c5c3ffaae8672cf2770c9a63615141ba5db773f" exitCode=0 Jul 20 02:51:12.685034 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.684664 2558 generic.go:356] "Generic (PLEG): container finished" podID="01b09630-9c37-419f-8ad8-e0b97aa6df82" containerID="454280bab7fcdcad8e564a92bac238ffdc6c89743cbe3660c537e50f70d4eea9" exitCode=0 Jul 20 02:51:12.685034 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.684701 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.685034 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.684698 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01b09630-9c37-419f-8ad8-e0b97aa6df82","Type":"ContainerDied","Data":"cdc12e352a5377a18c6f5cdf4ccd7d3c07d7c80b8f1f409f7924d88106b9db86"} Jul 20 02:51:12.685034 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.684837 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01b09630-9c37-419f-8ad8-e0b97aa6df82","Type":"ContainerDied","Data":"d3c3f2319689f0f448374b987914f00d7e950b93bb253eadc00efd75f1269bfa"} Jul 20 02:51:12.685034 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.684865 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01b09630-9c37-419f-8ad8-e0b97aa6df82","Type":"ContainerDied","Data":"0ff20bc1a0992a582fa8ad26c0f2ef2c54a35f27c097825bba7d5c4c55e370c8"} Jul 20 02:51:12.685034 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.684893 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01b09630-9c37-419f-8ad8-e0b97aa6df82","Type":"ContainerDied","Data":"259ed9af1282abb0563757e4b43308b21f71b13e55f3e5c80f3ad1a5e2154d6f"} Jul 20 02:51:12.685034 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.684907 2558 scope.go:117] "RemoveContainer" containerID="cdc12e352a5377a18c6f5cdf4ccd7d3c07d7c80b8f1f409f7924d88106b9db86" Jul 20 02:51:12.685034 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.684917 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01b09630-9c37-419f-8ad8-e0b97aa6df82","Type":"ContainerDied","Data":"931c8bb914b43643f2cbd7630c5c3ffaae8672cf2770c9a63615141ba5db773f"} Jul 20 02:51:12.685034 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.684941 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01b09630-9c37-419f-8ad8-e0b97aa6df82","Type":"ContainerDied","Data":"454280bab7fcdcad8e564a92bac238ffdc6c89743cbe3660c537e50f70d4eea9"} Jul 20 02:51:12.685034 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.684964 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01b09630-9c37-419f-8ad8-e0b97aa6df82","Type":"ContainerDied","Data":"77e070f963eab3b96bd8125b995ec469331c23f5d6e1225d7714ab712d75579f"} Jul 20 02:51:12.694066 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.694052 2558 scope.go:117] "RemoveContainer" containerID="d3c3f2319689f0f448374b987914f00d7e950b93bb253eadc00efd75f1269bfa" Jul 20 02:51:12.702913 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.702893 2558 scope.go:117] "RemoveContainer" containerID="0ff20bc1a0992a582fa8ad26c0f2ef2c54a35f27c097825bba7d5c4c55e370c8" Jul 20 02:51:12.708695 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.708672 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 20 02:51:12.709849 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.709836 2558 scope.go:117] "RemoveContainer" containerID="259ed9af1282abb0563757e4b43308b21f71b13e55f3e5c80f3ad1a5e2154d6f" Jul 20 02:51:12.714920 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.714898 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 20 02:51:12.717040 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.717026 2558 scope.go:117] "RemoveContainer" containerID="931c8bb914b43643f2cbd7630c5c3ffaae8672cf2770c9a63615141ba5db773f" Jul 20 02:51:12.723514 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.723495 2558 scope.go:117] "RemoveContainer" containerID="454280bab7fcdcad8e564a92bac238ffdc6c89743cbe3660c537e50f70d4eea9" Jul 20 02:51:12.730564 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.730549 2558 scope.go:117] "RemoveContainer" containerID="3394177568c88e299ac9ea0feab4432f10d6b7a2ebc05bed2a457ab6993f3b8e" Jul 20 02:51:12.736705 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.736687 2558 scope.go:117] "RemoveContainer" containerID="cdc12e352a5377a18c6f5cdf4ccd7d3c07d7c80b8f1f409f7924d88106b9db86" Jul 20 02:51:12.736936 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:51:12.736918 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdc12e352a5377a18c6f5cdf4ccd7d3c07d7c80b8f1f409f7924d88106b9db86\": container with ID starting with cdc12e352a5377a18c6f5cdf4ccd7d3c07d7c80b8f1f409f7924d88106b9db86 not found: ID does not exist" containerID="cdc12e352a5377a18c6f5cdf4ccd7d3c07d7c80b8f1f409f7924d88106b9db86" Jul 20 02:51:12.736977 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.736945 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdc12e352a5377a18c6f5cdf4ccd7d3c07d7c80b8f1f409f7924d88106b9db86"} err="failed to get container status \"cdc12e352a5377a18c6f5cdf4ccd7d3c07d7c80b8f1f409f7924d88106b9db86\": rpc error: code = NotFound desc = could not find container \"cdc12e352a5377a18c6f5cdf4ccd7d3c07d7c80b8f1f409f7924d88106b9db86\": container with ID starting with cdc12e352a5377a18c6f5cdf4ccd7d3c07d7c80b8f1f409f7924d88106b9db86 not found: ID does not exist" Jul 20 02:51:12.736977 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.736962 2558 scope.go:117] "RemoveContainer" containerID="d3c3f2319689f0f448374b987914f00d7e950b93bb253eadc00efd75f1269bfa" Jul 20 02:51:12.737232 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:51:12.737216 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3c3f2319689f0f448374b987914f00d7e950b93bb253eadc00efd75f1269bfa\": container with ID starting with d3c3f2319689f0f448374b987914f00d7e950b93bb253eadc00efd75f1269bfa not found: ID does not exist" containerID="d3c3f2319689f0f448374b987914f00d7e950b93bb253eadc00efd75f1269bfa" Jul 20 02:51:12.737232 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.737235 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3c3f2319689f0f448374b987914f00d7e950b93bb253eadc00efd75f1269bfa"} err="failed to get container status \"d3c3f2319689f0f448374b987914f00d7e950b93bb253eadc00efd75f1269bfa\": rpc error: code = NotFound desc = could not find container \"d3c3f2319689f0f448374b987914f00d7e950b93bb253eadc00efd75f1269bfa\": container with ID starting with d3c3f2319689f0f448374b987914f00d7e950b93bb253eadc00efd75f1269bfa not found: ID does not exist" Jul 20 02:51:12.737372 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.737247 2558 scope.go:117] "RemoveContainer" containerID="0ff20bc1a0992a582fa8ad26c0f2ef2c54a35f27c097825bba7d5c4c55e370c8" Jul 20 02:51:12.737504 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:51:12.737483 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ff20bc1a0992a582fa8ad26c0f2ef2c54a35f27c097825bba7d5c4c55e370c8\": container with ID starting with 0ff20bc1a0992a582fa8ad26c0f2ef2c54a35f27c097825bba7d5c4c55e370c8 not found: ID does not exist" containerID="0ff20bc1a0992a582fa8ad26c0f2ef2c54a35f27c097825bba7d5c4c55e370c8" Jul 20 02:51:12.737542 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.737514 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ff20bc1a0992a582fa8ad26c0f2ef2c54a35f27c097825bba7d5c4c55e370c8"} err="failed to get container status \"0ff20bc1a0992a582fa8ad26c0f2ef2c54a35f27c097825bba7d5c4c55e370c8\": rpc error: code = NotFound desc = could not find container \"0ff20bc1a0992a582fa8ad26c0f2ef2c54a35f27c097825bba7d5c4c55e370c8\": container with ID starting with 0ff20bc1a0992a582fa8ad26c0f2ef2c54a35f27c097825bba7d5c4c55e370c8 not found: ID does not exist" Jul 20 02:51:12.737542 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.737536 2558 scope.go:117] "RemoveContainer" containerID="259ed9af1282abb0563757e4b43308b21f71b13e55f3e5c80f3ad1a5e2154d6f" Jul 20 02:51:12.737822 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:51:12.737796 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"259ed9af1282abb0563757e4b43308b21f71b13e55f3e5c80f3ad1a5e2154d6f\": container with ID starting with 259ed9af1282abb0563757e4b43308b21f71b13e55f3e5c80f3ad1a5e2154d6f not found: ID does not exist" containerID="259ed9af1282abb0563757e4b43308b21f71b13e55f3e5c80f3ad1a5e2154d6f" Jul 20 02:51:12.737962 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.737826 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"259ed9af1282abb0563757e4b43308b21f71b13e55f3e5c80f3ad1a5e2154d6f"} err="failed to get container status \"259ed9af1282abb0563757e4b43308b21f71b13e55f3e5c80f3ad1a5e2154d6f\": rpc error: code = NotFound desc = could not find container \"259ed9af1282abb0563757e4b43308b21f71b13e55f3e5c80f3ad1a5e2154d6f\": container with ID starting with 259ed9af1282abb0563757e4b43308b21f71b13e55f3e5c80f3ad1a5e2154d6f not found: ID does not exist" Jul 20 02:51:12.737962 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.737849 2558 scope.go:117] "RemoveContainer" containerID="931c8bb914b43643f2cbd7630c5c3ffaae8672cf2770c9a63615141ba5db773f" Jul 20 02:51:12.738232 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:51:12.738213 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"931c8bb914b43643f2cbd7630c5c3ffaae8672cf2770c9a63615141ba5db773f\": container with ID starting with 931c8bb914b43643f2cbd7630c5c3ffaae8672cf2770c9a63615141ba5db773f not found: ID does not exist" containerID="931c8bb914b43643f2cbd7630c5c3ffaae8672cf2770c9a63615141ba5db773f" Jul 20 02:51:12.738331 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.738240 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"931c8bb914b43643f2cbd7630c5c3ffaae8672cf2770c9a63615141ba5db773f"} err="failed to get container status \"931c8bb914b43643f2cbd7630c5c3ffaae8672cf2770c9a63615141ba5db773f\": rpc error: code = NotFound desc = could not find container \"931c8bb914b43643f2cbd7630c5c3ffaae8672cf2770c9a63615141ba5db773f\": container with ID starting with 931c8bb914b43643f2cbd7630c5c3ffaae8672cf2770c9a63615141ba5db773f not found: ID does not exist" Jul 20 02:51:12.738331 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.738258 2558 scope.go:117] "RemoveContainer" containerID="454280bab7fcdcad8e564a92bac238ffdc6c89743cbe3660c537e50f70d4eea9" Jul 20 02:51:12.738519 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:51:12.738502 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"454280bab7fcdcad8e564a92bac238ffdc6c89743cbe3660c537e50f70d4eea9\": container with ID starting with 454280bab7fcdcad8e564a92bac238ffdc6c89743cbe3660c537e50f70d4eea9 not found: ID does not exist" containerID="454280bab7fcdcad8e564a92bac238ffdc6c89743cbe3660c537e50f70d4eea9" Jul 20 02:51:12.738588 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.738535 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"454280bab7fcdcad8e564a92bac238ffdc6c89743cbe3660c537e50f70d4eea9"} err="failed to get container status \"454280bab7fcdcad8e564a92bac238ffdc6c89743cbe3660c537e50f70d4eea9\": rpc error: code = NotFound desc = could not find container \"454280bab7fcdcad8e564a92bac238ffdc6c89743cbe3660c537e50f70d4eea9\": container with ID starting with 454280bab7fcdcad8e564a92bac238ffdc6c89743cbe3660c537e50f70d4eea9 not found: ID does not exist" Jul 20 02:51:12.738588 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.738559 2558 scope.go:117] "RemoveContainer" containerID="3394177568c88e299ac9ea0feab4432f10d6b7a2ebc05bed2a457ab6993f3b8e" Jul 20 02:51:12.738783 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:51:12.738766 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3394177568c88e299ac9ea0feab4432f10d6b7a2ebc05bed2a457ab6993f3b8e\": container with ID starting with 3394177568c88e299ac9ea0feab4432f10d6b7a2ebc05bed2a457ab6993f3b8e not found: ID does not exist" containerID="3394177568c88e299ac9ea0feab4432f10d6b7a2ebc05bed2a457ab6993f3b8e" Jul 20 02:51:12.738824 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.738788 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3394177568c88e299ac9ea0feab4432f10d6b7a2ebc05bed2a457ab6993f3b8e"} err="failed to get container status \"3394177568c88e299ac9ea0feab4432f10d6b7a2ebc05bed2a457ab6993f3b8e\": rpc error: code = NotFound desc = could not find container \"3394177568c88e299ac9ea0feab4432f10d6b7a2ebc05bed2a457ab6993f3b8e\": container with ID starting with 3394177568c88e299ac9ea0feab4432f10d6b7a2ebc05bed2a457ab6993f3b8e not found: ID does not exist" Jul 20 02:51:12.738824 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.738803 2558 scope.go:117] "RemoveContainer" containerID="cdc12e352a5377a18c6f5cdf4ccd7d3c07d7c80b8f1f409f7924d88106b9db86" Jul 20 02:51:12.739022 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.739002 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdc12e352a5377a18c6f5cdf4ccd7d3c07d7c80b8f1f409f7924d88106b9db86"} err="failed to get container status \"cdc12e352a5377a18c6f5cdf4ccd7d3c07d7c80b8f1f409f7924d88106b9db86\": rpc error: code = NotFound desc = could not find container \"cdc12e352a5377a18c6f5cdf4ccd7d3c07d7c80b8f1f409f7924d88106b9db86\": container with ID starting with cdc12e352a5377a18c6f5cdf4ccd7d3c07d7c80b8f1f409f7924d88106b9db86 not found: ID does not exist" Jul 20 02:51:12.739071 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.739025 2558 scope.go:117] "RemoveContainer" containerID="d3c3f2319689f0f448374b987914f00d7e950b93bb253eadc00efd75f1269bfa" Jul 20 02:51:12.739248 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.739224 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3c3f2319689f0f448374b987914f00d7e950b93bb253eadc00efd75f1269bfa"} err="failed to get container status \"d3c3f2319689f0f448374b987914f00d7e950b93bb253eadc00efd75f1269bfa\": rpc error: code = NotFound desc = could not find container \"d3c3f2319689f0f448374b987914f00d7e950b93bb253eadc00efd75f1269bfa\": container with ID starting with d3c3f2319689f0f448374b987914f00d7e950b93bb253eadc00efd75f1269bfa not found: ID does not exist" Jul 20 02:51:12.739297 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.739250 2558 scope.go:117] "RemoveContainer" containerID="0ff20bc1a0992a582fa8ad26c0f2ef2c54a35f27c097825bba7d5c4c55e370c8" Jul 20 02:51:12.739459 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.739430 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ff20bc1a0992a582fa8ad26c0f2ef2c54a35f27c097825bba7d5c4c55e370c8"} err="failed to get container status \"0ff20bc1a0992a582fa8ad26c0f2ef2c54a35f27c097825bba7d5c4c55e370c8\": rpc error: code = NotFound desc = could not find container \"0ff20bc1a0992a582fa8ad26c0f2ef2c54a35f27c097825bba7d5c4c55e370c8\": container with ID starting with 0ff20bc1a0992a582fa8ad26c0f2ef2c54a35f27c097825bba7d5c4c55e370c8 not found: ID does not exist" Jul 20 02:51:12.739536 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.739462 2558 scope.go:117] "RemoveContainer" containerID="259ed9af1282abb0563757e4b43308b21f71b13e55f3e5c80f3ad1a5e2154d6f" Jul 20 02:51:12.739579 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.739534 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 20 02:51:12.739690 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.739673 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"259ed9af1282abb0563757e4b43308b21f71b13e55f3e5c80f3ad1a5e2154d6f"} err="failed to get container status \"259ed9af1282abb0563757e4b43308b21f71b13e55f3e5c80f3ad1a5e2154d6f\": rpc error: code = NotFound desc = could not find container \"259ed9af1282abb0563757e4b43308b21f71b13e55f3e5c80f3ad1a5e2154d6f\": container with ID starting with 259ed9af1282abb0563757e4b43308b21f71b13e55f3e5c80f3ad1a5e2154d6f not found: ID does not exist" Jul 20 02:51:12.739732 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.739690 2558 scope.go:117] "RemoveContainer" containerID="931c8bb914b43643f2cbd7630c5c3ffaae8672cf2770c9a63615141ba5db773f" Jul 20 02:51:12.739822 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.739807 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="537fa96e-a0c2-48ff-877b-5aca6c1e40f7" containerName="registry" Jul 20 02:51:12.739866 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.739824 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="537fa96e-a0c2-48ff-877b-5aca6c1e40f7" containerName="registry" Jul 20 02:51:12.739866 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.739834 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01b09630-9c37-419f-8ad8-e0b97aa6df82" containerName="kube-rbac-proxy-thanos" Jul 20 02:51:12.739866 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.739842 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="01b09630-9c37-419f-8ad8-e0b97aa6df82" containerName="kube-rbac-proxy-thanos" Jul 20 02:51:12.739866 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.739860 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01b09630-9c37-419f-8ad8-e0b97aa6df82" containerName="kube-rbac-proxy" Jul 20 02:51:12.740005 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.739869 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="01b09630-9c37-419f-8ad8-e0b97aa6df82" containerName="kube-rbac-proxy" Jul 20 02:51:12.740005 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.739879 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01b09630-9c37-419f-8ad8-e0b97aa6df82" containerName="init-config-reloader" Jul 20 02:51:12.740005 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.739888 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="01b09630-9c37-419f-8ad8-e0b97aa6df82" containerName="init-config-reloader" Jul 20 02:51:12.740005 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.739897 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01b09630-9c37-419f-8ad8-e0b97aa6df82" containerName="config-reloader" Jul 20 02:51:12.740005 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.739897 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"931c8bb914b43643f2cbd7630c5c3ffaae8672cf2770c9a63615141ba5db773f"} err="failed to get container status \"931c8bb914b43643f2cbd7630c5c3ffaae8672cf2770c9a63615141ba5db773f\": rpc error: code = NotFound desc = could not find container \"931c8bb914b43643f2cbd7630c5c3ffaae8672cf2770c9a63615141ba5db773f\": container with ID starting with 931c8bb914b43643f2cbd7630c5c3ffaae8672cf2770c9a63615141ba5db773f not found: ID does not exist" Jul 20 02:51:12.740005 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.739915 2558 scope.go:117] "RemoveContainer" containerID="454280bab7fcdcad8e564a92bac238ffdc6c89743cbe3660c537e50f70d4eea9" Jul 20 02:51:12.740005 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.739902 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="01b09630-9c37-419f-8ad8-e0b97aa6df82" containerName="config-reloader" Jul 20 02:51:12.740005 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.739958 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95a990cb-fd40-4d72-8a18-fc3ae2930904" containerName="console" Jul 20 02:51:12.740005 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.739966 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="95a990cb-fd40-4d72-8a18-fc3ae2930904" containerName="console" Jul 20 02:51:12.740005 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.739977 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01b09630-9c37-419f-8ad8-e0b97aa6df82" containerName="thanos-sidecar" Jul 20 02:51:12.740005 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.739985 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="01b09630-9c37-419f-8ad8-e0b97aa6df82" containerName="thanos-sidecar" Jul 20 02:51:12.740005 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.739996 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01b09630-9c37-419f-8ad8-e0b97aa6df82" containerName="prometheus" Jul 20 02:51:12.740005 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.740006 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="01b09630-9c37-419f-8ad8-e0b97aa6df82" containerName="prometheus" Jul 20 02:51:12.740005 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.740013 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01b09630-9c37-419f-8ad8-e0b97aa6df82" containerName="kube-rbac-proxy-web" Jul 20 02:51:12.740515 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.740020 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="01b09630-9c37-419f-8ad8-e0b97aa6df82" containerName="kube-rbac-proxy-web" Jul 20 02:51:12.740515 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.740123 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="537fa96e-a0c2-48ff-877b-5aca6c1e40f7" containerName="registry" Jul 20 02:51:12.740515 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.740134 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="95a990cb-fd40-4d72-8a18-fc3ae2930904" containerName="console" Jul 20 02:51:12.740515 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.740140 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="01b09630-9c37-419f-8ad8-e0b97aa6df82" containerName="kube-rbac-proxy" Jul 20 02:51:12.740515 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.740148 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="01b09630-9c37-419f-8ad8-e0b97aa6df82" containerName="config-reloader" Jul 20 02:51:12.740515 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.740155 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="01b09630-9c37-419f-8ad8-e0b97aa6df82" containerName="prometheus" Jul 20 02:51:12.740515 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.740164 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="01b09630-9c37-419f-8ad8-e0b97aa6df82" containerName="kube-rbac-proxy-web" Jul 20 02:51:12.740515 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.740172 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="01b09630-9c37-419f-8ad8-e0b97aa6df82" containerName="kube-rbac-proxy-thanos" Jul 20 02:51:12.740515 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.740182 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="01b09630-9c37-419f-8ad8-e0b97aa6df82" containerName="thanos-sidecar" Jul 20 02:51:12.740515 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.740194 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"454280bab7fcdcad8e564a92bac238ffdc6c89743cbe3660c537e50f70d4eea9"} err="failed to get container status \"454280bab7fcdcad8e564a92bac238ffdc6c89743cbe3660c537e50f70d4eea9\": rpc error: code = NotFound desc = could not find container \"454280bab7fcdcad8e564a92bac238ffdc6c89743cbe3660c537e50f70d4eea9\": container with ID starting with 454280bab7fcdcad8e564a92bac238ffdc6c89743cbe3660c537e50f70d4eea9 not found: ID does not exist" Jul 20 02:51:12.740515 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.740212 2558 scope.go:117] "RemoveContainer" containerID="3394177568c88e299ac9ea0feab4432f10d6b7a2ebc05bed2a457ab6993f3b8e" Jul 20 02:51:12.740515 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.740475 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3394177568c88e299ac9ea0feab4432f10d6b7a2ebc05bed2a457ab6993f3b8e"} err="failed to get container status \"3394177568c88e299ac9ea0feab4432f10d6b7a2ebc05bed2a457ab6993f3b8e\": rpc error: code = NotFound desc = could not find container \"3394177568c88e299ac9ea0feab4432f10d6b7a2ebc05bed2a457ab6993f3b8e\": container with ID starting with 3394177568c88e299ac9ea0feab4432f10d6b7a2ebc05bed2a457ab6993f3b8e not found: ID does not exist" Jul 20 02:51:12.740515 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.740492 2558 scope.go:117] "RemoveContainer" containerID="cdc12e352a5377a18c6f5cdf4ccd7d3c07d7c80b8f1f409f7924d88106b9db86" Jul 20 02:51:12.740996 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.740698 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdc12e352a5377a18c6f5cdf4ccd7d3c07d7c80b8f1f409f7924d88106b9db86"} err="failed to get container status \"cdc12e352a5377a18c6f5cdf4ccd7d3c07d7c80b8f1f409f7924d88106b9db86\": rpc error: code = NotFound desc = could not find container \"cdc12e352a5377a18c6f5cdf4ccd7d3c07d7c80b8f1f409f7924d88106b9db86\": container with ID starting with cdc12e352a5377a18c6f5cdf4ccd7d3c07d7c80b8f1f409f7924d88106b9db86 not found: ID does not exist" Jul 20 02:51:12.740996 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.740718 2558 scope.go:117] "RemoveContainer" containerID="d3c3f2319689f0f448374b987914f00d7e950b93bb253eadc00efd75f1269bfa" Jul 20 02:51:12.740996 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.740943 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3c3f2319689f0f448374b987914f00d7e950b93bb253eadc00efd75f1269bfa"} err="failed to get container status \"d3c3f2319689f0f448374b987914f00d7e950b93bb253eadc00efd75f1269bfa\": rpc error: code = NotFound desc = could not find container \"d3c3f2319689f0f448374b987914f00d7e950b93bb253eadc00efd75f1269bfa\": container with ID starting with d3c3f2319689f0f448374b987914f00d7e950b93bb253eadc00efd75f1269bfa not found: ID does not exist" Jul 20 02:51:12.740996 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.740960 2558 scope.go:117] "RemoveContainer" containerID="0ff20bc1a0992a582fa8ad26c0f2ef2c54a35f27c097825bba7d5c4c55e370c8" Jul 20 02:51:12.741196 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.741172 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ff20bc1a0992a582fa8ad26c0f2ef2c54a35f27c097825bba7d5c4c55e370c8"} err="failed to get container status \"0ff20bc1a0992a582fa8ad26c0f2ef2c54a35f27c097825bba7d5c4c55e370c8\": rpc error: code = NotFound desc = could not find container \"0ff20bc1a0992a582fa8ad26c0f2ef2c54a35f27c097825bba7d5c4c55e370c8\": container with ID starting with 0ff20bc1a0992a582fa8ad26c0f2ef2c54a35f27c097825bba7d5c4c55e370c8 not found: ID does not exist" Jul 20 02:51:12.741231 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.741198 2558 scope.go:117] "RemoveContainer" containerID="259ed9af1282abb0563757e4b43308b21f71b13e55f3e5c80f3ad1a5e2154d6f" Jul 20 02:51:12.741415 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.741398 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"259ed9af1282abb0563757e4b43308b21f71b13e55f3e5c80f3ad1a5e2154d6f"} err="failed to get container status \"259ed9af1282abb0563757e4b43308b21f71b13e55f3e5c80f3ad1a5e2154d6f\": rpc error: code = NotFound desc = could not find container \"259ed9af1282abb0563757e4b43308b21f71b13e55f3e5c80f3ad1a5e2154d6f\": container with ID starting with 259ed9af1282abb0563757e4b43308b21f71b13e55f3e5c80f3ad1a5e2154d6f not found: ID does not exist" Jul 20 02:51:12.741464 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.741416 2558 scope.go:117] "RemoveContainer" containerID="931c8bb914b43643f2cbd7630c5c3ffaae8672cf2770c9a63615141ba5db773f" Jul 20 02:51:12.741655 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.741631 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"931c8bb914b43643f2cbd7630c5c3ffaae8672cf2770c9a63615141ba5db773f"} err="failed to get container status \"931c8bb914b43643f2cbd7630c5c3ffaae8672cf2770c9a63615141ba5db773f\": rpc error: code = NotFound desc = could not find container \"931c8bb914b43643f2cbd7630c5c3ffaae8672cf2770c9a63615141ba5db773f\": container with ID starting with 931c8bb914b43643f2cbd7630c5c3ffaae8672cf2770c9a63615141ba5db773f not found: ID does not exist" Jul 20 02:51:12.741655 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.741651 2558 scope.go:117] "RemoveContainer" containerID="454280bab7fcdcad8e564a92bac238ffdc6c89743cbe3660c537e50f70d4eea9" Jul 20 02:51:12.741861 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.741842 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"454280bab7fcdcad8e564a92bac238ffdc6c89743cbe3660c537e50f70d4eea9"} err="failed to get container status \"454280bab7fcdcad8e564a92bac238ffdc6c89743cbe3660c537e50f70d4eea9\": rpc error: code = NotFound desc = could not find container \"454280bab7fcdcad8e564a92bac238ffdc6c89743cbe3660c537e50f70d4eea9\": container with ID starting with 454280bab7fcdcad8e564a92bac238ffdc6c89743cbe3660c537e50f70d4eea9 not found: ID does not exist" Jul 20 02:51:12.741861 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.741858 2558 scope.go:117] "RemoveContainer" containerID="3394177568c88e299ac9ea0feab4432f10d6b7a2ebc05bed2a457ab6993f3b8e" Jul 20 02:51:12.742078 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.742061 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3394177568c88e299ac9ea0feab4432f10d6b7a2ebc05bed2a457ab6993f3b8e"} err="failed to get container status \"3394177568c88e299ac9ea0feab4432f10d6b7a2ebc05bed2a457ab6993f3b8e\": rpc error: code = NotFound desc = could not find container \"3394177568c88e299ac9ea0feab4432f10d6b7a2ebc05bed2a457ab6993f3b8e\": container with ID starting with 3394177568c88e299ac9ea0feab4432f10d6b7a2ebc05bed2a457ab6993f3b8e not found: ID does not exist" Jul 20 02:51:12.742078 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.742077 2558 scope.go:117] "RemoveContainer" containerID="cdc12e352a5377a18c6f5cdf4ccd7d3c07d7c80b8f1f409f7924d88106b9db86" Jul 20 02:51:12.742299 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.742281 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdc12e352a5377a18c6f5cdf4ccd7d3c07d7c80b8f1f409f7924d88106b9db86"} err="failed to get container status \"cdc12e352a5377a18c6f5cdf4ccd7d3c07d7c80b8f1f409f7924d88106b9db86\": rpc error: code = NotFound desc = could not find container \"cdc12e352a5377a18c6f5cdf4ccd7d3c07d7c80b8f1f409f7924d88106b9db86\": container with ID starting with cdc12e352a5377a18c6f5cdf4ccd7d3c07d7c80b8f1f409f7924d88106b9db86 not found: ID does not exist" Jul 20 02:51:12.742344 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.742300 2558 scope.go:117] "RemoveContainer" containerID="d3c3f2319689f0f448374b987914f00d7e950b93bb253eadc00efd75f1269bfa" Jul 20 02:51:12.742516 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.742498 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3c3f2319689f0f448374b987914f00d7e950b93bb253eadc00efd75f1269bfa"} err="failed to get container status \"d3c3f2319689f0f448374b987914f00d7e950b93bb253eadc00efd75f1269bfa\": rpc error: code = NotFound desc = could not find container \"d3c3f2319689f0f448374b987914f00d7e950b93bb253eadc00efd75f1269bfa\": container with ID starting with d3c3f2319689f0f448374b987914f00d7e950b93bb253eadc00efd75f1269bfa not found: ID does not exist" Jul 20 02:51:12.742588 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.742518 2558 scope.go:117] "RemoveContainer" containerID="0ff20bc1a0992a582fa8ad26c0f2ef2c54a35f27c097825bba7d5c4c55e370c8" Jul 20 02:51:12.742719 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.742703 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ff20bc1a0992a582fa8ad26c0f2ef2c54a35f27c097825bba7d5c4c55e370c8"} err="failed to get container status \"0ff20bc1a0992a582fa8ad26c0f2ef2c54a35f27c097825bba7d5c4c55e370c8\": rpc error: code = NotFound desc = could not find container \"0ff20bc1a0992a582fa8ad26c0f2ef2c54a35f27c097825bba7d5c4c55e370c8\": container with ID starting with 0ff20bc1a0992a582fa8ad26c0f2ef2c54a35f27c097825bba7d5c4c55e370c8 not found: ID does not exist" Jul 20 02:51:12.742780 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.742721 2558 scope.go:117] "RemoveContainer" containerID="259ed9af1282abb0563757e4b43308b21f71b13e55f3e5c80f3ad1a5e2154d6f" Jul 20 02:51:12.742924 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.742908 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"259ed9af1282abb0563757e4b43308b21f71b13e55f3e5c80f3ad1a5e2154d6f"} err="failed to get container status \"259ed9af1282abb0563757e4b43308b21f71b13e55f3e5c80f3ad1a5e2154d6f\": rpc error: code = NotFound desc = could not find container \"259ed9af1282abb0563757e4b43308b21f71b13e55f3e5c80f3ad1a5e2154d6f\": container with ID starting with 259ed9af1282abb0563757e4b43308b21f71b13e55f3e5c80f3ad1a5e2154d6f not found: ID does not exist" Jul 20 02:51:12.742982 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.742925 2558 scope.go:117] "RemoveContainer" containerID="931c8bb914b43643f2cbd7630c5c3ffaae8672cf2770c9a63615141ba5db773f" Jul 20 02:51:12.743253 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.743235 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"931c8bb914b43643f2cbd7630c5c3ffaae8672cf2770c9a63615141ba5db773f"} err="failed to get container status \"931c8bb914b43643f2cbd7630c5c3ffaae8672cf2770c9a63615141ba5db773f\": rpc error: code = NotFound desc = could not find container \"931c8bb914b43643f2cbd7630c5c3ffaae8672cf2770c9a63615141ba5db773f\": container with ID starting with 931c8bb914b43643f2cbd7630c5c3ffaae8672cf2770c9a63615141ba5db773f not found: ID does not exist" Jul 20 02:51:12.743307 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.743255 2558 scope.go:117] "RemoveContainer" containerID="454280bab7fcdcad8e564a92bac238ffdc6c89743cbe3660c537e50f70d4eea9" Jul 20 02:51:12.743419 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.743400 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"454280bab7fcdcad8e564a92bac238ffdc6c89743cbe3660c537e50f70d4eea9"} err="failed to get container status \"454280bab7fcdcad8e564a92bac238ffdc6c89743cbe3660c537e50f70d4eea9\": rpc error: code = NotFound desc = could not find container \"454280bab7fcdcad8e564a92bac238ffdc6c89743cbe3660c537e50f70d4eea9\": container with ID starting with 454280bab7fcdcad8e564a92bac238ffdc6c89743cbe3660c537e50f70d4eea9 not found: ID does not exist" Jul 20 02:51:12.743458 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.743420 2558 scope.go:117] "RemoveContainer" containerID="3394177568c88e299ac9ea0feab4432f10d6b7a2ebc05bed2a457ab6993f3b8e" Jul 20 02:51:12.743663 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.743642 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3394177568c88e299ac9ea0feab4432f10d6b7a2ebc05bed2a457ab6993f3b8e"} err="failed to get container status \"3394177568c88e299ac9ea0feab4432f10d6b7a2ebc05bed2a457ab6993f3b8e\": rpc error: code = NotFound desc = could not find container \"3394177568c88e299ac9ea0feab4432f10d6b7a2ebc05bed2a457ab6993f3b8e\": container with ID starting with 3394177568c88e299ac9ea0feab4432f10d6b7a2ebc05bed2a457ab6993f3b8e not found: ID does not exist" Jul 20 02:51:12.743663 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.743662 2558 scope.go:117] "RemoveContainer" containerID="cdc12e352a5377a18c6f5cdf4ccd7d3c07d7c80b8f1f409f7924d88106b9db86" Jul 20 02:51:12.743828 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.743810 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdc12e352a5377a18c6f5cdf4ccd7d3c07d7c80b8f1f409f7924d88106b9db86"} err="failed to get container status \"cdc12e352a5377a18c6f5cdf4ccd7d3c07d7c80b8f1f409f7924d88106b9db86\": rpc error: code = NotFound desc = could not find container \"cdc12e352a5377a18c6f5cdf4ccd7d3c07d7c80b8f1f409f7924d88106b9db86\": container with ID starting with cdc12e352a5377a18c6f5cdf4ccd7d3c07d7c80b8f1f409f7924d88106b9db86 not found: ID does not exist" Jul 20 02:51:12.743874 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.743829 2558 scope.go:117] "RemoveContainer" containerID="d3c3f2319689f0f448374b987914f00d7e950b93bb253eadc00efd75f1269bfa" Jul 20 02:51:12.744002 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.743987 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3c3f2319689f0f448374b987914f00d7e950b93bb253eadc00efd75f1269bfa"} err="failed to get container status \"d3c3f2319689f0f448374b987914f00d7e950b93bb253eadc00efd75f1269bfa\": rpc error: code = NotFound desc = could not find container \"d3c3f2319689f0f448374b987914f00d7e950b93bb253eadc00efd75f1269bfa\": container with ID starting with d3c3f2319689f0f448374b987914f00d7e950b93bb253eadc00efd75f1269bfa not found: ID does not exist" Jul 20 02:51:12.744042 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.744002 2558 scope.go:117] "RemoveContainer" containerID="0ff20bc1a0992a582fa8ad26c0f2ef2c54a35f27c097825bba7d5c4c55e370c8" Jul 20 02:51:12.744240 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.744225 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ff20bc1a0992a582fa8ad26c0f2ef2c54a35f27c097825bba7d5c4c55e370c8"} err="failed to get container status \"0ff20bc1a0992a582fa8ad26c0f2ef2c54a35f27c097825bba7d5c4c55e370c8\": rpc error: code = NotFound desc = could not find container \"0ff20bc1a0992a582fa8ad26c0f2ef2c54a35f27c097825bba7d5c4c55e370c8\": container with ID starting with 0ff20bc1a0992a582fa8ad26c0f2ef2c54a35f27c097825bba7d5c4c55e370c8 not found: ID does not exist" Jul 20 02:51:12.744288 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.744241 2558 scope.go:117] "RemoveContainer" containerID="259ed9af1282abb0563757e4b43308b21f71b13e55f3e5c80f3ad1a5e2154d6f" Jul 20 02:51:12.744418 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.744402 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"259ed9af1282abb0563757e4b43308b21f71b13e55f3e5c80f3ad1a5e2154d6f"} err="failed to get container status \"259ed9af1282abb0563757e4b43308b21f71b13e55f3e5c80f3ad1a5e2154d6f\": rpc error: code = NotFound desc = could not find container \"259ed9af1282abb0563757e4b43308b21f71b13e55f3e5c80f3ad1a5e2154d6f\": container with ID starting with 259ed9af1282abb0563757e4b43308b21f71b13e55f3e5c80f3ad1a5e2154d6f not found: ID does not exist" Jul 20 02:51:12.744458 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.744418 2558 scope.go:117] "RemoveContainer" containerID="931c8bb914b43643f2cbd7630c5c3ffaae8672cf2770c9a63615141ba5db773f" Jul 20 02:51:12.744640 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.744624 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"931c8bb914b43643f2cbd7630c5c3ffaae8672cf2770c9a63615141ba5db773f"} err="failed to get container status \"931c8bb914b43643f2cbd7630c5c3ffaae8672cf2770c9a63615141ba5db773f\": rpc error: code = NotFound desc = could not find container \"931c8bb914b43643f2cbd7630c5c3ffaae8672cf2770c9a63615141ba5db773f\": container with ID starting with 931c8bb914b43643f2cbd7630c5c3ffaae8672cf2770c9a63615141ba5db773f not found: ID does not exist" Jul 20 02:51:12.744640 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.744640 2558 scope.go:117] "RemoveContainer" containerID="454280bab7fcdcad8e564a92bac238ffdc6c89743cbe3660c537e50f70d4eea9" Jul 20 02:51:12.744806 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.744790 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"454280bab7fcdcad8e564a92bac238ffdc6c89743cbe3660c537e50f70d4eea9"} err="failed to get container status \"454280bab7fcdcad8e564a92bac238ffdc6c89743cbe3660c537e50f70d4eea9\": rpc error: code = NotFound desc = could not find container \"454280bab7fcdcad8e564a92bac238ffdc6c89743cbe3660c537e50f70d4eea9\": container with ID starting with 454280bab7fcdcad8e564a92bac238ffdc6c89743cbe3660c537e50f70d4eea9 not found: ID does not exist" Jul 20 02:51:12.744806 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.744806 2558 scope.go:117] "RemoveContainer" containerID="3394177568c88e299ac9ea0feab4432f10d6b7a2ebc05bed2a457ab6993f3b8e" Jul 20 02:51:12.745001 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.744986 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3394177568c88e299ac9ea0feab4432f10d6b7a2ebc05bed2a457ab6993f3b8e"} err="failed to get container status \"3394177568c88e299ac9ea0feab4432f10d6b7a2ebc05bed2a457ab6993f3b8e\": rpc error: code = NotFound desc = could not find container \"3394177568c88e299ac9ea0feab4432f10d6b7a2ebc05bed2a457ab6993f3b8e\": container with ID starting with 3394177568c88e299ac9ea0feab4432f10d6b7a2ebc05bed2a457ab6993f3b8e not found: ID does not exist" Jul 20 02:51:12.745049 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.745008 2558 scope.go:117] "RemoveContainer" containerID="cdc12e352a5377a18c6f5cdf4ccd7d3c07d7c80b8f1f409f7924d88106b9db86" Jul 20 02:51:12.745177 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.745162 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdc12e352a5377a18c6f5cdf4ccd7d3c07d7c80b8f1f409f7924d88106b9db86"} err="failed to get container status \"cdc12e352a5377a18c6f5cdf4ccd7d3c07d7c80b8f1f409f7924d88106b9db86\": rpc error: code = NotFound desc = could not find container \"cdc12e352a5377a18c6f5cdf4ccd7d3c07d7c80b8f1f409f7924d88106b9db86\": container with ID starting with cdc12e352a5377a18c6f5cdf4ccd7d3c07d7c80b8f1f409f7924d88106b9db86 not found: ID does not exist" Jul 20 02:51:12.745225 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.745177 2558 scope.go:117] "RemoveContainer" containerID="d3c3f2319689f0f448374b987914f00d7e950b93bb253eadc00efd75f1269bfa" Jul 20 02:51:12.745319 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.745305 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.745373 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.745339 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3c3f2319689f0f448374b987914f00d7e950b93bb253eadc00efd75f1269bfa"} err="failed to get container status \"d3c3f2319689f0f448374b987914f00d7e950b93bb253eadc00efd75f1269bfa\": rpc error: code = NotFound desc = could not find container \"d3c3f2319689f0f448374b987914f00d7e950b93bb253eadc00efd75f1269bfa\": container with ID starting with d3c3f2319689f0f448374b987914f00d7e950b93bb253eadc00efd75f1269bfa not found: ID does not exist" Jul 20 02:51:12.745373 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.745363 2558 scope.go:117] "RemoveContainer" containerID="0ff20bc1a0992a582fa8ad26c0f2ef2c54a35f27c097825bba7d5c4c55e370c8" Jul 20 02:51:12.745762 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.745556 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ff20bc1a0992a582fa8ad26c0f2ef2c54a35f27c097825bba7d5c4c55e370c8"} err="failed to get container status \"0ff20bc1a0992a582fa8ad26c0f2ef2c54a35f27c097825bba7d5c4c55e370c8\": rpc error: code = NotFound desc = could not find container \"0ff20bc1a0992a582fa8ad26c0f2ef2c54a35f27c097825bba7d5c4c55e370c8\": container with ID starting with 0ff20bc1a0992a582fa8ad26c0f2ef2c54a35f27c097825bba7d5c4c55e370c8 not found: ID does not exist" Jul 20 02:51:12.745762 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.745580 2558 scope.go:117] "RemoveContainer" containerID="259ed9af1282abb0563757e4b43308b21f71b13e55f3e5c80f3ad1a5e2154d6f" Jul 20 02:51:12.745955 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.745917 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"259ed9af1282abb0563757e4b43308b21f71b13e55f3e5c80f3ad1a5e2154d6f"} err="failed to get container status \"259ed9af1282abb0563757e4b43308b21f71b13e55f3e5c80f3ad1a5e2154d6f\": rpc error: code = NotFound desc = could not find container \"259ed9af1282abb0563757e4b43308b21f71b13e55f3e5c80f3ad1a5e2154d6f\": container with ID starting with 259ed9af1282abb0563757e4b43308b21f71b13e55f3e5c80f3ad1a5e2154d6f not found: ID does not exist" Jul 20 02:51:12.745955 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.745939 2558 scope.go:117] "RemoveContainer" containerID="931c8bb914b43643f2cbd7630c5c3ffaae8672cf2770c9a63615141ba5db773f" Jul 20 02:51:12.746200 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.746179 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"931c8bb914b43643f2cbd7630c5c3ffaae8672cf2770c9a63615141ba5db773f"} err="failed to get container status \"931c8bb914b43643f2cbd7630c5c3ffaae8672cf2770c9a63615141ba5db773f\": rpc error: code = NotFound desc = could not find container \"931c8bb914b43643f2cbd7630c5c3ffaae8672cf2770c9a63615141ba5db773f\": container with ID starting with 931c8bb914b43643f2cbd7630c5c3ffaae8672cf2770c9a63615141ba5db773f not found: ID does not exist" Jul 20 02:51:12.746269 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.746203 2558 scope.go:117] "RemoveContainer" containerID="454280bab7fcdcad8e564a92bac238ffdc6c89743cbe3660c537e50f70d4eea9" Jul 20 02:51:12.746456 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.746437 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"454280bab7fcdcad8e564a92bac238ffdc6c89743cbe3660c537e50f70d4eea9"} err="failed to get container status \"454280bab7fcdcad8e564a92bac238ffdc6c89743cbe3660c537e50f70d4eea9\": rpc error: code = NotFound desc = could not find container \"454280bab7fcdcad8e564a92bac238ffdc6c89743cbe3660c537e50f70d4eea9\": container with ID starting with 454280bab7fcdcad8e564a92bac238ffdc6c89743cbe3660c537e50f70d4eea9 not found: ID does not exist" Jul 20 02:51:12.746521 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.746456 2558 scope.go:117] "RemoveContainer" containerID="3394177568c88e299ac9ea0feab4432f10d6b7a2ebc05bed2a457ab6993f3b8e" Jul 20 02:51:12.746685 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.746668 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3394177568c88e299ac9ea0feab4432f10d6b7a2ebc05bed2a457ab6993f3b8e"} err="failed to get container status \"3394177568c88e299ac9ea0feab4432f10d6b7a2ebc05bed2a457ab6993f3b8e\": rpc error: code = NotFound desc = could not find container \"3394177568c88e299ac9ea0feab4432f10d6b7a2ebc05bed2a457ab6993f3b8e\": container with ID starting with 3394177568c88e299ac9ea0feab4432f10d6b7a2ebc05bed2a457ab6993f3b8e not found: ID does not exist" Jul 20 02:51:12.748082 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.748043 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 20 02:51:12.748298 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.748178 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 20 02:51:12.748402 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.748393 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fok20bnkg872k\"" Jul 20 02:51:12.748453 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.748393 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 20 02:51:12.748453 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.748447 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 20 02:51:12.748585 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.748511 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 20 02:51:12.748585 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.748534 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-c5zkf\"" Jul 20 02:51:12.748759 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.748745 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 20 02:51:12.748759 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.748754 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 20 02:51:12.748880 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.748764 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 20 02:51:12.748880 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.748776 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 20 02:51:12.748880 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.748809 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 20 02:51:12.749494 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.749476 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 20 02:51:12.749657 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.749623 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 20 02:51:12.749746 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.749639 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 20 02:51:12.752245 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.752220 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 20 02:51:12.754446 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.754421 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 20 02:51:12.756522 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.756504 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 20 02:51:12.868906 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.868870 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d7586ee8-a6a4-4488-bd31-e447d1dedf33-config\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.869272 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.868910 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d7586ee8-a6a4-4488-bd31-e447d1dedf33-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.869272 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.868937 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d7586ee8-a6a4-4488-bd31-e447d1dedf33-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.869272 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.869009 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d7586ee8-a6a4-4488-bd31-e447d1dedf33-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.869272 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.869039 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d7586ee8-a6a4-4488-bd31-e447d1dedf33-web-config\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.869272 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.869059 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d7586ee8-a6a4-4488-bd31-e447d1dedf33-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.869272 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.869075 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7586ee8-a6a4-4488-bd31-e447d1dedf33-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.869272 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.869124 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d7586ee8-a6a4-4488-bd31-e447d1dedf33-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.869272 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.869154 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vw9s\" (UniqueName: \"kubernetes.io/projected/d7586ee8-a6a4-4488-bd31-e447d1dedf33-kube-api-access-6vw9s\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.869272 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.869180 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7586ee8-a6a4-4488-bd31-e447d1dedf33-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.869272 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.869203 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d7586ee8-a6a4-4488-bd31-e447d1dedf33-config-out\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.869272 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.869222 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d7586ee8-a6a4-4488-bd31-e447d1dedf33-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.869272 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.869262 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d7586ee8-a6a4-4488-bd31-e447d1dedf33-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.869719 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.869293 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d7586ee8-a6a4-4488-bd31-e447d1dedf33-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.869719 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.869319 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d7586ee8-a6a4-4488-bd31-e447d1dedf33-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.869719 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.869350 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7586ee8-a6a4-4488-bd31-e447d1dedf33-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.869719 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.869376 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d7586ee8-a6a4-4488-bd31-e447d1dedf33-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.869719 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.869402 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d7586ee8-a6a4-4488-bd31-e447d1dedf33-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.869719 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.869447 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d7586ee8-a6a4-4488-bd31-e447d1dedf33-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.869719 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.869475 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d7586ee8-a6a4-4488-bd31-e447d1dedf33-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.970400 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.970312 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d7586ee8-a6a4-4488-bd31-e447d1dedf33-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.970400 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.970354 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7586ee8-a6a4-4488-bd31-e447d1dedf33-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.970400 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.970371 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d7586ee8-a6a4-4488-bd31-e447d1dedf33-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.970642 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.970475 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d7586ee8-a6a4-4488-bd31-e447d1dedf33-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.970642 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.970522 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d7586ee8-a6a4-4488-bd31-e447d1dedf33-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.970642 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.970542 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d7586ee8-a6a4-4488-bd31-e447d1dedf33-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.970642 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.970567 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d7586ee8-a6a4-4488-bd31-e447d1dedf33-config\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.970642 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.970583 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d7586ee8-a6a4-4488-bd31-e447d1dedf33-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.970642 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.970610 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d7586ee8-a6a4-4488-bd31-e447d1dedf33-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.970930 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.970647 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d7586ee8-a6a4-4488-bd31-e447d1dedf33-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.970930 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.970671 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d7586ee8-a6a4-4488-bd31-e447d1dedf33-web-config\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.970930 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.970696 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d7586ee8-a6a4-4488-bd31-e447d1dedf33-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.970930 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.970719 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7586ee8-a6a4-4488-bd31-e447d1dedf33-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.970930 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.970746 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d7586ee8-a6a4-4488-bd31-e447d1dedf33-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.970930 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.970774 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vw9s\" (UniqueName: \"kubernetes.io/projected/d7586ee8-a6a4-4488-bd31-e447d1dedf33-kube-api-access-6vw9s\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.970930 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.970806 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7586ee8-a6a4-4488-bd31-e447d1dedf33-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.970930 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.970837 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d7586ee8-a6a4-4488-bd31-e447d1dedf33-config-out\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.970930 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.970863 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d7586ee8-a6a4-4488-bd31-e447d1dedf33-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.970930 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.970888 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d7586ee8-a6a4-4488-bd31-e447d1dedf33-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.970930 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.970918 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d7586ee8-a6a4-4488-bd31-e447d1dedf33-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.971463 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.971266 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d7586ee8-a6a4-4488-bd31-e447d1dedf33-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.971463 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.971300 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d7586ee8-a6a4-4488-bd31-e447d1dedf33-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.971564 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.971476 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7586ee8-a6a4-4488-bd31-e447d1dedf33-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.971864 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.971840 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d7586ee8-a6a4-4488-bd31-e447d1dedf33-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.972133 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.972111 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d7586ee8-a6a4-4488-bd31-e447d1dedf33-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.975068 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.974023 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7586ee8-a6a4-4488-bd31-e447d1dedf33-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.975068 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.974592 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d7586ee8-a6a4-4488-bd31-e447d1dedf33-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.975068 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.974835 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d7586ee8-a6a4-4488-bd31-e447d1dedf33-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.975068 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.974934 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d7586ee8-a6a4-4488-bd31-e447d1dedf33-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.975068 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.974997 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d7586ee8-a6a4-4488-bd31-e447d1dedf33-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.975399 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.975143 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d7586ee8-a6a4-4488-bd31-e447d1dedf33-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.975399 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.975253 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d7586ee8-a6a4-4488-bd31-e447d1dedf33-config-out\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.975399 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.975145 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d7586ee8-a6a4-4488-bd31-e447d1dedf33-web-config\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.975399 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.975381 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d7586ee8-a6a4-4488-bd31-e447d1dedf33-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.975704 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.975679 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d7586ee8-a6a4-4488-bd31-e447d1dedf33-config\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.975779 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.975731 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7586ee8-a6a4-4488-bd31-e447d1dedf33-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.976229 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.976196 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d7586ee8-a6a4-4488-bd31-e447d1dedf33-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.977049 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.977027 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d7586ee8-a6a4-4488-bd31-e447d1dedf33-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.977149 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.977071 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d7586ee8-a6a4-4488-bd31-e447d1dedf33-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:12.980316 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:12.980297 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vw9s\" (UniqueName: \"kubernetes.io/projected/d7586ee8-a6a4-4488-bd31-e447d1dedf33-kube-api-access-6vw9s\") pod \"prometheus-k8s-0\" (UID: \"d7586ee8-a6a4-4488-bd31-e447d1dedf33\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:13.054836 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:13.054805 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:13.192892 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:13.192864 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 20 02:51:13.194865 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:51:13.194835 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7586ee8_a6a4_4488_bd31_e447d1dedf33.slice/crio-81eb7a30114b306736b20a6ec8afc3ed1eca78a0ac43f53057a5b961e5433b69 WatchSource:0}: Error finding container 81eb7a30114b306736b20a6ec8afc3ed1eca78a0ac43f53057a5b961e5433b69: Status 404 returned error can't find the container with id 81eb7a30114b306736b20a6ec8afc3ed1eca78a0ac43f53057a5b961e5433b69 Jul 20 02:51:13.422834 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:13.422800 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01b09630-9c37-419f-8ad8-e0b97aa6df82" path="/var/lib/kubelet/pods/01b09630-9c37-419f-8ad8-e0b97aa6df82/volumes" Jul 20 02:51:13.689416 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:13.689379 2558 generic.go:356] "Generic (PLEG): container finished" podID="d7586ee8-a6a4-4488-bd31-e447d1dedf33" containerID="8778437afe44e8ac11a3024d14d5a2694563c1dee5f723b1036954fcd4b5cb77" exitCode=0 Jul 20 02:51:13.689567 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:13.689466 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d7586ee8-a6a4-4488-bd31-e447d1dedf33","Type":"ContainerDied","Data":"8778437afe44e8ac11a3024d14d5a2694563c1dee5f723b1036954fcd4b5cb77"} Jul 20 02:51:13.689567 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:13.689533 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d7586ee8-a6a4-4488-bd31-e447d1dedf33","Type":"ContainerStarted","Data":"81eb7a30114b306736b20a6ec8afc3ed1eca78a0ac43f53057a5b961e5433b69"} Jul 20 02:51:14.697916 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:14.697886 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d7586ee8-a6a4-4488-bd31-e447d1dedf33","Type":"ContainerStarted","Data":"cd7036640c4fc47101316822ba0adbe8e0586a8136da9517d273d659f8991b2f"} Jul 20 02:51:14.698312 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:14.697945 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d7586ee8-a6a4-4488-bd31-e447d1dedf33","Type":"ContainerStarted","Data":"5668948df5bae7a1cc46d875278e52aa7bec1dbd63a71a802bcd7ae419541cf6"} Jul 20 02:51:14.698312 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:14.697974 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d7586ee8-a6a4-4488-bd31-e447d1dedf33","Type":"ContainerStarted","Data":"a4979bde9c4cbceb1c7bb3b798097c20c69ca5dacfee571b7c273312fe1c0355"} Jul 20 02:51:14.698312 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:14.697998 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d7586ee8-a6a4-4488-bd31-e447d1dedf33","Type":"ContainerStarted","Data":"730b1af304fc3fcd8cc08c770e925c3e370860c7c2087be8c7f7bcb25606d6cd"} Jul 20 02:51:14.698312 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:14.698020 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d7586ee8-a6a4-4488-bd31-e447d1dedf33","Type":"ContainerStarted","Data":"2b6e35d1af588aec13d43ce55dafa9940f746b96b16607f55ec93617bcbc0b2e"} Jul 20 02:51:14.698312 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:14.698042 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d7586ee8-a6a4-4488-bd31-e447d1dedf33","Type":"ContainerStarted","Data":"a38acaffc585f49addb151df26ac7d5ab867347a1823ed9229ae7f3f021ba387"} Jul 20 02:51:14.725390 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:14.725319 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.705954905 podStartE2EDuration="2.72530486s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.349955ms" isStatefulPod=false podCreationTimestamp="2026-07-20 02:51:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:51:14.72326804 +0000 UTC m=+369.922517180" watchObservedRunningTime="2026-07-20 02:51:14.72530486 +0000 UTC m=+369.924554069" Jul 20 02:51:18.055799 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:18.055761 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:51:19.121403 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:19.121354 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d70f5103-7dc9-4279-9781-38e395da435b-metrics-certs\") pod \"network-metrics-daemon-9x7hv\" (UID: \"d70f5103-7dc9-4279-9781-38e395da435b\") " pod="openshift-multus/network-metrics-daemon-9x7hv" Jul 20 02:51:19.123809 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:19.123784 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d70f5103-7dc9-4279-9781-38e395da435b-metrics-certs\") pod \"network-metrics-daemon-9x7hv\" (UID: \"d70f5103-7dc9-4279-9781-38e395da435b\") " pod="openshift-multus/network-metrics-daemon-9x7hv" Jul 20 02:51:19.222376 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:19.222343 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8vd4r\"" Jul 20 02:51:19.230259 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:19.230227 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9x7hv" Jul 20 02:51:19.356227 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:19.356197 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9x7hv"] Jul 20 02:51:19.359518 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:51:19.359492 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd70f5103_7dc9_4279_9781_38e395da435b.slice/crio-a35b1f9fb9e9f9694ae0510deb432c749e0a247d6e85c6e7a03bbf84e44abcb2 WatchSource:0}: Error finding container a35b1f9fb9e9f9694ae0510deb432c749e0a247d6e85c6e7a03bbf84e44abcb2: Status 404 returned error can't find the container with id a35b1f9fb9e9f9694ae0510deb432c749e0a247d6e85c6e7a03bbf84e44abcb2 Jul 20 02:51:19.361410 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:19.361392 2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 20 02:51:19.717845 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:19.717812 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9x7hv" event={"ID":"d70f5103-7dc9-4279-9781-38e395da435b","Type":"ContainerStarted","Data":"a35b1f9fb9e9f9694ae0510deb432c749e0a247d6e85c6e7a03bbf84e44abcb2"} Jul 20 02:51:20.723447 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:20.723414 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9x7hv" event={"ID":"d70f5103-7dc9-4279-9781-38e395da435b","Type":"ContainerStarted","Data":"bfbe2a79629fd18fb9b7dbce85015e7842d299afb8e2408455e5035f78daa373"} Jul 20 02:51:20.723861 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:20.723493 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9x7hv" event={"ID":"d70f5103-7dc9-4279-9781-38e395da435b","Type":"ContainerStarted","Data":"20234a53e7480eb5c1b9cf838d7d22d539db1a11c31ee35425dac16c666065d1"} Jul 20 02:51:20.737783 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:51:20.737732 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9x7hv" podStartSLOduration=374.83770793 podStartE2EDuration="6m15.737716149s" totalImagesPullingTime="900.008219ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:45:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:51:20.737171414 +0000 UTC m=+375.936420560" watchObservedRunningTime="2026-07-20 02:51:20.737716149 +0000 UTC m=+375.936965295" Jul 20 02:52:13.055675 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:52:13.055637 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:52:13.071151 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:52:13.071126 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:52:13.934565 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:52:13.934529 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 20 02:53:59.061658 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:53:59.061575 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-dx2dc"] Jul 20 02:53:59.064689 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:53:59.064673 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-dx2dc" Jul 20 02:53:59.067179 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:53:59.067156 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 20 02:53:59.068267 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:53:59.068248 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 20 02:53:59.068267 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:53:59.068261 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-hbb25\"" Jul 20 02:53:59.072087 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:53:59.072067 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-dx2dc"] Jul 20 02:53:59.111315 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:53:59.111283 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0b75de0f-c0d9-4bb7-823d-5d6083303e9e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-dx2dc\" (UID: \"0b75de0f-c0d9-4bb7-823d-5d6083303e9e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-dx2dc" Jul 20 02:53:59.111427 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:53:59.111327 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftzlj\" (UniqueName: \"kubernetes.io/projected/0b75de0f-c0d9-4bb7-823d-5d6083303e9e-kube-api-access-ftzlj\") pod \"cert-manager-cainjector-559cdd8c87-dx2dc\" (UID: \"0b75de0f-c0d9-4bb7-823d-5d6083303e9e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-dx2dc" Jul 20 02:53:59.212382 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:53:59.212344 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0b75de0f-c0d9-4bb7-823d-5d6083303e9e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-dx2dc\" (UID: \"0b75de0f-c0d9-4bb7-823d-5d6083303e9e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-dx2dc" Jul 20 02:53:59.212547 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:53:59.212389 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftzlj\" (UniqueName: \"kubernetes.io/projected/0b75de0f-c0d9-4bb7-823d-5d6083303e9e-kube-api-access-ftzlj\") pod \"cert-manager-cainjector-559cdd8c87-dx2dc\" (UID: \"0b75de0f-c0d9-4bb7-823d-5d6083303e9e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-dx2dc" Jul 20 02:53:59.220173 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:53:59.220145 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0b75de0f-c0d9-4bb7-823d-5d6083303e9e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-dx2dc\" (UID: \"0b75de0f-c0d9-4bb7-823d-5d6083303e9e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-dx2dc" Jul 20 02:53:59.220448 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:53:59.220424 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftzlj\" (UniqueName: \"kubernetes.io/projected/0b75de0f-c0d9-4bb7-823d-5d6083303e9e-kube-api-access-ftzlj\") pod \"cert-manager-cainjector-559cdd8c87-dx2dc\" (UID: \"0b75de0f-c0d9-4bb7-823d-5d6083303e9e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-dx2dc" Jul 20 02:53:59.374547 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:53:59.374517 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-dx2dc" Jul 20 02:53:59.498867 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:53:59.498844 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-dx2dc"] Jul 20 02:53:59.501343 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:53:59.501318 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b75de0f_c0d9_4bb7_823d_5d6083303e9e.slice/crio-50ce69928f1ff10b3b334f20eebe508a71bd5e5c9ebea92dbbc19fea169b6aad WatchSource:0}: Error finding container 50ce69928f1ff10b3b334f20eebe508a71bd5e5c9ebea92dbbc19fea169b6aad: Status 404 returned error can't find the container with id 50ce69928f1ff10b3b334f20eebe508a71bd5e5c9ebea92dbbc19fea169b6aad Jul 20 02:54:00.299933 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:54:00.299888 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-dx2dc" event={"ID":"0b75de0f-c0d9-4bb7-823d-5d6083303e9e","Type":"ContainerStarted","Data":"50ce69928f1ff10b3b334f20eebe508a71bd5e5c9ebea92dbbc19fea169b6aad"} Jul 20 02:54:03.315196 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:54:03.315161 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-dx2dc" event={"ID":"0b75de0f-c0d9-4bb7-823d-5d6083303e9e","Type":"ContainerStarted","Data":"e133a92e4ae500f09d8317122d430b18fbd00cb282c5d93cb2cbbb8ebe5fc31e"} Jul 20 02:54:03.334706 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:54:03.334653 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-dx2dc" podStartSLOduration=1.308686523 podStartE2EDuration="4.334637983s" totalImagesPullingTime="3.02595146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:53:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:54:03.333807628 +0000 UTC m=+538.533056776" watchObservedRunningTime="2026-07-20 02:54:03.334637983 +0000 UTC m=+538.533887130" Jul 20 02:54:07.750557 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:54:07.750522 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-mc6tg"] Jul 20 02:54:07.753830 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:54:07.753809 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-mc6tg" Jul 20 02:54:07.756452 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:54:07.756432 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-97cbg\"" Jul 20 02:54:07.761000 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:54:07.760976 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-mc6tg"] Jul 20 02:54:07.780623 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:54:07.780591 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8h9c\" (UniqueName: \"kubernetes.io/projected/59c94727-9b73-4fa3-b451-eae7a656681f-kube-api-access-g8h9c\") pod \"cert-manager-55b4d954d5-mc6tg\" (UID: \"59c94727-9b73-4fa3-b451-eae7a656681f\") " pod="cert-manager/cert-manager-55b4d954d5-mc6tg" Jul 20 02:54:07.780753 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:54:07.780631 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/59c94727-9b73-4fa3-b451-eae7a656681f-bound-sa-token\") pod \"cert-manager-55b4d954d5-mc6tg\" (UID: \"59c94727-9b73-4fa3-b451-eae7a656681f\") " pod="cert-manager/cert-manager-55b4d954d5-mc6tg" Jul 20 02:54:07.881639 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:54:07.881605 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8h9c\" (UniqueName: \"kubernetes.io/projected/59c94727-9b73-4fa3-b451-eae7a656681f-kube-api-access-g8h9c\") pod \"cert-manager-55b4d954d5-mc6tg\" (UID: \"59c94727-9b73-4fa3-b451-eae7a656681f\") " pod="cert-manager/cert-manager-55b4d954d5-mc6tg" Jul 20 02:54:07.881639 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:54:07.881646 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/59c94727-9b73-4fa3-b451-eae7a656681f-bound-sa-token\") pod \"cert-manager-55b4d954d5-mc6tg\" (UID: \"59c94727-9b73-4fa3-b451-eae7a656681f\") " pod="cert-manager/cert-manager-55b4d954d5-mc6tg" Jul 20 02:54:07.889796 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:54:07.889762 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/59c94727-9b73-4fa3-b451-eae7a656681f-bound-sa-token\") pod \"cert-manager-55b4d954d5-mc6tg\" (UID: \"59c94727-9b73-4fa3-b451-eae7a656681f\") " pod="cert-manager/cert-manager-55b4d954d5-mc6tg" Jul 20 02:54:07.889937 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:54:07.889919 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8h9c\" (UniqueName: \"kubernetes.io/projected/59c94727-9b73-4fa3-b451-eae7a656681f-kube-api-access-g8h9c\") pod \"cert-manager-55b4d954d5-mc6tg\" (UID: \"59c94727-9b73-4fa3-b451-eae7a656681f\") " pod="cert-manager/cert-manager-55b4d954d5-mc6tg" Jul 20 02:54:08.063434 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:54:08.063342 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-mc6tg" Jul 20 02:54:08.187646 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:54:08.187462 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-mc6tg"] Jul 20 02:54:08.190145 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:54:08.190112 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59c94727_9b73_4fa3_b451_eae7a656681f.slice/crio-a45a2cde3553f25d5b9642f31f40ef9674c19bfad26c04cfa41597ff96b9f38b WatchSource:0}: Error finding container a45a2cde3553f25d5b9642f31f40ef9674c19bfad26c04cfa41597ff96b9f38b: Status 404 returned error can't find the container with id a45a2cde3553f25d5b9642f31f40ef9674c19bfad26c04cfa41597ff96b9f38b Jul 20 02:54:08.339286 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:54:08.339202 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-mc6tg" event={"ID":"59c94727-9b73-4fa3-b451-eae7a656681f","Type":"ContainerStarted","Data":"a530a7fe0ce2613e1a10dff32d0ebb1b59a4b2a67f5dd6297235e86be9666be2"} Jul 20 02:54:08.339286 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:54:08.339242 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-mc6tg" event={"ID":"59c94727-9b73-4fa3-b451-eae7a656681f","Type":"ContainerStarted","Data":"a45a2cde3553f25d5b9642f31f40ef9674c19bfad26c04cfa41597ff96b9f38b"} Jul 20 02:54:08.353700 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:54:08.353652 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-mc6tg" podStartSLOduration=1.353636703 podStartE2EDuration="1.353636703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:54:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:54:08.353592587 +0000 UTC m=+543.552841735" watchObservedRunningTime="2026-07-20 02:54:08.353636703 +0000 UTC m=+543.552885849" Jul 20 02:54:34.849090 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:54:34.849039 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f8fbd65f7-gshjj"] Jul 20 02:54:34.852323 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:54:34.852305 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5f8fbd65f7-gshjj" Jul 20 02:54:34.855815 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:54:34.855794 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 20 02:54:34.856124 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:54:34.856085 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 20 02:54:34.856242 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:54:34.856088 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-lp5sr\"" Jul 20 02:54:34.856242 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:54:34.856142 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 20 02:54:34.856242 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:54:34.856193 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 20 02:54:34.870029 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:54:34.870006 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f8fbd65f7-gshjj"] Jul 20 02:54:34.995204 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:54:34.995170 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7231656-ed40-406a-8346-c9675cab218c-tmp\") pod \"opendatahub-operator-controller-manager-5f8fbd65f7-gshjj\" (UID: \"f7231656-ed40-406a-8346-c9675cab218c\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8fbd65f7-gshjj" Jul 20 02:54:34.995204 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:54:34.995204 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f7231656-ed40-406a-8346-c9675cab218c-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f8fbd65f7-gshjj\" (UID: \"f7231656-ed40-406a-8346-c9675cab218c\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8fbd65f7-gshjj" Jul 20 02:54:34.995403 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:54:34.995292 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4kx9\" (UniqueName: \"kubernetes.io/projected/f7231656-ed40-406a-8346-c9675cab218c-kube-api-access-z4kx9\") pod \"opendatahub-operator-controller-manager-5f8fbd65f7-gshjj\" (UID: \"f7231656-ed40-406a-8346-c9675cab218c\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8fbd65f7-gshjj" Jul 20 02:54:34.995403 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:54:34.995349 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f7231656-ed40-406a-8346-c9675cab218c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f8fbd65f7-gshjj\" (UID: \"f7231656-ed40-406a-8346-c9675cab218c\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8fbd65f7-gshjj" Jul 20 02:54:34.995403 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:54:34.995379 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f7231656-ed40-406a-8346-c9675cab218c-manifests\") pod \"opendatahub-operator-controller-manager-5f8fbd65f7-gshjj\" (UID: \"f7231656-ed40-406a-8346-c9675cab218c\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8fbd65f7-gshjj" Jul 20 02:54:35.096206 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:54:35.096172 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4kx9\" (UniqueName: \"kubernetes.io/projected/f7231656-ed40-406a-8346-c9675cab218c-kube-api-access-z4kx9\") pod \"opendatahub-operator-controller-manager-5f8fbd65f7-gshjj\" (UID: \"f7231656-ed40-406a-8346-c9675cab218c\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8fbd65f7-gshjj" Jul 20 02:54:35.096364 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:54:35.096214 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f7231656-ed40-406a-8346-c9675cab218c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f8fbd65f7-gshjj\" (UID: \"f7231656-ed40-406a-8346-c9675cab218c\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8fbd65f7-gshjj" Jul 20 02:54:35.096364 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:54:35.096237 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f7231656-ed40-406a-8346-c9675cab218c-manifests\") pod \"opendatahub-operator-controller-manager-5f8fbd65f7-gshjj\" (UID: \"f7231656-ed40-406a-8346-c9675cab218c\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8fbd65f7-gshjj" Jul 20 02:54:35.096364 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:54:35.096294 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7231656-ed40-406a-8346-c9675cab218c-tmp\") pod \"opendatahub-operator-controller-manager-5f8fbd65f7-gshjj\" (UID: \"f7231656-ed40-406a-8346-c9675cab218c\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8fbd65f7-gshjj" Jul 20 02:54:35.096364 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:54:35.096310 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f7231656-ed40-406a-8346-c9675cab218c-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f8fbd65f7-gshjj\" (UID: \"f7231656-ed40-406a-8346-c9675cab218c\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8fbd65f7-gshjj" Jul 20 02:54:35.096773 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:54:35.096746 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f7231656-ed40-406a-8346-c9675cab218c-manifests\") pod \"opendatahub-operator-controller-manager-5f8fbd65f7-gshjj\" (UID: \"f7231656-ed40-406a-8346-c9675cab218c\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8fbd65f7-gshjj" Jul 20 02:54:35.098736 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:54:35.098714 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7231656-ed40-406a-8346-c9675cab218c-tmp\") pod \"opendatahub-operator-controller-manager-5f8fbd65f7-gshjj\" (UID: \"f7231656-ed40-406a-8346-c9675cab218c\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8fbd65f7-gshjj" Jul 20 02:54:35.098896 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:54:35.098877 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f7231656-ed40-406a-8346-c9675cab218c-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f8fbd65f7-gshjj\" (UID: \"f7231656-ed40-406a-8346-c9675cab218c\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8fbd65f7-gshjj" Jul 20 02:54:35.098957 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:54:35.098875 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f7231656-ed40-406a-8346-c9675cab218c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f8fbd65f7-gshjj\" (UID: \"f7231656-ed40-406a-8346-c9675cab218c\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8fbd65f7-gshjj" Jul 20 02:54:35.103944 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:54:35.103889 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4kx9\" (UniqueName: \"kubernetes.io/projected/f7231656-ed40-406a-8346-c9675cab218c-kube-api-access-z4kx9\") pod \"opendatahub-operator-controller-manager-5f8fbd65f7-gshjj\" (UID: \"f7231656-ed40-406a-8346-c9675cab218c\") " pod="opendatahub/opendatahub-operator-controller-manager-5f8fbd65f7-gshjj" Jul 20 02:54:35.163041 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:54:35.163013 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5f8fbd65f7-gshjj" Jul 20 02:54:35.297604 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:54:35.297575 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f8fbd65f7-gshjj"] Jul 20 02:54:35.301797 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:54:35.301763 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7231656_ed40_406a_8346_c9675cab218c.slice/crio-14937695ffa8267a1b00fec4cd8787e27ad6ceae67f35497da6f5db81f84758d WatchSource:0}: Error finding container 14937695ffa8267a1b00fec4cd8787e27ad6ceae67f35497da6f5db81f84758d: Status 404 returned error can't find the container with id 14937695ffa8267a1b00fec4cd8787e27ad6ceae67f35497da6f5db81f84758d Jul 20 02:54:35.439043 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:54:35.439007 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f8fbd65f7-gshjj" event={"ID":"f7231656-ed40-406a-8346-c9675cab218c","Type":"ContainerStarted","Data":"14937695ffa8267a1b00fec4cd8787e27ad6ceae67f35497da6f5db81f84758d"} Jul 20 02:54:38.455130 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:54:38.455070 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f8fbd65f7-gshjj" event={"ID":"f7231656-ed40-406a-8346-c9675cab218c","Type":"ContainerStarted","Data":"0b6773f8716ffaa1a6e8931e8ecc07f9822367582ed597eaf09e25bc45436b51"} Jul 20 02:54:39.461271 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:54:39.461237 2558 generic.go:356] "Generic (PLEG): container finished" podID="f7231656-ed40-406a-8346-c9675cab218c" containerID="0b6773f8716ffaa1a6e8931e8ecc07f9822367582ed597eaf09e25bc45436b51" exitCode=0 Jul 20 02:54:39.461271 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:54:39.461278 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f8fbd65f7-gshjj" event={"ID":"f7231656-ed40-406a-8346-c9675cab218c","Type":"ContainerDied","Data":"0b6773f8716ffaa1a6e8931e8ecc07f9822367582ed597eaf09e25bc45436b51"} Jul 20 02:54:40.468498 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:54:40.468455 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f8fbd65f7-gshjj" event={"ID":"f7231656-ed40-406a-8346-c9675cab218c","Type":"ContainerStarted","Data":"bcf5fdc2fdd8018fab864a46d638a821ef0a252c3a9b642a5601cfba2f5005d4"} Jul 20 02:54:40.468937 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:54:40.468600 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5f8fbd65f7-gshjj" Jul 20 02:54:40.492168 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:54:40.492117 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5f8fbd65f7-gshjj" podStartSLOduration=2.17103411 podStartE2EDuration="6.492083109s" totalImagesPullingTime="3.303429906s" totalInitContainerRuntime="1.017619093s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:54:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:54:40.491333387 +0000 UTC m=+575.690582536" watchObservedRunningTime="2026-07-20 02:54:40.492083109 +0000 UTC m=+575.691332320" Jul 20 02:54:51.474441 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:54:51.474409 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5f8fbd65f7-gshjj" Jul 20 02:55:02.561820 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:02.561785 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788lhvx"] Jul 20 02:55:02.565122 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:02.565105 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788lhvx" Jul 20 02:55:02.568857 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:02.568832 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 20 02:55:02.568857 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:02.568852 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 20 02:55:02.569049 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:02.568935 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-2n7bs\"" Jul 20 02:55:02.569337 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:02.569319 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 20 02:55:02.595121 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:02.595078 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788lhvx"] Jul 20 02:55:02.614789 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:02.614758 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f8ddfdf-af55-4672-9e10-2f4851dacd18-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6788lhvx\" (UID: \"1f8ddfdf-af55-4672-9e10-2f4851dacd18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788lhvx" Jul 20 02:55:02.614897 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:02.614796 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f8ddfdf-af55-4672-9e10-2f4851dacd18-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6788lhvx\" (UID: \"1f8ddfdf-af55-4672-9e10-2f4851dacd18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788lhvx" Jul 20 02:55:02.614897 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:02.614868 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f8ddfdf-af55-4672-9e10-2f4851dacd18-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6788lhvx\" (UID: \"1f8ddfdf-af55-4672-9e10-2f4851dacd18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788lhvx" Jul 20 02:55:02.614965 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:02.614895 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f8ddfdf-af55-4672-9e10-2f4851dacd18-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6788lhvx\" (UID: \"1f8ddfdf-af55-4672-9e10-2f4851dacd18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788lhvx" Jul 20 02:55:02.614965 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:02.614927 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f8ddfdf-af55-4672-9e10-2f4851dacd18-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6788lhvx\" (UID: \"1f8ddfdf-af55-4672-9e10-2f4851dacd18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788lhvx" Jul 20 02:55:02.614965 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:02.614951 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f8ddfdf-af55-4672-9e10-2f4851dacd18-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6788lhvx\" (UID: \"1f8ddfdf-af55-4672-9e10-2f4851dacd18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788lhvx" Jul 20 02:55:02.615076 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:02.614978 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn78s\" (UniqueName: \"kubernetes.io/projected/1f8ddfdf-af55-4672-9e10-2f4851dacd18-kube-api-access-bn78s\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6788lhvx\" (UID: \"1f8ddfdf-af55-4672-9e10-2f4851dacd18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788lhvx" Jul 20 02:55:02.615076 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:02.615001 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f8ddfdf-af55-4672-9e10-2f4851dacd18-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6788lhvx\" (UID: \"1f8ddfdf-af55-4672-9e10-2f4851dacd18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788lhvx" Jul 20 02:55:02.615076 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:02.615059 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f8ddfdf-af55-4672-9e10-2f4851dacd18-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6788lhvx\" (UID: \"1f8ddfdf-af55-4672-9e10-2f4851dacd18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788lhvx" Jul 20 02:55:02.715788 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:02.715753 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f8ddfdf-af55-4672-9e10-2f4851dacd18-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6788lhvx\" (UID: \"1f8ddfdf-af55-4672-9e10-2f4851dacd18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788lhvx" Jul 20 02:55:02.715970 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:02.715799 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f8ddfdf-af55-4672-9e10-2f4851dacd18-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6788lhvx\" (UID: \"1f8ddfdf-af55-4672-9e10-2f4851dacd18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788lhvx" Jul 20 02:55:02.715970 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:02.715822 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f8ddfdf-af55-4672-9e10-2f4851dacd18-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6788lhvx\" (UID: \"1f8ddfdf-af55-4672-9e10-2f4851dacd18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788lhvx" Jul 20 02:55:02.715970 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:02.715942 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f8ddfdf-af55-4672-9e10-2f4851dacd18-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6788lhvx\" (UID: \"1f8ddfdf-af55-4672-9e10-2f4851dacd18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788lhvx" Jul 20 02:55:02.716178 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:02.715997 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f8ddfdf-af55-4672-9e10-2f4851dacd18-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6788lhvx\" (UID: \"1f8ddfdf-af55-4672-9e10-2f4851dacd18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788lhvx" Jul 20 02:55:02.716178 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:02.716029 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn78s\" (UniqueName: \"kubernetes.io/projected/1f8ddfdf-af55-4672-9e10-2f4851dacd18-kube-api-access-bn78s\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6788lhvx\" (UID: \"1f8ddfdf-af55-4672-9e10-2f4851dacd18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788lhvx" Jul 20 02:55:02.716178 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:02.716066 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f8ddfdf-af55-4672-9e10-2f4851dacd18-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6788lhvx\" (UID: \"1f8ddfdf-af55-4672-9e10-2f4851dacd18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788lhvx" Jul 20 02:55:02.716178 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:02.716150 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f8ddfdf-af55-4672-9e10-2f4851dacd18-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6788lhvx\" (UID: \"1f8ddfdf-af55-4672-9e10-2f4851dacd18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788lhvx" Jul 20 02:55:02.716386 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:02.716198 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f8ddfdf-af55-4672-9e10-2f4851dacd18-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6788lhvx\" (UID: \"1f8ddfdf-af55-4672-9e10-2f4851dacd18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788lhvx" Jul 20 02:55:02.716386 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:02.716350 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f8ddfdf-af55-4672-9e10-2f4851dacd18-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6788lhvx\" (UID: \"1f8ddfdf-af55-4672-9e10-2f4851dacd18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788lhvx" Jul 20 02:55:02.716565 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:02.716541 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f8ddfdf-af55-4672-9e10-2f4851dacd18-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6788lhvx\" (UID: \"1f8ddfdf-af55-4672-9e10-2f4851dacd18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788lhvx" Jul 20 02:55:02.716723 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:02.716571 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f8ddfdf-af55-4672-9e10-2f4851dacd18-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6788lhvx\" (UID: \"1f8ddfdf-af55-4672-9e10-2f4851dacd18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788lhvx" Jul 20 02:55:02.716723 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:02.716602 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f8ddfdf-af55-4672-9e10-2f4851dacd18-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6788lhvx\" (UID: \"1f8ddfdf-af55-4672-9e10-2f4851dacd18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788lhvx" Jul 20 02:55:02.716859 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:02.716832 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f8ddfdf-af55-4672-9e10-2f4851dacd18-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6788lhvx\" (UID: \"1f8ddfdf-af55-4672-9e10-2f4851dacd18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788lhvx" Jul 20 02:55:02.718205 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:02.718182 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f8ddfdf-af55-4672-9e10-2f4851dacd18-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6788lhvx\" (UID: \"1f8ddfdf-af55-4672-9e10-2f4851dacd18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788lhvx" Jul 20 02:55:02.718433 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:02.718417 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f8ddfdf-af55-4672-9e10-2f4851dacd18-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6788lhvx\" (UID: \"1f8ddfdf-af55-4672-9e10-2f4851dacd18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788lhvx" Jul 20 02:55:02.727815 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:02.727782 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f8ddfdf-af55-4672-9e10-2f4851dacd18-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6788lhvx\" (UID: \"1f8ddfdf-af55-4672-9e10-2f4851dacd18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788lhvx" Jul 20 02:55:02.728084 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:02.728063 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn78s\" (UniqueName: \"kubernetes.io/projected/1f8ddfdf-af55-4672-9e10-2f4851dacd18-kube-api-access-bn78s\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6788lhvx\" (UID: \"1f8ddfdf-af55-4672-9e10-2f4851dacd18\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788lhvx" Jul 20 02:55:02.875458 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:02.875421 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788lhvx" Jul 20 02:55:03.025474 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:03.025449 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788lhvx"] Jul 20 02:55:03.027795 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:55:03.027765 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f8ddfdf_af55_4672_9e10_2f4851dacd18.slice/crio-efb117d910a0587528fbdba8ca4438676383578c575e10499b73478340dbe9c6 WatchSource:0}: Error finding container efb117d910a0587528fbdba8ca4438676383578c575e10499b73478340dbe9c6: Status 404 returned error can't find the container with id efb117d910a0587528fbdba8ca4438676383578c575e10499b73478340dbe9c6 Jul 20 02:55:03.169209 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:03.169126 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-66bc978c4c-b8hqq"] Jul 20 02:55:03.173638 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:03.173616 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-66bc978c4c-b8hqq" Jul 20 02:55:03.176359 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:03.176336 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 20 02:55:03.176359 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:03.176341 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-tnjpf\"" Jul 20 02:55:03.176505 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:03.176420 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 20 02:55:03.176606 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:03.176594 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 20 02:55:03.176642 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:03.176600 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 20 02:55:03.177671 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:03.177651 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 20 02:55:03.181710 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:03.181641 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-66bc978c4c-b8hqq"] Jul 20 02:55:03.219957 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:03.219921 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b9f6f098-9594-4a0a-aac2-6fa4fb073b85-metrics-cert\") pod \"lws-controller-manager-66bc978c4c-b8hqq\" (UID: \"b9f6f098-9594-4a0a-aac2-6fa4fb073b85\") " pod="openshift-lws-operator/lws-controller-manager-66bc978c4c-b8hqq" Jul 20 02:55:03.220138 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:03.219994 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b9f6f098-9594-4a0a-aac2-6fa4fb073b85-manager-config\") pod \"lws-controller-manager-66bc978c4c-b8hqq\" (UID: \"b9f6f098-9594-4a0a-aac2-6fa4fb073b85\") " pod="openshift-lws-operator/lws-controller-manager-66bc978c4c-b8hqq" Jul 20 02:55:03.220138 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:03.220070 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ltq6\" (UniqueName: \"kubernetes.io/projected/b9f6f098-9594-4a0a-aac2-6fa4fb073b85-kube-api-access-8ltq6\") pod \"lws-controller-manager-66bc978c4c-b8hqq\" (UID: \"b9f6f098-9594-4a0a-aac2-6fa4fb073b85\") " pod="openshift-lws-operator/lws-controller-manager-66bc978c4c-b8hqq" Jul 20 02:55:03.220138 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:03.220115 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b9f6f098-9594-4a0a-aac2-6fa4fb073b85-cert\") pod \"lws-controller-manager-66bc978c4c-b8hqq\" (UID: \"b9f6f098-9594-4a0a-aac2-6fa4fb073b85\") " pod="openshift-lws-operator/lws-controller-manager-66bc978c4c-b8hqq" Jul 20 02:55:03.321337 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:03.321301 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ltq6\" (UniqueName: \"kubernetes.io/projected/b9f6f098-9594-4a0a-aac2-6fa4fb073b85-kube-api-access-8ltq6\") pod \"lws-controller-manager-66bc978c4c-b8hqq\" (UID: \"b9f6f098-9594-4a0a-aac2-6fa4fb073b85\") " pod="openshift-lws-operator/lws-controller-manager-66bc978c4c-b8hqq" Jul 20 02:55:03.321522 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:03.321343 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b9f6f098-9594-4a0a-aac2-6fa4fb073b85-cert\") pod \"lws-controller-manager-66bc978c4c-b8hqq\" (UID: \"b9f6f098-9594-4a0a-aac2-6fa4fb073b85\") " pod="openshift-lws-operator/lws-controller-manager-66bc978c4c-b8hqq" Jul 20 02:55:03.321522 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:03.321376 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b9f6f098-9594-4a0a-aac2-6fa4fb073b85-metrics-cert\") pod \"lws-controller-manager-66bc978c4c-b8hqq\" (UID: \"b9f6f098-9594-4a0a-aac2-6fa4fb073b85\") " pod="openshift-lws-operator/lws-controller-manager-66bc978c4c-b8hqq" Jul 20 02:55:03.321522 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:03.321401 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b9f6f098-9594-4a0a-aac2-6fa4fb073b85-manager-config\") pod \"lws-controller-manager-66bc978c4c-b8hqq\" (UID: \"b9f6f098-9594-4a0a-aac2-6fa4fb073b85\") " pod="openshift-lws-operator/lws-controller-manager-66bc978c4c-b8hqq" Jul 20 02:55:03.322160 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:03.322132 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b9f6f098-9594-4a0a-aac2-6fa4fb073b85-manager-config\") pod \"lws-controller-manager-66bc978c4c-b8hqq\" (UID: \"b9f6f098-9594-4a0a-aac2-6fa4fb073b85\") " pod="openshift-lws-operator/lws-controller-manager-66bc978c4c-b8hqq" Jul 20 02:55:03.323899 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:03.323879 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b9f6f098-9594-4a0a-aac2-6fa4fb073b85-metrics-cert\") pod \"lws-controller-manager-66bc978c4c-b8hqq\" (UID: \"b9f6f098-9594-4a0a-aac2-6fa4fb073b85\") " pod="openshift-lws-operator/lws-controller-manager-66bc978c4c-b8hqq" Jul 20 02:55:03.324012 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:03.323995 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b9f6f098-9594-4a0a-aac2-6fa4fb073b85-cert\") pod \"lws-controller-manager-66bc978c4c-b8hqq\" (UID: \"b9f6f098-9594-4a0a-aac2-6fa4fb073b85\") " pod="openshift-lws-operator/lws-controller-manager-66bc978c4c-b8hqq" Jul 20 02:55:03.330416 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:03.330397 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ltq6\" (UniqueName: \"kubernetes.io/projected/b9f6f098-9594-4a0a-aac2-6fa4fb073b85-kube-api-access-8ltq6\") pod \"lws-controller-manager-66bc978c4c-b8hqq\" (UID: \"b9f6f098-9594-4a0a-aac2-6fa4fb073b85\") " pod="openshift-lws-operator/lws-controller-manager-66bc978c4c-b8hqq" Jul 20 02:55:03.484365 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:03.484262 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-66bc978c4c-b8hqq" Jul 20 02:55:03.561541 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:03.561505 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788lhvx" event={"ID":"1f8ddfdf-af55-4672-9e10-2f4851dacd18","Type":"ContainerStarted","Data":"efb117d910a0587528fbdba8ca4438676383578c575e10499b73478340dbe9c6"} Jul 20 02:55:03.631991 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:03.631960 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-66bc978c4c-b8hqq"] Jul 20 02:55:03.635305 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:55:03.635270 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9f6f098_9594_4a0a_aac2_6fa4fb073b85.slice/crio-cc6e5e520d76e2026201571b930d098183704b438c829cf0f73e2d4c678658a3 WatchSource:0}: Error finding container cc6e5e520d76e2026201571b930d098183704b438c829cf0f73e2d4c678658a3: Status 404 returned error can't find the container with id cc6e5e520d76e2026201571b930d098183704b438c829cf0f73e2d4c678658a3 Jul 20 02:55:04.580281 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:04.580241 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-66bc978c4c-b8hqq" event={"ID":"b9f6f098-9594-4a0a-aac2-6fa4fb073b85","Type":"ContainerStarted","Data":"cc6e5e520d76e2026201571b930d098183704b438c829cf0f73e2d4c678658a3"} Jul 20 02:55:07.236403 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:07.236358 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 02:55:07.236840 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:07.236458 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 02:55:07.236840 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:07.236512 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 02:55:07.594952 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:07.594846 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788lhvx" event={"ID":"1f8ddfdf-af55-4672-9e10-2f4851dacd18","Type":"ContainerStarted","Data":"97205a2ad782d147fcf2423a34026657d32673f77bf5841d6b947f656ab702f3"} Jul 20 02:55:07.596572 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:07.596544 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-66bc978c4c-b8hqq" event={"ID":"b9f6f098-9594-4a0a-aac2-6fa4fb073b85","Type":"ContainerStarted","Data":"3c4a707095492f3d8d6f4b26dc165504b5c417447478abace7f7f67472bbcd2b"} Jul 20 02:55:07.596699 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:07.596619 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-66bc978c4c-b8hqq" Jul 20 02:55:07.612861 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:07.612811 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788lhvx" podStartSLOduration=1.406451167 podStartE2EDuration="5.612791827s" totalImagesPullingTime="4.20634066s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:55:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:55:07.612041488 +0000 UTC m=+602.811290635" watchObservedRunningTime="2026-07-20 02:55:07.612791827 +0000 UTC m=+602.812040974" Jul 20 02:55:07.628571 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:07.628527 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-66bc978c4c-b8hqq" podStartSLOduration=0.977838538 podStartE2EDuration="4.628513098s" totalImagesPullingTime="3.65067456s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:55:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:55:07.627812092 +0000 UTC m=+602.827061250" watchObservedRunningTime="2026-07-20 02:55:07.628513098 +0000 UTC m=+602.827762294" Jul 20 02:55:07.875829 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:07.875795 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788lhvx" Jul 20 02:55:08.881345 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:08.881314 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788lhvx" Jul 20 02:55:09.606155 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:09.606129 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788lhvx" Jul 20 02:55:09.607025 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:09.607001 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788lhvx" Jul 20 02:55:18.604060 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:55:18.604030 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-66bc978c4c-b8hqq" Jul 20 02:56:39.402711 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:56:39.402672 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8z8lx"] Jul 20 02:56:39.405907 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:56:39.405888 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8z8lx" Jul 20 02:56:39.408621 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:56:39.408589 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 20 02:56:39.408986 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:56:39.408787 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-tpsxl\"" Jul 20 02:56:39.409937 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:56:39.409916 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 20 02:56:39.414592 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:56:39.414567 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8z8lx"] Jul 20 02:56:39.518960 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:56:39.518929 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnkwc\" (UniqueName: \"kubernetes.io/projected/c17e70d0-f944-4684-98fe-60339b2ebf17-kube-api-access-tnkwc\") pod \"limitador-operator-controller-manager-859d7b78bd-8z8lx\" (UID: \"c17e70d0-f944-4684-98fe-60339b2ebf17\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8z8lx" Jul 20 02:56:39.619814 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:56:39.619783 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnkwc\" (UniqueName: \"kubernetes.io/projected/c17e70d0-f944-4684-98fe-60339b2ebf17-kube-api-access-tnkwc\") pod \"limitador-operator-controller-manager-859d7b78bd-8z8lx\" (UID: \"c17e70d0-f944-4684-98fe-60339b2ebf17\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8z8lx" Jul 20 02:56:39.628286 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:56:39.628263 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnkwc\" (UniqueName: \"kubernetes.io/projected/c17e70d0-f944-4684-98fe-60339b2ebf17-kube-api-access-tnkwc\") pod \"limitador-operator-controller-manager-859d7b78bd-8z8lx\" (UID: \"c17e70d0-f944-4684-98fe-60339b2ebf17\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8z8lx" Jul 20 02:56:39.717184 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:56:39.717076 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8z8lx" Jul 20 02:56:39.856364 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:56:39.856256 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8z8lx"] Jul 20 02:56:39.859353 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:56:39.859319 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc17e70d0_f944_4684_98fe_60339b2ebf17.slice/crio-e695b3f3fae7e9ee24c033763756acf06a9b32bdb2f6b273cec60da0576d08aa WatchSource:0}: Error finding container e695b3f3fae7e9ee24c033763756acf06a9b32bdb2f6b273cec60da0576d08aa: Status 404 returned error can't find the container with id e695b3f3fae7e9ee24c033763756acf06a9b32bdb2f6b273cec60da0576d08aa Jul 20 02:56:39.861839 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:56:39.861823 2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 20 02:56:39.983736 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:56:39.983654 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8z8lx" event={"ID":"c17e70d0-f944-4684-98fe-60339b2ebf17","Type":"ContainerStarted","Data":"e695b3f3fae7e9ee24c033763756acf06a9b32bdb2f6b273cec60da0576d08aa"} Jul 20 02:56:42.000418 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:56:42.000327 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8z8lx" event={"ID":"c17e70d0-f944-4684-98fe-60339b2ebf17","Type":"ContainerStarted","Data":"4bfb933a1eeba19d98628c499166449497c5cfca05a8a204570726b72992f156"} Jul 20 02:56:42.000770 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:56:42.000535 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8z8lx" Jul 20 02:56:42.017115 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:56:42.017061 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8z8lx" podStartSLOduration=1.21110734 podStartE2EDuration="3.017047211s" totalImagesPullingTime="1.805939871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:56:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:56:42.015562211 +0000 UTC m=+697.214811371" watchObservedRunningTime="2026-07-20 02:56:42.017047211 +0000 UTC m=+697.216296359" Jul 20 02:56:51.631545 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:56:51.631508 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8z8lx"] Jul 20 02:56:51.631934 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:56:51.631802 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8z8lx" podUID="c17e70d0-f944-4684-98fe-60339b2ebf17" containerName="manager" containerID="cri-o://4bfb933a1eeba19d98628c499166449497c5cfca05a8a204570726b72992f156" gracePeriod=2 Jul 20 02:56:51.633751 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:56:51.633726 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8z8lx" Jul 20 02:56:51.638811 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:56:51.638787 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8z8lx"] Jul 20 02:56:51.640939 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:56:51.640903 2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-8z8lx\" is forbidden: User \"system:node:ip-10-0-142-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-139.ec2.internal' and this object" podUID="c17e70d0-f944-4684-98fe-60339b2ebf17" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8z8lx" Jul 20 02:56:51.655854 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:56:51.655825 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ffzz5"] Jul 20 02:56:51.656252 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:56:51.656234 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c17e70d0-f944-4684-98fe-60339b2ebf17" containerName="manager" Jul 20 02:56:51.656348 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:56:51.656256 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c17e70d0-f944-4684-98fe-60339b2ebf17" containerName="manager" Jul 20 02:56:51.656406 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:56:51.656356 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c17e70d0-f944-4684-98fe-60339b2ebf17" containerName="manager" Jul 20 02:56:51.659644 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:56:51.659622 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ffzz5" Jul 20 02:56:51.661962 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:56:51.661929 2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-8z8lx\" is forbidden: User \"system:node:ip-10-0-142-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-139.ec2.internal' and this object" podUID="c17e70d0-f944-4684-98fe-60339b2ebf17" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8z8lx" Jul 20 02:56:51.673156 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:56:51.673123 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ffzz5"] Jul 20 02:56:51.818119 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:56:51.818054 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzcmv\" (UniqueName: \"kubernetes.io/projected/2f67d3a8-4192-412d-a969-61ec3f7351bc-kube-api-access-nzcmv\") pod \"limitador-operator-controller-manager-859d7b78bd-ffzz5\" (UID: \"2f67d3a8-4192-412d-a969-61ec3f7351bc\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ffzz5" Jul 20 02:56:51.890496 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:56:51.890442 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8z8lx" Jul 20 02:56:51.893071 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:56:51.893039 2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-8z8lx\" is forbidden: User \"system:node:ip-10-0-142-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-139.ec2.internal' and this object" podUID="c17e70d0-f944-4684-98fe-60339b2ebf17" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8z8lx" Jul 20 02:56:51.919449 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:56:51.919422 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzcmv\" (UniqueName: \"kubernetes.io/projected/2f67d3a8-4192-412d-a969-61ec3f7351bc-kube-api-access-nzcmv\") pod \"limitador-operator-controller-manager-859d7b78bd-ffzz5\" (UID: \"2f67d3a8-4192-412d-a969-61ec3f7351bc\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ffzz5" Jul 20 02:56:51.927859 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:56:51.927836 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzcmv\" (UniqueName: \"kubernetes.io/projected/2f67d3a8-4192-412d-a969-61ec3f7351bc-kube-api-access-nzcmv\") pod \"limitador-operator-controller-manager-859d7b78bd-ffzz5\" (UID: \"2f67d3a8-4192-412d-a969-61ec3f7351bc\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ffzz5" Jul 20 02:56:52.015033 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:56:52.015002 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ffzz5" Jul 20 02:56:52.020128 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:56:52.020088 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tnkwc\" (UniqueName: \"kubernetes.io/projected/c17e70d0-f944-4684-98fe-60339b2ebf17-kube-api-access-tnkwc\") pod \"c17e70d0-f944-4684-98fe-60339b2ebf17\" (UID: \"c17e70d0-f944-4684-98fe-60339b2ebf17\") " Jul 20 02:56:52.022217 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:56:52.022194 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c17e70d0-f944-4684-98fe-60339b2ebf17-kube-api-access-tnkwc" (OuterVolumeSpecName: "kube-api-access-tnkwc") pod "c17e70d0-f944-4684-98fe-60339b2ebf17" (UID: "c17e70d0-f944-4684-98fe-60339b2ebf17"). InnerVolumeSpecName "kube-api-access-tnkwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 02:56:52.045570 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:56:52.045543 2558 generic.go:356] "Generic (PLEG): container finished" podID="c17e70d0-f944-4684-98fe-60339b2ebf17" containerID="4bfb933a1eeba19d98628c499166449497c5cfca05a8a204570726b72992f156" exitCode=0 Jul 20 02:56:52.045705 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:56:52.045583 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8z8lx" Jul 20 02:56:52.045705 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:56:52.045635 2558 scope.go:117] "RemoveContainer" containerID="4bfb933a1eeba19d98628c499166449497c5cfca05a8a204570726b72992f156" Jul 20 02:56:52.048191 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:56:52.048160 2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-8z8lx\" is forbidden: User \"system:node:ip-10-0-142-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-139.ec2.internal' and this object" podUID="c17e70d0-f944-4684-98fe-60339b2ebf17" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8z8lx" Jul 20 02:56:52.055351 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:56:52.055334 2558 scope.go:117] "RemoveContainer" containerID="4bfb933a1eeba19d98628c499166449497c5cfca05a8a204570726b72992f156" Jul 20 02:56:52.055617 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:56:52.055599 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bfb933a1eeba19d98628c499166449497c5cfca05a8a204570726b72992f156\": container with ID starting with 4bfb933a1eeba19d98628c499166449497c5cfca05a8a204570726b72992f156 not found: ID does not exist" containerID="4bfb933a1eeba19d98628c499166449497c5cfca05a8a204570726b72992f156" Jul 20 02:56:52.055695 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:56:52.055629 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bfb933a1eeba19d98628c499166449497c5cfca05a8a204570726b72992f156"} err="failed to get container status \"4bfb933a1eeba19d98628c499166449497c5cfca05a8a204570726b72992f156\": rpc error: code = NotFound desc = could not find container \"4bfb933a1eeba19d98628c499166449497c5cfca05a8a204570726b72992f156\": container with ID starting with 4bfb933a1eeba19d98628c499166449497c5cfca05a8a204570726b72992f156 not found: ID does not exist" Jul 20 02:56:52.055895 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:56:52.055876 2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-8z8lx\" is forbidden: User \"system:node:ip-10-0-142-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-139.ec2.internal' and this object" podUID="c17e70d0-f944-4684-98fe-60339b2ebf17" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8z8lx" Jul 20 02:56:52.121668 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:56:52.121631 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnkwc\" (UniqueName: \"kubernetes.io/projected/c17e70d0-f944-4684-98fe-60339b2ebf17-kube-api-access-tnkwc\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 02:56:52.144077 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:56:52.144052 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ffzz5"] Jul 20 02:56:52.146364 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:56:52.146336 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f67d3a8_4192_412d_a969_61ec3f7351bc.slice/crio-bd536236219e451c8374c1da0abca54da936fc996dc3772d8194a6040bda67e0 WatchSource:0}: Error finding container bd536236219e451c8374c1da0abca54da936fc996dc3772d8194a6040bda67e0: Status 404 returned error can't find the container with id bd536236219e451c8374c1da0abca54da936fc996dc3772d8194a6040bda67e0 Jul 20 02:56:53.051254 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:56:53.051212 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ffzz5" event={"ID":"2f67d3a8-4192-412d-a969-61ec3f7351bc","Type":"ContainerStarted","Data":"8e44e4e5f6efdf8da1e3981e12018f129b2b1afa212398aa2905556e3f0d43e9"} Jul 20 02:56:53.051709 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:56:53.051273 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ffzz5" event={"ID":"2f67d3a8-4192-412d-a969-61ec3f7351bc","Type":"ContainerStarted","Data":"bd536236219e451c8374c1da0abca54da936fc996dc3772d8194a6040bda67e0"} Jul 20 02:56:53.051709 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:56:53.051397 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ffzz5" Jul 20 02:56:53.053673 ip-10-0-142-139 kubenswrapper[2558]: E0720 02:56:53.053649 2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-8z8lx\" is forbidden: User \"system:node:ip-10-0-142-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-139.ec2.internal' and this object" podUID="c17e70d0-f944-4684-98fe-60339b2ebf17" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8z8lx" Jul 20 02:56:53.075289 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:56:53.075243 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ffzz5" podStartSLOduration=2.075226867 podStartE2EDuration="2.075226867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:56:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:56:53.073370963 +0000 UTC m=+708.272620109" watchObservedRunningTime="2026-07-20 02:56:53.075226867 +0000 UTC m=+708.274476014" Jul 20 02:56:53.424822 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:56:53.424790 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c17e70d0-f944-4684-98fe-60339b2ebf17" path="/var/lib/kubelet/pods/c17e70d0-f944-4684-98fe-60339b2ebf17/volumes" Jul 20 02:57:04.059448 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:04.059411 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ffzz5" Jul 20 02:57:31.497510 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:31.497476 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-7kwrp"] Jul 20 02:57:31.500581 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:31.500561 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-7kwrp" Jul 20 02:57:31.503181 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:31.503162 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-pg2bx\"" Jul 20 02:57:31.512512 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:31.512490 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-7kwrp"] Jul 20 02:57:31.551639 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:31.551610 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffxqh\" (UniqueName: \"kubernetes.io/projected/c22bf050-6558-428c-9c72-4c90c60e62ce-kube-api-access-ffxqh\") pod \"maas-default-gateway-openshift-default-5896d4dd49-7kwrp\" (UID: \"c22bf050-6558-428c-9c72-4c90c60e62ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-7kwrp" Jul 20 02:57:31.551811 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:31.551646 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c22bf050-6558-428c-9c72-4c90c60e62ce-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-7kwrp\" (UID: \"c22bf050-6558-428c-9c72-4c90c60e62ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-7kwrp" Jul 20 02:57:31.551811 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:31.551680 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c22bf050-6558-428c-9c72-4c90c60e62ce-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-7kwrp\" (UID: \"c22bf050-6558-428c-9c72-4c90c60e62ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-7kwrp" Jul 20 02:57:31.551811 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:31.551710 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c22bf050-6558-428c-9c72-4c90c60e62ce-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-7kwrp\" (UID: \"c22bf050-6558-428c-9c72-4c90c60e62ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-7kwrp" Jul 20 02:57:31.551811 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:31.551748 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c22bf050-6558-428c-9c72-4c90c60e62ce-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-7kwrp\" (UID: \"c22bf050-6558-428c-9c72-4c90c60e62ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-7kwrp" Jul 20 02:57:31.551989 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:31.551831 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c22bf050-6558-428c-9c72-4c90c60e62ce-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-7kwrp\" (UID: \"c22bf050-6558-428c-9c72-4c90c60e62ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-7kwrp" Jul 20 02:57:31.551989 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:31.551881 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c22bf050-6558-428c-9c72-4c90c60e62ce-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-7kwrp\" (UID: \"c22bf050-6558-428c-9c72-4c90c60e62ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-7kwrp" Jul 20 02:57:31.551989 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:31.551908 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c22bf050-6558-428c-9c72-4c90c60e62ce-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-7kwrp\" (UID: \"c22bf050-6558-428c-9c72-4c90c60e62ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-7kwrp" Jul 20 02:57:31.551989 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:31.551961 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c22bf050-6558-428c-9c72-4c90c60e62ce-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-7kwrp\" (UID: \"c22bf050-6558-428c-9c72-4c90c60e62ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-7kwrp" Jul 20 02:57:31.653154 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:31.653090 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c22bf050-6558-428c-9c72-4c90c60e62ce-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-7kwrp\" (UID: \"c22bf050-6558-428c-9c72-4c90c60e62ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-7kwrp" Jul 20 02:57:31.653348 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:31.653174 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c22bf050-6558-428c-9c72-4c90c60e62ce-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-7kwrp\" (UID: \"c22bf050-6558-428c-9c72-4c90c60e62ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-7kwrp" Jul 20 02:57:31.653348 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:31.653262 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c22bf050-6558-428c-9c72-4c90c60e62ce-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-7kwrp\" (UID: \"c22bf050-6558-428c-9c72-4c90c60e62ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-7kwrp" Jul 20 02:57:31.653348 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:31.653279 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c22bf050-6558-428c-9c72-4c90c60e62ce-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-7kwrp\" (UID: \"c22bf050-6558-428c-9c72-4c90c60e62ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-7kwrp" Jul 20 02:57:31.653348 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:31.653299 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c22bf050-6558-428c-9c72-4c90c60e62ce-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-7kwrp\" (UID: \"c22bf050-6558-428c-9c72-4c90c60e62ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-7kwrp" Jul 20 02:57:31.653348 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:31.653315 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffxqh\" (UniqueName: \"kubernetes.io/projected/c22bf050-6558-428c-9c72-4c90c60e62ce-kube-api-access-ffxqh\") pod \"maas-default-gateway-openshift-default-5896d4dd49-7kwrp\" (UID: \"c22bf050-6558-428c-9c72-4c90c60e62ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-7kwrp" Jul 20 02:57:31.653348 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:31.653334 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c22bf050-6558-428c-9c72-4c90c60e62ce-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-7kwrp\" (UID: \"c22bf050-6558-428c-9c72-4c90c60e62ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-7kwrp" Jul 20 02:57:31.653732 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:31.653373 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c22bf050-6558-428c-9c72-4c90c60e62ce-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-7kwrp\" (UID: \"c22bf050-6558-428c-9c72-4c90c60e62ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-7kwrp" Jul 20 02:57:31.653732 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:31.653482 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c22bf050-6558-428c-9c72-4c90c60e62ce-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-7kwrp\" (UID: \"c22bf050-6558-428c-9c72-4c90c60e62ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-7kwrp" Jul 20 02:57:31.653732 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:31.653515 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c22bf050-6558-428c-9c72-4c90c60e62ce-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-7kwrp\" (UID: \"c22bf050-6558-428c-9c72-4c90c60e62ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-7kwrp" Jul 20 02:57:31.653732 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:31.653580 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c22bf050-6558-428c-9c72-4c90c60e62ce-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-7kwrp\" (UID: \"c22bf050-6558-428c-9c72-4c90c60e62ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-7kwrp" Jul 20 02:57:31.653948 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:31.653817 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c22bf050-6558-428c-9c72-4c90c60e62ce-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-7kwrp\" (UID: \"c22bf050-6558-428c-9c72-4c90c60e62ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-7kwrp" Jul 20 02:57:31.653948 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:31.653873 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c22bf050-6558-428c-9c72-4c90c60e62ce-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-7kwrp\" (UID: \"c22bf050-6558-428c-9c72-4c90c60e62ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-7kwrp" Jul 20 02:57:31.653948 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:31.653914 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c22bf050-6558-428c-9c72-4c90c60e62ce-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-7kwrp\" (UID: \"c22bf050-6558-428c-9c72-4c90c60e62ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-7kwrp" Jul 20 02:57:31.655694 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:31.655669 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c22bf050-6558-428c-9c72-4c90c60e62ce-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-7kwrp\" (UID: \"c22bf050-6558-428c-9c72-4c90c60e62ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-7kwrp" Jul 20 02:57:31.655898 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:31.655878 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c22bf050-6558-428c-9c72-4c90c60e62ce-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-7kwrp\" (UID: \"c22bf050-6558-428c-9c72-4c90c60e62ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-7kwrp" Jul 20 02:57:31.661127 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:31.661088 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffxqh\" (UniqueName: \"kubernetes.io/projected/c22bf050-6558-428c-9c72-4c90c60e62ce-kube-api-access-ffxqh\") pod \"maas-default-gateway-openshift-default-5896d4dd49-7kwrp\" (UID: \"c22bf050-6558-428c-9c72-4c90c60e62ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-7kwrp" Jul 20 02:57:31.661222 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:31.661156 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c22bf050-6558-428c-9c72-4c90c60e62ce-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-7kwrp\" (UID: \"c22bf050-6558-428c-9c72-4c90c60e62ce\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-7kwrp" Jul 20 02:57:31.811803 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:31.811709 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-7kwrp" Jul 20 02:57:31.948948 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:31.948920 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-7kwrp"] Jul 20 02:57:31.951114 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:57:31.951077 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc22bf050_6558_428c_9c72_4c90c60e62ce.slice/crio-6bd44fbfefc44f2de44e3887c2347220712840f506d1c93e6787cae93fa0b4ae WatchSource:0}: Error finding container 6bd44fbfefc44f2de44e3887c2347220712840f506d1c93e6787cae93fa0b4ae: Status 404 returned error can't find the container with id 6bd44fbfefc44f2de44e3887c2347220712840f506d1c93e6787cae93fa0b4ae Jul 20 02:57:31.953616 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:31.953585 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 02:57:31.953685 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:31.953646 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 02:57:31.953685 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:31.953674 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 02:57:32.219465 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:32.219428 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-7kwrp" event={"ID":"c22bf050-6558-428c-9c72-4c90c60e62ce","Type":"ContainerStarted","Data":"c1463fe51614f59384bb21c458664cd6397e693b7ca0f8ce23c8e25a4db8b1f7"} Jul 20 02:57:32.219670 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:32.219484 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-7kwrp" event={"ID":"c22bf050-6558-428c-9c72-4c90c60e62ce","Type":"ContainerStarted","Data":"6bd44fbfefc44f2de44e3887c2347220712840f506d1c93e6787cae93fa0b4ae"} Jul 20 02:57:32.238177 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:32.238126 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-7kwrp" podStartSLOduration=1.238107984 podStartE2EDuration="1.238107984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:57:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:57:32.236162227 +0000 UTC m=+747.435411384" watchObservedRunningTime="2026-07-20 02:57:32.238107984 +0000 UTC m=+747.437357121" Jul 20 02:57:32.812640 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:32.812603 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-7kwrp" Jul 20 02:57:33.521748 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:33.521706 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 02:57:33.525400 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:33.525376 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-jfbcx" Jul 20 02:57:33.528074 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:33.528048 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 20 02:57:33.528225 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:33.528108 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2nxdn\"" Jul 20 02:57:33.534011 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:33.533986 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 02:57:33.560043 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:33.560015 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 02:57:33.671051 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:33.671021 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cab70675-976c-4960-bfdc-88655db88924-config-file\") pod \"limitador-limitador-54d6d988d5-jfbcx\" (UID: \"cab70675-976c-4960-bfdc-88655db88924\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jfbcx" Jul 20 02:57:33.671051 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:33.671056 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nknzg\" (UniqueName: \"kubernetes.io/projected/cab70675-976c-4960-bfdc-88655db88924-kube-api-access-nknzg\") pod \"limitador-limitador-54d6d988d5-jfbcx\" (UID: \"cab70675-976c-4960-bfdc-88655db88924\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jfbcx" Jul 20 02:57:33.771810 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:33.771722 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cab70675-976c-4960-bfdc-88655db88924-config-file\") pod \"limitador-limitador-54d6d988d5-jfbcx\" (UID: \"cab70675-976c-4960-bfdc-88655db88924\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jfbcx" Jul 20 02:57:33.771810 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:33.771765 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nknzg\" (UniqueName: \"kubernetes.io/projected/cab70675-976c-4960-bfdc-88655db88924-kube-api-access-nknzg\") pod \"limitador-limitador-54d6d988d5-jfbcx\" (UID: \"cab70675-976c-4960-bfdc-88655db88924\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jfbcx" Jul 20 02:57:33.772471 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:33.772451 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cab70675-976c-4960-bfdc-88655db88924-config-file\") pod \"limitador-limitador-54d6d988d5-jfbcx\" (UID: \"cab70675-976c-4960-bfdc-88655db88924\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jfbcx" Jul 20 02:57:33.779547 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:33.779516 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nknzg\" (UniqueName: \"kubernetes.io/projected/cab70675-976c-4960-bfdc-88655db88924-kube-api-access-nknzg\") pod \"limitador-limitador-54d6d988d5-jfbcx\" (UID: \"cab70675-976c-4960-bfdc-88655db88924\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jfbcx" Jul 20 02:57:33.817156 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:33.817131 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-7kwrp" Jul 20 02:57:33.836881 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:33.836849 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-jfbcx" Jul 20 02:57:33.965535 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:33.965508 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 02:57:33.967934 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:57:33.967911 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcab70675_976c_4960_bfdc_88655db88924.slice/crio-2af06ef5541405b4a9f6f27e57d270e07648a2f95e9b88b6225c4a58353da893 WatchSource:0}: Error finding container 2af06ef5541405b4a9f6f27e57d270e07648a2f95e9b88b6225c4a58353da893: Status 404 returned error can't find the container with id 2af06ef5541405b4a9f6f27e57d270e07648a2f95e9b88b6225c4a58353da893 Jul 20 02:57:34.229646 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:34.229610 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-jfbcx" event={"ID":"cab70675-976c-4960-bfdc-88655db88924","Type":"ContainerStarted","Data":"2af06ef5541405b4a9f6f27e57d270e07648a2f95e9b88b6225c4a58353da893"} Jul 20 02:57:34.229859 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:34.229843 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-7kwrp" Jul 20 02:57:34.230857 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:34.230838 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-7kwrp" Jul 20 02:57:37.247914 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:37.247882 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-jfbcx" event={"ID":"cab70675-976c-4960-bfdc-88655db88924","Type":"ContainerStarted","Data":"98cdbe6616266b4ecad98b754c5d44f36dcb27fe1a4a4aa947669cdae633baa0"} Jul 20 02:57:37.248403 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:37.248010 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-jfbcx" Jul 20 02:57:37.264299 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:37.264242 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-jfbcx" podStartSLOduration=1.796230139 podStartE2EDuration="4.264226677s" totalImagesPullingTime="2.467996538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:57:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:57:37.263188766 +0000 UTC m=+752.462437938" watchObservedRunningTime="2026-07-20 02:57:37.264226677 +0000 UTC m=+752.463475824" Jul 20 02:57:48.255867 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:57:48.255834 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-jfbcx" Jul 20 02:58:45.174953 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:58:45.174877 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 02:58:52.585253 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:58:52.585170 2558 generic.go:356] "Generic (PLEG): container finished" podID="c22bf050-6558-428c-9c72-4c90c60e62ce" containerID="c1463fe51614f59384bb21c458664cd6397e693b7ca0f8ce23c8e25a4db8b1f7" exitCode=0 Jul 20 02:58:52.585253 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:58:52.585227 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-7kwrp" event={"ID":"c22bf050-6558-428c-9c72-4c90c60e62ce","Type":"ContainerDied","Data":"c1463fe51614f59384bb21c458664cd6397e693b7ca0f8ce23c8e25a4db8b1f7"} Jul 20 02:58:52.585675 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:58:52.585599 2558 scope.go:117] "RemoveContainer" containerID="c1463fe51614f59384bb21c458664cd6397e693b7ca0f8ce23c8e25a4db8b1f7" Jul 20 02:58:52.586736 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:58:52.586706 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 02:58:52.586805 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:58:52.586788 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 02:58:52.586856 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:58:52.586841 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 02:58:53.591050 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:58:53.591018 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-7kwrp" event={"ID":"c22bf050-6558-428c-9c72-4c90c60e62ce","Type":"ContainerStarted","Data":"43eefe01678438f1b518cab628cadc902b31e922a1cc91d03e2e7a35cd50a915"} Jul 20 02:58:53.812217 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:58:53.812165 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-7kwrp" Jul 20 02:58:53.812435 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:58:53.812413 2558 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-7kwrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Jul 20 02:58:53.812481 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:58:53.812460 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-7kwrp" podUID="c22bf050-6558-428c-9c72-4c90c60e62ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Jul 20 02:58:54.812425 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:58:54.812389 2558 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-7kwrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Jul 20 02:58:54.812806 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:58:54.812443 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-7kwrp" podUID="c22bf050-6558-428c-9c72-4c90c60e62ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Jul 20 02:58:55.812325 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:58:55.812287 2558 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-7kwrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Jul 20 02:58:55.812505 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:58:55.812346 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-7kwrp" podUID="c22bf050-6558-428c-9c72-4c90c60e62ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Jul 20 02:58:56.812973 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:58:56.812942 2558 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-7kwrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Jul 20 02:58:56.813370 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:58:56.813010 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-7kwrp" podUID="c22bf050-6558-428c-9c72-4c90c60e62ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Jul 20 02:58:57.816127 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:58:57.816083 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-7kwrp" Jul 20 02:58:57.816497 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:58:57.816444 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-7kwrp" Jul 20 02:58:57.817241 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:58:57.817219 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-7kwrp" Jul 20 02:59:06.522958 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:06.522919 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 02:59:25.733346 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:25.733312 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 02:59:44.230771 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:44.230734 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-zvj77"] Jul 20 02:59:44.234006 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:44.233987 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zvj77" Jul 20 02:59:44.236751 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:44.236730 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 20 02:59:44.236861 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:44.236761 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-6tcpl\"" Jul 20 02:59:44.236861 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:44.236768 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 20 02:59:44.236989 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:44.236832 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 20 02:59:44.242772 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:44.242752 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-zvj77"] Jul 20 02:59:44.379051 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:44.379013 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df024b7e-1cac-4263-bdb7-634ce1c66984-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zvj77\" (UID: \"df024b7e-1cac-4263-bdb7-634ce1c66984\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zvj77" Jul 20 02:59:44.379051 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:44.379052 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df024b7e-1cac-4263-bdb7-634ce1c66984-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zvj77\" (UID: \"df024b7e-1cac-4263-bdb7-634ce1c66984\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zvj77" Jul 20 02:59:44.379319 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:44.379163 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfs6x\" (UniqueName: \"kubernetes.io/projected/df024b7e-1cac-4263-bdb7-634ce1c66984-kube-api-access-qfs6x\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zvj77\" (UID: \"df024b7e-1cac-4263-bdb7-634ce1c66984\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zvj77" Jul 20 02:59:44.379319 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:44.379254 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df024b7e-1cac-4263-bdb7-634ce1c66984-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zvj77\" (UID: \"df024b7e-1cac-4263-bdb7-634ce1c66984\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zvj77" Jul 20 02:59:44.379408 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:44.379326 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df024b7e-1cac-4263-bdb7-634ce1c66984-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zvj77\" (UID: \"df024b7e-1cac-4263-bdb7-634ce1c66984\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zvj77" Jul 20 02:59:44.379408 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:44.379359 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df024b7e-1cac-4263-bdb7-634ce1c66984-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zvj77\" (UID: \"df024b7e-1cac-4263-bdb7-634ce1c66984\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zvj77" Jul 20 02:59:44.480449 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:44.480411 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df024b7e-1cac-4263-bdb7-634ce1c66984-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zvj77\" (UID: \"df024b7e-1cac-4263-bdb7-634ce1c66984\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zvj77" Jul 20 02:59:44.480624 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:44.480467 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df024b7e-1cac-4263-bdb7-634ce1c66984-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zvj77\" (UID: \"df024b7e-1cac-4263-bdb7-634ce1c66984\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zvj77" Jul 20 02:59:44.480624 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:44.480577 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df024b7e-1cac-4263-bdb7-634ce1c66984-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zvj77\" (UID: \"df024b7e-1cac-4263-bdb7-634ce1c66984\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zvj77" Jul 20 02:59:44.480713 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:44.480638 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df024b7e-1cac-4263-bdb7-634ce1c66984-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zvj77\" (UID: \"df024b7e-1cac-4263-bdb7-634ce1c66984\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zvj77" Jul 20 02:59:44.480713 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:44.480659 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df024b7e-1cac-4263-bdb7-634ce1c66984-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zvj77\" (UID: \"df024b7e-1cac-4263-bdb7-634ce1c66984\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zvj77" Jul 20 02:59:44.480713 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:44.480696 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfs6x\" (UniqueName: \"kubernetes.io/projected/df024b7e-1cac-4263-bdb7-634ce1c66984-kube-api-access-qfs6x\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zvj77\" (UID: \"df024b7e-1cac-4263-bdb7-634ce1c66984\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zvj77" Jul 20 02:59:44.480883 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:44.480823 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df024b7e-1cac-4263-bdb7-634ce1c66984-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zvj77\" (UID: \"df024b7e-1cac-4263-bdb7-634ce1c66984\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zvj77" Jul 20 02:59:44.480973 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:44.480950 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df024b7e-1cac-4263-bdb7-634ce1c66984-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zvj77\" (UID: \"df024b7e-1cac-4263-bdb7-634ce1c66984\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zvj77" Jul 20 02:59:44.481032 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:44.481012 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df024b7e-1cac-4263-bdb7-634ce1c66984-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zvj77\" (UID: \"df024b7e-1cac-4263-bdb7-634ce1c66984\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zvj77" Jul 20 02:59:44.482955 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:44.482933 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df024b7e-1cac-4263-bdb7-634ce1c66984-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zvj77\" (UID: \"df024b7e-1cac-4263-bdb7-634ce1c66984\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zvj77" Jul 20 02:59:44.483236 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:44.483220 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df024b7e-1cac-4263-bdb7-634ce1c66984-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zvj77\" (UID: \"df024b7e-1cac-4263-bdb7-634ce1c66984\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zvj77" Jul 20 02:59:44.490247 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:44.490220 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfs6x\" (UniqueName: \"kubernetes.io/projected/df024b7e-1cac-4263-bdb7-634ce1c66984-kube-api-access-qfs6x\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zvj77\" (UID: \"df024b7e-1cac-4263-bdb7-634ce1c66984\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zvj77" Jul 20 02:59:44.544997 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:44.544961 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zvj77" Jul 20 02:59:44.676070 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:44.676043 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-zvj77"] Jul 20 02:59:44.677896 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:59:44.677867 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf024b7e_1cac_4263_bdb7_634ce1c66984.slice/crio-8e55635a0ae6819fc96d831a2ddd6c7f0e4982877f20cdc86b82d3cf308b8b5c WatchSource:0}: Error finding container 8e55635a0ae6819fc96d831a2ddd6c7f0e4982877f20cdc86b82d3cf308b8b5c: Status 404 returned error can't find the container with id 8e55635a0ae6819fc96d831a2ddd6c7f0e4982877f20cdc86b82d3cf308b8b5c Jul 20 02:59:44.818196 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:44.818086 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zvj77" event={"ID":"df024b7e-1cac-4263-bdb7-634ce1c66984","Type":"ContainerStarted","Data":"8e55635a0ae6819fc96d831a2ddd6c7f0e4982877f20cdc86b82d3cf308b8b5c"} Jul 20 02:59:45.320351 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:45.320314 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 02:59:45.824377 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:45.824340 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zvj77" event={"ID":"df024b7e-1cac-4263-bdb7-634ce1c66984","Type":"ContainerStarted","Data":"f3e2fb592248b66275db3021164c246e4ea78416ed52947e0d48bd3eeb9a2cde"} Jul 20 02:59:45.840508 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:45.840458 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zvj77" podStartSLOduration=0.844002051 podStartE2EDuration="1.840440006s" totalImagesPullingTime="996.437955ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:59:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:59:45.839829831 +0000 UTC m=+881.039078978" watchObservedRunningTime="2026-07-20 02:59:45.840440006 +0000 UTC m=+881.039689150" Jul 20 02:59:54.545726 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:54.545680 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zvj77" Jul 20 02:59:54.546136 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:54.545751 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zvj77" Jul 20 02:59:54.558384 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:54.558358 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zvj77" Jul 20 02:59:54.875493 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:54.875466 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zvj77" Jul 20 02:59:56.826146 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:56.826036 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd2qjf"] Jul 20 02:59:56.830004 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:56.829979 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd2qjf" Jul 20 02:59:56.832481 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:56.832460 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 20 02:59:56.839432 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:56.839408 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd2qjf"] Jul 20 02:59:56.892217 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:56.892189 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b316a49-33b7-4525-b37a-a0e0bf2ef0af-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd2qjf\" (UID: \"4b316a49-33b7-4525-b37a-a0e0bf2ef0af\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd2qjf" Jul 20 02:59:56.892372 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:56.892232 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b316a49-33b7-4525-b37a-a0e0bf2ef0af-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd2qjf\" (UID: \"4b316a49-33b7-4525-b37a-a0e0bf2ef0af\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd2qjf" Jul 20 02:59:56.892372 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:56.892257 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b316a49-33b7-4525-b37a-a0e0bf2ef0af-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd2qjf\" (UID: \"4b316a49-33b7-4525-b37a-a0e0bf2ef0af\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd2qjf" Jul 20 02:59:56.892453 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:56.892391 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wt4l\" (UniqueName: \"kubernetes.io/projected/4b316a49-33b7-4525-b37a-a0e0bf2ef0af-kube-api-access-4wt4l\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd2qjf\" (UID: \"4b316a49-33b7-4525-b37a-a0e0bf2ef0af\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd2qjf" Jul 20 02:59:56.892507 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:56.892486 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b316a49-33b7-4525-b37a-a0e0bf2ef0af-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd2qjf\" (UID: \"4b316a49-33b7-4525-b37a-a0e0bf2ef0af\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd2qjf" Jul 20 02:59:56.892545 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:56.892534 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b316a49-33b7-4525-b37a-a0e0bf2ef0af-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd2qjf\" (UID: \"4b316a49-33b7-4525-b37a-a0e0bf2ef0af\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd2qjf" Jul 20 02:59:56.993385 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:56.993344 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b316a49-33b7-4525-b37a-a0e0bf2ef0af-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd2qjf\" (UID: \"4b316a49-33b7-4525-b37a-a0e0bf2ef0af\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd2qjf" Jul 20 02:59:56.993552 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:56.993396 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b316a49-33b7-4525-b37a-a0e0bf2ef0af-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd2qjf\" (UID: \"4b316a49-33b7-4525-b37a-a0e0bf2ef0af\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd2qjf" Jul 20 02:59:56.993552 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:56.993455 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b316a49-33b7-4525-b37a-a0e0bf2ef0af-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd2qjf\" (UID: \"4b316a49-33b7-4525-b37a-a0e0bf2ef0af\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd2qjf" Jul 20 02:59:56.993552 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:56.993488 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b316a49-33b7-4525-b37a-a0e0bf2ef0af-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd2qjf\" (UID: \"4b316a49-33b7-4525-b37a-a0e0bf2ef0af\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd2qjf" Jul 20 02:59:56.993720 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:56.993608 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b316a49-33b7-4525-b37a-a0e0bf2ef0af-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd2qjf\" (UID: \"4b316a49-33b7-4525-b37a-a0e0bf2ef0af\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd2qjf" Jul 20 02:59:56.993720 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:56.993690 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wt4l\" (UniqueName: \"kubernetes.io/projected/4b316a49-33b7-4525-b37a-a0e0bf2ef0af-kube-api-access-4wt4l\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd2qjf\" (UID: \"4b316a49-33b7-4525-b37a-a0e0bf2ef0af\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd2qjf" Jul 20 02:59:56.993840 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:56.993819 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b316a49-33b7-4525-b37a-a0e0bf2ef0af-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd2qjf\" (UID: \"4b316a49-33b7-4525-b37a-a0e0bf2ef0af\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd2qjf" Jul 20 02:59:56.993898 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:56.993858 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b316a49-33b7-4525-b37a-a0e0bf2ef0af-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd2qjf\" (UID: \"4b316a49-33b7-4525-b37a-a0e0bf2ef0af\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd2qjf" Jul 20 02:59:56.993955 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:56.993920 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b316a49-33b7-4525-b37a-a0e0bf2ef0af-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd2qjf\" (UID: \"4b316a49-33b7-4525-b37a-a0e0bf2ef0af\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd2qjf" Jul 20 02:59:56.995652 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:56.995635 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b316a49-33b7-4525-b37a-a0e0bf2ef0af-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd2qjf\" (UID: \"4b316a49-33b7-4525-b37a-a0e0bf2ef0af\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd2qjf" Jul 20 02:59:56.996134 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:56.996114 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b316a49-33b7-4525-b37a-a0e0bf2ef0af-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd2qjf\" (UID: \"4b316a49-33b7-4525-b37a-a0e0bf2ef0af\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd2qjf" Jul 20 02:59:57.001830 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:57.001808 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wt4l\" (UniqueName: \"kubernetes.io/projected/4b316a49-33b7-4525-b37a-a0e0bf2ef0af-kube-api-access-4wt4l\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd2qjf\" (UID: \"4b316a49-33b7-4525-b37a-a0e0bf2ef0af\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd2qjf" Jul 20 02:59:57.158884 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:57.158852 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd2qjf" Jul 20 02:59:57.290954 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:57.290922 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd2qjf"] Jul 20 02:59:57.294046 ip-10-0-142-139 kubenswrapper[2558]: W0720 02:59:57.294020 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b316a49_33b7_4525_b37a_a0e0bf2ef0af.slice/crio-026c3a2407eb320763e872b4e439fa198f12941f25658bef61970be76da01d29 WatchSource:0}: Error finding container 026c3a2407eb320763e872b4e439fa198f12941f25658bef61970be76da01d29: Status 404 returned error can't find the container with id 026c3a2407eb320763e872b4e439fa198f12941f25658bef61970be76da01d29 Jul 20 02:59:57.878908 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:57.878870 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd2qjf" event={"ID":"4b316a49-33b7-4525-b37a-a0e0bf2ef0af","Type":"ContainerStarted","Data":"db63c52f96e9cfbf3bba0599426c81a53c34bafbd91553e5c921efcd37e02ceb"} Jul 20 02:59:57.879340 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:57.878932 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd2qjf" event={"ID":"4b316a49-33b7-4525-b37a-a0e0bf2ef0af","Type":"ContainerStarted","Data":"026c3a2407eb320763e872b4e439fa198f12941f25658bef61970be76da01d29"} Jul 20 02:59:57.895017 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:57.894968 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd2qjf" podStartSLOduration=1.7218825039999999 podStartE2EDuration="1.894951512s" totalImagesPullingTime="173.069008ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 02:59:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 02:59:57.894137437 +0000 UTC m=+893.093386575" watchObservedRunningTime="2026-07-20 02:59:57.894951512 +0000 UTC m=+893.094200658" Jul 20 02:59:58.417315 ip-10-0-142-139 kubenswrapper[2558]: I0720 02:59:58.417281 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:00:04.331432 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:00:04.331394 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:00:07.160024 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:00:07.159985 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd2qjf" Jul 20 03:00:07.160424 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:00:07.160077 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd2qjf" Jul 20 03:00:07.172213 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:00:07.172186 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd2qjf" Jul 20 03:00:07.934196 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:00:07.934165 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd2qjf" Jul 20 03:00:15.921585 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:00:15.921550 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:02:02.025055 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:02:02.025019 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:02:13.820366 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:02:13.820329 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:02:23.227977 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:02:23.227934 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:04:05.322016 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:04:05.321978 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:04:13.522772 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:04:13.522739 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:04:45.729371 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:04:45.729289 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:04:55.324845 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:04:55.324806 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:05:05.622495 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:05:05.622462 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:05:13.222207 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:05:13.222171 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:05:24.922785 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:05:24.922752 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:05:35.425054 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:05:35.425025 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:05:48.218141 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:05:48.218086 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:05:58.120757 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:05:58.120678 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:06:19.819235 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:06:19.819197 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:06:30.616924 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:06:30.616886 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:06:54.123059 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:06:54.123017 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:07:08.524921 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:07:08.524886 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:07:20.524312 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:07:20.524277 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:07:32.822636 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:07:32.822554 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:07:47.026325 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:07:47.026288 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:09:48.219110 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:09:48.219056 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:09:59.934031 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:09:59.933993 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:10:09.023111 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:10:09.023055 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:10:21.020879 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:10:21.020841 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:10:31.416223 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:10:31.416134 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:10:43.221498 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:10:43.219645 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:10:51.628760 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:10:51.628719 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:11:02.322951 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:11:02.322918 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:11:13.118550 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:11:13.118510 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:11:22.348033 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:11:22.347997 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:11:33.314396 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:11:33.314358 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:11:55.220402 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:11:55.220364 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:12:04.622320 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:12:04.622234 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:12:20.535277 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:12:20.535244 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-2eb3e7a8-kserve-6785c64f9-7nsnh"] Jul 20 03:12:20.538816 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:12:20.538794 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2eb3e7a8-kserve-6785c64f9-7nsnh" Jul 20 03:12:20.541255 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:12:20.541234 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-2eb3e7a8-kserve-self-signed-certs\"" Jul 20 03:12:20.553040 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:12:20.553016 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-2eb3e7a8-kserve-6785c64f9-7nsnh"] Jul 20 03:12:20.656422 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:12:20.656391 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a-tmp-dir\") pod \"e2e-same-id-a-2eb3e7a8-kserve-6785c64f9-7nsnh\" (UID: \"77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a\") " pod="llm/e2e-same-id-a-2eb3e7a8-kserve-6785c64f9-7nsnh" Jul 20 03:12:20.656589 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:12:20.656445 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xbtl\" (UniqueName: \"kubernetes.io/projected/77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a-kube-api-access-4xbtl\") pod \"e2e-same-id-a-2eb3e7a8-kserve-6785c64f9-7nsnh\" (UID: \"77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a\") " pod="llm/e2e-same-id-a-2eb3e7a8-kserve-6785c64f9-7nsnh" Jul 20 03:12:20.656589 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:12:20.656486 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a-dshm\") pod \"e2e-same-id-a-2eb3e7a8-kserve-6785c64f9-7nsnh\" (UID: \"77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a\") " pod="llm/e2e-same-id-a-2eb3e7a8-kserve-6785c64f9-7nsnh" Jul 20 03:12:20.656589 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:12:20.656564 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a-home\") pod \"e2e-same-id-a-2eb3e7a8-kserve-6785c64f9-7nsnh\" (UID: \"77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a\") " pod="llm/e2e-same-id-a-2eb3e7a8-kserve-6785c64f9-7nsnh" Jul 20 03:12:20.656589 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:12:20.656581 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a-tls-certs\") pod \"e2e-same-id-a-2eb3e7a8-kserve-6785c64f9-7nsnh\" (UID: \"77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a\") " pod="llm/e2e-same-id-a-2eb3e7a8-kserve-6785c64f9-7nsnh" Jul 20 03:12:20.656726 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:12:20.656602 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a-model-cache\") pod \"e2e-same-id-a-2eb3e7a8-kserve-6785c64f9-7nsnh\" (UID: \"77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a\") " pod="llm/e2e-same-id-a-2eb3e7a8-kserve-6785c64f9-7nsnh" Jul 20 03:12:20.757145 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:12:20.757086 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a-tmp-dir\") pod \"e2e-same-id-a-2eb3e7a8-kserve-6785c64f9-7nsnh\" (UID: \"77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a\") " pod="llm/e2e-same-id-a-2eb3e7a8-kserve-6785c64f9-7nsnh" Jul 20 03:12:20.757313 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:12:20.757177 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xbtl\" (UniqueName: \"kubernetes.io/projected/77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a-kube-api-access-4xbtl\") pod \"e2e-same-id-a-2eb3e7a8-kserve-6785c64f9-7nsnh\" (UID: \"77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a\") " pod="llm/e2e-same-id-a-2eb3e7a8-kserve-6785c64f9-7nsnh" Jul 20 03:12:20.757313 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:12:20.757199 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a-dshm\") pod \"e2e-same-id-a-2eb3e7a8-kserve-6785c64f9-7nsnh\" (UID: \"77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a\") " pod="llm/e2e-same-id-a-2eb3e7a8-kserve-6785c64f9-7nsnh" Jul 20 03:12:20.757313 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:12:20.757234 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a-home\") pod \"e2e-same-id-a-2eb3e7a8-kserve-6785c64f9-7nsnh\" (UID: \"77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a\") " pod="llm/e2e-same-id-a-2eb3e7a8-kserve-6785c64f9-7nsnh" Jul 20 03:12:20.757474 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:12:20.757358 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a-tls-certs\") pod \"e2e-same-id-a-2eb3e7a8-kserve-6785c64f9-7nsnh\" (UID: \"77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a\") " pod="llm/e2e-same-id-a-2eb3e7a8-kserve-6785c64f9-7nsnh" Jul 20 03:12:20.757474 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:12:20.757414 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a-model-cache\") pod \"e2e-same-id-a-2eb3e7a8-kserve-6785c64f9-7nsnh\" (UID: \"77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a\") " pod="llm/e2e-same-id-a-2eb3e7a8-kserve-6785c64f9-7nsnh" Jul 20 03:12:20.757570 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:12:20.757511 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a-tmp-dir\") pod \"e2e-same-id-a-2eb3e7a8-kserve-6785c64f9-7nsnh\" (UID: \"77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a\") " pod="llm/e2e-same-id-a-2eb3e7a8-kserve-6785c64f9-7nsnh" Jul 20 03:12:20.757624 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:12:20.757589 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a-home\") pod \"e2e-same-id-a-2eb3e7a8-kserve-6785c64f9-7nsnh\" (UID: \"77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a\") " pod="llm/e2e-same-id-a-2eb3e7a8-kserve-6785c64f9-7nsnh" Jul 20 03:12:20.757775 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:12:20.757755 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a-model-cache\") pod \"e2e-same-id-a-2eb3e7a8-kserve-6785c64f9-7nsnh\" (UID: \"77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a\") " pod="llm/e2e-same-id-a-2eb3e7a8-kserve-6785c64f9-7nsnh" Jul 20 03:12:20.759536 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:12:20.759518 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a-dshm\") pod \"e2e-same-id-a-2eb3e7a8-kserve-6785c64f9-7nsnh\" (UID: \"77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a\") " pod="llm/e2e-same-id-a-2eb3e7a8-kserve-6785c64f9-7nsnh" Jul 20 03:12:20.759879 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:12:20.759858 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a-tls-certs\") pod \"e2e-same-id-a-2eb3e7a8-kserve-6785c64f9-7nsnh\" (UID: \"77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a\") " pod="llm/e2e-same-id-a-2eb3e7a8-kserve-6785c64f9-7nsnh" Jul 20 03:12:20.764344 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:12:20.764324 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xbtl\" (UniqueName: \"kubernetes.io/projected/77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a-kube-api-access-4xbtl\") pod \"e2e-same-id-a-2eb3e7a8-kserve-6785c64f9-7nsnh\" (UID: \"77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a\") " pod="llm/e2e-same-id-a-2eb3e7a8-kserve-6785c64f9-7nsnh" Jul 20 03:12:20.848700 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:12:20.848614 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2eb3e7a8-kserve-6785c64f9-7nsnh" Jul 20 03:12:20.981194 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:12:20.981144 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-2eb3e7a8-kserve-6785c64f9-7nsnh"] Jul 20 03:12:20.983408 ip-10-0-142-139 kubenswrapper[2558]: W0720 03:12:20.983379 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod77dd6ff9_4ac7_4eb5_98e4_1d71bc65ed9a.slice/crio-03ede2766bcf73bab20514456879da73a3e758c0608c3840566458baf14748a2 WatchSource:0}: Error finding container 03ede2766bcf73bab20514456879da73a3e758c0608c3840566458baf14748a2: Status 404 returned error can't find the container with id 03ede2766bcf73bab20514456879da73a3e758c0608c3840566458baf14748a2 Jul 20 03:12:21.224691 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:12:21.224649 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2eb3e7a8-kserve-6785c64f9-7nsnh" event={"ID":"77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a","Type":"ContainerStarted","Data":"189ce349080c8cff72eec0c4ab58b421e3f22c91ea59c9345fda06d313754f2c"} Jul 20 03:12:21.224868 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:12:21.224699 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2eb3e7a8-kserve-6785c64f9-7nsnh" event={"ID":"77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a","Type":"ContainerStarted","Data":"03ede2766bcf73bab20514456879da73a3e758c0608c3840566458baf14748a2"} Jul 20 03:12:21.241546 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:12:21.241497 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-2eb3e7a8-kserve-6785c64f9-7nsnh" podStartSLOduration=1.241480277 podStartE2EDuration="1.241480277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 03:12:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 03:12:21.240241887 +0000 UTC m=+1636.439491048" watchObservedRunningTime="2026-07-20 03:12:21.241480277 +0000 UTC m=+1636.440729423" Jul 20 03:12:23.718438 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:12:23.718405 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:12:30.849131 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:12:30.849001 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-2eb3e7a8-kserve-6785c64f9-7nsnh" Jul 20 03:12:30.849131 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:12:30.849070 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-2eb3e7a8-kserve-6785c64f9-7nsnh" Jul 20 03:12:30.862037 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:12:30.862012 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-2eb3e7a8-kserve-6785c64f9-7nsnh" Jul 20 03:12:31.279045 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:12:31.279007 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-2eb3e7a8-kserve-6785c64f9-7nsnh" Jul 20 03:12:48.874948 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:12:48.874915 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:13:26.826113 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:13:26.826015 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-2eb3e7a8-kserve-6785c64f9-7nsnh"] Jul 20 03:13:30.025713 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:13:30.025677 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:13:41.846837 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:13:41.846798 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-2eb3e7a8-kserve-6785c64f9-7nsnh" podUID="77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a" containerName="main" containerID="cri-o://189ce349080c8cff72eec0c4ab58b421e3f22c91ea59c9345fda06d313754f2c" gracePeriod=45 Jul 20 03:13:42.095527 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:13:42.095505 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2eb3e7a8-kserve-6785c64f9-7nsnh" Jul 20 03:13:42.286181 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:13:42.286147 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4xbtl\" (UniqueName: \"kubernetes.io/projected/77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a-kube-api-access-4xbtl\") pod \"77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a\" (UID: \"77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a\") " Jul 20 03:13:42.286379 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:13:42.286188 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a-tmp-dir\") pod \"77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a\" (UID: \"77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a\") " Jul 20 03:13:42.286379 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:13:42.286239 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a-dshm\") pod \"77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a\" (UID: \"77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a\") " Jul 20 03:13:42.286379 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:13:42.286253 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a-home\") pod \"77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a\" (UID: \"77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a\") " Jul 20 03:13:42.286379 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:13:42.286273 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a-tls-certs\") pod \"77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a\" (UID: \"77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a\") " Jul 20 03:13:42.286379 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:13:42.286291 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a-model-cache\") pod \"77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a\" (UID: \"77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a\") " Jul 20 03:13:42.286641 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:13:42.286553 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a-home" (OuterVolumeSpecName: "home") pod "77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a" (UID: "77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:13:42.286641 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:13:42.286592 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a" (UID: "77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:13:42.286641 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:13:42.286624 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a-model-cache" (OuterVolumeSpecName: "model-cache") pod "77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a" (UID: "77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:13:42.288519 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:13:42.288494 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a-kube-api-access-4xbtl" (OuterVolumeSpecName: "kube-api-access-4xbtl") pod "77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a" (UID: "77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a"). InnerVolumeSpecName "kube-api-access-4xbtl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 03:13:42.288622 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:13:42.288528 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a" (UID: "77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 03:13:42.288622 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:13:42.288535 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a-dshm" (OuterVolumeSpecName: "dshm") pod "77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a" (UID: "77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:13:42.387247 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:13:42.387215 2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a-dshm\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:13:42.387247 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:13:42.387244 2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a-home\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:13:42.387247 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:13:42.387254 2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a-tls-certs\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:13:42.387448 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:13:42.387263 2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a-model-cache\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:13:42.387448 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:13:42.387272 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xbtl\" (UniqueName: \"kubernetes.io/projected/77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a-kube-api-access-4xbtl\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:13:42.387448 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:13:42.387283 2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a-tmp-dir\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:13:42.596567 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:13:42.596488 2558 generic.go:356] "Generic (PLEG): container finished" podID="77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a" containerID="189ce349080c8cff72eec0c4ab58b421e3f22c91ea59c9345fda06d313754f2c" exitCode=0 Jul 20 03:13:42.596567 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:13:42.596531 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2eb3e7a8-kserve-6785c64f9-7nsnh" Jul 20 03:13:42.596745 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:13:42.596569 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2eb3e7a8-kserve-6785c64f9-7nsnh" event={"ID":"77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a","Type":"ContainerDied","Data":"189ce349080c8cff72eec0c4ab58b421e3f22c91ea59c9345fda06d313754f2c"} Jul 20 03:13:42.596745 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:13:42.596613 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2eb3e7a8-kserve-6785c64f9-7nsnh" event={"ID":"77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a","Type":"ContainerDied","Data":"03ede2766bcf73bab20514456879da73a3e758c0608c3840566458baf14748a2"} Jul 20 03:13:42.596745 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:13:42.596636 2558 scope.go:117] "RemoveContainer" containerID="189ce349080c8cff72eec0c4ab58b421e3f22c91ea59c9345fda06d313754f2c" Jul 20 03:13:42.605256 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:13:42.605238 2558 scope.go:117] "RemoveContainer" containerID="189ce349080c8cff72eec0c4ab58b421e3f22c91ea59c9345fda06d313754f2c" Jul 20 03:13:42.605507 ip-10-0-142-139 kubenswrapper[2558]: E0720 03:13:42.605491 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"189ce349080c8cff72eec0c4ab58b421e3f22c91ea59c9345fda06d313754f2c\": container with ID starting with 189ce349080c8cff72eec0c4ab58b421e3f22c91ea59c9345fda06d313754f2c not found: ID does not exist" containerID="189ce349080c8cff72eec0c4ab58b421e3f22c91ea59c9345fda06d313754f2c" Jul 20 03:13:42.605556 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:13:42.605516 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"189ce349080c8cff72eec0c4ab58b421e3f22c91ea59c9345fda06d313754f2c"} err="failed to get container status \"189ce349080c8cff72eec0c4ab58b421e3f22c91ea59c9345fda06d313754f2c\": rpc error: code = NotFound desc = could not find container \"189ce349080c8cff72eec0c4ab58b421e3f22c91ea59c9345fda06d313754f2c\": container with ID starting with 189ce349080c8cff72eec0c4ab58b421e3f22c91ea59c9345fda06d313754f2c not found: ID does not exist" Jul 20 03:13:42.616854 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:13:42.616833 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-2eb3e7a8-kserve-6785c64f9-7nsnh"] Jul 20 03:13:42.622800 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:13:42.622778 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-2eb3e7a8-kserve-6785c64f9-7nsnh"] Jul 20 03:13:43.425331 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:13:43.425291 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a" path="/var/lib/kubelet/pods/77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a/volumes" Jul 20 03:13:45.727676 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:13:45.727636 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:13:52.227906 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:13:52.227868 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:14:08.521279 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:14:08.521239 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:14:17.723634 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:14:17.723595 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:14:27.517903 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:14:27.517865 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:14:35.024993 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:14:35.024953 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:14:46.320731 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:14:46.320694 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:14:55.820490 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:14:55.820455 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:15:00.136533 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:15:00.136449 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29741955-7qppl"] Jul 20 03:15:00.136894 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:15:00.136818 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a" containerName="main" Jul 20 03:15:00.136894 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:15:00.136833 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a" containerName="main" Jul 20 03:15:00.136894 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:15:00.136891 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="77dd6ff9-4ac7-4eb5-98e4-1d71bc65ed9a" containerName="main" Jul 20 03:15:00.139862 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:15:00.139843 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741955-7qppl" Jul 20 03:15:00.142472 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:15:00.142448 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hlzh6\"" Jul 20 03:15:00.146314 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:15:00.146285 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29741955-7qppl"] Jul 20 03:15:00.151360 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:15:00.151337 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741955-7qppl" Jul 20 03:15:00.491658 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:15:00.491580 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29741955-7qppl"] Jul 20 03:15:00.494937 ip-10-0-142-139 kubenswrapper[2558]: W0720 03:15:00.494909 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99c52a19_7be6_44d6_968a_a6dad23d6ffa.slice/crio-2bd7880aa115471cf436f7c37acfe7d54951e72b0c72d40cac6d7a0b9228b92b WatchSource:0}: Error finding container 2bd7880aa115471cf436f7c37acfe7d54951e72b0c72d40cac6d7a0b9228b92b: Status 404 returned error can't find the container with id 2bd7880aa115471cf436f7c37acfe7d54951e72b0c72d40cac6d7a0b9228b92b Jul 20 03:15:00.496815 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:15:00.496798 2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 20 03:15:00.951717 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:15:00.951680 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741955-7qppl" event={"ID":"99c52a19-7be6-44d6-968a-a6dad23d6ffa","Type":"ContainerStarted","Data":"2bd7880aa115471cf436f7c37acfe7d54951e72b0c72d40cac6d7a0b9228b92b"} Jul 20 03:15:03.975600 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:15:03.975492 2558 generic.go:356] "Generic (PLEG): container finished" podID="99c52a19-7be6-44d6-968a-a6dad23d6ffa" containerID="6c36499bb7d2b7fafae4f03ffa1f52aa974f43e60d43aac123238840485f719e" exitCode=0 Jul 20 03:15:03.976053 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:15:03.975652 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741955-7qppl" event={"ID":"99c52a19-7be6-44d6-968a-a6dad23d6ffa","Type":"ContainerDied","Data":"6c36499bb7d2b7fafae4f03ffa1f52aa974f43e60d43aac123238840485f719e"} Jul 20 03:15:05.109062 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:15:05.109034 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741955-7qppl" Jul 20 03:15:05.985867 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:15:05.985839 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741955-7qppl" Jul 20 03:15:05.985867 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:15:05.985864 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741955-7qppl" event={"ID":"99c52a19-7be6-44d6-968a-a6dad23d6ffa","Type":"ContainerDied","Data":"2bd7880aa115471cf436f7c37acfe7d54951e72b0c72d40cac6d7a0b9228b92b"} Jul 20 03:15:05.986079 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:15:05.985907 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bd7880aa115471cf436f7c37acfe7d54951e72b0c72d40cac6d7a0b9228b92b" Jul 20 03:15:09.219043 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:15:09.219006 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:15:19.515924 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:15:19.515892 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:15:34.421111 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:15:34.421067 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:15:43.129035 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:15:43.128992 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:15:55.917450 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:15:55.917411 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:16:12.029871 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:16:12.029836 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:16:24.027614 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:16:24.027562 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:16:35.621366 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:16:35.621282 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:16:48.015968 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:16:48.015929 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:17:00.534161 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:00.534127 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:17:06.694354 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:06.694317 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m"] Jul 20 03:17:06.694810 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:06.694786 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99c52a19-7be6-44d6-968a-a6dad23d6ffa" containerName="cleanup" Jul 20 03:17:06.694810 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:06.694800 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="99c52a19-7be6-44d6-968a-a6dad23d6ffa" containerName="cleanup" Jul 20 03:17:06.694880 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:06.694866 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="99c52a19-7be6-44d6-968a-a6dad23d6ffa" containerName="cleanup" Jul 20 03:17:06.697945 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:06.697928 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m" Jul 20 03:17:06.700786 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:06.700763 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-33118b8e-openshift-default-dockercfg-97m8p\"" Jul 20 03:17:06.719308 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:06.719285 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m"] Jul 20 03:17:06.811812 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:06.811775 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f9hd\" (UniqueName: \"kubernetes.io/projected/22c49739-59d1-4228-a94b-8e23ef05978d-kube-api-access-7f9hd\") pod \"e2e-ait-33118b8e-openshift-default-69f75595f7-9946m\" (UID: \"22c49739-59d1-4228-a94b-8e23ef05978d\") " pod="openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m" Jul 20 03:17:06.812016 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:06.811822 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22c49739-59d1-4228-a94b-8e23ef05978d-workload-certs\") pod \"e2e-ait-33118b8e-openshift-default-69f75595f7-9946m\" (UID: \"22c49739-59d1-4228-a94b-8e23ef05978d\") " pod="openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m" Jul 20 03:17:06.812016 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:06.811848 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22c49739-59d1-4228-a94b-8e23ef05978d-credential-socket\") pod \"e2e-ait-33118b8e-openshift-default-69f75595f7-9946m\" (UID: \"22c49739-59d1-4228-a94b-8e23ef05978d\") " pod="openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m" Jul 20 03:17:06.812016 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:06.811958 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22c49739-59d1-4228-a94b-8e23ef05978d-istio-data\") pod \"e2e-ait-33118b8e-openshift-default-69f75595f7-9946m\" (UID: \"22c49739-59d1-4228-a94b-8e23ef05978d\") " pod="openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m" Jul 20 03:17:06.812016 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:06.811991 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22c49739-59d1-4228-a94b-8e23ef05978d-istio-podinfo\") pod \"e2e-ait-33118b8e-openshift-default-69f75595f7-9946m\" (UID: \"22c49739-59d1-4228-a94b-8e23ef05978d\") " pod="openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m" Jul 20 03:17:06.812204 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:06.812077 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22c49739-59d1-4228-a94b-8e23ef05978d-istiod-ca-cert\") pod \"e2e-ait-33118b8e-openshift-default-69f75595f7-9946m\" (UID: \"22c49739-59d1-4228-a94b-8e23ef05978d\") " pod="openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m" Jul 20 03:17:06.812204 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:06.812128 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22c49739-59d1-4228-a94b-8e23ef05978d-istio-envoy\") pod \"e2e-ait-33118b8e-openshift-default-69f75595f7-9946m\" (UID: \"22c49739-59d1-4228-a94b-8e23ef05978d\") " pod="openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m" Jul 20 03:17:06.812204 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:06.812161 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22c49739-59d1-4228-a94b-8e23ef05978d-workload-socket\") pod \"e2e-ait-33118b8e-openshift-default-69f75595f7-9946m\" (UID: \"22c49739-59d1-4228-a94b-8e23ef05978d\") " pod="openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m" Jul 20 03:17:06.812204 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:06.812199 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22c49739-59d1-4228-a94b-8e23ef05978d-istio-token\") pod \"e2e-ait-33118b8e-openshift-default-69f75595f7-9946m\" (UID: \"22c49739-59d1-4228-a94b-8e23ef05978d\") " pod="openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m" Jul 20 03:17:06.913121 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:06.913064 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22c49739-59d1-4228-a94b-8e23ef05978d-istiod-ca-cert\") pod \"e2e-ait-33118b8e-openshift-default-69f75595f7-9946m\" (UID: \"22c49739-59d1-4228-a94b-8e23ef05978d\") " pod="openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m" Jul 20 03:17:06.913313 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:06.913132 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22c49739-59d1-4228-a94b-8e23ef05978d-istio-envoy\") pod \"e2e-ait-33118b8e-openshift-default-69f75595f7-9946m\" (UID: \"22c49739-59d1-4228-a94b-8e23ef05978d\") " pod="openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m" Jul 20 03:17:06.913313 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:06.913167 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22c49739-59d1-4228-a94b-8e23ef05978d-workload-socket\") pod \"e2e-ait-33118b8e-openshift-default-69f75595f7-9946m\" (UID: \"22c49739-59d1-4228-a94b-8e23ef05978d\") " pod="openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m" Jul 20 03:17:06.913313 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:06.913195 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22c49739-59d1-4228-a94b-8e23ef05978d-istio-token\") pod \"e2e-ait-33118b8e-openshift-default-69f75595f7-9946m\" (UID: \"22c49739-59d1-4228-a94b-8e23ef05978d\") " pod="openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m" Jul 20 03:17:06.913313 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:06.913231 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f9hd\" (UniqueName: \"kubernetes.io/projected/22c49739-59d1-4228-a94b-8e23ef05978d-kube-api-access-7f9hd\") pod \"e2e-ait-33118b8e-openshift-default-69f75595f7-9946m\" (UID: \"22c49739-59d1-4228-a94b-8e23ef05978d\") " pod="openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m" Jul 20 03:17:06.913313 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:06.913264 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22c49739-59d1-4228-a94b-8e23ef05978d-workload-certs\") pod \"e2e-ait-33118b8e-openshift-default-69f75595f7-9946m\" (UID: \"22c49739-59d1-4228-a94b-8e23ef05978d\") " pod="openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m" Jul 20 03:17:06.913313 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:06.913299 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22c49739-59d1-4228-a94b-8e23ef05978d-credential-socket\") pod \"e2e-ait-33118b8e-openshift-default-69f75595f7-9946m\" (UID: \"22c49739-59d1-4228-a94b-8e23ef05978d\") " pod="openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m" Jul 20 03:17:06.913612 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:06.913354 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22c49739-59d1-4228-a94b-8e23ef05978d-istio-data\") pod \"e2e-ait-33118b8e-openshift-default-69f75595f7-9946m\" (UID: \"22c49739-59d1-4228-a94b-8e23ef05978d\") " pod="openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m" Jul 20 03:17:06.913612 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:06.913379 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22c49739-59d1-4228-a94b-8e23ef05978d-istio-podinfo\") pod \"e2e-ait-33118b8e-openshift-default-69f75595f7-9946m\" (UID: \"22c49739-59d1-4228-a94b-8e23ef05978d\") " pod="openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m" Jul 20 03:17:06.913771 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:06.913750 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22c49739-59d1-4228-a94b-8e23ef05978d-workload-socket\") pod \"e2e-ait-33118b8e-openshift-default-69f75595f7-9946m\" (UID: \"22c49739-59d1-4228-a94b-8e23ef05978d\") " pod="openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m" Jul 20 03:17:06.913825 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:06.913793 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22c49739-59d1-4228-a94b-8e23ef05978d-credential-socket\") pod \"e2e-ait-33118b8e-openshift-default-69f75595f7-9946m\" (UID: \"22c49739-59d1-4228-a94b-8e23ef05978d\") " pod="openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m" Jul 20 03:17:06.913825 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:06.913809 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22c49739-59d1-4228-a94b-8e23ef05978d-istio-data\") pod \"e2e-ait-33118b8e-openshift-default-69f75595f7-9946m\" (UID: \"22c49739-59d1-4228-a94b-8e23ef05978d\") " pod="openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m" Jul 20 03:17:06.913898 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:06.913888 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22c49739-59d1-4228-a94b-8e23ef05978d-workload-certs\") pod \"e2e-ait-33118b8e-openshift-default-69f75595f7-9946m\" (UID: \"22c49739-59d1-4228-a94b-8e23ef05978d\") " pod="openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m" Jul 20 03:17:06.913945 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:06.913926 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22c49739-59d1-4228-a94b-8e23ef05978d-istiod-ca-cert\") pod \"e2e-ait-33118b8e-openshift-default-69f75595f7-9946m\" (UID: \"22c49739-59d1-4228-a94b-8e23ef05978d\") " pod="openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m" Jul 20 03:17:06.915588 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:06.915563 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22c49739-59d1-4228-a94b-8e23ef05978d-istio-envoy\") pod \"e2e-ait-33118b8e-openshift-default-69f75595f7-9946m\" (UID: \"22c49739-59d1-4228-a94b-8e23ef05978d\") " pod="openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m" Jul 20 03:17:06.915986 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:06.915968 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22c49739-59d1-4228-a94b-8e23ef05978d-istio-podinfo\") pod \"e2e-ait-33118b8e-openshift-default-69f75595f7-9946m\" (UID: \"22c49739-59d1-4228-a94b-8e23ef05978d\") " pod="openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m" Jul 20 03:17:06.922033 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:06.922010 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22c49739-59d1-4228-a94b-8e23ef05978d-istio-token\") pod \"e2e-ait-33118b8e-openshift-default-69f75595f7-9946m\" (UID: \"22c49739-59d1-4228-a94b-8e23ef05978d\") " pod="openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m" Jul 20 03:17:06.922336 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:06.922308 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f9hd\" (UniqueName: \"kubernetes.io/projected/22c49739-59d1-4228-a94b-8e23ef05978d-kube-api-access-7f9hd\") pod \"e2e-ait-33118b8e-openshift-default-69f75595f7-9946m\" (UID: \"22c49739-59d1-4228-a94b-8e23ef05978d\") " pod="openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m" Jul 20 03:17:07.008159 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:07.008030 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m" Jul 20 03:17:07.150268 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:07.150242 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m"] Jul 20 03:17:07.152563 ip-10-0-142-139 kubenswrapper[2558]: W0720 03:17:07.152537 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22c49739_59d1_4228_a94b_8e23ef05978d.slice/crio-70c781ce9ff631f59829f05fb1ae828eef959f37e0fdc5536b06b99ae2b54319 WatchSource:0}: Error finding container 70c781ce9ff631f59829f05fb1ae828eef959f37e0fdc5536b06b99ae2b54319: Status 404 returned error can't find the container with id 70c781ce9ff631f59829f05fb1ae828eef959f37e0fdc5536b06b99ae2b54319 Jul 20 03:17:07.154655 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:07.154627 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:17:07.154710 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:07.154687 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:17:07.154949 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:07.154713 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:17:07.540676 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:07.540643 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m" event={"ID":"22c49739-59d1-4228-a94b-8e23ef05978d","Type":"ContainerStarted","Data":"6452c40a71a40f1174587aee1645cae51de5523aafe6cc2aec66664a88e6b449"} Jul 20 03:17:07.540840 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:07.540693 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m" event={"ID":"22c49739-59d1-4228-a94b-8e23ef05978d","Type":"ContainerStarted","Data":"70c781ce9ff631f59829f05fb1ae828eef959f37e0fdc5536b06b99ae2b54319"} Jul 20 03:17:07.561413 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:07.561362 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m" podStartSLOduration=1.561346814 podStartE2EDuration="1.561346814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 03:17:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 03:17:07.55972412 +0000 UTC m=+1922.758973268" watchObservedRunningTime="2026-07-20 03:17:07.561346814 +0000 UTC m=+1922.760596039" Jul 20 03:17:08.008413 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:08.008373 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m" Jul 20 03:17:08.013231 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:08.013207 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m" Jul 20 03:17:08.545164 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:08.545131 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m" Jul 20 03:17:08.546074 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:08.546056 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m" Jul 20 03:17:14.219840 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:14.219803 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:17:22.613995 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:22.613902 2558 generic.go:356] "Generic (PLEG): container finished" podID="22c49739-59d1-4228-a94b-8e23ef05978d" containerID="6452c40a71a40f1174587aee1645cae51de5523aafe6cc2aec66664a88e6b449" exitCode=0 Jul 20 03:17:22.613995 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:22.613977 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m" event={"ID":"22c49739-59d1-4228-a94b-8e23ef05978d","Type":"ContainerDied","Data":"6452c40a71a40f1174587aee1645cae51de5523aafe6cc2aec66664a88e6b449"} Jul 20 03:17:22.614474 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:22.614394 2558 scope.go:117] "RemoveContainer" containerID="6452c40a71a40f1174587aee1645cae51de5523aafe6cc2aec66664a88e6b449" Jul 20 03:17:22.615445 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:22.615419 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:17:22.615516 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:22.615499 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:17:22.615593 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:22.615570 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:17:23.546068 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:23.546029 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m" Jul 20 03:17:23.620903 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:23.620862 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m" event={"ID":"22c49739-59d1-4228-a94b-8e23ef05978d","Type":"ContainerStarted","Data":"50045a44f2d4ed101717eb945c0012c4d89640860c747f411124e2b8f642d2d6"} Jul 20 03:17:24.008582 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:24.008538 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m" Jul 20 03:17:24.009808 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:24.009782 2558 patch_prober.go:28] interesting pod/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Jul 20 03:17:24.009938 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:24.009862 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m" podUID="22c49739-59d1-4228-a94b-8e23ef05978d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Jul 20 03:17:25.009022 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:25.008985 2558 patch_prober.go:28] interesting pod/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Jul 20 03:17:25.009435 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:25.009036 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m" podUID="22c49739-59d1-4228-a94b-8e23ef05978d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Jul 20 03:17:26.008675 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:26.008637 2558 patch_prober.go:28] interesting pod/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Jul 20 03:17:26.008859 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:26.008704 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m" podUID="22c49739-59d1-4228-a94b-8e23ef05978d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Jul 20 03:17:27.009299 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:27.009257 2558 patch_prober.go:28] interesting pod/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Jul 20 03:17:27.009764 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:27.009330 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m" podUID="22c49739-59d1-4228-a94b-8e23ef05978d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Jul 20 03:17:28.013142 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:28.013108 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m" Jul 20 03:17:28.013600 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:28.013406 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m" Jul 20 03:17:28.014039 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:28.014020 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m" Jul 20 03:17:34.727834 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:34.727794 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33118b8e-28msv"] Jul 20 03:17:34.731763 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:34.731738 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33118b8e-28msv" Jul 20 03:17:34.734257 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:34.734238 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 20 03:17:34.734347 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:34.734293 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hlzh6\"" Jul 20 03:17:34.738066 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:34.738038 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33118b8e-28msv"] Jul 20 03:17:34.876139 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:34.876072 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f0228d1-4904-459b-a693-e70ce5455423-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-33118b8e-28msv\" (UID: \"6f0228d1-4904-459b-a693-e70ce5455423\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33118b8e-28msv" Jul 20 03:17:34.977413 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:34.977372 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f0228d1-4904-459b-a693-e70ce5455423-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-33118b8e-28msv\" (UID: \"6f0228d1-4904-459b-a693-e70ce5455423\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33118b8e-28msv" Jul 20 03:17:34.978038 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:34.977990 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f0228d1-4904-459b-a693-e70ce5455423-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-33118b8e-28msv\" (UID: \"6f0228d1-4904-459b-a693-e70ce5455423\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33118b8e-28msv" Jul 20 03:17:35.043567 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:35.043522 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33118b8e-28msv" Jul 20 03:17:35.170988 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:35.170963 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33118b8e-28msv"] Jul 20 03:17:35.173011 ip-10-0-142-139 kubenswrapper[2558]: W0720 03:17:35.172988 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f0228d1_4904_459b_a693_e70ce5455423.slice/crio-cfffa3c03caa753cdd220699284cdf85a6c94c88f63b2713ff448bdd264ed0e2 WatchSource:0}: Error finding container cfffa3c03caa753cdd220699284cdf85a6c94c88f63b2713ff448bdd264ed0e2: Status 404 returned error can't find the container with id cfffa3c03caa753cdd220699284cdf85a6c94c88f63b2713ff448bdd264ed0e2 Jul 20 03:17:35.677572 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:35.677533 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33118b8e-28msv" event={"ID":"6f0228d1-4904-459b-a693-e70ce5455423","Type":"ContainerStarted","Data":"a577786330492de120457758b9a2cdd9b3a6c76678b0ef3ea24797cc9b653cbe"} Jul 20 03:17:35.677717 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:35.677591 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33118b8e-28msv" event={"ID":"6f0228d1-4904-459b-a693-e70ce5455423","Type":"ContainerStarted","Data":"cfffa3c03caa753cdd220699284cdf85a6c94c88f63b2713ff448bdd264ed0e2"} Jul 20 03:17:35.692691 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:35.692642 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33118b8e-28msv" podStartSLOduration=1.69262428 podStartE2EDuration="1.69262428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 03:17:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 03:17:35.691612888 +0000 UTC m=+1950.890862036" watchObservedRunningTime="2026-07-20 03:17:35.69262428 +0000 UTC m=+1950.891873427" Jul 20 03:17:36.682925 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:36.682895 2558 generic.go:356] "Generic (PLEG): container finished" podID="6f0228d1-4904-459b-a693-e70ce5455423" containerID="a577786330492de120457758b9a2cdd9b3a6c76678b0ef3ea24797cc9b653cbe" exitCode=0 Jul 20 03:17:36.683392 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:36.682966 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33118b8e-28msv" event={"ID":"6f0228d1-4904-459b-a693-e70ce5455423","Type":"ContainerDied","Data":"a577786330492de120457758b9a2cdd9b3a6c76678b0ef3ea24797cc9b653cbe"} Jul 20 03:17:37.817938 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:37.817915 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33118b8e-28msv" Jul 20 03:17:38.006081 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:38.005995 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f0228d1-4904-459b-a693-e70ce5455423-maas-api-service-ca\") pod \"6f0228d1-4904-459b-a693-e70ce5455423\" (UID: \"6f0228d1-4904-459b-a693-e70ce5455423\") " Jul 20 03:17:38.006414 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:38.006394 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f0228d1-4904-459b-a693-e70ce5455423-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6f0228d1-4904-459b-a693-e70ce5455423" (UID: "6f0228d1-4904-459b-a693-e70ce5455423"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 03:17:38.107673 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:38.107643 2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f0228d1-4904-459b-a693-e70ce5455423-maas-api-service-ca\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:17:38.693661 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:38.693634 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33118b8e-28msv" Jul 20 03:17:38.693806 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:38.693658 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33118b8e-28msv" event={"ID":"6f0228d1-4904-459b-a693-e70ce5455423","Type":"ContainerDied","Data":"cfffa3c03caa753cdd220699284cdf85a6c94c88f63b2713ff448bdd264ed0e2"} Jul 20 03:17:38.693806 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:38.693706 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfffa3c03caa753cdd220699284cdf85a6c94c88f63b2713ff448bdd264ed0e2" Jul 20 03:17:48.319780 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:48.319738 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:17:49.804850 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:49.804816 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33118b8e-28msv"] Jul 20 03:17:49.809541 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:49.809502 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33118b8e-28msv"] Jul 20 03:17:49.955064 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:49.955028 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m"] Jul 20 03:17:49.955346 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:49.955290 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m" podUID="22c49739-59d1-4228-a94b-8e23ef05978d" containerName="istio-proxy" containerID="cri-o://50045a44f2d4ed101717eb945c0012c4d89640860c747f411124e2b8f642d2d6" gracePeriod=30 Jul 20 03:17:50.990091 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:50.990053 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp"] Jul 20 03:17:50.990626 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:50.990607 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f0228d1-4904-459b-a693-e70ce5455423" containerName="revoke-keys" Jul 20 03:17:50.990678 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:50.990631 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f0228d1-4904-459b-a693-e70ce5455423" containerName="revoke-keys" Jul 20 03:17:50.990750 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:50.990739 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f0228d1-4904-459b-a693-e70ce5455423" containerName="revoke-keys" Jul 20 03:17:50.995133 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:50.995112 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp" Jul 20 03:17:50.997933 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:50.997911 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-53dce522-gw-openshift-default-dockercfg-jmmqk\"" Jul 20 03:17:51.005213 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:51.005187 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp"] Jul 20 03:17:51.016011 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:51.015920 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d10b552-61a5-4fdb-9b1a-e2ef5af447dc-istio-data\") pod \"e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp\" (UID: \"5d10b552-61a5-4fdb-9b1a-e2ef5af447dc\") " pod="openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp" Jul 20 03:17:51.016151 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:51.016073 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d10b552-61a5-4fdb-9b1a-e2ef5af447dc-workload-certs\") pod \"e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp\" (UID: \"5d10b552-61a5-4fdb-9b1a-e2ef5af447dc\") " pod="openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp" Jul 20 03:17:51.016284 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:51.016266 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d10b552-61a5-4fdb-9b1a-e2ef5af447dc-credential-socket\") pod \"e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp\" (UID: \"5d10b552-61a5-4fdb-9b1a-e2ef5af447dc\") " pod="openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp" Jul 20 03:17:51.016323 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:51.016299 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d10b552-61a5-4fdb-9b1a-e2ef5af447dc-istio-podinfo\") pod \"e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp\" (UID: \"5d10b552-61a5-4fdb-9b1a-e2ef5af447dc\") " pod="openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp" Jul 20 03:17:51.016368 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:51.016343 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d10b552-61a5-4fdb-9b1a-e2ef5af447dc-istiod-ca-cert\") pod \"e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp\" (UID: \"5d10b552-61a5-4fdb-9b1a-e2ef5af447dc\") " pod="openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp" Jul 20 03:17:51.016415 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:51.016376 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d10b552-61a5-4fdb-9b1a-e2ef5af447dc-istio-envoy\") pod \"e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp\" (UID: \"5d10b552-61a5-4fdb-9b1a-e2ef5af447dc\") " pod="openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp" Jul 20 03:17:51.016583 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:51.016568 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d10b552-61a5-4fdb-9b1a-e2ef5af447dc-workload-socket\") pod \"e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp\" (UID: \"5d10b552-61a5-4fdb-9b1a-e2ef5af447dc\") " pod="openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp" Jul 20 03:17:51.016654 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:51.016633 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d10b552-61a5-4fdb-9b1a-e2ef5af447dc-istio-token\") pod \"e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp\" (UID: \"5d10b552-61a5-4fdb-9b1a-e2ef5af447dc\") " pod="openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp" Jul 20 03:17:51.016780 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:51.016671 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4nq2\" (UniqueName: \"kubernetes.io/projected/5d10b552-61a5-4fdb-9b1a-e2ef5af447dc-kube-api-access-t4nq2\") pod \"e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp\" (UID: \"5d10b552-61a5-4fdb-9b1a-e2ef5af447dc\") " pod="openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp" Jul 20 03:17:51.118082 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:51.118045 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d10b552-61a5-4fdb-9b1a-e2ef5af447dc-istio-envoy\") pod \"e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp\" (UID: \"5d10b552-61a5-4fdb-9b1a-e2ef5af447dc\") " pod="openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp" Jul 20 03:17:51.118288 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:51.118117 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d10b552-61a5-4fdb-9b1a-e2ef5af447dc-workload-socket\") pod \"e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp\" (UID: \"5d10b552-61a5-4fdb-9b1a-e2ef5af447dc\") " pod="openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp" Jul 20 03:17:51.118288 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:51.118164 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d10b552-61a5-4fdb-9b1a-e2ef5af447dc-istio-token\") pod \"e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp\" (UID: \"5d10b552-61a5-4fdb-9b1a-e2ef5af447dc\") " pod="openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp" Jul 20 03:17:51.118288 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:51.118184 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4nq2\" (UniqueName: \"kubernetes.io/projected/5d10b552-61a5-4fdb-9b1a-e2ef5af447dc-kube-api-access-t4nq2\") pod \"e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp\" (UID: \"5d10b552-61a5-4fdb-9b1a-e2ef5af447dc\") " pod="openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp" Jul 20 03:17:51.118288 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:51.118217 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d10b552-61a5-4fdb-9b1a-e2ef5af447dc-istio-data\") pod \"e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp\" (UID: \"5d10b552-61a5-4fdb-9b1a-e2ef5af447dc\") " pod="openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp" Jul 20 03:17:51.118288 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:51.118239 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d10b552-61a5-4fdb-9b1a-e2ef5af447dc-workload-certs\") pod \"e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp\" (UID: \"5d10b552-61a5-4fdb-9b1a-e2ef5af447dc\") " pod="openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp" Jul 20 03:17:51.118288 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:51.118272 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d10b552-61a5-4fdb-9b1a-e2ef5af447dc-credential-socket\") pod \"e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp\" (UID: \"5d10b552-61a5-4fdb-9b1a-e2ef5af447dc\") " pod="openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp" Jul 20 03:17:51.118988 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:51.118295 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d10b552-61a5-4fdb-9b1a-e2ef5af447dc-istio-podinfo\") pod \"e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp\" (UID: \"5d10b552-61a5-4fdb-9b1a-e2ef5af447dc\") " pod="openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp" Jul 20 03:17:51.118988 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:51.118341 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d10b552-61a5-4fdb-9b1a-e2ef5af447dc-istiod-ca-cert\") pod \"e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp\" (UID: \"5d10b552-61a5-4fdb-9b1a-e2ef5af447dc\") " pod="openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp" Jul 20 03:17:51.118988 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:51.118694 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d10b552-61a5-4fdb-9b1a-e2ef5af447dc-workload-certs\") pod \"e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp\" (UID: \"5d10b552-61a5-4fdb-9b1a-e2ef5af447dc\") " pod="openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp" Jul 20 03:17:51.118988 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:51.118770 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d10b552-61a5-4fdb-9b1a-e2ef5af447dc-istio-data\") pod \"e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp\" (UID: \"5d10b552-61a5-4fdb-9b1a-e2ef5af447dc\") " pod="openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp" Jul 20 03:17:51.118988 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:51.118799 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d10b552-61a5-4fdb-9b1a-e2ef5af447dc-credential-socket\") pod \"e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp\" (UID: \"5d10b552-61a5-4fdb-9b1a-e2ef5af447dc\") " pod="openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp" Jul 20 03:17:51.118988 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:51.118899 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d10b552-61a5-4fdb-9b1a-e2ef5af447dc-workload-socket\") pod \"e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp\" (UID: \"5d10b552-61a5-4fdb-9b1a-e2ef5af447dc\") " pod="openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp" Jul 20 03:17:51.119218 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:51.119010 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d10b552-61a5-4fdb-9b1a-e2ef5af447dc-istiod-ca-cert\") pod \"e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp\" (UID: \"5d10b552-61a5-4fdb-9b1a-e2ef5af447dc\") " pod="openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp" Jul 20 03:17:51.121021 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:51.120996 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d10b552-61a5-4fdb-9b1a-e2ef5af447dc-istio-envoy\") pod \"e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp\" (UID: \"5d10b552-61a5-4fdb-9b1a-e2ef5af447dc\") " pod="openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp" Jul 20 03:17:51.121338 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:51.121317 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d10b552-61a5-4fdb-9b1a-e2ef5af447dc-istio-podinfo\") pod \"e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp\" (UID: \"5d10b552-61a5-4fdb-9b1a-e2ef5af447dc\") " pod="openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp" Jul 20 03:17:51.127020 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:51.126996 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d10b552-61a5-4fdb-9b1a-e2ef5af447dc-istio-token\") pod \"e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp\" (UID: \"5d10b552-61a5-4fdb-9b1a-e2ef5af447dc\") " pod="openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp" Jul 20 03:17:51.127266 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:51.127235 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4nq2\" (UniqueName: \"kubernetes.io/projected/5d10b552-61a5-4fdb-9b1a-e2ef5af447dc-kube-api-access-t4nq2\") pod \"e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp\" (UID: \"5d10b552-61a5-4fdb-9b1a-e2ef5af447dc\") " pod="openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp" Jul 20 03:17:51.308583 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:51.308485 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp" Jul 20 03:17:51.425065 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:51.425031 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f0228d1-4904-459b-a693-e70ce5455423" path="/var/lib/kubelet/pods/6f0228d1-4904-459b-a693-e70ce5455423/volumes" Jul 20 03:17:51.447566 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:51.447540 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp"] Jul 20 03:17:51.449624 ip-10-0-142-139 kubenswrapper[2558]: W0720 03:17:51.449599 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d10b552_61a5_4fdb_9b1a_e2ef5af447dc.slice/crio-c4a554499754881e89ed3f0f8a9b6031a15021587b58fef31f415897b562be95 WatchSource:0}: Error finding container c4a554499754881e89ed3f0f8a9b6031a15021587b58fef31f415897b562be95: Status 404 returned error can't find the container with id c4a554499754881e89ed3f0f8a9b6031a15021587b58fef31f415897b562be95 Jul 20 03:17:51.452187 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:51.452154 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:17:51.452289 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:51.452229 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:17:51.452289 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:51.452275 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:17:51.752500 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:51.752462 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp" event={"ID":"5d10b552-61a5-4fdb-9b1a-e2ef5af447dc","Type":"ContainerStarted","Data":"25caf47238ccb25876389f0550bd02f20c9cbf6e9b85d722c35a324a74d106af"} Jul 20 03:17:51.752718 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:51.752522 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp" event={"ID":"5d10b552-61a5-4fdb-9b1a-e2ef5af447dc","Type":"ContainerStarted","Data":"c4a554499754881e89ed3f0f8a9b6031a15021587b58fef31f415897b562be95"} Jul 20 03:17:51.770045 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:51.769991 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp" podStartSLOduration=1.769969881 podStartE2EDuration="1.769969881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 03:17:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 03:17:51.769372511 +0000 UTC m=+1966.968621659" watchObservedRunningTime="2026-07-20 03:17:51.769969881 +0000 UTC m=+1966.969219028" Jul 20 03:17:52.309628 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:52.309588 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp" Jul 20 03:17:52.314899 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:52.314868 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp" Jul 20 03:17:52.757503 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:52.757476 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp" Jul 20 03:17:52.758487 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:52.758466 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp" Jul 20 03:17:55.204659 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:55.204629 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m" Jul 20 03:17:55.255804 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:55.255727 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22c49739-59d1-4228-a94b-8e23ef05978d-istio-data\") pod \"22c49739-59d1-4228-a94b-8e23ef05978d\" (UID: \"22c49739-59d1-4228-a94b-8e23ef05978d\") " Jul 20 03:17:55.255976 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:55.255803 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22c49739-59d1-4228-a94b-8e23ef05978d-istio-podinfo\") pod \"22c49739-59d1-4228-a94b-8e23ef05978d\" (UID: \"22c49739-59d1-4228-a94b-8e23ef05978d\") " Jul 20 03:17:55.255976 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:55.255839 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7f9hd\" (UniqueName: \"kubernetes.io/projected/22c49739-59d1-4228-a94b-8e23ef05978d-kube-api-access-7f9hd\") pod \"22c49739-59d1-4228-a94b-8e23ef05978d\" (UID: \"22c49739-59d1-4228-a94b-8e23ef05978d\") " Jul 20 03:17:55.255976 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:55.255873 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22c49739-59d1-4228-a94b-8e23ef05978d-istiod-ca-cert\") pod \"22c49739-59d1-4228-a94b-8e23ef05978d\" (UID: \"22c49739-59d1-4228-a94b-8e23ef05978d\") " Jul 20 03:17:55.256199 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:55.256002 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22c49739-59d1-4228-a94b-8e23ef05978d-istio-envoy\") pod \"22c49739-59d1-4228-a94b-8e23ef05978d\" (UID: \"22c49739-59d1-4228-a94b-8e23ef05978d\") " Jul 20 03:17:55.256199 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:55.256049 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22c49739-59d1-4228-a94b-8e23ef05978d-workload-certs\") pod \"22c49739-59d1-4228-a94b-8e23ef05978d\" (UID: \"22c49739-59d1-4228-a94b-8e23ef05978d\") " Jul 20 03:17:55.256199 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:55.256078 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22c49739-59d1-4228-a94b-8e23ef05978d-istio-token\") pod \"22c49739-59d1-4228-a94b-8e23ef05978d\" (UID: \"22c49739-59d1-4228-a94b-8e23ef05978d\") " Jul 20 03:17:55.256199 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:55.256141 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22c49739-59d1-4228-a94b-8e23ef05978d-workload-socket\") pod \"22c49739-59d1-4228-a94b-8e23ef05978d\" (UID: \"22c49739-59d1-4228-a94b-8e23ef05978d\") " Jul 20 03:17:55.256199 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:55.256184 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22c49739-59d1-4228-a94b-8e23ef05978d-credential-socket\") pod \"22c49739-59d1-4228-a94b-8e23ef05978d\" (UID: \"22c49739-59d1-4228-a94b-8e23ef05978d\") " Jul 20 03:17:55.256446 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:55.256315 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22c49739-59d1-4228-a94b-8e23ef05978d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "22c49739-59d1-4228-a94b-8e23ef05978d" (UID: "22c49739-59d1-4228-a94b-8e23ef05978d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:17:55.256527 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:55.256506 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22c49739-59d1-4228-a94b-8e23ef05978d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "22c49739-59d1-4228-a94b-8e23ef05978d" (UID: "22c49739-59d1-4228-a94b-8e23ef05978d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:17:55.256590 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:55.256558 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22c49739-59d1-4228-a94b-8e23ef05978d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "22c49739-59d1-4228-a94b-8e23ef05978d" (UID: "22c49739-59d1-4228-a94b-8e23ef05978d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 03:17:55.256590 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:55.256568 2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22c49739-59d1-4228-a94b-8e23ef05978d-workload-certs\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:17:55.256722 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:55.256686 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22c49739-59d1-4228-a94b-8e23ef05978d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "22c49739-59d1-4228-a94b-8e23ef05978d" (UID: "22c49739-59d1-4228-a94b-8e23ef05978d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:17:55.256722 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:55.256707 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22c49739-59d1-4228-a94b-8e23ef05978d-istio-data" (OuterVolumeSpecName: "istio-data") pod "22c49739-59d1-4228-a94b-8e23ef05978d" (UID: "22c49739-59d1-4228-a94b-8e23ef05978d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:17:55.258234 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:55.258212 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22c49739-59d1-4228-a94b-8e23ef05978d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "22c49739-59d1-4228-a94b-8e23ef05978d" (UID: "22c49739-59d1-4228-a94b-8e23ef05978d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 20 03:17:55.258448 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:55.258412 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22c49739-59d1-4228-a94b-8e23ef05978d-istio-token" (OuterVolumeSpecName: "istio-token") pod "22c49739-59d1-4228-a94b-8e23ef05978d" (UID: "22c49739-59d1-4228-a94b-8e23ef05978d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 03:17:55.258535 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:55.258487 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22c49739-59d1-4228-a94b-8e23ef05978d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "22c49739-59d1-4228-a94b-8e23ef05978d" (UID: "22c49739-59d1-4228-a94b-8e23ef05978d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:17:55.258535 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:55.258492 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22c49739-59d1-4228-a94b-8e23ef05978d-kube-api-access-7f9hd" (OuterVolumeSpecName: "kube-api-access-7f9hd") pod "22c49739-59d1-4228-a94b-8e23ef05978d" (UID: "22c49739-59d1-4228-a94b-8e23ef05978d"). InnerVolumeSpecName "kube-api-access-7f9hd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 03:17:55.357751 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:55.357712 2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22c49739-59d1-4228-a94b-8e23ef05978d-istio-data\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:17:55.357751 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:55.357743 2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22c49739-59d1-4228-a94b-8e23ef05978d-istio-podinfo\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:17:55.357751 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:55.357753 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f9hd\" (UniqueName: \"kubernetes.io/projected/22c49739-59d1-4228-a94b-8e23ef05978d-kube-api-access-7f9hd\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:17:55.357751 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:55.357761 2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22c49739-59d1-4228-a94b-8e23ef05978d-istiod-ca-cert\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:17:55.358309 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:55.357769 2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22c49739-59d1-4228-a94b-8e23ef05978d-istio-envoy\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:17:55.358309 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:55.357777 2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22c49739-59d1-4228-a94b-8e23ef05978d-istio-token\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:17:55.358309 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:55.357785 2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22c49739-59d1-4228-a94b-8e23ef05978d-workload-socket\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:17:55.358309 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:55.357792 2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22c49739-59d1-4228-a94b-8e23ef05978d-credential-socket\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:17:55.778303 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:55.778262 2558 generic.go:356] "Generic (PLEG): container finished" podID="22c49739-59d1-4228-a94b-8e23ef05978d" containerID="50045a44f2d4ed101717eb945c0012c4d89640860c747f411124e2b8f642d2d6" exitCode=0 Jul 20 03:17:55.778483 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:55.778298 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m" event={"ID":"22c49739-59d1-4228-a94b-8e23ef05978d","Type":"ContainerDied","Data":"50045a44f2d4ed101717eb945c0012c4d89640860c747f411124e2b8f642d2d6"} Jul 20 03:17:55.778483 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:55.778330 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m" Jul 20 03:17:55.778483 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:55.778383 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m" event={"ID":"22c49739-59d1-4228-a94b-8e23ef05978d","Type":"ContainerDied","Data":"70c781ce9ff631f59829f05fb1ae828eef959f37e0fdc5536b06b99ae2b54319"} Jul 20 03:17:55.778483 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:55.778405 2558 scope.go:117] "RemoveContainer" containerID="50045a44f2d4ed101717eb945c0012c4d89640860c747f411124e2b8f642d2d6" Jul 20 03:17:55.787074 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:55.787050 2558 scope.go:117] "RemoveContainer" containerID="6452c40a71a40f1174587aee1645cae51de5523aafe6cc2aec66664a88e6b449" Jul 20 03:17:55.795292 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:55.795265 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m"] Jul 20 03:17:55.795695 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:55.795675 2558 scope.go:117] "RemoveContainer" containerID="50045a44f2d4ed101717eb945c0012c4d89640860c747f411124e2b8f642d2d6" Jul 20 03:17:55.795938 ip-10-0-142-139 kubenswrapper[2558]: E0720 03:17:55.795922 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50045a44f2d4ed101717eb945c0012c4d89640860c747f411124e2b8f642d2d6\": container with ID starting with 50045a44f2d4ed101717eb945c0012c4d89640860c747f411124e2b8f642d2d6 not found: ID does not exist" containerID="50045a44f2d4ed101717eb945c0012c4d89640860c747f411124e2b8f642d2d6" Jul 20 03:17:55.795989 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:55.795945 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50045a44f2d4ed101717eb945c0012c4d89640860c747f411124e2b8f642d2d6"} err="failed to get container status \"50045a44f2d4ed101717eb945c0012c4d89640860c747f411124e2b8f642d2d6\": rpc error: code = NotFound desc = could not find container \"50045a44f2d4ed101717eb945c0012c4d89640860c747f411124e2b8f642d2d6\": container with ID starting with 50045a44f2d4ed101717eb945c0012c4d89640860c747f411124e2b8f642d2d6 not found: ID does not exist" Jul 20 03:17:55.795989 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:55.795963 2558 scope.go:117] "RemoveContainer" containerID="6452c40a71a40f1174587aee1645cae51de5523aafe6cc2aec66664a88e6b449" Jul 20 03:17:55.796225 ip-10-0-142-139 kubenswrapper[2558]: E0720 03:17:55.796203 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6452c40a71a40f1174587aee1645cae51de5523aafe6cc2aec66664a88e6b449\": container with ID starting with 6452c40a71a40f1174587aee1645cae51de5523aafe6cc2aec66664a88e6b449 not found: ID does not exist" containerID="6452c40a71a40f1174587aee1645cae51de5523aafe6cc2aec66664a88e6b449" Jul 20 03:17:55.796290 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:55.796235 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6452c40a71a40f1174587aee1645cae51de5523aafe6cc2aec66664a88e6b449"} err="failed to get container status \"6452c40a71a40f1174587aee1645cae51de5523aafe6cc2aec66664a88e6b449\": rpc error: code = NotFound desc = could not find container \"6452c40a71a40f1174587aee1645cae51de5523aafe6cc2aec66664a88e6b449\": container with ID starting with 6452c40a71a40f1174587aee1645cae51de5523aafe6cc2aec66664a88e6b449 not found: ID does not exist" Jul 20 03:17:55.799029 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:55.799008 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-33118b8e-openshift-default-69f75595f7-9946m"] Jul 20 03:17:57.424036 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:57.423965 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22c49739-59d1-4228-a94b-8e23ef05978d" path="/var/lib/kubelet/pods/22c49739-59d1-4228-a94b-8e23ef05978d/volumes" Jul 20 03:17:59.920158 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:17:59.920115 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:18:07.759003 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:07.758963 2558 patch_prober.go:28] interesting pod/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Jul 20 03:18:07.759387 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:07.759023 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp" podUID="5d10b552-61a5-4fdb-9b1a-e2ef5af447dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Jul 20 03:18:07.835394 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:07.835307 2558 generic.go:356] "Generic (PLEG): container finished" podID="5d10b552-61a5-4fdb-9b1a-e2ef5af447dc" containerID="25caf47238ccb25876389f0550bd02f20c9cbf6e9b85d722c35a324a74d106af" exitCode=0 Jul 20 03:18:07.835394 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:07.835379 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp" event={"ID":"5d10b552-61a5-4fdb-9b1a-e2ef5af447dc","Type":"ContainerDied","Data":"25caf47238ccb25876389f0550bd02f20c9cbf6e9b85d722c35a324a74d106af"} Jul 20 03:18:07.835776 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:07.835762 2558 scope.go:117] "RemoveContainer" containerID="25caf47238ccb25876389f0550bd02f20c9cbf6e9b85d722c35a324a74d106af" Jul 20 03:18:07.836790 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:07.836765 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:18:07.836851 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:07.836838 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:18:07.836886 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:07.836873 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:18:08.842038 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:08.842001 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp" event={"ID":"5d10b552-61a5-4fdb-9b1a-e2ef5af447dc","Type":"ContainerStarted","Data":"ab137116734419fe518fc7c1707c93078b47a8b909e8bff85c0fda5881437f93"} Jul 20 03:18:09.309464 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:09.309417 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp" Jul 20 03:18:09.309689 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:09.309661 2558 patch_prober.go:28] interesting pod/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Jul 20 03:18:09.309840 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:09.309728 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp" podUID="5d10b552-61a5-4fdb-9b1a-e2ef5af447dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Jul 20 03:18:10.309040 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:10.309008 2558 patch_prober.go:28] interesting pod/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Jul 20 03:18:10.309452 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:10.309062 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp" podUID="5d10b552-61a5-4fdb-9b1a-e2ef5af447dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Jul 20 03:18:11.309293 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:11.309257 2558 patch_prober.go:28] interesting pod/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Jul 20 03:18:11.309678 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:11.309330 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp" podUID="5d10b552-61a5-4fdb-9b1a-e2ef5af447dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Jul 20 03:18:12.309706 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:12.309668 2558 patch_prober.go:28] interesting pod/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Jul 20 03:18:12.310136 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:12.309741 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp" podUID="5d10b552-61a5-4fdb-9b1a-e2ef5af447dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Jul 20 03:18:13.312721 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:13.312690 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp" Jul 20 03:18:13.313210 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:13.313066 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp" Jul 20 03:18:13.313828 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:13.313808 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp" Jul 20 03:18:18.231539 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:18.231499 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-53dce522-9prcz"] Jul 20 03:18:18.231949 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:18.231852 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22c49739-59d1-4228-a94b-8e23ef05978d" containerName="istio-proxy" Jul 20 03:18:18.231949 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:18.231864 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="22c49739-59d1-4228-a94b-8e23ef05978d" containerName="istio-proxy" Jul 20 03:18:18.231949 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:18.231871 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22c49739-59d1-4228-a94b-8e23ef05978d" containerName="istio-proxy" Jul 20 03:18:18.231949 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:18.231877 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="22c49739-59d1-4228-a94b-8e23ef05978d" containerName="istio-proxy" Jul 20 03:18:18.231949 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:18.231950 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="22c49739-59d1-4228-a94b-8e23ef05978d" containerName="istio-proxy" Jul 20 03:18:18.232168 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:18.231959 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="22c49739-59d1-4228-a94b-8e23ef05978d" containerName="istio-proxy" Jul 20 03:18:18.234930 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:18.234913 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-53dce522-9prcz" Jul 20 03:18:18.237302 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:18.237279 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 20 03:18:18.237420 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:18.237320 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hlzh6\"" Jul 20 03:18:18.244442 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:18.241950 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-53dce522-9prcz"] Jul 20 03:18:18.383314 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:18.383277 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc8ef5c4-fe12-4894-88b5-eb10bc562c25-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-53dce522-9prcz\" (UID: \"bc8ef5c4-fe12-4894-88b5-eb10bc562c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-53dce522-9prcz" Jul 20 03:18:18.484046 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:18.483960 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc8ef5c4-fe12-4894-88b5-eb10bc562c25-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-53dce522-9prcz\" (UID: \"bc8ef5c4-fe12-4894-88b5-eb10bc562c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-53dce522-9prcz" Jul 20 03:18:18.484596 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:18.484575 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc8ef5c4-fe12-4894-88b5-eb10bc562c25-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-53dce522-9prcz\" (UID: \"bc8ef5c4-fe12-4894-88b5-eb10bc562c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-53dce522-9prcz" Jul 20 03:18:18.548404 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:18.548373 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-53dce522-9prcz" Jul 20 03:18:18.677613 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:18.677570 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-53dce522-9prcz"] Jul 20 03:18:18.678369 ip-10-0-142-139 kubenswrapper[2558]: W0720 03:18:18.678344 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc8ef5c4_fe12_4894_88b5_eb10bc562c25.slice/crio-77efa7ec6b103689dce0c703a24b390c1616590bb1bc76bdff674e99d2bb4b21 WatchSource:0}: Error finding container 77efa7ec6b103689dce0c703a24b390c1616590bb1bc76bdff674e99d2bb4b21: Status 404 returned error can't find the container with id 77efa7ec6b103689dce0c703a24b390c1616590bb1bc76bdff674e99d2bb4b21 Jul 20 03:18:18.894990 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:18.894957 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-53dce522-9prcz" event={"ID":"bc8ef5c4-fe12-4894-88b5-eb10bc562c25","Type":"ContainerStarted","Data":"0921925f21a458d8633ffe0bceb620024c252c35a24d3ead860a2a17b1f829d0"} Jul 20 03:18:18.895135 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:18.895011 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-53dce522-9prcz" event={"ID":"bc8ef5c4-fe12-4894-88b5-eb10bc562c25","Type":"ContainerStarted","Data":"77efa7ec6b103689dce0c703a24b390c1616590bb1bc76bdff674e99d2bb4b21"} Jul 20 03:18:18.909312 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:18.909266 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-53dce522-9prcz" podStartSLOduration=0.909251403 podStartE2EDuration="909.251403ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 03:18:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 03:18:18.90790377 +0000 UTC m=+1994.107152920" watchObservedRunningTime="2026-07-20 03:18:18.909251403 +0000 UTC m=+1994.108500549" Jul 20 03:18:19.902609 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:19.902572 2558 generic.go:356] "Generic (PLEG): container finished" podID="bc8ef5c4-fe12-4894-88b5-eb10bc562c25" containerID="0921925f21a458d8633ffe0bceb620024c252c35a24d3ead860a2a17b1f829d0" exitCode=0 Jul 20 03:18:19.903175 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:19.902632 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-53dce522-9prcz" event={"ID":"bc8ef5c4-fe12-4894-88b5-eb10bc562c25","Type":"ContainerDied","Data":"0921925f21a458d8633ffe0bceb620024c252c35a24d3ead860a2a17b1f829d0"} Jul 20 03:18:21.036926 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:21.036900 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-53dce522-9prcz" Jul 20 03:18:21.107056 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:21.107022 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc8ef5c4-fe12-4894-88b5-eb10bc562c25-maas-api-service-ca\") pod \"bc8ef5c4-fe12-4894-88b5-eb10bc562c25\" (UID: \"bc8ef5c4-fe12-4894-88b5-eb10bc562c25\") " Jul 20 03:18:21.107440 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:21.107415 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc8ef5c4-fe12-4894-88b5-eb10bc562c25-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bc8ef5c4-fe12-4894-88b5-eb10bc562c25" (UID: "bc8ef5c4-fe12-4894-88b5-eb10bc562c25"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 03:18:21.207751 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:21.207670 2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc8ef5c4-fe12-4894-88b5-eb10bc562c25-maas-api-service-ca\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:18:21.914288 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:21.914253 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-53dce522-9prcz" event={"ID":"bc8ef5c4-fe12-4894-88b5-eb10bc562c25","Type":"ContainerDied","Data":"77efa7ec6b103689dce0c703a24b390c1616590bb1bc76bdff674e99d2bb4b21"} Jul 20 03:18:21.914288 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:21.914282 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-53dce522-9prcz" Jul 20 03:18:21.914505 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:21.914329 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77efa7ec6b103689dce0c703a24b390c1616590bb1bc76bdff674e99d2bb4b21" Jul 20 03:18:31.715078 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:31.715043 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:18:33.307738 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:33.307701 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-53dce522-9prcz"] Jul 20 03:18:33.309637 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:33.309614 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-53dce522-9prcz"] Jul 20 03:18:33.430426 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:33.430390 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc8ef5c4-fe12-4894-88b5-eb10bc562c25" path="/var/lib/kubelet/pods/bc8ef5c4-fe12-4894-88b5-eb10bc562c25/volumes" Jul 20 03:18:33.446904 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:33.446867 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp"] Jul 20 03:18:33.447227 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:33.447170 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp" podUID="5d10b552-61a5-4fdb-9b1a-e2ef5af447dc" containerName="istio-proxy" containerID="cri-o://ab137116734419fe518fc7c1707c93078b47a8b909e8bff85c0fda5881437f93" gracePeriod=30 Jul 20 03:18:34.217792 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:34.217751 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v"] Jul 20 03:18:34.218126 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:34.218112 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc8ef5c4-fe12-4894-88b5-eb10bc562c25" containerName="revoke-keys" Jul 20 03:18:34.218126 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:34.218126 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc8ef5c4-fe12-4894-88b5-eb10bc562c25" containerName="revoke-keys" Jul 20 03:18:34.218250 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:34.218191 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc8ef5c4-fe12-4894-88b5-eb10bc562c25" containerName="revoke-keys" Jul 20 03:18:34.222044 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:34.222022 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v" Jul 20 03:18:34.224829 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:34.224811 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-dd7b8c7f-openshift-default-dockercfg-9fmnv\"" Jul 20 03:18:34.238612 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:34.238585 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v"] Jul 20 03:18:34.308333 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:34.308300 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ec4ce53-e616-4171-a5b8-0972046b2429-istio-data\") pod \"e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v\" (UID: \"7ec4ce53-e616-4171-a5b8-0972046b2429\") " pod="openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v" Jul 20 03:18:34.308333 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:34.308343 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ec4ce53-e616-4171-a5b8-0972046b2429-istio-podinfo\") pod \"e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v\" (UID: \"7ec4ce53-e616-4171-a5b8-0972046b2429\") " pod="openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v" Jul 20 03:18:34.308908 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:34.308380 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znc4z\" (UniqueName: \"kubernetes.io/projected/7ec4ce53-e616-4171-a5b8-0972046b2429-kube-api-access-znc4z\") pod \"e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v\" (UID: \"7ec4ce53-e616-4171-a5b8-0972046b2429\") " pod="openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v" Jul 20 03:18:34.308908 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:34.308411 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ec4ce53-e616-4171-a5b8-0972046b2429-istio-token\") pod \"e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v\" (UID: \"7ec4ce53-e616-4171-a5b8-0972046b2429\") " pod="openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v" Jul 20 03:18:34.308908 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:34.308494 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ec4ce53-e616-4171-a5b8-0972046b2429-istio-envoy\") pod \"e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v\" (UID: \"7ec4ce53-e616-4171-a5b8-0972046b2429\") " pod="openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v" Jul 20 03:18:34.308908 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:34.308571 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ec4ce53-e616-4171-a5b8-0972046b2429-credential-socket\") pod \"e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v\" (UID: \"7ec4ce53-e616-4171-a5b8-0972046b2429\") " pod="openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v" Jul 20 03:18:34.308908 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:34.308617 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ec4ce53-e616-4171-a5b8-0972046b2429-istiod-ca-cert\") pod \"e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v\" (UID: \"7ec4ce53-e616-4171-a5b8-0972046b2429\") " pod="openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v" Jul 20 03:18:34.308908 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:34.308648 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ec4ce53-e616-4171-a5b8-0972046b2429-workload-certs\") pod \"e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v\" (UID: \"7ec4ce53-e616-4171-a5b8-0972046b2429\") " pod="openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v" Jul 20 03:18:34.308908 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:34.308701 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ec4ce53-e616-4171-a5b8-0972046b2429-workload-socket\") pod \"e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v\" (UID: \"7ec4ce53-e616-4171-a5b8-0972046b2429\") " pod="openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v" Jul 20 03:18:34.409773 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:34.409736 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ec4ce53-e616-4171-a5b8-0972046b2429-istiod-ca-cert\") pod \"e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v\" (UID: \"7ec4ce53-e616-4171-a5b8-0972046b2429\") " pod="openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v" Jul 20 03:18:34.409970 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:34.409778 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ec4ce53-e616-4171-a5b8-0972046b2429-workload-certs\") pod \"e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v\" (UID: \"7ec4ce53-e616-4171-a5b8-0972046b2429\") " pod="openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v" Jul 20 03:18:34.409970 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:34.409823 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ec4ce53-e616-4171-a5b8-0972046b2429-workload-socket\") pod \"e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v\" (UID: \"7ec4ce53-e616-4171-a5b8-0972046b2429\") " pod="openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v" Jul 20 03:18:34.409970 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:34.409863 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ec4ce53-e616-4171-a5b8-0972046b2429-istio-data\") pod \"e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v\" (UID: \"7ec4ce53-e616-4171-a5b8-0972046b2429\") " pod="openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v" Jul 20 03:18:34.409970 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:34.409901 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ec4ce53-e616-4171-a5b8-0972046b2429-istio-podinfo\") pod \"e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v\" (UID: \"7ec4ce53-e616-4171-a5b8-0972046b2429\") " pod="openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v" Jul 20 03:18:34.409970 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:34.409940 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znc4z\" (UniqueName: \"kubernetes.io/projected/7ec4ce53-e616-4171-a5b8-0972046b2429-kube-api-access-znc4z\") pod \"e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v\" (UID: \"7ec4ce53-e616-4171-a5b8-0972046b2429\") " pod="openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v" Jul 20 03:18:34.409970 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:34.409968 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ec4ce53-e616-4171-a5b8-0972046b2429-istio-token\") pod \"e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v\" (UID: \"7ec4ce53-e616-4171-a5b8-0972046b2429\") " pod="openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v" Jul 20 03:18:34.410320 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:34.410001 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ec4ce53-e616-4171-a5b8-0972046b2429-istio-envoy\") pod \"e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v\" (UID: \"7ec4ce53-e616-4171-a5b8-0972046b2429\") " pod="openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v" Jul 20 03:18:34.410320 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:34.410059 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ec4ce53-e616-4171-a5b8-0972046b2429-credential-socket\") pod \"e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v\" (UID: \"7ec4ce53-e616-4171-a5b8-0972046b2429\") " pod="openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v" Jul 20 03:18:34.410404 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:34.410320 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ec4ce53-e616-4171-a5b8-0972046b2429-workload-socket\") pod \"e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v\" (UID: \"7ec4ce53-e616-4171-a5b8-0972046b2429\") " pod="openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v" Jul 20 03:18:34.410518 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:34.410500 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ec4ce53-e616-4171-a5b8-0972046b2429-workload-certs\") pod \"e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v\" (UID: \"7ec4ce53-e616-4171-a5b8-0972046b2429\") " pod="openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v" Jul 20 03:18:34.410573 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:34.410523 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ec4ce53-e616-4171-a5b8-0972046b2429-istio-data\") pod \"e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v\" (UID: \"7ec4ce53-e616-4171-a5b8-0972046b2429\") " pod="openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v" Jul 20 03:18:34.410607 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:34.410588 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ec4ce53-e616-4171-a5b8-0972046b2429-credential-socket\") pod \"e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v\" (UID: \"7ec4ce53-e616-4171-a5b8-0972046b2429\") " pod="openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v" Jul 20 03:18:34.410607 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:34.410592 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ec4ce53-e616-4171-a5b8-0972046b2429-istiod-ca-cert\") pod \"e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v\" (UID: \"7ec4ce53-e616-4171-a5b8-0972046b2429\") " pod="openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v" Jul 20 03:18:34.412429 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:34.412408 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ec4ce53-e616-4171-a5b8-0972046b2429-istio-envoy\") pod \"e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v\" (UID: \"7ec4ce53-e616-4171-a5b8-0972046b2429\") " pod="openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v" Jul 20 03:18:34.412592 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:34.412574 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ec4ce53-e616-4171-a5b8-0972046b2429-istio-podinfo\") pod \"e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v\" (UID: \"7ec4ce53-e616-4171-a5b8-0972046b2429\") " pod="openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v" Jul 20 03:18:34.422255 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:34.422226 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ec4ce53-e616-4171-a5b8-0972046b2429-istio-token\") pod \"e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v\" (UID: \"7ec4ce53-e616-4171-a5b8-0972046b2429\") " pod="openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v" Jul 20 03:18:34.422638 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:34.422618 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znc4z\" (UniqueName: \"kubernetes.io/projected/7ec4ce53-e616-4171-a5b8-0972046b2429-kube-api-access-znc4z\") pod \"e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v\" (UID: \"7ec4ce53-e616-4171-a5b8-0972046b2429\") " pod="openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v" Jul 20 03:18:34.531979 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:34.531898 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v" Jul 20 03:18:34.666984 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:34.666961 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v"] Jul 20 03:18:34.668679 ip-10-0-142-139 kubenswrapper[2558]: W0720 03:18:34.668652 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ec4ce53_e616_4171_a5b8_0972046b2429.slice/crio-7568601245d321e9f53c42c3cc89da117097494d8b08cd46d5dd2e633a701dcb WatchSource:0}: Error finding container 7568601245d321e9f53c42c3cc89da117097494d8b08cd46d5dd2e633a701dcb: Status 404 returned error can't find the container with id 7568601245d321e9f53c42c3cc89da117097494d8b08cd46d5dd2e633a701dcb Jul 20 03:18:34.670818 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:34.670786 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:18:34.670892 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:34.670851 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:18:34.670892 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:34.670877 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:18:34.975730 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:34.975697 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v" event={"ID":"7ec4ce53-e616-4171-a5b8-0972046b2429","Type":"ContainerStarted","Data":"38f53cc696a5727a5f45add36923a0730bbd83ed2787f7eda694ee2545accf1a"} Jul 20 03:18:34.975915 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:34.975748 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v" event={"ID":"7ec4ce53-e616-4171-a5b8-0972046b2429","Type":"ContainerStarted","Data":"7568601245d321e9f53c42c3cc89da117097494d8b08cd46d5dd2e633a701dcb"} Jul 20 03:18:34.995397 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:34.995349 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v" podStartSLOduration=0.995332102 podStartE2EDuration="995.332102ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 03:18:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 03:18:34.994567612 +0000 UTC m=+2010.193816759" watchObservedRunningTime="2026-07-20 03:18:34.995332102 +0000 UTC m=+2010.194581248" Jul 20 03:18:35.532315 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:35.532281 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v" Jul 20 03:18:35.537006 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:35.536978 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v" Jul 20 03:18:35.981527 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:35.981497 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v" Jul 20 03:18:35.982575 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:35.982555 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v" Jul 20 03:18:38.686408 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:38.686384 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp" Jul 20 03:18:38.749305 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:38.749226 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d10b552-61a5-4fdb-9b1a-e2ef5af447dc-workload-socket\") pod \"5d10b552-61a5-4fdb-9b1a-e2ef5af447dc\" (UID: \"5d10b552-61a5-4fdb-9b1a-e2ef5af447dc\") " Jul 20 03:18:38.749305 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:38.749263 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d10b552-61a5-4fdb-9b1a-e2ef5af447dc-istio-token\") pod \"5d10b552-61a5-4fdb-9b1a-e2ef5af447dc\" (UID: \"5d10b552-61a5-4fdb-9b1a-e2ef5af447dc\") " Jul 20 03:18:38.749514 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:38.749316 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d10b552-61a5-4fdb-9b1a-e2ef5af447dc-credential-socket\") pod \"5d10b552-61a5-4fdb-9b1a-e2ef5af447dc\" (UID: \"5d10b552-61a5-4fdb-9b1a-e2ef5af447dc\") " Jul 20 03:18:38.749514 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:38.749332 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d10b552-61a5-4fdb-9b1a-e2ef5af447dc-istiod-ca-cert\") pod \"5d10b552-61a5-4fdb-9b1a-e2ef5af447dc\" (UID: \"5d10b552-61a5-4fdb-9b1a-e2ef5af447dc\") " Jul 20 03:18:38.749514 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:38.749352 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4nq2\" (UniqueName: \"kubernetes.io/projected/5d10b552-61a5-4fdb-9b1a-e2ef5af447dc-kube-api-access-t4nq2\") pod \"5d10b552-61a5-4fdb-9b1a-e2ef5af447dc\" (UID: \"5d10b552-61a5-4fdb-9b1a-e2ef5af447dc\") " Jul 20 03:18:38.749514 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:38.749379 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d10b552-61a5-4fdb-9b1a-e2ef5af447dc-istio-envoy\") pod \"5d10b552-61a5-4fdb-9b1a-e2ef5af447dc\" (UID: \"5d10b552-61a5-4fdb-9b1a-e2ef5af447dc\") " Jul 20 03:18:38.749514 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:38.749425 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d10b552-61a5-4fdb-9b1a-e2ef5af447dc-workload-certs\") pod \"5d10b552-61a5-4fdb-9b1a-e2ef5af447dc\" (UID: \"5d10b552-61a5-4fdb-9b1a-e2ef5af447dc\") " Jul 20 03:18:38.749514 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:38.749448 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d10b552-61a5-4fdb-9b1a-e2ef5af447dc-istio-podinfo\") pod \"5d10b552-61a5-4fdb-9b1a-e2ef5af447dc\" (UID: \"5d10b552-61a5-4fdb-9b1a-e2ef5af447dc\") " Jul 20 03:18:38.749514 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:38.749504 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d10b552-61a5-4fdb-9b1a-e2ef5af447dc-istio-data\") pod \"5d10b552-61a5-4fdb-9b1a-e2ef5af447dc\" (UID: \"5d10b552-61a5-4fdb-9b1a-e2ef5af447dc\") " Jul 20 03:18:38.749877 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:38.749549 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d10b552-61a5-4fdb-9b1a-e2ef5af447dc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5d10b552-61a5-4fdb-9b1a-e2ef5af447dc" (UID: "5d10b552-61a5-4fdb-9b1a-e2ef5af447dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:18:38.749877 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:38.749651 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d10b552-61a5-4fdb-9b1a-e2ef5af447dc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5d10b552-61a5-4fdb-9b1a-e2ef5af447dc" (UID: "5d10b552-61a5-4fdb-9b1a-e2ef5af447dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:18:38.749877 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:38.749697 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d10b552-61a5-4fdb-9b1a-e2ef5af447dc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5d10b552-61a5-4fdb-9b1a-e2ef5af447dc" (UID: "5d10b552-61a5-4fdb-9b1a-e2ef5af447dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:18:38.749877 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:38.749774 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d10b552-61a5-4fdb-9b1a-e2ef5af447dc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5d10b552-61a5-4fdb-9b1a-e2ef5af447dc" (UID: "5d10b552-61a5-4fdb-9b1a-e2ef5af447dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 03:18:38.750060 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:38.749886 2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d10b552-61a5-4fdb-9b1a-e2ef5af447dc-workload-socket\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:18:38.750060 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:38.749905 2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d10b552-61a5-4fdb-9b1a-e2ef5af447dc-credential-socket\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:18:38.750060 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:38.749918 2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d10b552-61a5-4fdb-9b1a-e2ef5af447dc-istiod-ca-cert\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:18:38.750060 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:38.749931 2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d10b552-61a5-4fdb-9b1a-e2ef5af447dc-workload-certs\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:18:38.750311 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:38.750288 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d10b552-61a5-4fdb-9b1a-e2ef5af447dc-istio-data" (OuterVolumeSpecName: "istio-data") pod "5d10b552-61a5-4fdb-9b1a-e2ef5af447dc" (UID: "5d10b552-61a5-4fdb-9b1a-e2ef5af447dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:18:38.751691 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:38.751661 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d10b552-61a5-4fdb-9b1a-e2ef5af447dc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5d10b552-61a5-4fdb-9b1a-e2ef5af447dc" (UID: "5d10b552-61a5-4fdb-9b1a-e2ef5af447dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:18:38.751826 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:38.751724 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d10b552-61a5-4fdb-9b1a-e2ef5af447dc-kube-api-access-t4nq2" (OuterVolumeSpecName: "kube-api-access-t4nq2") pod "5d10b552-61a5-4fdb-9b1a-e2ef5af447dc" (UID: "5d10b552-61a5-4fdb-9b1a-e2ef5af447dc"). InnerVolumeSpecName "kube-api-access-t4nq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 03:18:38.751940 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:38.751925 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d10b552-61a5-4fdb-9b1a-e2ef5af447dc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5d10b552-61a5-4fdb-9b1a-e2ef5af447dc" (UID: "5d10b552-61a5-4fdb-9b1a-e2ef5af447dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 20 03:18:38.752037 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:38.752023 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d10b552-61a5-4fdb-9b1a-e2ef5af447dc-istio-token" (OuterVolumeSpecName: "istio-token") pod "5d10b552-61a5-4fdb-9b1a-e2ef5af447dc" (UID: "5d10b552-61a5-4fdb-9b1a-e2ef5af447dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 03:18:38.850772 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:38.850736 2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d10b552-61a5-4fdb-9b1a-e2ef5af447dc-istio-envoy\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:18:38.850772 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:38.850766 2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d10b552-61a5-4fdb-9b1a-e2ef5af447dc-istio-podinfo\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:18:38.850772 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:38.850775 2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d10b552-61a5-4fdb-9b1a-e2ef5af447dc-istio-data\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:18:38.850996 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:38.850784 2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d10b552-61a5-4fdb-9b1a-e2ef5af447dc-istio-token\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:18:38.850996 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:38.850792 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4nq2\" (UniqueName: \"kubernetes.io/projected/5d10b552-61a5-4fdb-9b1a-e2ef5af447dc-kube-api-access-t4nq2\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:18:38.998998 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:38.998958 2558 generic.go:356] "Generic (PLEG): container finished" podID="5d10b552-61a5-4fdb-9b1a-e2ef5af447dc" containerID="ab137116734419fe518fc7c1707c93078b47a8b909e8bff85c0fda5881437f93" exitCode=0 Jul 20 03:18:38.999273 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:38.999014 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp" Jul 20 03:18:38.999273 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:38.999040 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp" event={"ID":"5d10b552-61a5-4fdb-9b1a-e2ef5af447dc","Type":"ContainerDied","Data":"ab137116734419fe518fc7c1707c93078b47a8b909e8bff85c0fda5881437f93"} Jul 20 03:18:38.999273 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:38.999118 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp" event={"ID":"5d10b552-61a5-4fdb-9b1a-e2ef5af447dc","Type":"ContainerDied","Data":"c4a554499754881e89ed3f0f8a9b6031a15021587b58fef31f415897b562be95"} Jul 20 03:18:38.999273 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:38.999145 2558 scope.go:117] "RemoveContainer" containerID="ab137116734419fe518fc7c1707c93078b47a8b909e8bff85c0fda5881437f93" Jul 20 03:18:39.008647 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:39.008618 2558 scope.go:117] "RemoveContainer" containerID="25caf47238ccb25876389f0550bd02f20c9cbf6e9b85d722c35a324a74d106af" Jul 20 03:18:39.016386 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:39.016363 2558 scope.go:117] "RemoveContainer" containerID="ab137116734419fe518fc7c1707c93078b47a8b909e8bff85c0fda5881437f93" Jul 20 03:18:39.016628 ip-10-0-142-139 kubenswrapper[2558]: E0720 03:18:39.016607 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab137116734419fe518fc7c1707c93078b47a8b909e8bff85c0fda5881437f93\": container with ID starting with ab137116734419fe518fc7c1707c93078b47a8b909e8bff85c0fda5881437f93 not found: ID does not exist" containerID="ab137116734419fe518fc7c1707c93078b47a8b909e8bff85c0fda5881437f93" Jul 20 03:18:39.016708 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:39.016640 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab137116734419fe518fc7c1707c93078b47a8b909e8bff85c0fda5881437f93"} err="failed to get container status \"ab137116734419fe518fc7c1707c93078b47a8b909e8bff85c0fda5881437f93\": rpc error: code = NotFound desc = could not find container \"ab137116734419fe518fc7c1707c93078b47a8b909e8bff85c0fda5881437f93\": container with ID starting with ab137116734419fe518fc7c1707c93078b47a8b909e8bff85c0fda5881437f93 not found: ID does not exist" Jul 20 03:18:39.016708 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:39.016663 2558 scope.go:117] "RemoveContainer" containerID="25caf47238ccb25876389f0550bd02f20c9cbf6e9b85d722c35a324a74d106af" Jul 20 03:18:39.016922 ip-10-0-142-139 kubenswrapper[2558]: E0720 03:18:39.016906 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25caf47238ccb25876389f0550bd02f20c9cbf6e9b85d722c35a324a74d106af\": container with ID starting with 25caf47238ccb25876389f0550bd02f20c9cbf6e9b85d722c35a324a74d106af not found: ID does not exist" containerID="25caf47238ccb25876389f0550bd02f20c9cbf6e9b85d722c35a324a74d106af" Jul 20 03:18:39.016963 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:39.016927 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25caf47238ccb25876389f0550bd02f20c9cbf6e9b85d722c35a324a74d106af"} err="failed to get container status \"25caf47238ccb25876389f0550bd02f20c9cbf6e9b85d722c35a324a74d106af\": rpc error: code = NotFound desc = could not find container \"25caf47238ccb25876389f0550bd02f20c9cbf6e9b85d722c35a324a74d106af\": container with ID starting with 25caf47238ccb25876389f0550bd02f20c9cbf6e9b85d722c35a324a74d106af not found: ID does not exist" Jul 20 03:18:39.020285 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:39.020263 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp"] Jul 20 03:18:39.025706 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:39.025686 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-53dce522-gw-openshift-default-79b4fc4c47-wz8kp"] Jul 20 03:18:39.424845 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:39.424813 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d10b552-61a5-4fdb-9b1a-e2ef5af447dc" path="/var/lib/kubelet/pods/5d10b552-61a5-4fdb-9b1a-e2ef5af447dc/volumes" Jul 20 03:18:42.620439 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:42.620403 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:18:50.982914 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:50.982823 2558 patch_prober.go:28] interesting pod/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 20 03:18:50.983335 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:50.982911 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v" podUID="7ec4ce53-e616-4171-a5b8-0972046b2429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 20 03:18:51.058057 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:51.058019 2558 generic.go:356] "Generic (PLEG): container finished" podID="7ec4ce53-e616-4171-a5b8-0972046b2429" containerID="38f53cc696a5727a5f45add36923a0730bbd83ed2787f7eda694ee2545accf1a" exitCode=0 Jul 20 03:18:51.058238 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:51.058111 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v" event={"ID":"7ec4ce53-e616-4171-a5b8-0972046b2429","Type":"ContainerDied","Data":"38f53cc696a5727a5f45add36923a0730bbd83ed2787f7eda694ee2545accf1a"} Jul 20 03:18:51.058555 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:51.058542 2558 scope.go:117] "RemoveContainer" containerID="38f53cc696a5727a5f45add36923a0730bbd83ed2787f7eda694ee2545accf1a" Jul 20 03:18:51.059547 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:51.059519 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:18:51.059646 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:51.059595 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:18:51.059646 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:51.059631 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:18:52.068997 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:52.068953 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v" event={"ID":"7ec4ce53-e616-4171-a5b8-0972046b2429","Type":"ContainerStarted","Data":"77f93d8a8b987f0b46951dbf0dca2aad2887efd0b52ffba63bbaf5e5df89c33b"} Jul 20 03:18:52.532528 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:52.532487 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v" Jul 20 03:18:52.532771 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:52.532748 2558 patch_prober.go:28] interesting pod/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 20 03:18:52.532861 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:52.532804 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v" podUID="7ec4ce53-e616-4171-a5b8-0972046b2429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 20 03:18:53.532440 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:53.532403 2558 patch_prober.go:28] interesting pod/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 20 03:18:53.532825 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:53.532458 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v" podUID="7ec4ce53-e616-4171-a5b8-0972046b2429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 20 03:18:54.533174 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:54.533134 2558 patch_prober.go:28] interesting pod/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 20 03:18:54.533580 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:54.533190 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v" podUID="7ec4ce53-e616-4171-a5b8-0972046b2429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 20 03:18:55.533170 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:55.533131 2558 patch_prober.go:28] interesting pod/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 20 03:18:55.533349 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:55.533187 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v" podUID="7ec4ce53-e616-4171-a5b8-0972046b2429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 20 03:18:56.537045 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:56.537016 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v" Jul 20 03:18:56.537452 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:56.537280 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v" Jul 20 03:18:56.537926 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:18:56.537909 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v" Jul 20 03:19:02.147445 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:02.147407 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd7b8c7f-9jlht"] Jul 20 03:19:02.147835 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:02.147763 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d10b552-61a5-4fdb-9b1a-e2ef5af447dc" containerName="istio-proxy" Jul 20 03:19:02.147835 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:02.147774 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d10b552-61a5-4fdb-9b1a-e2ef5af447dc" containerName="istio-proxy" Jul 20 03:19:02.147835 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:02.147792 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d10b552-61a5-4fdb-9b1a-e2ef5af447dc" containerName="istio-proxy" Jul 20 03:19:02.147835 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:02.147797 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d10b552-61a5-4fdb-9b1a-e2ef5af447dc" containerName="istio-proxy" Jul 20 03:19:02.147964 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:02.147860 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d10b552-61a5-4fdb-9b1a-e2ef5af447dc" containerName="istio-proxy" Jul 20 03:19:02.152657 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:02.152618 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd7b8c7f-9jlht" Jul 20 03:19:02.156460 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:02.156425 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 20 03:19:02.156715 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:02.156692 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hlzh6\"" Jul 20 03:19:02.158455 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:02.158431 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd7b8c7f-9jlht"] Jul 20 03:19:02.269825 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:02.269780 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d99b8db6-7c40-4655-9885-0da03df7a278-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-dd7b8c7f-9jlht\" (UID: \"d99b8db6-7c40-4655-9885-0da03df7a278\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd7b8c7f-9jlht" Jul 20 03:19:02.371250 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:02.371214 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d99b8db6-7c40-4655-9885-0da03df7a278-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-dd7b8c7f-9jlht\" (UID: \"d99b8db6-7c40-4655-9885-0da03df7a278\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd7b8c7f-9jlht" Jul 20 03:19:02.371889 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:02.371863 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d99b8db6-7c40-4655-9885-0da03df7a278-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-dd7b8c7f-9jlht\" (UID: \"d99b8db6-7c40-4655-9885-0da03df7a278\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd7b8c7f-9jlht" Jul 20 03:19:02.466632 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:02.466544 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd7b8c7f-9jlht" Jul 20 03:19:02.597365 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:02.597332 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd7b8c7f-9jlht"] Jul 20 03:19:02.598945 ip-10-0-142-139 kubenswrapper[2558]: W0720 03:19:02.598917 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd99b8db6_7c40_4655_9885_0da03df7a278.slice/crio-811e4fb2124a7839dd9f230f37f9eaad798e0fa66d27b4f9bb396c2caff85ed9 WatchSource:0}: Error finding container 811e4fb2124a7839dd9f230f37f9eaad798e0fa66d27b4f9bb396c2caff85ed9: Status 404 returned error can't find the container with id 811e4fb2124a7839dd9f230f37f9eaad798e0fa66d27b4f9bb396c2caff85ed9 Jul 20 03:19:03.131854 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:03.131816 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd7b8c7f-9jlht" event={"ID":"d99b8db6-7c40-4655-9885-0da03df7a278","Type":"ContainerStarted","Data":"bfdfff95c886c590c6fc6dc1b485648ff2f0d1e0d9de08350045f8725151bd2b"} Jul 20 03:19:03.132120 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:03.131877 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd7b8c7f-9jlht" event={"ID":"d99b8db6-7c40-4655-9885-0da03df7a278","Type":"ContainerStarted","Data":"811e4fb2124a7839dd9f230f37f9eaad798e0fa66d27b4f9bb396c2caff85ed9"} Jul 20 03:19:03.148753 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:03.148687 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd7b8c7f-9jlht" podStartSLOduration=1.148665321 podStartE2EDuration="1.148665321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 03:19:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 03:19:03.146307449 +0000 UTC m=+2038.345556635" watchObservedRunningTime="2026-07-20 03:19:03.148665321 +0000 UTC m=+2038.347914471" Jul 20 03:19:04.139383 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:04.139341 2558 generic.go:356] "Generic (PLEG): container finished" podID="d99b8db6-7c40-4655-9885-0da03df7a278" containerID="bfdfff95c886c590c6fc6dc1b485648ff2f0d1e0d9de08350045f8725151bd2b" exitCode=0 Jul 20 03:19:04.139580 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:04.139429 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd7b8c7f-9jlht" event={"ID":"d99b8db6-7c40-4655-9885-0da03df7a278","Type":"ContainerDied","Data":"bfdfff95c886c590c6fc6dc1b485648ff2f0d1e0d9de08350045f8725151bd2b"} Jul 20 03:19:05.276375 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:05.276351 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd7b8c7f-9jlht" Jul 20 03:19:05.397983 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:05.397894 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d99b8db6-7c40-4655-9885-0da03df7a278-maas-api-service-ca\") pod \"d99b8db6-7c40-4655-9885-0da03df7a278\" (UID: \"d99b8db6-7c40-4655-9885-0da03df7a278\") " Jul 20 03:19:05.398302 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:05.398276 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d99b8db6-7c40-4655-9885-0da03df7a278-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d99b8db6-7c40-4655-9885-0da03df7a278" (UID: "d99b8db6-7c40-4655-9885-0da03df7a278"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 03:19:05.498894 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:05.498863 2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d99b8db6-7c40-4655-9885-0da03df7a278-maas-api-service-ca\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:19:06.149944 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:06.149912 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd7b8c7f-9jlht" event={"ID":"d99b8db6-7c40-4655-9885-0da03df7a278","Type":"ContainerDied","Data":"811e4fb2124a7839dd9f230f37f9eaad798e0fa66d27b4f9bb396c2caff85ed9"} Jul 20 03:19:06.150140 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:06.149961 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="811e4fb2124a7839dd9f230f37f9eaad798e0fa66d27b4f9bb396c2caff85ed9" Jul 20 03:19:06.150140 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:06.149978 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd7b8c7f-9jlht" Jul 20 03:19:15.520669 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:15.520634 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:19:17.216756 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:17.216718 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd7b8c7f-9jlht"] Jul 20 03:19:17.218822 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:17.218800 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-dd7b8c7f-9jlht"] Jul 20 03:19:17.425000 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:17.424964 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d99b8db6-7c40-4655-9885-0da03df7a278" path="/var/lib/kubelet/pods/d99b8db6-7c40-4655-9885-0da03df7a278/volumes" Jul 20 03:19:18.430362 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:18.430330 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v"] Jul 20 03:19:18.430795 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:18.430557 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v" podUID="7ec4ce53-e616-4171-a5b8-0972046b2429" containerName="istio-proxy" containerID="cri-o://77f93d8a8b987f0b46951dbf0dca2aad2887efd0b52ffba63bbaf5e5df89c33b" gracePeriod=30 Jul 20 03:19:19.173062 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:19.173027 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj"] Jul 20 03:19:19.173411 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:19.173396 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d99b8db6-7c40-4655-9885-0da03df7a278" containerName="revoke-keys" Jul 20 03:19:19.173411 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:19.173412 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d99b8db6-7c40-4655-9885-0da03df7a278" containerName="revoke-keys" Jul 20 03:19:19.173538 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:19.173471 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d10b552-61a5-4fdb-9b1a-e2ef5af447dc" containerName="istio-proxy" Jul 20 03:19:19.173538 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:19.173482 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d99b8db6-7c40-4655-9885-0da03df7a278" containerName="revoke-keys" Jul 20 03:19:19.179645 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:19.179608 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj" Jul 20 03:19:19.183190 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:19.183165 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-45fa7d8a-openshift-default-dockercfg-rch7p\"" Jul 20 03:19:19.188940 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:19.188912 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj"] Jul 20 03:19:19.231546 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:19.231512 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/912c6f86-e47b-4479-b85d-5583fb47eb02-istiod-ca-cert\") pod \"e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj\" (UID: \"912c6f86-e47b-4479-b85d-5583fb47eb02\") " pod="openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj" Jul 20 03:19:19.231735 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:19.231561 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/912c6f86-e47b-4479-b85d-5583fb47eb02-istio-podinfo\") pod \"e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj\" (UID: \"912c6f86-e47b-4479-b85d-5583fb47eb02\") " pod="openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj" Jul 20 03:19:19.231735 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:19.231591 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/912c6f86-e47b-4479-b85d-5583fb47eb02-credential-socket\") pod \"e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj\" (UID: \"912c6f86-e47b-4479-b85d-5583fb47eb02\") " pod="openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj" Jul 20 03:19:19.231735 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:19.231678 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/912c6f86-e47b-4479-b85d-5583fb47eb02-istio-token\") pod \"e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj\" (UID: \"912c6f86-e47b-4479-b85d-5583fb47eb02\") " pod="openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj" Jul 20 03:19:19.231735 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:19.231702 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nszzh\" (UniqueName: \"kubernetes.io/projected/912c6f86-e47b-4479-b85d-5583fb47eb02-kube-api-access-nszzh\") pod \"e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj\" (UID: \"912c6f86-e47b-4479-b85d-5583fb47eb02\") " pod="openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj" Jul 20 03:19:19.231913 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:19.231733 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/912c6f86-e47b-4479-b85d-5583fb47eb02-workload-socket\") pod \"e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj\" (UID: \"912c6f86-e47b-4479-b85d-5583fb47eb02\") " pod="openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj" Jul 20 03:19:19.231913 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:19.231756 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/912c6f86-e47b-4479-b85d-5583fb47eb02-istio-envoy\") pod \"e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj\" (UID: \"912c6f86-e47b-4479-b85d-5583fb47eb02\") " pod="openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj" Jul 20 03:19:19.231913 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:19.231843 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/912c6f86-e47b-4479-b85d-5583fb47eb02-workload-certs\") pod \"e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj\" (UID: \"912c6f86-e47b-4479-b85d-5583fb47eb02\") " pod="openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj" Jul 20 03:19:19.231913 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:19.231883 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/912c6f86-e47b-4479-b85d-5583fb47eb02-istio-data\") pod \"e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj\" (UID: \"912c6f86-e47b-4479-b85d-5583fb47eb02\") " pod="openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj" Jul 20 03:19:19.333138 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:19.333088 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/912c6f86-e47b-4479-b85d-5583fb47eb02-workload-socket\") pod \"e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj\" (UID: \"912c6f86-e47b-4479-b85d-5583fb47eb02\") " pod="openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj" Jul 20 03:19:19.333330 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:19.333146 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/912c6f86-e47b-4479-b85d-5583fb47eb02-istio-envoy\") pod \"e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj\" (UID: \"912c6f86-e47b-4479-b85d-5583fb47eb02\") " pod="openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj" Jul 20 03:19:19.333330 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:19.333193 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/912c6f86-e47b-4479-b85d-5583fb47eb02-workload-certs\") pod \"e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj\" (UID: \"912c6f86-e47b-4479-b85d-5583fb47eb02\") " pod="openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj" Jul 20 03:19:19.333330 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:19.333217 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/912c6f86-e47b-4479-b85d-5583fb47eb02-istio-data\") pod \"e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj\" (UID: \"912c6f86-e47b-4479-b85d-5583fb47eb02\") " pod="openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj" Jul 20 03:19:19.333330 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:19.333311 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/912c6f86-e47b-4479-b85d-5583fb47eb02-istiod-ca-cert\") pod \"e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj\" (UID: \"912c6f86-e47b-4479-b85d-5583fb47eb02\") " pod="openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj" Jul 20 03:19:19.333565 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:19.333366 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/912c6f86-e47b-4479-b85d-5583fb47eb02-istio-podinfo\") pod \"e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj\" (UID: \"912c6f86-e47b-4479-b85d-5583fb47eb02\") " pod="openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj" Jul 20 03:19:19.333565 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:19.333398 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/912c6f86-e47b-4479-b85d-5583fb47eb02-credential-socket\") pod \"e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj\" (UID: \"912c6f86-e47b-4479-b85d-5583fb47eb02\") " pod="openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj" Jul 20 03:19:19.333565 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:19.333471 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/912c6f86-e47b-4479-b85d-5583fb47eb02-istio-token\") pod \"e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj\" (UID: \"912c6f86-e47b-4479-b85d-5583fb47eb02\") " pod="openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj" Jul 20 03:19:19.333565 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:19.333495 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nszzh\" (UniqueName: \"kubernetes.io/projected/912c6f86-e47b-4479-b85d-5583fb47eb02-kube-api-access-nszzh\") pod \"e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj\" (UID: \"912c6f86-e47b-4479-b85d-5583fb47eb02\") " pod="openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj" Jul 20 03:19:19.333779 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:19.333573 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/912c6f86-e47b-4479-b85d-5583fb47eb02-workload-socket\") pod \"e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj\" (UID: \"912c6f86-e47b-4479-b85d-5583fb47eb02\") " pod="openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj" Jul 20 03:19:19.333978 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:19.333945 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/912c6f86-e47b-4479-b85d-5583fb47eb02-istio-data\") pod \"e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj\" (UID: \"912c6f86-e47b-4479-b85d-5583fb47eb02\") " pod="openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj" Jul 20 03:19:19.334074 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:19.333996 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/912c6f86-e47b-4479-b85d-5583fb47eb02-workload-certs\") pod \"e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj\" (UID: \"912c6f86-e47b-4479-b85d-5583fb47eb02\") " pod="openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj" Jul 20 03:19:19.334150 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:19.334114 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/912c6f86-e47b-4479-b85d-5583fb47eb02-credential-socket\") pod \"e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj\" (UID: \"912c6f86-e47b-4479-b85d-5583fb47eb02\") " pod="openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj" Jul 20 03:19:19.334425 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:19.334404 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/912c6f86-e47b-4479-b85d-5583fb47eb02-istiod-ca-cert\") pod \"e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj\" (UID: \"912c6f86-e47b-4479-b85d-5583fb47eb02\") " pod="openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj" Jul 20 03:19:19.335674 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:19.335647 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/912c6f86-e47b-4479-b85d-5583fb47eb02-istio-envoy\") pod \"e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj\" (UID: \"912c6f86-e47b-4479-b85d-5583fb47eb02\") " pod="openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj" Jul 20 03:19:19.335987 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:19.335966 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/912c6f86-e47b-4479-b85d-5583fb47eb02-istio-podinfo\") pod \"e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj\" (UID: \"912c6f86-e47b-4479-b85d-5583fb47eb02\") " pod="openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj" Jul 20 03:19:19.341813 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:19.341793 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/912c6f86-e47b-4479-b85d-5583fb47eb02-istio-token\") pod \"e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj\" (UID: \"912c6f86-e47b-4479-b85d-5583fb47eb02\") " pod="openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj" Jul 20 03:19:19.342271 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:19.342250 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nszzh\" (UniqueName: \"kubernetes.io/projected/912c6f86-e47b-4479-b85d-5583fb47eb02-kube-api-access-nszzh\") pod \"e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj\" (UID: \"912c6f86-e47b-4479-b85d-5583fb47eb02\") " pod="openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj" Jul 20 03:19:19.495210 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:19.495129 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj" Jul 20 03:19:19.632044 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:19.632014 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj"] Jul 20 03:19:19.633944 ip-10-0-142-139 kubenswrapper[2558]: W0720 03:19:19.633919 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod912c6f86_e47b_4479_b85d_5583fb47eb02.slice/crio-d726d0faf385b9c104a29e20ba514b32b9263de2cab82d1c0baa25f777a81c65 WatchSource:0}: Error finding container d726d0faf385b9c104a29e20ba514b32b9263de2cab82d1c0baa25f777a81c65: Status 404 returned error can't find the container with id d726d0faf385b9c104a29e20ba514b32b9263de2cab82d1c0baa25f777a81c65 Jul 20 03:19:19.636164 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:19.636128 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:19:19.636271 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:19.636212 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:19:19.636271 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:19.636252 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:19:20.219482 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:20.219441 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj" event={"ID":"912c6f86-e47b-4479-b85d-5583fb47eb02","Type":"ContainerStarted","Data":"4e8dd39c7aae43310ece16a97d5e03967db95e2b09c5ec113c17ad18ce7e8ea1"} Jul 20 03:19:20.219708 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:20.219505 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj" event={"ID":"912c6f86-e47b-4479-b85d-5583fb47eb02","Type":"ContainerStarted","Data":"d726d0faf385b9c104a29e20ba514b32b9263de2cab82d1c0baa25f777a81c65"} Jul 20 03:19:20.239376 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:20.239328 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj" podStartSLOduration=1.239312384 podStartE2EDuration="1.239312384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 03:19:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 03:19:20.238285649 +0000 UTC m=+2055.437534797" watchObservedRunningTime="2026-07-20 03:19:20.239312384 +0000 UTC m=+2055.438561530" Jul 20 03:19:20.495960 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:20.495880 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj" Jul 20 03:19:20.500658 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:20.500631 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj" Jul 20 03:19:21.224010 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:21.223978 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj" Jul 20 03:19:21.224980 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:21.224963 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj" Jul 20 03:19:23.687802 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:23.687776 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v" Jul 20 03:19:23.775890 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:23.775807 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-znc4z\" (UniqueName: \"kubernetes.io/projected/7ec4ce53-e616-4171-a5b8-0972046b2429-kube-api-access-znc4z\") pod \"7ec4ce53-e616-4171-a5b8-0972046b2429\" (UID: \"7ec4ce53-e616-4171-a5b8-0972046b2429\") " Jul 20 03:19:23.775890 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:23.775856 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ec4ce53-e616-4171-a5b8-0972046b2429-credential-socket\") pod \"7ec4ce53-e616-4171-a5b8-0972046b2429\" (UID: \"7ec4ce53-e616-4171-a5b8-0972046b2429\") " Jul 20 03:19:23.775890 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:23.775884 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ec4ce53-e616-4171-a5b8-0972046b2429-istio-podinfo\") pod \"7ec4ce53-e616-4171-a5b8-0972046b2429\" (UID: \"7ec4ce53-e616-4171-a5b8-0972046b2429\") " Jul 20 03:19:23.776183 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:23.775949 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ec4ce53-e616-4171-a5b8-0972046b2429-istio-data\") pod \"7ec4ce53-e616-4171-a5b8-0972046b2429\" (UID: \"7ec4ce53-e616-4171-a5b8-0972046b2429\") " Jul 20 03:19:23.776183 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:23.775975 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ec4ce53-e616-4171-a5b8-0972046b2429-istio-token\") pod \"7ec4ce53-e616-4171-a5b8-0972046b2429\" (UID: \"7ec4ce53-e616-4171-a5b8-0972046b2429\") " Jul 20 03:19:23.776183 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:23.776011 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ec4ce53-e616-4171-a5b8-0972046b2429-istio-envoy\") pod \"7ec4ce53-e616-4171-a5b8-0972046b2429\" (UID: \"7ec4ce53-e616-4171-a5b8-0972046b2429\") " Jul 20 03:19:23.776183 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:23.776034 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ec4ce53-e616-4171-a5b8-0972046b2429-istiod-ca-cert\") pod \"7ec4ce53-e616-4171-a5b8-0972046b2429\" (UID: \"7ec4ce53-e616-4171-a5b8-0972046b2429\") " Jul 20 03:19:23.776183 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:23.776118 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ec4ce53-e616-4171-a5b8-0972046b2429-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7ec4ce53-e616-4171-a5b8-0972046b2429" (UID: "7ec4ce53-e616-4171-a5b8-0972046b2429"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:19:23.776183 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:23.776145 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ec4ce53-e616-4171-a5b8-0972046b2429-workload-certs\") pod \"7ec4ce53-e616-4171-a5b8-0972046b2429\" (UID: \"7ec4ce53-e616-4171-a5b8-0972046b2429\") " Jul 20 03:19:23.776183 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:23.776178 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ec4ce53-e616-4171-a5b8-0972046b2429-workload-socket\") pod \"7ec4ce53-e616-4171-a5b8-0972046b2429\" (UID: \"7ec4ce53-e616-4171-a5b8-0972046b2429\") " Jul 20 03:19:23.776552 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:23.776370 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ec4ce53-e616-4171-a5b8-0972046b2429-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7ec4ce53-e616-4171-a5b8-0972046b2429" (UID: "7ec4ce53-e616-4171-a5b8-0972046b2429"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:19:23.776552 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:23.776499 2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ec4ce53-e616-4171-a5b8-0972046b2429-workload-certs\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:19:23.776552 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:23.776546 2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ec4ce53-e616-4171-a5b8-0972046b2429-credential-socket\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:19:23.776717 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:23.776553 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ec4ce53-e616-4171-a5b8-0972046b2429-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7ec4ce53-e616-4171-a5b8-0972046b2429" (UID: "7ec4ce53-e616-4171-a5b8-0972046b2429"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:19:23.776773 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:23.776746 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ec4ce53-e616-4171-a5b8-0972046b2429-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7ec4ce53-e616-4171-a5b8-0972046b2429" (UID: "7ec4ce53-e616-4171-a5b8-0972046b2429"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 03:19:23.776954 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:23.776925 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ec4ce53-e616-4171-a5b8-0972046b2429-istio-data" (OuterVolumeSpecName: "istio-data") pod "7ec4ce53-e616-4171-a5b8-0972046b2429" (UID: "7ec4ce53-e616-4171-a5b8-0972046b2429"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:19:23.778149 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:23.778125 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ec4ce53-e616-4171-a5b8-0972046b2429-kube-api-access-znc4z" (OuterVolumeSpecName: "kube-api-access-znc4z") pod "7ec4ce53-e616-4171-a5b8-0972046b2429" (UID: "7ec4ce53-e616-4171-a5b8-0972046b2429"). InnerVolumeSpecName "kube-api-access-znc4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 03:19:23.778301 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:23.778280 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ec4ce53-e616-4171-a5b8-0972046b2429-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7ec4ce53-e616-4171-a5b8-0972046b2429" (UID: "7ec4ce53-e616-4171-a5b8-0972046b2429"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 20 03:19:23.778424 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:23.778346 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ec4ce53-e616-4171-a5b8-0972046b2429-istio-token" (OuterVolumeSpecName: "istio-token") pod "7ec4ce53-e616-4171-a5b8-0972046b2429" (UID: "7ec4ce53-e616-4171-a5b8-0972046b2429"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 03:19:23.778424 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:23.778364 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ec4ce53-e616-4171-a5b8-0972046b2429-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7ec4ce53-e616-4171-a5b8-0972046b2429" (UID: "7ec4ce53-e616-4171-a5b8-0972046b2429"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:19:23.877967 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:23.877926 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znc4z\" (UniqueName: \"kubernetes.io/projected/7ec4ce53-e616-4171-a5b8-0972046b2429-kube-api-access-znc4z\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:19:23.877967 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:23.877958 2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ec4ce53-e616-4171-a5b8-0972046b2429-istio-podinfo\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:19:23.877967 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:23.877968 2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ec4ce53-e616-4171-a5b8-0972046b2429-istio-data\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:19:23.877967 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:23.877977 2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ec4ce53-e616-4171-a5b8-0972046b2429-istio-token\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:19:23.878282 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:23.877984 2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ec4ce53-e616-4171-a5b8-0972046b2429-istio-envoy\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:19:23.878282 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:23.877993 2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ec4ce53-e616-4171-a5b8-0972046b2429-istiod-ca-cert\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:19:23.878282 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:23.878001 2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ec4ce53-e616-4171-a5b8-0972046b2429-workload-socket\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:19:24.240538 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:24.240500 2558 generic.go:356] "Generic (PLEG): container finished" podID="7ec4ce53-e616-4171-a5b8-0972046b2429" containerID="77f93d8a8b987f0b46951dbf0dca2aad2887efd0b52ffba63bbaf5e5df89c33b" exitCode=0 Jul 20 03:19:24.240712 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:24.240551 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v" Jul 20 03:19:24.240712 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:24.240579 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v" event={"ID":"7ec4ce53-e616-4171-a5b8-0972046b2429","Type":"ContainerDied","Data":"77f93d8a8b987f0b46951dbf0dca2aad2887efd0b52ffba63bbaf5e5df89c33b"} Jul 20 03:19:24.240712 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:24.240637 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v" event={"ID":"7ec4ce53-e616-4171-a5b8-0972046b2429","Type":"ContainerDied","Data":"7568601245d321e9f53c42c3cc89da117097494d8b08cd46d5dd2e633a701dcb"} Jul 20 03:19:24.240712 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:24.240660 2558 scope.go:117] "RemoveContainer" containerID="77f93d8a8b987f0b46951dbf0dca2aad2887efd0b52ffba63bbaf5e5df89c33b" Jul 20 03:19:24.250574 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:24.250557 2558 scope.go:117] "RemoveContainer" containerID="38f53cc696a5727a5f45add36923a0730bbd83ed2787f7eda694ee2545accf1a" Jul 20 03:19:24.258552 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:24.258538 2558 scope.go:117] "RemoveContainer" containerID="77f93d8a8b987f0b46951dbf0dca2aad2887efd0b52ffba63bbaf5e5df89c33b" Jul 20 03:19:24.258793 ip-10-0-142-139 kubenswrapper[2558]: E0720 03:19:24.258771 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77f93d8a8b987f0b46951dbf0dca2aad2887efd0b52ffba63bbaf5e5df89c33b\": container with ID starting with 77f93d8a8b987f0b46951dbf0dca2aad2887efd0b52ffba63bbaf5e5df89c33b not found: ID does not exist" containerID="77f93d8a8b987f0b46951dbf0dca2aad2887efd0b52ffba63bbaf5e5df89c33b" Jul 20 03:19:24.258900 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:24.258800 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77f93d8a8b987f0b46951dbf0dca2aad2887efd0b52ffba63bbaf5e5df89c33b"} err="failed to get container status \"77f93d8a8b987f0b46951dbf0dca2aad2887efd0b52ffba63bbaf5e5df89c33b\": rpc error: code = NotFound desc = could not find container \"77f93d8a8b987f0b46951dbf0dca2aad2887efd0b52ffba63bbaf5e5df89c33b\": container with ID starting with 77f93d8a8b987f0b46951dbf0dca2aad2887efd0b52ffba63bbaf5e5df89c33b not found: ID does not exist" Jul 20 03:19:24.258900 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:24.258817 2558 scope.go:117] "RemoveContainer" containerID="38f53cc696a5727a5f45add36923a0730bbd83ed2787f7eda694ee2545accf1a" Jul 20 03:19:24.259066 ip-10-0-142-139 kubenswrapper[2558]: E0720 03:19:24.259049 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38f53cc696a5727a5f45add36923a0730bbd83ed2787f7eda694ee2545accf1a\": container with ID starting with 38f53cc696a5727a5f45add36923a0730bbd83ed2787f7eda694ee2545accf1a not found: ID does not exist" containerID="38f53cc696a5727a5f45add36923a0730bbd83ed2787f7eda694ee2545accf1a" Jul 20 03:19:24.259135 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:24.259072 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38f53cc696a5727a5f45add36923a0730bbd83ed2787f7eda694ee2545accf1a"} err="failed to get container status \"38f53cc696a5727a5f45add36923a0730bbd83ed2787f7eda694ee2545accf1a\": rpc error: code = NotFound desc = could not find container \"38f53cc696a5727a5f45add36923a0730bbd83ed2787f7eda694ee2545accf1a\": container with ID starting with 38f53cc696a5727a5f45add36923a0730bbd83ed2787f7eda694ee2545accf1a not found: ID does not exist" Jul 20 03:19:24.263821 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:24.263798 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v"] Jul 20 03:19:24.269984 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:24.269961 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-dd7b8c7f-openshift-default-757bbfcdf7-vf95v"] Jul 20 03:19:25.424679 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:25.424648 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ec4ce53-e616-4171-a5b8-0972046b2429" path="/var/lib/kubelet/pods/7ec4ce53-e616-4171-a5b8-0972046b2429/volumes" Jul 20 03:19:26.625798 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:26.625760 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:19:35.304581 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:35.304545 2558 generic.go:356] "Generic (PLEG): container finished" podID="912c6f86-e47b-4479-b85d-5583fb47eb02" containerID="4e8dd39c7aae43310ece16a97d5e03967db95e2b09c5ec113c17ad18ce7e8ea1" exitCode=0 Jul 20 03:19:35.305021 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:35.304621 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj" event={"ID":"912c6f86-e47b-4479-b85d-5583fb47eb02","Type":"ContainerDied","Data":"4e8dd39c7aae43310ece16a97d5e03967db95e2b09c5ec113c17ad18ce7e8ea1"} Jul 20 03:19:35.305021 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:35.305000 2558 scope.go:117] "RemoveContainer" containerID="4e8dd39c7aae43310ece16a97d5e03967db95e2b09c5ec113c17ad18ce7e8ea1" Jul 20 03:19:35.306051 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:35.306023 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:19:35.306180 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:35.306124 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:19:35.306180 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:35.306173 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:19:36.224440 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:36.224404 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj" Jul 20 03:19:36.311244 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:36.311207 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj" event={"ID":"912c6f86-e47b-4479-b85d-5583fb47eb02","Type":"ContainerStarted","Data":"a213e705283c3b2bfa2677730dda29623df61234e274a6894455bf6515403f9c"} Jul 20 03:19:36.496223 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:36.496138 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj" Jul 20 03:19:36.497453 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:36.497424 2558 patch_prober.go:28] interesting pod/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 20 03:19:36.497569 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:36.497491 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj" podUID="912c6f86-e47b-4479-b85d-5583fb47eb02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 20 03:19:37.496383 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:37.496345 2558 patch_prober.go:28] interesting pod/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 20 03:19:37.496789 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:37.496442 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj" podUID="912c6f86-e47b-4479-b85d-5583fb47eb02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 20 03:19:38.496363 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:38.496325 2558 patch_prober.go:28] interesting pod/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 20 03:19:38.496545 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:38.496380 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj" podUID="912c6f86-e47b-4479-b85d-5583fb47eb02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 20 03:19:39.496309 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:39.496272 2558 patch_prober.go:28] interesting pod/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 20 03:19:39.496487 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:39.496327 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj" podUID="912c6f86-e47b-4479-b85d-5583fb47eb02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 20 03:19:40.499309 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:40.499279 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj" Jul 20 03:19:40.499664 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:40.499624 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj" Jul 20 03:19:40.500370 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:40.500350 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj" Jul 20 03:19:46.534990 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:46.534952 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-45fa7d8a-sxqv6"] Jul 20 03:19:46.535443 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:46.535346 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ec4ce53-e616-4171-a5b8-0972046b2429" containerName="istio-proxy" Jul 20 03:19:46.535443 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:46.535358 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ec4ce53-e616-4171-a5b8-0972046b2429" containerName="istio-proxy" Jul 20 03:19:46.535443 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:46.535367 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ec4ce53-e616-4171-a5b8-0972046b2429" containerName="istio-proxy" Jul 20 03:19:46.535443 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:46.535373 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ec4ce53-e616-4171-a5b8-0972046b2429" containerName="istio-proxy" Jul 20 03:19:46.535443 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:46.535422 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ec4ce53-e616-4171-a5b8-0972046b2429" containerName="istio-proxy" Jul 20 03:19:46.538371 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:46.538354 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-45fa7d8a-sxqv6" Jul 20 03:19:46.540976 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:46.540956 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 20 03:19:46.541111 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:46.540959 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hlzh6\"" Jul 20 03:19:46.545066 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:46.545042 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-45fa7d8a-sxqv6"] Jul 20 03:19:46.684617 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:46.684576 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04ed506d-770a-44c7-84bd-c0f659104784-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-45fa7d8a-sxqv6\" (UID: \"04ed506d-770a-44c7-84bd-c0f659104784\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-45fa7d8a-sxqv6" Jul 20 03:19:46.785625 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:46.785514 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04ed506d-770a-44c7-84bd-c0f659104784-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-45fa7d8a-sxqv6\" (UID: \"04ed506d-770a-44c7-84bd-c0f659104784\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-45fa7d8a-sxqv6" Jul 20 03:19:46.786199 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:46.786179 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04ed506d-770a-44c7-84bd-c0f659104784-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-45fa7d8a-sxqv6\" (UID: \"04ed506d-770a-44c7-84bd-c0f659104784\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-45fa7d8a-sxqv6" Jul 20 03:19:46.849183 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:46.849151 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-45fa7d8a-sxqv6" Jul 20 03:19:47.000895 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:47.000858 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-45fa7d8a-sxqv6"] Jul 20 03:19:47.003603 ip-10-0-142-139 kubenswrapper[2558]: W0720 03:19:47.003567 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04ed506d_770a_44c7_84bd_c0f659104784.slice/crio-c97ae18255df7c1a49d9b793457835f477947b97972d612d3245eaf1f93173f4 WatchSource:0}: Error finding container c97ae18255df7c1a49d9b793457835f477947b97972d612d3245eaf1f93173f4: Status 404 returned error can't find the container with id c97ae18255df7c1a49d9b793457835f477947b97972d612d3245eaf1f93173f4 Jul 20 03:19:47.366797 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:47.366762 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-45fa7d8a-sxqv6" event={"ID":"04ed506d-770a-44c7-84bd-c0f659104784","Type":"ContainerStarted","Data":"503892aded9c4015d28b87bf1453c1b59522ef39323f96eb5f37b6444e539407"} Jul 20 03:19:47.366986 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:47.366822 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-45fa7d8a-sxqv6" event={"ID":"04ed506d-770a-44c7-84bd-c0f659104784","Type":"ContainerStarted","Data":"c97ae18255df7c1a49d9b793457835f477947b97972d612d3245eaf1f93173f4"} Jul 20 03:19:47.383353 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:47.383291 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-45fa7d8a-sxqv6" podStartSLOduration=1.383276415 podStartE2EDuration="1.383276415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 03:19:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 03:19:47.382023414 +0000 UTC m=+2082.581272561" watchObservedRunningTime="2026-07-20 03:19:47.383276415 +0000 UTC m=+2082.582525611" Jul 20 03:19:48.373680 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:48.373638 2558 generic.go:356] "Generic (PLEG): container finished" podID="04ed506d-770a-44c7-84bd-c0f659104784" containerID="503892aded9c4015d28b87bf1453c1b59522ef39323f96eb5f37b6444e539407" exitCode=0 Jul 20 03:19:48.374174 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:48.373709 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-45fa7d8a-sxqv6" event={"ID":"04ed506d-770a-44c7-84bd-c0f659104784","Type":"ContainerDied","Data":"503892aded9c4015d28b87bf1453c1b59522ef39323f96eb5f37b6444e539407"} Jul 20 03:19:49.510747 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:49.510720 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-45fa7d8a-sxqv6" Jul 20 03:19:49.610863 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:49.610824 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04ed506d-770a-44c7-84bd-c0f659104784-maas-api-service-ca\") pod \"04ed506d-770a-44c7-84bd-c0f659104784\" (UID: \"04ed506d-770a-44c7-84bd-c0f659104784\") " Jul 20 03:19:49.611212 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:49.611189 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04ed506d-770a-44c7-84bd-c0f659104784-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "04ed506d-770a-44c7-84bd-c0f659104784" (UID: "04ed506d-770a-44c7-84bd-c0f659104784"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 03:19:49.712146 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:49.712020 2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04ed506d-770a-44c7-84bd-c0f659104784-maas-api-service-ca\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:19:50.384356 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:50.384318 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-45fa7d8a-sxqv6" event={"ID":"04ed506d-770a-44c7-84bd-c0f659104784","Type":"ContainerDied","Data":"c97ae18255df7c1a49d9b793457835f477947b97972d612d3245eaf1f93173f4"} Jul 20 03:19:50.384356 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:50.384341 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-45fa7d8a-sxqv6" Jul 20 03:19:50.384566 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:19:50.384366 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c97ae18255df7c1a49d9b793457835f477947b97972d612d3245eaf1f93173f4" Jul 20 03:20:00.022716 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:20:00.022681 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:20:01.623299 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:20:01.623265 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-45fa7d8a-sxqv6"] Jul 20 03:20:01.626434 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:20:01.626406 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-45fa7d8a-sxqv6"] Jul 20 03:20:01.779343 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:20:01.779307 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj"] Jul 20 03:20:01.779570 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:20:01.779541 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj" podUID="912c6f86-e47b-4479-b85d-5583fb47eb02" containerName="istio-proxy" containerID="cri-o://a213e705283c3b2bfa2677730dda29623df61234e274a6894455bf6515403f9c" gracePeriod=30 Jul 20 03:20:03.424231 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:20:03.424200 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04ed506d-770a-44c7-84bd-c0f659104784" path="/var/lib/kubelet/pods/04ed506d-770a-44c7-84bd-c0f659104784/volumes" Jul 20 03:20:07.034552 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:20:07.034526 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj" Jul 20 03:20:07.063811 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:20:07.063742 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/912c6f86-e47b-4479-b85d-5583fb47eb02-istio-envoy\") pod \"912c6f86-e47b-4479-b85d-5583fb47eb02\" (UID: \"912c6f86-e47b-4479-b85d-5583fb47eb02\") " Jul 20 03:20:07.063811 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:20:07.063776 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/912c6f86-e47b-4479-b85d-5583fb47eb02-istio-data\") pod \"912c6f86-e47b-4479-b85d-5583fb47eb02\" (UID: \"912c6f86-e47b-4479-b85d-5583fb47eb02\") " Jul 20 03:20:07.063811 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:20:07.063807 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/912c6f86-e47b-4479-b85d-5583fb47eb02-istio-token\") pod \"912c6f86-e47b-4479-b85d-5583fb47eb02\" (UID: \"912c6f86-e47b-4479-b85d-5583fb47eb02\") " Jul 20 03:20:07.064064 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:20:07.063821 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/912c6f86-e47b-4479-b85d-5583fb47eb02-workload-socket\") pod \"912c6f86-e47b-4479-b85d-5583fb47eb02\" (UID: \"912c6f86-e47b-4479-b85d-5583fb47eb02\") " Jul 20 03:20:07.064064 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:20:07.063871 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/912c6f86-e47b-4479-b85d-5583fb47eb02-workload-certs\") pod \"912c6f86-e47b-4479-b85d-5583fb47eb02\" (UID: \"912c6f86-e47b-4479-b85d-5583fb47eb02\") " Jul 20 03:20:07.064064 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:20:07.063906 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nszzh\" (UniqueName: \"kubernetes.io/projected/912c6f86-e47b-4479-b85d-5583fb47eb02-kube-api-access-nszzh\") pod \"912c6f86-e47b-4479-b85d-5583fb47eb02\" (UID: \"912c6f86-e47b-4479-b85d-5583fb47eb02\") " Jul 20 03:20:07.064064 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:20:07.063968 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/912c6f86-e47b-4479-b85d-5583fb47eb02-istiod-ca-cert\") pod \"912c6f86-e47b-4479-b85d-5583fb47eb02\" (UID: \"912c6f86-e47b-4479-b85d-5583fb47eb02\") " Jul 20 03:20:07.064064 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:20:07.064021 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/912c6f86-e47b-4479-b85d-5583fb47eb02-credential-socket\") pod \"912c6f86-e47b-4479-b85d-5583fb47eb02\" (UID: \"912c6f86-e47b-4479-b85d-5583fb47eb02\") " Jul 20 03:20:07.064064 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:20:07.064048 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/912c6f86-e47b-4479-b85d-5583fb47eb02-istio-podinfo\") pod \"912c6f86-e47b-4479-b85d-5583fb47eb02\" (UID: \"912c6f86-e47b-4479-b85d-5583fb47eb02\") " Jul 20 03:20:07.064408 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:20:07.064234 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/912c6f86-e47b-4479-b85d-5583fb47eb02-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "912c6f86-e47b-4479-b85d-5583fb47eb02" (UID: "912c6f86-e47b-4479-b85d-5583fb47eb02"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:20:07.064469 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:20:07.064407 2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/912c6f86-e47b-4479-b85d-5583fb47eb02-workload-socket\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:20:07.064830 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:20:07.064622 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/912c6f86-e47b-4479-b85d-5583fb47eb02-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "912c6f86-e47b-4479-b85d-5583fb47eb02" (UID: "912c6f86-e47b-4479-b85d-5583fb47eb02"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:20:07.064830 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:20:07.064678 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/912c6f86-e47b-4479-b85d-5583fb47eb02-istio-data" (OuterVolumeSpecName: "istio-data") pod "912c6f86-e47b-4479-b85d-5583fb47eb02" (UID: "912c6f86-e47b-4479-b85d-5583fb47eb02"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:20:07.065024 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:20:07.064879 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/912c6f86-e47b-4479-b85d-5583fb47eb02-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "912c6f86-e47b-4479-b85d-5583fb47eb02" (UID: "912c6f86-e47b-4479-b85d-5583fb47eb02"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:20:07.065024 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:20:07.064923 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/912c6f86-e47b-4479-b85d-5583fb47eb02-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "912c6f86-e47b-4479-b85d-5583fb47eb02" (UID: "912c6f86-e47b-4479-b85d-5583fb47eb02"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 03:20:07.066167 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:20:07.066137 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/912c6f86-e47b-4479-b85d-5583fb47eb02-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "912c6f86-e47b-4479-b85d-5583fb47eb02" (UID: "912c6f86-e47b-4479-b85d-5583fb47eb02"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:20:07.066430 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:20:07.066403 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/912c6f86-e47b-4479-b85d-5583fb47eb02-kube-api-access-nszzh" (OuterVolumeSpecName: "kube-api-access-nszzh") pod "912c6f86-e47b-4479-b85d-5583fb47eb02" (UID: "912c6f86-e47b-4479-b85d-5583fb47eb02"). InnerVolumeSpecName "kube-api-access-nszzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 03:20:07.066512 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:20:07.066451 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/912c6f86-e47b-4479-b85d-5583fb47eb02-istio-token" (OuterVolumeSpecName: "istio-token") pod "912c6f86-e47b-4479-b85d-5583fb47eb02" (UID: "912c6f86-e47b-4479-b85d-5583fb47eb02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 03:20:07.066592 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:20:07.066577 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/912c6f86-e47b-4479-b85d-5583fb47eb02-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "912c6f86-e47b-4479-b85d-5583fb47eb02" (UID: "912c6f86-e47b-4479-b85d-5583fb47eb02"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 20 03:20:07.165738 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:20:07.165707 2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/912c6f86-e47b-4479-b85d-5583fb47eb02-istio-token\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:20:07.165738 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:20:07.165733 2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/912c6f86-e47b-4479-b85d-5583fb47eb02-workload-certs\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:20:07.165738 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:20:07.165742 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nszzh\" (UniqueName: \"kubernetes.io/projected/912c6f86-e47b-4479-b85d-5583fb47eb02-kube-api-access-nszzh\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:20:07.165971 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:20:07.165751 2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/912c6f86-e47b-4479-b85d-5583fb47eb02-istiod-ca-cert\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:20:07.165971 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:20:07.165762 2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/912c6f86-e47b-4479-b85d-5583fb47eb02-credential-socket\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:20:07.165971 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:20:07.165772 2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/912c6f86-e47b-4479-b85d-5583fb47eb02-istio-podinfo\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:20:07.165971 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:20:07.165781 2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/912c6f86-e47b-4479-b85d-5583fb47eb02-istio-envoy\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:20:07.165971 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:20:07.165789 2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/912c6f86-e47b-4479-b85d-5583fb47eb02-istio-data\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:20:07.460387 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:20:07.460356 2558 generic.go:356] "Generic (PLEG): container finished" podID="912c6f86-e47b-4479-b85d-5583fb47eb02" containerID="a213e705283c3b2bfa2677730dda29623df61234e274a6894455bf6515403f9c" exitCode=0 Jul 20 03:20:07.460558 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:20:07.460402 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj" Jul 20 03:20:07.460558 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:20:07.460445 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj" event={"ID":"912c6f86-e47b-4479-b85d-5583fb47eb02","Type":"ContainerDied","Data":"a213e705283c3b2bfa2677730dda29623df61234e274a6894455bf6515403f9c"} Jul 20 03:20:07.460558 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:20:07.460506 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj" event={"ID":"912c6f86-e47b-4479-b85d-5583fb47eb02","Type":"ContainerDied","Data":"d726d0faf385b9c104a29e20ba514b32b9263de2cab82d1c0baa25f777a81c65"} Jul 20 03:20:07.460558 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:20:07.460535 2558 scope.go:117] "RemoveContainer" containerID="a213e705283c3b2bfa2677730dda29623df61234e274a6894455bf6515403f9c" Jul 20 03:20:07.470846 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:20:07.470824 2558 scope.go:117] "RemoveContainer" containerID="4e8dd39c7aae43310ece16a97d5e03967db95e2b09c5ec113c17ad18ce7e8ea1" Jul 20 03:20:07.478747 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:20:07.478725 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj"] Jul 20 03:20:07.480532 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:20:07.480510 2558 scope.go:117] "RemoveContainer" containerID="a213e705283c3b2bfa2677730dda29623df61234e274a6894455bf6515403f9c" Jul 20 03:20:07.480914 ip-10-0-142-139 kubenswrapper[2558]: E0720 03:20:07.480858 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a213e705283c3b2bfa2677730dda29623df61234e274a6894455bf6515403f9c\": container with ID starting with a213e705283c3b2bfa2677730dda29623df61234e274a6894455bf6515403f9c not found: ID does not exist" containerID="a213e705283c3b2bfa2677730dda29623df61234e274a6894455bf6515403f9c" Jul 20 03:20:07.481024 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:20:07.480915 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a213e705283c3b2bfa2677730dda29623df61234e274a6894455bf6515403f9c"} err="failed to get container status \"a213e705283c3b2bfa2677730dda29623df61234e274a6894455bf6515403f9c\": rpc error: code = NotFound desc = could not find container \"a213e705283c3b2bfa2677730dda29623df61234e274a6894455bf6515403f9c\": container with ID starting with a213e705283c3b2bfa2677730dda29623df61234e274a6894455bf6515403f9c not found: ID does not exist" Jul 20 03:20:07.481024 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:20:07.480932 2558 scope.go:117] "RemoveContainer" containerID="4e8dd39c7aae43310ece16a97d5e03967db95e2b09c5ec113c17ad18ce7e8ea1" Jul 20 03:20:07.481288 ip-10-0-142-139 kubenswrapper[2558]: E0720 03:20:07.481201 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e8dd39c7aae43310ece16a97d5e03967db95e2b09c5ec113c17ad18ce7e8ea1\": container with ID starting with 4e8dd39c7aae43310ece16a97d5e03967db95e2b09c5ec113c17ad18ce7e8ea1 not found: ID does not exist" containerID="4e8dd39c7aae43310ece16a97d5e03967db95e2b09c5ec113c17ad18ce7e8ea1" Jul 20 03:20:07.481288 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:20:07.481222 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e8dd39c7aae43310ece16a97d5e03967db95e2b09c5ec113c17ad18ce7e8ea1"} err="failed to get container status \"4e8dd39c7aae43310ece16a97d5e03967db95e2b09c5ec113c17ad18ce7e8ea1\": rpc error: code = NotFound desc = could not find container \"4e8dd39c7aae43310ece16a97d5e03967db95e2b09c5ec113c17ad18ce7e8ea1\": container with ID starting with 4e8dd39c7aae43310ece16a97d5e03967db95e2b09c5ec113c17ad18ce7e8ea1 not found: ID does not exist" Jul 20 03:20:07.482687 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:20:07.482665 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-45fa7d8a-openshift-default-758665bb79-98nmj"] Jul 20 03:20:09.424790 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:20:09.424754 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="912c6f86-e47b-4479-b85d-5583fb47eb02" path="/var/lib/kubelet/pods/912c6f86-e47b-4479-b85d-5583fb47eb02/volumes" Jul 20 03:21:32.323444 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:32.323409 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:21:42.223558 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:42.223522 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:21:44.705984 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:44.705941 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5"] Jul 20 03:21:44.706482 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:44.706466 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04ed506d-770a-44c7-84bd-c0f659104784" containerName="revoke-keys" Jul 20 03:21:44.706532 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:44.706486 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="04ed506d-770a-44c7-84bd-c0f659104784" containerName="revoke-keys" Jul 20 03:21:44.706532 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:44.706522 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="912c6f86-e47b-4479-b85d-5583fb47eb02" containerName="istio-proxy" Jul 20 03:21:44.706532 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:44.706530 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="912c6f86-e47b-4479-b85d-5583fb47eb02" containerName="istio-proxy" Jul 20 03:21:44.706642 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:44.706547 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="912c6f86-e47b-4479-b85d-5583fb47eb02" containerName="istio-proxy" Jul 20 03:21:44.706642 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:44.706556 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="912c6f86-e47b-4479-b85d-5583fb47eb02" containerName="istio-proxy" Jul 20 03:21:44.706642 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:44.706632 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="912c6f86-e47b-4479-b85d-5583fb47eb02" containerName="istio-proxy" Jul 20 03:21:44.706744 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:44.706644 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="912c6f86-e47b-4479-b85d-5583fb47eb02" containerName="istio-proxy" Jul 20 03:21:44.706744 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:44.706654 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="04ed506d-770a-44c7-84bd-c0f659104784" containerName="revoke-keys" Jul 20 03:21:44.706744 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:44.706665 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ec4ce53-e616-4171-a5b8-0972046b2429" containerName="istio-proxy" Jul 20 03:21:44.710390 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:44.710370 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5" Jul 20 03:21:44.712996 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:44.712976 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-1cc4bbf4-openshift-default-dockercfg-2xfwd\"" Jul 20 03:21:44.732944 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:44.732921 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5"] Jul 20 03:21:44.828642 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:44.828605 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh6vp\" (UniqueName: \"kubernetes.io/projected/6f837206-26e4-4f74-9dad-e6272c2a95be-kube-api-access-sh6vp\") pod \"e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5\" (UID: \"6f837206-26e4-4f74-9dad-e6272c2a95be\") " pod="openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5" Jul 20 03:21:44.828832 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:44.828654 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f837206-26e4-4f74-9dad-e6272c2a95be-workload-socket\") pod \"e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5\" (UID: \"6f837206-26e4-4f74-9dad-e6272c2a95be\") " pod="openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5" Jul 20 03:21:44.828832 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:44.828676 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f837206-26e4-4f74-9dad-e6272c2a95be-istio-podinfo\") pod \"e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5\" (UID: \"6f837206-26e4-4f74-9dad-e6272c2a95be\") " pod="openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5" Jul 20 03:21:44.828832 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:44.828719 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f837206-26e4-4f74-9dad-e6272c2a95be-istiod-ca-cert\") pod \"e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5\" (UID: \"6f837206-26e4-4f74-9dad-e6272c2a95be\") " pod="openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5" Jul 20 03:21:44.828832 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:44.828768 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f837206-26e4-4f74-9dad-e6272c2a95be-workload-certs\") pod \"e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5\" (UID: \"6f837206-26e4-4f74-9dad-e6272c2a95be\") " pod="openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5" Jul 20 03:21:44.828832 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:44.828789 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f837206-26e4-4f74-9dad-e6272c2a95be-istio-envoy\") pod \"e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5\" (UID: \"6f837206-26e4-4f74-9dad-e6272c2a95be\") " pod="openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5" Jul 20 03:21:44.828832 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:44.828831 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f837206-26e4-4f74-9dad-e6272c2a95be-credential-socket\") pod \"e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5\" (UID: \"6f837206-26e4-4f74-9dad-e6272c2a95be\") " pod="openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5" Jul 20 03:21:44.829064 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:44.828855 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f837206-26e4-4f74-9dad-e6272c2a95be-istio-data\") pod \"e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5\" (UID: \"6f837206-26e4-4f74-9dad-e6272c2a95be\") " pod="openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5" Jul 20 03:21:44.829064 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:44.828873 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f837206-26e4-4f74-9dad-e6272c2a95be-istio-token\") pod \"e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5\" (UID: \"6f837206-26e4-4f74-9dad-e6272c2a95be\") " pod="openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5" Jul 20 03:21:44.929748 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:44.929718 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f837206-26e4-4f74-9dad-e6272c2a95be-workload-socket\") pod \"e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5\" (UID: \"6f837206-26e4-4f74-9dad-e6272c2a95be\") " pod="openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5" Jul 20 03:21:44.929748 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:44.929754 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f837206-26e4-4f74-9dad-e6272c2a95be-istio-podinfo\") pod \"e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5\" (UID: \"6f837206-26e4-4f74-9dad-e6272c2a95be\") " pod="openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5" Jul 20 03:21:44.930027 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:44.929779 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f837206-26e4-4f74-9dad-e6272c2a95be-istiod-ca-cert\") pod \"e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5\" (UID: \"6f837206-26e4-4f74-9dad-e6272c2a95be\") " pod="openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5" Jul 20 03:21:44.930027 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:44.929798 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f837206-26e4-4f74-9dad-e6272c2a95be-workload-certs\") pod \"e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5\" (UID: \"6f837206-26e4-4f74-9dad-e6272c2a95be\") " pod="openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5" Jul 20 03:21:44.930027 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:44.929961 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f837206-26e4-4f74-9dad-e6272c2a95be-istio-envoy\") pod \"e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5\" (UID: \"6f837206-26e4-4f74-9dad-e6272c2a95be\") " pod="openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5" Jul 20 03:21:44.930027 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:44.930018 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f837206-26e4-4f74-9dad-e6272c2a95be-credential-socket\") pod \"e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5\" (UID: \"6f837206-26e4-4f74-9dad-e6272c2a95be\") " pod="openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5" Jul 20 03:21:44.930289 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:44.930056 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f837206-26e4-4f74-9dad-e6272c2a95be-istio-data\") pod \"e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5\" (UID: \"6f837206-26e4-4f74-9dad-e6272c2a95be\") " pod="openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5" Jul 20 03:21:44.930289 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:44.930082 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f837206-26e4-4f74-9dad-e6272c2a95be-istio-token\") pod \"e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5\" (UID: \"6f837206-26e4-4f74-9dad-e6272c2a95be\") " pod="openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5" Jul 20 03:21:44.930289 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:44.930221 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh6vp\" (UniqueName: \"kubernetes.io/projected/6f837206-26e4-4f74-9dad-e6272c2a95be-kube-api-access-sh6vp\") pod \"e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5\" (UID: \"6f837206-26e4-4f74-9dad-e6272c2a95be\") " pod="openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5" Jul 20 03:21:44.930444 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:44.930299 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f837206-26e4-4f74-9dad-e6272c2a95be-workload-certs\") pod \"e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5\" (UID: \"6f837206-26e4-4f74-9dad-e6272c2a95be\") " pod="openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5" Jul 20 03:21:44.930444 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:44.930397 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f837206-26e4-4f74-9dad-e6272c2a95be-credential-socket\") pod \"e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5\" (UID: \"6f837206-26e4-4f74-9dad-e6272c2a95be\") " pod="openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5" Jul 20 03:21:44.930562 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:44.930463 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f837206-26e4-4f74-9dad-e6272c2a95be-workload-socket\") pod \"e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5\" (UID: \"6f837206-26e4-4f74-9dad-e6272c2a95be\") " pod="openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5" Jul 20 03:21:44.930562 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:44.930486 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f837206-26e4-4f74-9dad-e6272c2a95be-istio-data\") pod \"e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5\" (UID: \"6f837206-26e4-4f74-9dad-e6272c2a95be\") " pod="openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5" Jul 20 03:21:44.930887 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:44.930870 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f837206-26e4-4f74-9dad-e6272c2a95be-istiod-ca-cert\") pod \"e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5\" (UID: \"6f837206-26e4-4f74-9dad-e6272c2a95be\") " pod="openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5" Jul 20 03:21:44.932387 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:44.932362 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f837206-26e4-4f74-9dad-e6272c2a95be-istio-envoy\") pod \"e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5\" (UID: \"6f837206-26e4-4f74-9dad-e6272c2a95be\") " pod="openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5" Jul 20 03:21:44.932855 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:44.932833 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f837206-26e4-4f74-9dad-e6272c2a95be-istio-podinfo\") pod \"e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5\" (UID: \"6f837206-26e4-4f74-9dad-e6272c2a95be\") " pod="openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5" Jul 20 03:21:44.938743 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:44.938720 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f837206-26e4-4f74-9dad-e6272c2a95be-istio-token\") pod \"e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5\" (UID: \"6f837206-26e4-4f74-9dad-e6272c2a95be\") " pod="openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5" Jul 20 03:21:44.939217 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:44.939195 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh6vp\" (UniqueName: \"kubernetes.io/projected/6f837206-26e4-4f74-9dad-e6272c2a95be-kube-api-access-sh6vp\") pod \"e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5\" (UID: \"6f837206-26e4-4f74-9dad-e6272c2a95be\") " pod="openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5" Jul 20 03:21:45.020312 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:45.020221 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5" Jul 20 03:21:45.188381 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:45.188336 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5"] Jul 20 03:21:45.197089 ip-10-0-142-139 kubenswrapper[2558]: W0720 03:21:45.194294 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f837206_26e4_4f74_9dad_e6272c2a95be.slice/crio-a153c29357fc1c398a9640ada9b890cda8d65051513df467a042e2a7a3bc7436 WatchSource:0}: Error finding container a153c29357fc1c398a9640ada9b890cda8d65051513df467a042e2a7a3bc7436: Status 404 returned error can't find the container with id a153c29357fc1c398a9640ada9b890cda8d65051513df467a042e2a7a3bc7436 Jul 20 03:21:45.200489 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:45.200445 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:21:45.200616 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:45.200541 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:21:45.200616 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:45.200584 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:21:45.920194 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:45.920162 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5" event={"ID":"6f837206-26e4-4f74-9dad-e6272c2a95be","Type":"ContainerStarted","Data":"868d2543176bc728cc3477d90110958c5f64d7fc271afc18ae3264b03649c326"} Jul 20 03:21:45.920581 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:45.920216 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5" event={"ID":"6f837206-26e4-4f74-9dad-e6272c2a95be","Type":"ContainerStarted","Data":"a153c29357fc1c398a9640ada9b890cda8d65051513df467a042e2a7a3bc7436"} Jul 20 03:21:45.941757 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:45.941713 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5" podStartSLOduration=1.941699512 podStartE2EDuration="1.941699512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 03:21:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 03:21:45.939819214 +0000 UTC m=+2201.139068362" watchObservedRunningTime="2026-07-20 03:21:45.941699512 +0000 UTC m=+2201.140948659" Jul 20 03:21:46.020965 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:46.020927 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5" Jul 20 03:21:46.025631 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:46.025604 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5" Jul 20 03:21:46.925058 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:46.925029 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5" Jul 20 03:21:46.925946 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:46.925928 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5" Jul 20 03:21:53.619591 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:53.619559 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:21:59.657934 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:59.657891 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1cc4bbf4-8qbps"] Jul 20 03:21:59.664068 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:59.664042 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1cc4bbf4-8qbps" Jul 20 03:21:59.667726 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:59.667349 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 20 03:21:59.667726 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:59.667585 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hlzh6\"" Jul 20 03:21:59.668307 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:59.668264 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1cc4bbf4-8qbps"] Jul 20 03:21:59.772152 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:59.772108 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7a4284e-bd9e-48b2-a24c-9c1d63c7e910-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-1cc4bbf4-8qbps\" (UID: \"c7a4284e-bd9e-48b2-a24c-9c1d63c7e910\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1cc4bbf4-8qbps" Jul 20 03:21:59.872909 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:59.872875 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7a4284e-bd9e-48b2-a24c-9c1d63c7e910-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-1cc4bbf4-8qbps\" (UID: \"c7a4284e-bd9e-48b2-a24c-9c1d63c7e910\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1cc4bbf4-8qbps" Jul 20 03:21:59.873607 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:59.873583 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7a4284e-bd9e-48b2-a24c-9c1d63c7e910-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-1cc4bbf4-8qbps\" (UID: \"c7a4284e-bd9e-48b2-a24c-9c1d63c7e910\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1cc4bbf4-8qbps" Jul 20 03:21:59.978730 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:21:59.978653 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1cc4bbf4-8qbps" Jul 20 03:22:00.115763 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:00.115735 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1cc4bbf4-8qbps"] Jul 20 03:22:00.118289 ip-10-0-142-139 kubenswrapper[2558]: W0720 03:22:00.118258 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7a4284e_bd9e_48b2_a24c_9c1d63c7e910.slice/crio-f3871b892efc1f2779a21403c667983886e3e2256f1c64c8e1b49e17b55b393a WatchSource:0}: Error finding container f3871b892efc1f2779a21403c667983886e3e2256f1c64c8e1b49e17b55b393a: Status 404 returned error can't find the container with id f3871b892efc1f2779a21403c667983886e3e2256f1c64c8e1b49e17b55b393a Jul 20 03:22:01.000406 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:01.000360 2558 generic.go:356] "Generic (PLEG): container finished" podID="c7a4284e-bd9e-48b2-a24c-9c1d63c7e910" containerID="ac79f62fbbc1fea800771531272a8591eeb04ae0e0c76a4fe6f0a777d7856833" exitCode=0 Jul 20 03:22:01.000784 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:01.000432 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1cc4bbf4-8qbps" event={"ID":"c7a4284e-bd9e-48b2-a24c-9c1d63c7e910","Type":"ContainerDied","Data":"ac79f62fbbc1fea800771531272a8591eeb04ae0e0c76a4fe6f0a777d7856833"} Jul 20 03:22:01.000784 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:01.000483 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1cc4bbf4-8qbps" event={"ID":"c7a4284e-bd9e-48b2-a24c-9c1d63c7e910","Type":"ContainerStarted","Data":"f3871b892efc1f2779a21403c667983886e3e2256f1c64c8e1b49e17b55b393a"} Jul 20 03:22:01.926960 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:01.926921 2558 patch_prober.go:28] interesting pod/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 20 03:22:01.927232 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:01.926986 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5" podUID="6f837206-26e4-4f74-9dad-e6272c2a95be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 20 03:22:02.006616 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:02.006583 2558 generic.go:356] "Generic (PLEG): container finished" podID="6f837206-26e4-4f74-9dad-e6272c2a95be" containerID="868d2543176bc728cc3477d90110958c5f64d7fc271afc18ae3264b03649c326" exitCode=0 Jul 20 03:22:02.007121 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:02.006661 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5" event={"ID":"6f837206-26e4-4f74-9dad-e6272c2a95be","Type":"ContainerDied","Data":"868d2543176bc728cc3477d90110958c5f64d7fc271afc18ae3264b03649c326"} Jul 20 03:22:02.007590 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:02.007159 2558 scope.go:117] "RemoveContainer" containerID="868d2543176bc728cc3477d90110958c5f64d7fc271afc18ae3264b03649c326" Jul 20 03:22:02.008250 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:02.008222 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:22:02.008365 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:02.008305 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:22:02.008365 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:02.008352 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:22:02.166662 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:02.166638 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1cc4bbf4-8qbps" Jul 20 03:22:02.294589 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:02.294511 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7a4284e-bd9e-48b2-a24c-9c1d63c7e910-maas-api-service-ca\") pod \"c7a4284e-bd9e-48b2-a24c-9c1d63c7e910\" (UID: \"c7a4284e-bd9e-48b2-a24c-9c1d63c7e910\") " Jul 20 03:22:02.294960 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:02.294933 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7a4284e-bd9e-48b2-a24c-9c1d63c7e910-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c7a4284e-bd9e-48b2-a24c-9c1d63c7e910" (UID: "c7a4284e-bd9e-48b2-a24c-9c1d63c7e910"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 03:22:02.396166 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:02.396128 2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7a4284e-bd9e-48b2-a24c-9c1d63c7e910-maas-api-service-ca\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:22:03.014684 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:03.014647 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5" event={"ID":"6f837206-26e4-4f74-9dad-e6272c2a95be","Type":"ContainerStarted","Data":"d769e78ef43a4a7abe8cdc8070036a1424aba21ba778dc4ecb53fa149f47525d"} Jul 20 03:22:03.015968 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:03.015950 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1cc4bbf4-8qbps" Jul 20 03:22:03.016057 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:03.015949 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1cc4bbf4-8qbps" event={"ID":"c7a4284e-bd9e-48b2-a24c-9c1d63c7e910","Type":"ContainerDied","Data":"f3871b892efc1f2779a21403c667983886e3e2256f1c64c8e1b49e17b55b393a"} Jul 20 03:22:03.016126 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:03.016076 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3871b892efc1f2779a21403c667983886e3e2256f1c64c8e1b49e17b55b393a" Jul 20 03:22:03.020565 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:03.020546 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5" Jul 20 03:22:03.020734 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:03.020713 2558 patch_prober.go:28] interesting pod/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 20 03:22:03.020773 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:03.020755 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5" podUID="6f837206-26e4-4f74-9dad-e6272c2a95be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 20 03:22:04.021076 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:04.021045 2558 patch_prober.go:28] interesting pod/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 20 03:22:04.021497 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:04.021124 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5" podUID="6f837206-26e4-4f74-9dad-e6272c2a95be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 20 03:22:05.021265 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:05.021227 2558 patch_prober.go:28] interesting pod/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 20 03:22:05.021656 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:05.021280 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5" podUID="6f837206-26e4-4f74-9dad-e6272c2a95be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 20 03:22:06.021541 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:06.021499 2558 patch_prober.go:28] interesting pod/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 20 03:22:06.021930 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:06.021555 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5" podUID="6f837206-26e4-4f74-9dad-e6272c2a95be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 20 03:22:07.024761 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:07.024732 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5" Jul 20 03:22:07.025173 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:07.025122 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5" Jul 20 03:22:07.025839 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:07.025819 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5" Jul 20 03:22:13.125082 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:13.125043 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:22:14.732397 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:14.732361 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1cc4bbf4-8qbps"] Jul 20 03:22:14.741059 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:14.737155 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1cc4bbf4-8qbps"] Jul 20 03:22:15.423859 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:15.423822 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7a4284e-bd9e-48b2-a24c-9c1d63c7e910" path="/var/lib/kubelet/pods/c7a4284e-bd9e-48b2-a24c-9c1d63c7e910/volumes" Jul 20 03:22:20.998014 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:20.997977 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5"] Jul 20 03:22:20.998510 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:20.998307 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5" podUID="6f837206-26e4-4f74-9dad-e6272c2a95be" containerName="istio-proxy" containerID="cri-o://d769e78ef43a4a7abe8cdc8070036a1424aba21ba778dc4ecb53fa149f47525d" gracePeriod=30 Jul 20 03:22:22.027673 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:22.027629 2558 patch_prober.go:28] interesting pod/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 20 03:22:22.028059 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:22.027706 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5" podUID="6f837206-26e4-4f74-9dad-e6272c2a95be" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 20 03:22:26.128741 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:26.128705 2558 generic.go:356] "Generic (PLEG): container finished" podID="6f837206-26e4-4f74-9dad-e6272c2a95be" containerID="d769e78ef43a4a7abe8cdc8070036a1424aba21ba778dc4ecb53fa149f47525d" exitCode=0 Jul 20 03:22:26.129240 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:26.128776 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5" event={"ID":"6f837206-26e4-4f74-9dad-e6272c2a95be","Type":"ContainerDied","Data":"d769e78ef43a4a7abe8cdc8070036a1424aba21ba778dc4ecb53fa149f47525d"} Jul 20 03:22:26.129240 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:26.128873 2558 scope.go:117] "RemoveContainer" containerID="868d2543176bc728cc3477d90110958c5f64d7fc271afc18ae3264b03649c326" Jul 20 03:22:26.242985 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:26.242964 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5" Jul 20 03:22:26.306004 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:26.305931 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f837206-26e4-4f74-9dad-e6272c2a95be-istio-data\") pod \"6f837206-26e4-4f74-9dad-e6272c2a95be\" (UID: \"6f837206-26e4-4f74-9dad-e6272c2a95be\") " Jul 20 03:22:26.306004 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:26.306002 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f837206-26e4-4f74-9dad-e6272c2a95be-workload-certs\") pod \"6f837206-26e4-4f74-9dad-e6272c2a95be\" (UID: \"6f837206-26e4-4f74-9dad-e6272c2a95be\") " Jul 20 03:22:26.306248 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:26.306033 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f837206-26e4-4f74-9dad-e6272c2a95be-istio-envoy\") pod \"6f837206-26e4-4f74-9dad-e6272c2a95be\" (UID: \"6f837206-26e4-4f74-9dad-e6272c2a95be\") " Jul 20 03:22:26.306248 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:26.306086 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f837206-26e4-4f74-9dad-e6272c2a95be-workload-socket\") pod \"6f837206-26e4-4f74-9dad-e6272c2a95be\" (UID: \"6f837206-26e4-4f74-9dad-e6272c2a95be\") " Jul 20 03:22:26.306248 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:26.306144 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f837206-26e4-4f74-9dad-e6272c2a95be-credential-socket\") pod \"6f837206-26e4-4f74-9dad-e6272c2a95be\" (UID: \"6f837206-26e4-4f74-9dad-e6272c2a95be\") " Jul 20 03:22:26.306248 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:26.306161 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f837206-26e4-4f74-9dad-e6272c2a95be-istio-token\") pod \"6f837206-26e4-4f74-9dad-e6272c2a95be\" (UID: \"6f837206-26e4-4f74-9dad-e6272c2a95be\") " Jul 20 03:22:26.306248 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:26.306181 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f837206-26e4-4f74-9dad-e6272c2a95be-istio-podinfo\") pod \"6f837206-26e4-4f74-9dad-e6272c2a95be\" (UID: \"6f837206-26e4-4f74-9dad-e6272c2a95be\") " Jul 20 03:22:26.306248 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:26.306197 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f837206-26e4-4f74-9dad-e6272c2a95be-istiod-ca-cert\") pod \"6f837206-26e4-4f74-9dad-e6272c2a95be\" (UID: \"6f837206-26e4-4f74-9dad-e6272c2a95be\") " Jul 20 03:22:26.306248 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:26.306215 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sh6vp\" (UniqueName: \"kubernetes.io/projected/6f837206-26e4-4f74-9dad-e6272c2a95be-kube-api-access-sh6vp\") pod \"6f837206-26e4-4f74-9dad-e6272c2a95be\" (UID: \"6f837206-26e4-4f74-9dad-e6272c2a95be\") " Jul 20 03:22:26.306601 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:26.306265 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f837206-26e4-4f74-9dad-e6272c2a95be-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6f837206-26e4-4f74-9dad-e6272c2a95be" (UID: "6f837206-26e4-4f74-9dad-e6272c2a95be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:22:26.306601 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:26.306442 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f837206-26e4-4f74-9dad-e6272c2a95be-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6f837206-26e4-4f74-9dad-e6272c2a95be" (UID: "6f837206-26e4-4f74-9dad-e6272c2a95be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:22:26.306709 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:26.306610 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f837206-26e4-4f74-9dad-e6272c2a95be-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6f837206-26e4-4f74-9dad-e6272c2a95be" (UID: "6f837206-26e4-4f74-9dad-e6272c2a95be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:22:26.306709 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:26.306680 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f837206-26e4-4f74-9dad-e6272c2a95be-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6f837206-26e4-4f74-9dad-e6272c2a95be" (UID: "6f837206-26e4-4f74-9dad-e6272c2a95be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 03:22:26.306885 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:26.306855 2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f837206-26e4-4f74-9dad-e6272c2a95be-workload-socket\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:22:26.306939 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:26.306883 2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f837206-26e4-4f74-9dad-e6272c2a95be-credential-socket\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:22:26.306939 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:26.306882 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f837206-26e4-4f74-9dad-e6272c2a95be-istio-data" (OuterVolumeSpecName: "istio-data") pod "6f837206-26e4-4f74-9dad-e6272c2a95be" (UID: "6f837206-26e4-4f74-9dad-e6272c2a95be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:22:26.306939 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:26.306900 2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f837206-26e4-4f74-9dad-e6272c2a95be-istiod-ca-cert\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:22:26.306939 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:26.306913 2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f837206-26e4-4f74-9dad-e6272c2a95be-workload-certs\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:22:26.308883 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:26.308849 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f837206-26e4-4f74-9dad-e6272c2a95be-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6f837206-26e4-4f74-9dad-e6272c2a95be" (UID: "6f837206-26e4-4f74-9dad-e6272c2a95be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:22:26.308883 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:26.308864 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f837206-26e4-4f74-9dad-e6272c2a95be-istio-token" (OuterVolumeSpecName: "istio-token") pod "6f837206-26e4-4f74-9dad-e6272c2a95be" (UID: "6f837206-26e4-4f74-9dad-e6272c2a95be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 03:22:26.309035 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:26.308903 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f837206-26e4-4f74-9dad-e6272c2a95be-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6f837206-26e4-4f74-9dad-e6272c2a95be" (UID: "6f837206-26e4-4f74-9dad-e6272c2a95be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 20 03:22:26.309035 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:26.308932 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f837206-26e4-4f74-9dad-e6272c2a95be-kube-api-access-sh6vp" (OuterVolumeSpecName: "kube-api-access-sh6vp") pod "6f837206-26e4-4f74-9dad-e6272c2a95be" (UID: "6f837206-26e4-4f74-9dad-e6272c2a95be"). InnerVolumeSpecName "kube-api-access-sh6vp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 03:22:26.407537 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:26.407505 2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f837206-26e4-4f74-9dad-e6272c2a95be-istio-data\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:22:26.407537 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:26.407529 2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f837206-26e4-4f74-9dad-e6272c2a95be-istio-envoy\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:22:26.407537 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:26.407537 2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f837206-26e4-4f74-9dad-e6272c2a95be-istio-token\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:22:26.407537 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:26.407546 2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f837206-26e4-4f74-9dad-e6272c2a95be-istio-podinfo\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:22:26.407803 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:26.407555 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh6vp\" (UniqueName: \"kubernetes.io/projected/6f837206-26e4-4f74-9dad-e6272c2a95be-kube-api-access-sh6vp\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:22:27.135589 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:27.135501 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5" Jul 20 03:22:27.135589 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:27.135541 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5" event={"ID":"6f837206-26e4-4f74-9dad-e6272c2a95be","Type":"ContainerDied","Data":"a153c29357fc1c398a9640ada9b890cda8d65051513df467a042e2a7a3bc7436"} Jul 20 03:22:27.136226 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:27.135607 2558 scope.go:117] "RemoveContainer" containerID="d769e78ef43a4a7abe8cdc8070036a1424aba21ba778dc4ecb53fa149f47525d" Jul 20 03:22:27.156712 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:27.156685 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5"] Jul 20 03:22:27.160395 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:27.160374 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-1cc4bbf4-openshift-default-5cc567d4c5-fdlc5"] Jul 20 03:22:27.425590 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:27.425497 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f837206-26e4-4f74-9dad-e6272c2a95be" path="/var/lib/kubelet/pods/6f837206-26e4-4f74-9dad-e6272c2a95be/volumes" Jul 20 03:22:40.597718 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:40.597680 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn"] Jul 20 03:22:40.598216 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:40.598195 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7a4284e-bd9e-48b2-a24c-9c1d63c7e910" containerName="revoke-keys" Jul 20 03:22:40.598272 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:40.598220 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7a4284e-bd9e-48b2-a24c-9c1d63c7e910" containerName="revoke-keys" Jul 20 03:22:40.598272 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:40.598247 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f837206-26e4-4f74-9dad-e6272c2a95be" containerName="istio-proxy" Jul 20 03:22:40.598272 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:40.598255 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f837206-26e4-4f74-9dad-e6272c2a95be" containerName="istio-proxy" Jul 20 03:22:40.598272 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:40.598270 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f837206-26e4-4f74-9dad-e6272c2a95be" containerName="istio-proxy" Jul 20 03:22:40.598404 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:40.598276 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f837206-26e4-4f74-9dad-e6272c2a95be" containerName="istio-proxy" Jul 20 03:22:40.598404 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:40.598337 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f837206-26e4-4f74-9dad-e6272c2a95be" containerName="istio-proxy" Jul 20 03:22:40.598404 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:40.598344 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f837206-26e4-4f74-9dad-e6272c2a95be" containerName="istio-proxy" Jul 20 03:22:40.598404 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:40.598350 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7a4284e-bd9e-48b2-a24c-9c1d63c7e910" containerName="revoke-keys" Jul 20 03:22:40.601239 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:40.601223 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn" Jul 20 03:22:40.604011 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:40.603985 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-27c7795f-openshift-default-dockercfg-nb6xx\"" Jul 20 03:22:40.615656 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:40.615627 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn"] Jul 20 03:22:40.734613 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:40.734575 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24da4649-0084-4aa8-8584-65fb1ce1180b-istiod-ca-cert\") pod \"e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn\" (UID: \"24da4649-0084-4aa8-8584-65fb1ce1180b\") " pod="openshift-ingress/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn" Jul 20 03:22:40.734613 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:40.734620 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24da4649-0084-4aa8-8584-65fb1ce1180b-credential-socket\") pod \"e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn\" (UID: \"24da4649-0084-4aa8-8584-65fb1ce1180b\") " pod="openshift-ingress/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn" Jul 20 03:22:40.734827 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:40.734694 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24da4649-0084-4aa8-8584-65fb1ce1180b-workload-certs\") pod \"e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn\" (UID: \"24da4649-0084-4aa8-8584-65fb1ce1180b\") " pod="openshift-ingress/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn" Jul 20 03:22:40.734827 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:40.734716 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24da4649-0084-4aa8-8584-65fb1ce1180b-istio-envoy\") pod \"e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn\" (UID: \"24da4649-0084-4aa8-8584-65fb1ce1180b\") " pod="openshift-ingress/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn" Jul 20 03:22:40.734827 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:40.734797 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24da4649-0084-4aa8-8584-65fb1ce1180b-istio-podinfo\") pod \"e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn\" (UID: \"24da4649-0084-4aa8-8584-65fb1ce1180b\") " pod="openshift-ingress/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn" Jul 20 03:22:40.734929 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:40.734832 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24da4649-0084-4aa8-8584-65fb1ce1180b-istio-token\") pod \"e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn\" (UID: \"24da4649-0084-4aa8-8584-65fb1ce1180b\") " pod="openshift-ingress/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn" Jul 20 03:22:40.734929 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:40.734863 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24da4649-0084-4aa8-8584-65fb1ce1180b-istio-data\") pod \"e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn\" (UID: \"24da4649-0084-4aa8-8584-65fb1ce1180b\") " pod="openshift-ingress/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn" Jul 20 03:22:40.734929 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:40.734888 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c492l\" (UniqueName: \"kubernetes.io/projected/24da4649-0084-4aa8-8584-65fb1ce1180b-kube-api-access-c492l\") pod \"e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn\" (UID: \"24da4649-0084-4aa8-8584-65fb1ce1180b\") " pod="openshift-ingress/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn" Jul 20 03:22:40.734929 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:40.734919 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24da4649-0084-4aa8-8584-65fb1ce1180b-workload-socket\") pod \"e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn\" (UID: \"24da4649-0084-4aa8-8584-65fb1ce1180b\") " pod="openshift-ingress/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn" Jul 20 03:22:40.835741 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:40.835701 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24da4649-0084-4aa8-8584-65fb1ce1180b-workload-certs\") pod \"e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn\" (UID: \"24da4649-0084-4aa8-8584-65fb1ce1180b\") " pod="openshift-ingress/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn" Jul 20 03:22:40.835741 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:40.835745 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24da4649-0084-4aa8-8584-65fb1ce1180b-istio-envoy\") pod \"e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn\" (UID: \"24da4649-0084-4aa8-8584-65fb1ce1180b\") " pod="openshift-ingress/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn" Jul 20 03:22:40.835997 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:40.835782 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24da4649-0084-4aa8-8584-65fb1ce1180b-istio-podinfo\") pod \"e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn\" (UID: \"24da4649-0084-4aa8-8584-65fb1ce1180b\") " pod="openshift-ingress/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn" Jul 20 03:22:40.835997 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:40.835816 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24da4649-0084-4aa8-8584-65fb1ce1180b-istio-token\") pod \"e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn\" (UID: \"24da4649-0084-4aa8-8584-65fb1ce1180b\") " pod="openshift-ingress/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn" Jul 20 03:22:40.835997 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:40.835863 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24da4649-0084-4aa8-8584-65fb1ce1180b-istio-data\") pod \"e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn\" (UID: \"24da4649-0084-4aa8-8584-65fb1ce1180b\") " pod="openshift-ingress/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn" Jul 20 03:22:40.835997 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:40.835898 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c492l\" (UniqueName: \"kubernetes.io/projected/24da4649-0084-4aa8-8584-65fb1ce1180b-kube-api-access-c492l\") pod \"e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn\" (UID: \"24da4649-0084-4aa8-8584-65fb1ce1180b\") " pod="openshift-ingress/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn" Jul 20 03:22:40.835997 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:40.835940 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24da4649-0084-4aa8-8584-65fb1ce1180b-workload-socket\") pod \"e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn\" (UID: \"24da4649-0084-4aa8-8584-65fb1ce1180b\") " pod="openshift-ingress/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn" Jul 20 03:22:40.835997 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:40.835972 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24da4649-0084-4aa8-8584-65fb1ce1180b-istiod-ca-cert\") pod \"e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn\" (UID: \"24da4649-0084-4aa8-8584-65fb1ce1180b\") " pod="openshift-ingress/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn" Jul 20 03:22:40.836342 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:40.836014 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24da4649-0084-4aa8-8584-65fb1ce1180b-credential-socket\") pod \"e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn\" (UID: \"24da4649-0084-4aa8-8584-65fb1ce1180b\") " pod="openshift-ingress/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn" Jul 20 03:22:40.836342 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:40.836188 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24da4649-0084-4aa8-8584-65fb1ce1180b-workload-certs\") pod \"e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn\" (UID: \"24da4649-0084-4aa8-8584-65fb1ce1180b\") " pod="openshift-ingress/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn" Jul 20 03:22:40.836441 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:40.836376 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24da4649-0084-4aa8-8584-65fb1ce1180b-workload-socket\") pod \"e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn\" (UID: \"24da4649-0084-4aa8-8584-65fb1ce1180b\") " pod="openshift-ingress/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn" Jul 20 03:22:40.836441 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:40.836394 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24da4649-0084-4aa8-8584-65fb1ce1180b-credential-socket\") pod \"e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn\" (UID: \"24da4649-0084-4aa8-8584-65fb1ce1180b\") " pod="openshift-ingress/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn" Jul 20 03:22:40.836551 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:40.836449 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24da4649-0084-4aa8-8584-65fb1ce1180b-istio-data\") pod \"e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn\" (UID: \"24da4649-0084-4aa8-8584-65fb1ce1180b\") " pod="openshift-ingress/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn" Jul 20 03:22:40.836686 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:40.836665 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24da4649-0084-4aa8-8584-65fb1ce1180b-istiod-ca-cert\") pod \"e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn\" (UID: \"24da4649-0084-4aa8-8584-65fb1ce1180b\") " pod="openshift-ingress/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn" Jul 20 03:22:40.838182 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:40.838160 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24da4649-0084-4aa8-8584-65fb1ce1180b-istio-envoy\") pod \"e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn\" (UID: \"24da4649-0084-4aa8-8584-65fb1ce1180b\") " pod="openshift-ingress/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn" Jul 20 03:22:40.838332 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:40.838316 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24da4649-0084-4aa8-8584-65fb1ce1180b-istio-podinfo\") pod \"e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn\" (UID: \"24da4649-0084-4aa8-8584-65fb1ce1180b\") " pod="openshift-ingress/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn" Jul 20 03:22:40.843692 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:40.843668 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24da4649-0084-4aa8-8584-65fb1ce1180b-istio-token\") pod \"e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn\" (UID: \"24da4649-0084-4aa8-8584-65fb1ce1180b\") " pod="openshift-ingress/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn" Jul 20 03:22:40.843790 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:40.843736 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c492l\" (UniqueName: \"kubernetes.io/projected/24da4649-0084-4aa8-8584-65fb1ce1180b-kube-api-access-c492l\") pod \"e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn\" (UID: \"24da4649-0084-4aa8-8584-65fb1ce1180b\") " pod="openshift-ingress/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn" Jul 20 03:22:40.911667 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:40.911616 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn" Jul 20 03:22:41.046241 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:41.046216 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn"] Jul 20 03:22:41.048490 ip-10-0-142-139 kubenswrapper[2558]: W0720 03:22:41.048459 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24da4649_0084_4aa8_8584_65fb1ce1180b.slice/crio-95dbdeb247a00ed22c604213dd0aba6fbcab776db57b061c8f37bbfcaef05708 WatchSource:0}: Error finding container 95dbdeb247a00ed22c604213dd0aba6fbcab776db57b061c8f37bbfcaef05708: Status 404 returned error can't find the container with id 95dbdeb247a00ed22c604213dd0aba6fbcab776db57b061c8f37bbfcaef05708 Jul 20 03:22:41.050617 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:41.050585 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:22:41.050704 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:41.050653 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:22:41.050704 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:41.050682 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:22:41.208331 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:41.208245 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn" event={"ID":"24da4649-0084-4aa8-8584-65fb1ce1180b","Type":"ContainerStarted","Data":"90f57f058f35cc2a47e6f7825d028785a02a7bf421c527381cb364642224256f"} Jul 20 03:22:41.208331 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:41.208308 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn" event={"ID":"24da4649-0084-4aa8-8584-65fb1ce1180b","Type":"ContainerStarted","Data":"95dbdeb247a00ed22c604213dd0aba6fbcab776db57b061c8f37bbfcaef05708"} Jul 20 03:22:41.226951 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:41.226896 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn" podStartSLOduration=1.226876371 podStartE2EDuration="1.226876371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 03:22:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 03:22:41.225768448 +0000 UTC m=+2256.425017611" watchObservedRunningTime="2026-07-20 03:22:41.226876371 +0000 UTC m=+2256.426125518" Jul 20 03:22:41.911866 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:41.911825 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn" Jul 20 03:22:42.916768 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:42.916728 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn" Jul 20 03:22:43.218234 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:43.218152 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn" Jul 20 03:22:43.219288 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:43.219269 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn" Jul 20 03:22:45.728646 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:45.728610 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:22:54.277046 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:54.277008 2558 generic.go:356] "Generic (PLEG): container finished" podID="24da4649-0084-4aa8-8584-65fb1ce1180b" containerID="90f57f058f35cc2a47e6f7825d028785a02a7bf421c527381cb364642224256f" exitCode=0 Jul 20 03:22:54.277521 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:54.277081 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn" event={"ID":"24da4649-0084-4aa8-8584-65fb1ce1180b","Type":"ContainerDied","Data":"90f57f058f35cc2a47e6f7825d028785a02a7bf421c527381cb364642224256f"} Jul 20 03:22:54.277521 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:54.277483 2558 scope.go:117] "RemoveContainer" containerID="90f57f058f35cc2a47e6f7825d028785a02a7bf421c527381cb364642224256f" Jul 20 03:22:54.278452 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:54.278421 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:22:54.278562 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:54.278506 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:22:54.278562 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:54.278551 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:22:55.283648 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:55.283602 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn" event={"ID":"24da4649-0084-4aa8-8584-65fb1ce1180b","Type":"ContainerStarted","Data":"a7d61a888bcfe0df5ccf9c5bfa7158c5f036011295c43e9f1bd167a43c54cd9d"} Jul 20 03:22:55.911794 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:55.911766 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn" Jul 20 03:22:55.917540 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:55.917495 2558 patch_prober.go:28] interesting pod/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 20 03:22:55.917722 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:55.917568 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn" podUID="24da4649-0084-4aa8-8584-65fb1ce1180b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 20 03:22:56.912420 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:56.912380 2558 patch_prober.go:28] interesting pod/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 20 03:22:56.912816 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:56.912450 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn" podUID="24da4649-0084-4aa8-8584-65fb1ce1180b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 20 03:22:57.912362 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:57.912323 2558 patch_prober.go:28] interesting pod/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 20 03:22:57.912551 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:57.912395 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn" podUID="24da4649-0084-4aa8-8584-65fb1ce1180b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 20 03:22:58.218853 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:58.218767 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn" Jul 20 03:22:58.912276 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:58.912241 2558 patch_prober.go:28] interesting pod/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 20 03:22:58.912450 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:58.912294 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn" podUID="24da4649-0084-4aa8-8584-65fb1ce1180b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 20 03:22:59.916224 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:59.916196 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn" Jul 20 03:22:59.917256 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:22:59.917234 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn" Jul 20 03:23:12.821319 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:23:12.821281 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:23:27.926933 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:23:27.926888 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:23:31.587070 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:23:31.587033 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27c7795f-dl6w9"] Jul 20 03:23:31.590476 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:23:31.590458 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27c7795f-dl6w9" Jul 20 03:23:31.594226 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:23:31.594202 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hlzh6\"" Jul 20 03:23:31.594809 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:23:31.594791 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 20 03:23:31.603748 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:23:31.603720 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27c7795f-dl6w9"] Jul 20 03:23:31.701728 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:23:31.701686 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4ebe95e-1718-4673-a3c1-bdc689da6e73-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-27c7795f-dl6w9\" (UID: \"b4ebe95e-1718-4673-a3c1-bdc689da6e73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27c7795f-dl6w9" Jul 20 03:23:31.802486 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:23:31.802453 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4ebe95e-1718-4673-a3c1-bdc689da6e73-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-27c7795f-dl6w9\" (UID: \"b4ebe95e-1718-4673-a3c1-bdc689da6e73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27c7795f-dl6w9" Jul 20 03:23:31.803039 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:23:31.803021 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4ebe95e-1718-4673-a3c1-bdc689da6e73-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-27c7795f-dl6w9\" (UID: \"b4ebe95e-1718-4673-a3c1-bdc689da6e73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27c7795f-dl6w9" Jul 20 03:23:31.899829 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:23:31.899797 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27c7795f-dl6w9" Jul 20 03:23:32.030562 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:23:32.030532 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27c7795f-dl6w9"] Jul 20 03:23:32.032614 ip-10-0-142-139 kubenswrapper[2558]: W0720 03:23:32.032583 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4ebe95e_1718_4673_a3c1_bdc689da6e73.slice/crio-2421ace0e8fd8a3b0d92eb71f5c8886c04d455771f78ec201664943ce24dce5d WatchSource:0}: Error finding container 2421ace0e8fd8a3b0d92eb71f5c8886c04d455771f78ec201664943ce24dce5d: Status 404 returned error can't find the container with id 2421ace0e8fd8a3b0d92eb71f5c8886c04d455771f78ec201664943ce24dce5d Jul 20 03:23:32.477048 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:23:32.477011 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27c7795f-dl6w9" event={"ID":"b4ebe95e-1718-4673-a3c1-bdc689da6e73","Type":"ContainerStarted","Data":"e6c63fa44602448bf57c1b4f3beb7b34ee7950c732fc68f6d098b641dd607981"} Jul 20 03:23:32.477231 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:23:32.477065 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27c7795f-dl6w9" event={"ID":"b4ebe95e-1718-4673-a3c1-bdc689da6e73","Type":"ContainerStarted","Data":"2421ace0e8fd8a3b0d92eb71f5c8886c04d455771f78ec201664943ce24dce5d"} Jul 20 03:23:32.493440 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:23:32.493397 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27c7795f-dl6w9" podStartSLOduration=1.493382708 podStartE2EDuration="1.493382708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 03:23:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 03:23:32.491603546 +0000 UTC m=+2307.690852694" watchObservedRunningTime="2026-07-20 03:23:32.493382708 +0000 UTC m=+2307.692631855" Jul 20 03:23:33.482541 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:23:33.482504 2558 generic.go:356] "Generic (PLEG): container finished" podID="b4ebe95e-1718-4673-a3c1-bdc689da6e73" containerID="e6c63fa44602448bf57c1b4f3beb7b34ee7950c732fc68f6d098b641dd607981" exitCode=0 Jul 20 03:23:33.482945 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:23:33.482552 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27c7795f-dl6w9" event={"ID":"b4ebe95e-1718-4673-a3c1-bdc689da6e73","Type":"ContainerDied","Data":"e6c63fa44602448bf57c1b4f3beb7b34ee7950c732fc68f6d098b641dd607981"} Jul 20 03:23:34.616755 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:23:34.616731 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27c7795f-dl6w9" Jul 20 03:23:34.728549 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:23:34.728520 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4ebe95e-1718-4673-a3c1-bdc689da6e73-maas-api-service-ca\") pod \"b4ebe95e-1718-4673-a3c1-bdc689da6e73\" (UID: \"b4ebe95e-1718-4673-a3c1-bdc689da6e73\") " Jul 20 03:23:34.728889 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:23:34.728866 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4ebe95e-1718-4673-a3c1-bdc689da6e73-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b4ebe95e-1718-4673-a3c1-bdc689da6e73" (UID: "b4ebe95e-1718-4673-a3c1-bdc689da6e73"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 03:23:34.829569 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:23:34.829484 2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4ebe95e-1718-4673-a3c1-bdc689da6e73-maas-api-service-ca\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:23:35.493425 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:23:35.493389 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27c7795f-dl6w9" event={"ID":"b4ebe95e-1718-4673-a3c1-bdc689da6e73","Type":"ContainerDied","Data":"2421ace0e8fd8a3b0d92eb71f5c8886c04d455771f78ec201664943ce24dce5d"} Jul 20 03:23:35.493425 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:23:35.493413 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27c7795f-dl6w9" Jul 20 03:23:35.493625 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:23:35.493449 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2421ace0e8fd8a3b0d92eb71f5c8886c04d455771f78ec201664943ce24dce5d" Jul 20 03:23:45.523992 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:23:45.523952 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:23:51.660521 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:23:51.660478 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27c7795f-dl6w9"] Jul 20 03:23:51.663415 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:23:51.663388 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-27c7795f-dl6w9"] Jul 20 03:23:53.423769 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:23:53.423733 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4ebe95e-1718-4673-a3c1-bdc689da6e73" path="/var/lib/kubelet/pods/b4ebe95e-1718-4673-a3c1-bdc689da6e73/volumes" Jul 20 03:23:54.220190 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:23:54.220157 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:24:05.525567 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:24:05.525488 2558 scope.go:117] "RemoveContainer" containerID="a577786330492de120457758b9a2cdd9b3a6c76678b0ef3ea24797cc9b653cbe" Jul 20 03:24:48.073004 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:24:48.072973 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn"] Jul 20 03:24:48.073836 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:24:48.073185 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn" podUID="24da4649-0084-4aa8-8584-65fb1ce1180b" containerName="istio-proxy" containerID="cri-o://a7d61a888bcfe0df5ccf9c5bfa7158c5f036011295c43e9f1bd167a43c54cd9d" gracePeriod=30 Jul 20 03:24:52.228633 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:24:52.228595 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:24:53.328276 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:24:53.328253 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn" Jul 20 03:24:53.402621 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:24:53.402583 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24da4649-0084-4aa8-8584-65fb1ce1180b-credential-socket\") pod \"24da4649-0084-4aa8-8584-65fb1ce1180b\" (UID: \"24da4649-0084-4aa8-8584-65fb1ce1180b\") " Jul 20 03:24:53.402787 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:24:53.402634 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24da4649-0084-4aa8-8584-65fb1ce1180b-istio-envoy\") pod \"24da4649-0084-4aa8-8584-65fb1ce1180b\" (UID: \"24da4649-0084-4aa8-8584-65fb1ce1180b\") " Jul 20 03:24:53.402787 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:24:53.402666 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24da4649-0084-4aa8-8584-65fb1ce1180b-istio-podinfo\") pod \"24da4649-0084-4aa8-8584-65fb1ce1180b\" (UID: \"24da4649-0084-4aa8-8584-65fb1ce1180b\") " Jul 20 03:24:53.402787 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:24:53.402726 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24da4649-0084-4aa8-8584-65fb1ce1180b-istiod-ca-cert\") pod \"24da4649-0084-4aa8-8584-65fb1ce1180b\" (UID: \"24da4649-0084-4aa8-8584-65fb1ce1180b\") " Jul 20 03:24:53.402787 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:24:53.402751 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24da4649-0084-4aa8-8584-65fb1ce1180b-istio-token\") pod \"24da4649-0084-4aa8-8584-65fb1ce1180b\" (UID: \"24da4649-0084-4aa8-8584-65fb1ce1180b\") " Jul 20 03:24:53.402992 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:24:53.402791 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24da4649-0084-4aa8-8584-65fb1ce1180b-workload-socket\") pod \"24da4649-0084-4aa8-8584-65fb1ce1180b\" (UID: \"24da4649-0084-4aa8-8584-65fb1ce1180b\") " Jul 20 03:24:53.402992 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:24:53.402818 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24da4649-0084-4aa8-8584-65fb1ce1180b-workload-certs\") pod \"24da4649-0084-4aa8-8584-65fb1ce1180b\" (UID: \"24da4649-0084-4aa8-8584-65fb1ce1180b\") " Jul 20 03:24:53.402992 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:24:53.402841 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24da4649-0084-4aa8-8584-65fb1ce1180b-istio-data\") pod \"24da4649-0084-4aa8-8584-65fb1ce1180b\" (UID: \"24da4649-0084-4aa8-8584-65fb1ce1180b\") " Jul 20 03:24:53.402992 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:24:53.402905 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c492l\" (UniqueName: \"kubernetes.io/projected/24da4649-0084-4aa8-8584-65fb1ce1180b-kube-api-access-c492l\") pod \"24da4649-0084-4aa8-8584-65fb1ce1180b\" (UID: \"24da4649-0084-4aa8-8584-65fb1ce1180b\") " Jul 20 03:24:53.402992 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:24:53.402906 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24da4649-0084-4aa8-8584-65fb1ce1180b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "24da4649-0084-4aa8-8584-65fb1ce1180b" (UID: "24da4649-0084-4aa8-8584-65fb1ce1180b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:24:53.403279 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:24:53.403082 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24da4649-0084-4aa8-8584-65fb1ce1180b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "24da4649-0084-4aa8-8584-65fb1ce1180b" (UID: "24da4649-0084-4aa8-8584-65fb1ce1180b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:24:53.403279 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:24:53.403131 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24da4649-0084-4aa8-8584-65fb1ce1180b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "24da4649-0084-4aa8-8584-65fb1ce1180b" (UID: "24da4649-0084-4aa8-8584-65fb1ce1180b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:24:53.403279 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:24:53.403157 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24da4649-0084-4aa8-8584-65fb1ce1180b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "24da4649-0084-4aa8-8584-65fb1ce1180b" (UID: "24da4649-0084-4aa8-8584-65fb1ce1180b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 03:24:53.403279 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:24:53.403275 2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24da4649-0084-4aa8-8584-65fb1ce1180b-istiod-ca-cert\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:24:53.403478 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:24:53.403293 2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24da4649-0084-4aa8-8584-65fb1ce1180b-workload-socket\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:24:53.403478 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:24:53.403305 2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24da4649-0084-4aa8-8584-65fb1ce1180b-workload-certs\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:24:53.403478 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:24:53.403318 2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24da4649-0084-4aa8-8584-65fb1ce1180b-credential-socket\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:24:53.403736 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:24:53.403711 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24da4649-0084-4aa8-8584-65fb1ce1180b-istio-data" (OuterVolumeSpecName: "istio-data") pod "24da4649-0084-4aa8-8584-65fb1ce1180b" (UID: "24da4649-0084-4aa8-8584-65fb1ce1180b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:24:53.404916 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:24:53.404892 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24da4649-0084-4aa8-8584-65fb1ce1180b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "24da4649-0084-4aa8-8584-65fb1ce1180b" (UID: "24da4649-0084-4aa8-8584-65fb1ce1180b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 20 03:24:53.405138 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:24:53.405117 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24da4649-0084-4aa8-8584-65fb1ce1180b-istio-token" (OuterVolumeSpecName: "istio-token") pod "24da4649-0084-4aa8-8584-65fb1ce1180b" (UID: "24da4649-0084-4aa8-8584-65fb1ce1180b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 03:24:53.405344 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:24:53.405324 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24da4649-0084-4aa8-8584-65fb1ce1180b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "24da4649-0084-4aa8-8584-65fb1ce1180b" (UID: "24da4649-0084-4aa8-8584-65fb1ce1180b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:24:53.405420 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:24:53.405397 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24da4649-0084-4aa8-8584-65fb1ce1180b-kube-api-access-c492l" (OuterVolumeSpecName: "kube-api-access-c492l") pod "24da4649-0084-4aa8-8584-65fb1ce1180b" (UID: "24da4649-0084-4aa8-8584-65fb1ce1180b"). InnerVolumeSpecName "kube-api-access-c492l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 03:24:53.504793 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:24:53.504759 2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24da4649-0084-4aa8-8584-65fb1ce1180b-istio-token\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:24:53.504793 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:24:53.504788 2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24da4649-0084-4aa8-8584-65fb1ce1180b-istio-data\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:24:53.504793 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:24:53.504797 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c492l\" (UniqueName: \"kubernetes.io/projected/24da4649-0084-4aa8-8584-65fb1ce1180b-kube-api-access-c492l\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:24:53.505032 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:24:53.504807 2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24da4649-0084-4aa8-8584-65fb1ce1180b-istio-envoy\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:24:53.505032 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:24:53.504816 2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24da4649-0084-4aa8-8584-65fb1ce1180b-istio-podinfo\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:24:53.875485 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:24:53.875451 2558 generic.go:356] "Generic (PLEG): container finished" podID="24da4649-0084-4aa8-8584-65fb1ce1180b" containerID="a7d61a888bcfe0df5ccf9c5bfa7158c5f036011295c43e9f1bd167a43c54cd9d" exitCode=0 Jul 20 03:24:53.875658 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:24:53.875506 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn" Jul 20 03:24:53.875658 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:24:53.875531 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn" event={"ID":"24da4649-0084-4aa8-8584-65fb1ce1180b","Type":"ContainerDied","Data":"a7d61a888bcfe0df5ccf9c5bfa7158c5f036011295c43e9f1bd167a43c54cd9d"} Jul 20 03:24:53.875658 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:24:53.875591 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn" event={"ID":"24da4649-0084-4aa8-8584-65fb1ce1180b","Type":"ContainerDied","Data":"95dbdeb247a00ed22c604213dd0aba6fbcab776db57b061c8f37bbfcaef05708"} Jul 20 03:24:53.875658 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:24:53.875614 2558 scope.go:117] "RemoveContainer" containerID="a7d61a888bcfe0df5ccf9c5bfa7158c5f036011295c43e9f1bd167a43c54cd9d" Jul 20 03:24:53.884360 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:24:53.884340 2558 scope.go:117] "RemoveContainer" containerID="90f57f058f35cc2a47e6f7825d028785a02a7bf421c527381cb364642224256f" Jul 20 03:24:53.892566 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:24:53.892549 2558 scope.go:117] "RemoveContainer" containerID="a7d61a888bcfe0df5ccf9c5bfa7158c5f036011295c43e9f1bd167a43c54cd9d" Jul 20 03:24:53.892811 ip-10-0-142-139 kubenswrapper[2558]: E0720 03:24:53.892793 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7d61a888bcfe0df5ccf9c5bfa7158c5f036011295c43e9f1bd167a43c54cd9d\": container with ID starting with a7d61a888bcfe0df5ccf9c5bfa7158c5f036011295c43e9f1bd167a43c54cd9d not found: ID does not exist" containerID="a7d61a888bcfe0df5ccf9c5bfa7158c5f036011295c43e9f1bd167a43c54cd9d" Jul 20 03:24:53.892865 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:24:53.892819 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7d61a888bcfe0df5ccf9c5bfa7158c5f036011295c43e9f1bd167a43c54cd9d"} err="failed to get container status \"a7d61a888bcfe0df5ccf9c5bfa7158c5f036011295c43e9f1bd167a43c54cd9d\": rpc error: code = NotFound desc = could not find container \"a7d61a888bcfe0df5ccf9c5bfa7158c5f036011295c43e9f1bd167a43c54cd9d\": container with ID starting with a7d61a888bcfe0df5ccf9c5bfa7158c5f036011295c43e9f1bd167a43c54cd9d not found: ID does not exist" Jul 20 03:24:53.892865 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:24:53.892837 2558 scope.go:117] "RemoveContainer" containerID="90f57f058f35cc2a47e6f7825d028785a02a7bf421c527381cb364642224256f" Jul 20 03:24:53.893065 ip-10-0-142-139 kubenswrapper[2558]: E0720 03:24:53.893048 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90f57f058f35cc2a47e6f7825d028785a02a7bf421c527381cb364642224256f\": container with ID starting with 90f57f058f35cc2a47e6f7825d028785a02a7bf421c527381cb364642224256f not found: ID does not exist" containerID="90f57f058f35cc2a47e6f7825d028785a02a7bf421c527381cb364642224256f" Jul 20 03:24:53.893181 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:24:53.893078 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90f57f058f35cc2a47e6f7825d028785a02a7bf421c527381cb364642224256f"} err="failed to get container status \"90f57f058f35cc2a47e6f7825d028785a02a7bf421c527381cb364642224256f\": rpc error: code = NotFound desc = could not find container \"90f57f058f35cc2a47e6f7825d028785a02a7bf421c527381cb364642224256f\": container with ID starting with 90f57f058f35cc2a47e6f7825d028785a02a7bf421c527381cb364642224256f not found: ID does not exist" Jul 20 03:24:53.908953 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:24:53.908931 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn"] Jul 20 03:24:53.929453 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:24:53.929419 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-27c7795f-openshift-default-657c8fbbc9-v6hbn"] Jul 20 03:24:55.424738 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:24:55.424705 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24da4649-0084-4aa8-8584-65fb1ce1180b" path="/var/lib/kubelet/pods/24da4649-0084-4aa8-8584-65fb1ce1180b/volumes" Jul 20 03:25:04.660861 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:04.660821 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:25:05.544783 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:05.544750 2558 scope.go:117] "RemoveContainer" containerID="0921925f21a458d8633ffe0bceb620024c252c35a24d3ead860a2a17b1f829d0" Jul 20 03:25:05.553365 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:05.553346 2558 scope.go:117] "RemoveContainer" containerID="bfdfff95c886c590c6fc6dc1b485648ff2f0d1e0d9de08350045f8725151bd2b" Jul 20 03:25:43.840801 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:43.840717 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs"] Jul 20 03:25:43.841433 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:43.841276 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24da4649-0084-4aa8-8584-65fb1ce1180b" containerName="istio-proxy" Jul 20 03:25:43.841433 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:43.841297 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="24da4649-0084-4aa8-8584-65fb1ce1180b" containerName="istio-proxy" Jul 20 03:25:43.841433 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:43.841322 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4ebe95e-1718-4673-a3c1-bdc689da6e73" containerName="revoke-keys" Jul 20 03:25:43.841433 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:43.841331 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4ebe95e-1718-4673-a3c1-bdc689da6e73" containerName="revoke-keys" Jul 20 03:25:43.841433 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:43.841345 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24da4649-0084-4aa8-8584-65fb1ce1180b" containerName="istio-proxy" Jul 20 03:25:43.841433 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:43.841353 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="24da4649-0084-4aa8-8584-65fb1ce1180b" containerName="istio-proxy" Jul 20 03:25:43.841731 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:43.841445 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="24da4649-0084-4aa8-8584-65fb1ce1180b" containerName="istio-proxy" Jul 20 03:25:43.841731 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:43.841458 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4ebe95e-1718-4673-a3c1-bdc689da6e73" containerName="revoke-keys" Jul 20 03:25:43.841731 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:43.841468 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="24da4649-0084-4aa8-8584-65fb1ce1180b" containerName="istio-proxy" Jul 20 03:25:43.844532 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:43.844504 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs" Jul 20 03:25:43.847253 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:43.847228 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-da51e5-openshift-default-dockercfg-9zsb2\"" Jul 20 03:25:43.859932 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:43.859907 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs"] Jul 20 03:25:43.924091 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:43.924059 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt77k\" (UniqueName: \"kubernetes.io/projected/61d4ee0d-7df1-4355-b38d-38478c52d141-kube-api-access-zt77k\") pod \"e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs\" (UID: \"61d4ee0d-7df1-4355-b38d-38478c52d141\") " pod="openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs" Jul 20 03:25:43.924091 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:43.924119 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d4ee0d-7df1-4355-b38d-38478c52d141-workload-socket\") pod \"e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs\" (UID: \"61d4ee0d-7df1-4355-b38d-38478c52d141\") " pod="openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs" Jul 20 03:25:43.924374 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:43.924143 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61d4ee0d-7df1-4355-b38d-38478c52d141-workload-certs\") pod \"e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs\" (UID: \"61d4ee0d-7df1-4355-b38d-38478c52d141\") " pod="openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs" Jul 20 03:25:43.924374 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:43.924186 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61d4ee0d-7df1-4355-b38d-38478c52d141-istio-token\") pod \"e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs\" (UID: \"61d4ee0d-7df1-4355-b38d-38478c52d141\") " pod="openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs" Jul 20 03:25:43.924374 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:43.924212 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61d4ee0d-7df1-4355-b38d-38478c52d141-istio-podinfo\") pod \"e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs\" (UID: \"61d4ee0d-7df1-4355-b38d-38478c52d141\") " pod="openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs" Jul 20 03:25:43.924374 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:43.924242 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61d4ee0d-7df1-4355-b38d-38478c52d141-istio-envoy\") pod \"e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs\" (UID: \"61d4ee0d-7df1-4355-b38d-38478c52d141\") " pod="openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs" Jul 20 03:25:43.924374 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:43.924268 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d4ee0d-7df1-4355-b38d-38478c52d141-credential-socket\") pod \"e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs\" (UID: \"61d4ee0d-7df1-4355-b38d-38478c52d141\") " pod="openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs" Jul 20 03:25:43.924374 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:43.924290 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61d4ee0d-7df1-4355-b38d-38478c52d141-istio-data\") pod \"e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs\" (UID: \"61d4ee0d-7df1-4355-b38d-38478c52d141\") " pod="openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs" Jul 20 03:25:43.924374 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:43.924357 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61d4ee0d-7df1-4355-b38d-38478c52d141-istiod-ca-cert\") pod \"e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs\" (UID: \"61d4ee0d-7df1-4355-b38d-38478c52d141\") " pod="openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs" Jul 20 03:25:44.024964 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:44.024932 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61d4ee0d-7df1-4355-b38d-38478c52d141-workload-certs\") pod \"e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs\" (UID: \"61d4ee0d-7df1-4355-b38d-38478c52d141\") " pod="openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs" Jul 20 03:25:44.024964 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:44.024977 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61d4ee0d-7df1-4355-b38d-38478c52d141-istio-token\") pod \"e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs\" (UID: \"61d4ee0d-7df1-4355-b38d-38478c52d141\") " pod="openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs" Jul 20 03:25:44.025225 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:44.025001 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61d4ee0d-7df1-4355-b38d-38478c52d141-istio-podinfo\") pod \"e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs\" (UID: \"61d4ee0d-7df1-4355-b38d-38478c52d141\") " pod="openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs" Jul 20 03:25:44.025225 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:44.025144 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61d4ee0d-7df1-4355-b38d-38478c52d141-istio-envoy\") pod \"e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs\" (UID: \"61d4ee0d-7df1-4355-b38d-38478c52d141\") " pod="openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs" Jul 20 03:25:44.025225 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:44.025194 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d4ee0d-7df1-4355-b38d-38478c52d141-credential-socket\") pod \"e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs\" (UID: \"61d4ee0d-7df1-4355-b38d-38478c52d141\") " pod="openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs" Jul 20 03:25:44.025341 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:44.025234 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61d4ee0d-7df1-4355-b38d-38478c52d141-istio-data\") pod \"e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs\" (UID: \"61d4ee0d-7df1-4355-b38d-38478c52d141\") " pod="openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs" Jul 20 03:25:44.025341 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:44.025276 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61d4ee0d-7df1-4355-b38d-38478c52d141-istiod-ca-cert\") pod \"e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs\" (UID: \"61d4ee0d-7df1-4355-b38d-38478c52d141\") " pod="openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs" Jul 20 03:25:44.025341 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:44.025316 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt77k\" (UniqueName: \"kubernetes.io/projected/61d4ee0d-7df1-4355-b38d-38478c52d141-kube-api-access-zt77k\") pod \"e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs\" (UID: \"61d4ee0d-7df1-4355-b38d-38478c52d141\") " pod="openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs" Jul 20 03:25:44.025481 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:44.025375 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d4ee0d-7df1-4355-b38d-38478c52d141-workload-socket\") pod \"e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs\" (UID: \"61d4ee0d-7df1-4355-b38d-38478c52d141\") " pod="openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs" Jul 20 03:25:44.025481 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:44.025415 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61d4ee0d-7df1-4355-b38d-38478c52d141-workload-certs\") pod \"e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs\" (UID: \"61d4ee0d-7df1-4355-b38d-38478c52d141\") " pod="openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs" Jul 20 03:25:44.025642 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:44.025619 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61d4ee0d-7df1-4355-b38d-38478c52d141-istio-data\") pod \"e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs\" (UID: \"61d4ee0d-7df1-4355-b38d-38478c52d141\") " pod="openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs" Jul 20 03:25:44.025687 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:44.025670 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d4ee0d-7df1-4355-b38d-38478c52d141-workload-socket\") pod \"e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs\" (UID: \"61d4ee0d-7df1-4355-b38d-38478c52d141\") " pod="openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs" Jul 20 03:25:44.025924 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:44.025896 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d4ee0d-7df1-4355-b38d-38478c52d141-credential-socket\") pod \"e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs\" (UID: \"61d4ee0d-7df1-4355-b38d-38478c52d141\") " pod="openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs" Jul 20 03:25:44.026279 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:44.026254 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61d4ee0d-7df1-4355-b38d-38478c52d141-istiod-ca-cert\") pod \"e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs\" (UID: \"61d4ee0d-7df1-4355-b38d-38478c52d141\") " pod="openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs" Jul 20 03:25:44.027950 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:44.027924 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61d4ee0d-7df1-4355-b38d-38478c52d141-istio-envoy\") pod \"e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs\" (UID: \"61d4ee0d-7df1-4355-b38d-38478c52d141\") " pod="openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs" Jul 20 03:25:44.028039 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:44.027929 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61d4ee0d-7df1-4355-b38d-38478c52d141-istio-podinfo\") pod \"e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs\" (UID: \"61d4ee0d-7df1-4355-b38d-38478c52d141\") " pod="openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs" Jul 20 03:25:44.038492 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:44.038468 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61d4ee0d-7df1-4355-b38d-38478c52d141-istio-token\") pod \"e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs\" (UID: \"61d4ee0d-7df1-4355-b38d-38478c52d141\") " pod="openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs" Jul 20 03:25:44.039195 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:44.039176 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt77k\" (UniqueName: \"kubernetes.io/projected/61d4ee0d-7df1-4355-b38d-38478c52d141-kube-api-access-zt77k\") pod \"e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs\" (UID: \"61d4ee0d-7df1-4355-b38d-38478c52d141\") " pod="openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs" Jul 20 03:25:44.158411 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:44.158380 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs" Jul 20 03:25:44.312112 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:44.312069 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs"] Jul 20 03:25:44.314561 ip-10-0-142-139 kubenswrapper[2558]: W0720 03:25:44.314533 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61d4ee0d_7df1_4355_b38d_38478c52d141.slice/crio-ec0cff27c1b06a42d46503c419caf9d296d55930bf77503f13116eb085ae412b WatchSource:0}: Error finding container ec0cff27c1b06a42d46503c419caf9d296d55930bf77503f13116eb085ae412b: Status 404 returned error can't find the container with id ec0cff27c1b06a42d46503c419caf9d296d55930bf77503f13116eb085ae412b Jul 20 03:25:44.316644 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:44.316612 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:25:44.316729 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:44.316679 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:25:44.316729 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:44.316707 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:25:45.118431 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:45.118399 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs" event={"ID":"61d4ee0d-7df1-4355-b38d-38478c52d141","Type":"ContainerStarted","Data":"7f3b3f9ce70cf777fbd6226a331cffe8bc183f4e2323345eca5f3939978b33ba"} Jul 20 03:25:45.118831 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:45.118449 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs" event={"ID":"61d4ee0d-7df1-4355-b38d-38478c52d141","Type":"ContainerStarted","Data":"ec0cff27c1b06a42d46503c419caf9d296d55930bf77503f13116eb085ae412b"} Jul 20 03:25:45.141824 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:45.141777 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs" podStartSLOduration=2.141762339 podStartE2EDuration="2.141762339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 03:25:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 03:25:45.140385163 +0000 UTC m=+2440.339634310" watchObservedRunningTime="2026-07-20 03:25:45.141762339 +0000 UTC m=+2440.341011486" Jul 20 03:25:45.159390 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:45.159363 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs" Jul 20 03:25:45.164161 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:45.164139 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs" Jul 20 03:25:46.123435 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:46.123405 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs" Jul 20 03:25:46.124490 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:46.124469 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs" Jul 20 03:25:52.301188 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:52.301153 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx"] Jul 20 03:25:52.304827 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:52.304807 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx" Jul 20 03:25:52.307726 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:52.307696 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-9fe24f-openshift-default-dockercfg-9nx6q\"" Jul 20 03:25:52.318064 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:52.318039 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx"] Jul 20 03:25:52.397063 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:52.397025 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfae4943-17f5-420d-810c-300c0c14e411-credential-socket\") pod \"e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx\" (UID: \"dfae4943-17f5-420d-810c-300c0c14e411\") " pod="openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx" Jul 20 03:25:52.397293 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:52.397080 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfae4943-17f5-420d-810c-300c0c14e411-istio-token\") pod \"e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx\" (UID: \"dfae4943-17f5-420d-810c-300c0c14e411\") " pod="openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx" Jul 20 03:25:52.397293 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:52.397125 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2vcd\" (UniqueName: \"kubernetes.io/projected/dfae4943-17f5-420d-810c-300c0c14e411-kube-api-access-z2vcd\") pod \"e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx\" (UID: \"dfae4943-17f5-420d-810c-300c0c14e411\") " pod="openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx" Jul 20 03:25:52.397293 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:52.397209 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfae4943-17f5-420d-810c-300c0c14e411-istio-podinfo\") pod \"e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx\" (UID: \"dfae4943-17f5-420d-810c-300c0c14e411\") " pod="openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx" Jul 20 03:25:52.397293 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:52.397245 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfae4943-17f5-420d-810c-300c0c14e411-istiod-ca-cert\") pod \"e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx\" (UID: \"dfae4943-17f5-420d-810c-300c0c14e411\") " pod="openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx" Jul 20 03:25:52.397569 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:52.397335 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfae4943-17f5-420d-810c-300c0c14e411-istio-envoy\") pod \"e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx\" (UID: \"dfae4943-17f5-420d-810c-300c0c14e411\") " pod="openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx" Jul 20 03:25:52.397569 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:52.397375 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfae4943-17f5-420d-810c-300c0c14e411-workload-certs\") pod \"e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx\" (UID: \"dfae4943-17f5-420d-810c-300c0c14e411\") " pod="openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx" Jul 20 03:25:52.397569 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:52.397401 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfae4943-17f5-420d-810c-300c0c14e411-workload-socket\") pod \"e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx\" (UID: \"dfae4943-17f5-420d-810c-300c0c14e411\") " pod="openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx" Jul 20 03:25:52.397569 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:52.397458 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfae4943-17f5-420d-810c-300c0c14e411-istio-data\") pod \"e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx\" (UID: \"dfae4943-17f5-420d-810c-300c0c14e411\") " pod="openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx" Jul 20 03:25:52.498918 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:52.498880 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfae4943-17f5-420d-810c-300c0c14e411-istio-envoy\") pod \"e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx\" (UID: \"dfae4943-17f5-420d-810c-300c0c14e411\") " pod="openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx" Jul 20 03:25:52.499161 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:52.498951 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfae4943-17f5-420d-810c-300c0c14e411-workload-certs\") pod \"e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx\" (UID: \"dfae4943-17f5-420d-810c-300c0c14e411\") " pod="openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx" Jul 20 03:25:52.499161 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:52.498985 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfae4943-17f5-420d-810c-300c0c14e411-workload-socket\") pod \"e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx\" (UID: \"dfae4943-17f5-420d-810c-300c0c14e411\") " pod="openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx" Jul 20 03:25:52.499161 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:52.499011 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfae4943-17f5-420d-810c-300c0c14e411-istio-data\") pod \"e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx\" (UID: \"dfae4943-17f5-420d-810c-300c0c14e411\") " pod="openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx" Jul 20 03:25:52.499161 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:52.499044 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfae4943-17f5-420d-810c-300c0c14e411-credential-socket\") pod \"e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx\" (UID: \"dfae4943-17f5-420d-810c-300c0c14e411\") " pod="openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx" Jul 20 03:25:52.499161 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:52.499142 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfae4943-17f5-420d-810c-300c0c14e411-istio-token\") pod \"e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx\" (UID: \"dfae4943-17f5-420d-810c-300c0c14e411\") " pod="openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx" Jul 20 03:25:52.499454 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:52.499165 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2vcd\" (UniqueName: \"kubernetes.io/projected/dfae4943-17f5-420d-810c-300c0c14e411-kube-api-access-z2vcd\") pod \"e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx\" (UID: \"dfae4943-17f5-420d-810c-300c0c14e411\") " pod="openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx" Jul 20 03:25:52.499454 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:52.499197 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfae4943-17f5-420d-810c-300c0c14e411-istio-podinfo\") pod \"e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx\" (UID: \"dfae4943-17f5-420d-810c-300c0c14e411\") " pod="openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx" Jul 20 03:25:52.499454 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:52.499221 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfae4943-17f5-420d-810c-300c0c14e411-istiod-ca-cert\") pod \"e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx\" (UID: \"dfae4943-17f5-420d-810c-300c0c14e411\") " pod="openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx" Jul 20 03:25:52.499589 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:52.499482 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfae4943-17f5-420d-810c-300c0c14e411-credential-socket\") pod \"e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx\" (UID: \"dfae4943-17f5-420d-810c-300c0c14e411\") " pod="openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx" Jul 20 03:25:52.499589 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:52.499521 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfae4943-17f5-420d-810c-300c0c14e411-workload-certs\") pod \"e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx\" (UID: \"dfae4943-17f5-420d-810c-300c0c14e411\") " pod="openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx" Jul 20 03:25:52.499732 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:52.499710 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfae4943-17f5-420d-810c-300c0c14e411-istio-data\") pod \"e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx\" (UID: \"dfae4943-17f5-420d-810c-300c0c14e411\") " pod="openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx" Jul 20 03:25:52.499854 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:52.499833 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfae4943-17f5-420d-810c-300c0c14e411-workload-socket\") pod \"e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx\" (UID: \"dfae4943-17f5-420d-810c-300c0c14e411\") " pod="openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx" Jul 20 03:25:52.499987 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:52.499968 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfae4943-17f5-420d-810c-300c0c14e411-istiod-ca-cert\") pod \"e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx\" (UID: \"dfae4943-17f5-420d-810c-300c0c14e411\") " pod="openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx" Jul 20 03:25:52.501274 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:52.501254 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfae4943-17f5-420d-810c-300c0c14e411-istio-envoy\") pod \"e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx\" (UID: \"dfae4943-17f5-420d-810c-300c0c14e411\") " pod="openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx" Jul 20 03:25:52.501721 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:52.501704 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfae4943-17f5-420d-810c-300c0c14e411-istio-podinfo\") pod \"e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx\" (UID: \"dfae4943-17f5-420d-810c-300c0c14e411\") " pod="openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx" Jul 20 03:25:52.507661 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:52.507634 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfae4943-17f5-420d-810c-300c0c14e411-istio-token\") pod \"e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx\" (UID: \"dfae4943-17f5-420d-810c-300c0c14e411\") " pod="openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx" Jul 20 03:25:52.508174 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:52.508156 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2vcd\" (UniqueName: \"kubernetes.io/projected/dfae4943-17f5-420d-810c-300c0c14e411-kube-api-access-z2vcd\") pod \"e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx\" (UID: \"dfae4943-17f5-420d-810c-300c0c14e411\") " pod="openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx" Jul 20 03:25:52.617950 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:52.617923 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx" Jul 20 03:25:52.755652 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:52.755626 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx"] Jul 20 03:25:52.757832 ip-10-0-142-139 kubenswrapper[2558]: W0720 03:25:52.757806 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfae4943_17f5_420d_810c_300c0c14e411.slice/crio-b34e88c1b2c378c9d0c7604ded2cd6abae9b1f404b9e5dc4cd6df9e9bee49d87 WatchSource:0}: Error finding container b34e88c1b2c378c9d0c7604ded2cd6abae9b1f404b9e5dc4cd6df9e9bee49d87: Status 404 returned error can't find the container with id b34e88c1b2c378c9d0c7604ded2cd6abae9b1f404b9e5dc4cd6df9e9bee49d87 Jul 20 03:25:52.760284 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:52.760251 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:25:52.760368 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:52.760320 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:25:52.760368 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:52.760350 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:25:53.158675 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:53.158637 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx" event={"ID":"dfae4943-17f5-420d-810c-300c0c14e411","Type":"ContainerStarted","Data":"2f1cdc01848b931cd78919d8d859696adceedfe9bc3708443751a5a1d078a006"} Jul 20 03:25:53.158883 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:53.158701 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx" event={"ID":"dfae4943-17f5-420d-810c-300c0c14e411","Type":"ContainerStarted","Data":"b34e88c1b2c378c9d0c7604ded2cd6abae9b1f404b9e5dc4cd6df9e9bee49d87"} Jul 20 03:25:53.183307 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:53.183241 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx" podStartSLOduration=1.183225116 podStartE2EDuration="1.183225116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 03:25:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 03:25:53.180821377 +0000 UTC m=+2448.380070526" watchObservedRunningTime="2026-07-20 03:25:53.183225116 +0000 UTC m=+2448.382474263" Jul 20 03:25:53.227946 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:53.227914 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:25:53.618940 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:53.618910 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx" Jul 20 03:25:53.623897 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:53.623873 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx" Jul 20 03:25:54.164375 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:54.164341 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx" Jul 20 03:25:54.165340 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:25:54.165315 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx" Jul 20 03:26:01.125453 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:26:01.125407 2558 patch_prober.go:28] interesting pod/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 20 03:26:01.125869 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:26:01.125466 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs" podUID="61d4ee0d-7df1-4355-b38d-38478c52d141" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 20 03:26:01.203253 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:26:01.203163 2558 generic.go:356] "Generic (PLEG): container finished" podID="61d4ee0d-7df1-4355-b38d-38478c52d141" containerID="7f3b3f9ce70cf777fbd6226a331cffe8bc183f4e2323345eca5f3939978b33ba" exitCode=0 Jul 20 03:26:01.203253 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:26:01.203241 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs" event={"ID":"61d4ee0d-7df1-4355-b38d-38478c52d141","Type":"ContainerDied","Data":"7f3b3f9ce70cf777fbd6226a331cffe8bc183f4e2323345eca5f3939978b33ba"} Jul 20 03:26:01.203642 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:26:01.203626 2558 scope.go:117] "RemoveContainer" containerID="7f3b3f9ce70cf777fbd6226a331cffe8bc183f4e2323345eca5f3939978b33ba" Jul 20 03:26:01.204571 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:26:01.204547 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:26:01.204622 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:26:01.204609 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:26:01.204663 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:26:01.204651 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:26:02.210239 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:26:02.210204 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs" event={"ID":"61d4ee0d-7df1-4355-b38d-38478c52d141","Type":"ContainerStarted","Data":"7edf63103135d42b6dd0216dc922c590b58eb6501e2a201fc00eb3a0afec86a3"} Jul 20 03:26:03.160128 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:26:03.160068 2558 patch_prober.go:28] interesting pod/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 20 03:26:03.160788 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:26:03.160419 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs" podUID="61d4ee0d-7df1-4355-b38d-38478c52d141" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 20 03:26:03.161013 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:26:03.159548 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs" Jul 20 03:26:04.159294 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:26:04.159250 2558 patch_prober.go:28] interesting pod/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 20 03:26:04.159788 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:26:04.159321 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs" podUID="61d4ee0d-7df1-4355-b38d-38478c52d141" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 20 03:26:05.160139 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:26:05.159634 2558 patch_prober.go:28] interesting pod/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 20 03:26:05.160139 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:26:05.159705 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs" podUID="61d4ee0d-7df1-4355-b38d-38478c52d141" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 20 03:26:05.526341 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:26:05.526246 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:26:05.581819 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:26:05.581785 2558 scope.go:117] "RemoveContainer" containerID="503892aded9c4015d28b87bf1453c1b59522ef39323f96eb5f37b6444e539407" Jul 20 03:26:06.159457 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:26:06.159420 2558 patch_prober.go:28] interesting pod/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 20 03:26:06.159661 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:26:06.159491 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs" podUID="61d4ee0d-7df1-4355-b38d-38478c52d141" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 20 03:26:07.163260 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:26:07.163177 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs" Jul 20 03:26:07.163745 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:26:07.163493 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs" Jul 20 03:26:07.164087 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:26:07.164068 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs" Jul 20 03:26:07.239766 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:26:07.239737 2558 generic.go:356] "Generic (PLEG): container finished" podID="dfae4943-17f5-420d-810c-300c0c14e411" containerID="2f1cdc01848b931cd78919d8d859696adceedfe9bc3708443751a5a1d078a006" exitCode=0 Jul 20 03:26:07.239941 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:26:07.239811 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx" event={"ID":"dfae4943-17f5-420d-810c-300c0c14e411","Type":"ContainerDied","Data":"2f1cdc01848b931cd78919d8d859696adceedfe9bc3708443751a5a1d078a006"} Jul 20 03:26:07.240368 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:26:07.240350 2558 scope.go:117] "RemoveContainer" containerID="2f1cdc01848b931cd78919d8d859696adceedfe9bc3708443751a5a1d078a006" Jul 20 03:26:07.241418 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:26:07.241389 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:26:07.241497 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:26:07.241480 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:26:07.241550 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:26:07.241527 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:26:08.246731 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:26:08.246688 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx" event={"ID":"dfae4943-17f5-420d-810c-300c0c14e411","Type":"ContainerStarted","Data":"e89d1035a168d842640194f34827062a8cfaaadf9f63b8db2c05466401c77aa9"} Jul 20 03:26:08.618325 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:26:08.618292 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx" Jul 20 03:26:08.619346 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:26:08.619321 2558 patch_prober.go:28] interesting pod/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 20 03:26:08.619481 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:26:08.619369 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx" podUID="dfae4943-17f5-420d-810c-300c0c14e411" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 20 03:26:09.167280 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:26:09.167194 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx" Jul 20 03:26:09.618812 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:26:09.618779 2558 patch_prober.go:28] interesting pod/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 20 03:26:09.619213 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:26:09.618849 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx" podUID="dfae4943-17f5-420d-810c-300c0c14e411" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 20 03:26:10.618549 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:26:10.618511 2558 patch_prober.go:28] interesting pod/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 20 03:26:10.618729 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:26:10.618575 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx" podUID="dfae4943-17f5-420d-810c-300c0c14e411" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 20 03:26:11.619364 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:26:11.619326 2558 patch_prober.go:28] interesting pod/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 20 03:26:11.619775 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:26:11.619387 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx" podUID="dfae4943-17f5-420d-810c-300c0c14e411" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 20 03:26:12.622827 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:26:12.622794 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx" Jul 20 03:26:12.623893 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:26:12.623874 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx" Jul 20 03:26:15.024681 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:26:15.024645 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:26:48.121561 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:26:48.121514 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:27:02.230751 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:02.230668 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:27:10.133856 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:10.133822 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-da51e5-wbcjp"] Jul 20 03:27:10.137501 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:10.137481 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-da51e5-wbcjp" Jul 20 03:27:10.140246 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:10.140226 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 20 03:27:10.140382 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:10.140255 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hlzh6\"" Jul 20 03:27:10.144393 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:10.144372 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-da51e5-wbcjp"] Jul 20 03:27:10.247265 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:10.247231 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f823576-8b89-4d14-ab28-90866da94eca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-da51e5-wbcjp\" (UID: \"9f823576-8b89-4d14-ab28-90866da94eca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-da51e5-wbcjp" Jul 20 03:27:10.348686 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:10.348651 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f823576-8b89-4d14-ab28-90866da94eca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-da51e5-wbcjp\" (UID: \"9f823576-8b89-4d14-ab28-90866da94eca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-da51e5-wbcjp" Jul 20 03:27:10.349330 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:10.349312 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f823576-8b89-4d14-ab28-90866da94eca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-da51e5-wbcjp\" (UID: \"9f823576-8b89-4d14-ab28-90866da94eca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-da51e5-wbcjp" Jul 20 03:27:10.449931 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:10.449852 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-da51e5-wbcjp" Jul 20 03:27:10.583526 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:10.583499 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-da51e5-wbcjp"] Jul 20 03:27:10.585635 ip-10-0-142-139 kubenswrapper[2558]: W0720 03:27:10.585605 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f823576_8b89_4d14_ab28_90866da94eca.slice/crio-6731a4a2791a95eb871bb529678e8187b59e62470890aaa994f3e5f6cfc7b93e WatchSource:0}: Error finding container 6731a4a2791a95eb871bb529678e8187b59e62470890aaa994f3e5f6cfc7b93e: Status 404 returned error can't find the container with id 6731a4a2791a95eb871bb529678e8187b59e62470890aaa994f3e5f6cfc7b93e Jul 20 03:27:11.557368 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:11.557280 2558 generic.go:356] "Generic (PLEG): container finished" podID="9f823576-8b89-4d14-ab28-90866da94eca" containerID="294ffdf9f1f0b748e4d747aaca17c93cbb257ff747138cf4b996320cbae77ced" exitCode=0 Jul 20 03:27:11.557368 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:11.557336 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-da51e5-wbcjp" event={"ID":"9f823576-8b89-4d14-ab28-90866da94eca","Type":"ContainerDied","Data":"294ffdf9f1f0b748e4d747aaca17c93cbb257ff747138cf4b996320cbae77ced"} Jul 20 03:27:11.557754 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:11.557375 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-da51e5-wbcjp" event={"ID":"9f823576-8b89-4d14-ab28-90866da94eca","Type":"ContainerStarted","Data":"6731a4a2791a95eb871bb529678e8187b59e62470890aaa994f3e5f6cfc7b93e"} Jul 20 03:27:12.694134 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:12.694080 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-da51e5-wbcjp" Jul 20 03:27:12.770154 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:12.770089 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f823576-8b89-4d14-ab28-90866da94eca-maas-api-service-ca\") pod \"9f823576-8b89-4d14-ab28-90866da94eca\" (UID: \"9f823576-8b89-4d14-ab28-90866da94eca\") " Jul 20 03:27:12.770454 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:12.770432 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f823576-8b89-4d14-ab28-90866da94eca-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9f823576-8b89-4d14-ab28-90866da94eca" (UID: "9f823576-8b89-4d14-ab28-90866da94eca"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 03:27:12.870895 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:12.870860 2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f823576-8b89-4d14-ab28-90866da94eca-maas-api-service-ca\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:27:13.569015 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:13.568972 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-da51e5-wbcjp" event={"ID":"9f823576-8b89-4d14-ab28-90866da94eca","Type":"ContainerDied","Data":"6731a4a2791a95eb871bb529678e8187b59e62470890aaa994f3e5f6cfc7b93e"} Jul 20 03:27:13.569202 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:13.569044 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6731a4a2791a95eb871bb529678e8187b59e62470890aaa994f3e5f6cfc7b93e" Jul 20 03:27:13.569202 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:13.568987 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-da51e5-wbcjp" Jul 20 03:27:15.325644 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:15.325608 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:27:23.821684 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:23.821651 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:27:25.274608 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:25.274565 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-da51e5-wbcjp"] Jul 20 03:27:25.276150 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:25.276131 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-da51e5-wbcjp"] Jul 20 03:27:25.424570 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:25.424537 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f823576-8b89-4d14-ab28-90866da94eca" path="/var/lib/kubelet/pods/9f823576-8b89-4d14-ab28-90866da94eca/volumes" Jul 20 03:27:31.677783 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:31.677749 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs"] Jul 20 03:27:31.678328 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:31.677987 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs" podUID="61d4ee0d-7df1-4355-b38d-38478c52d141" containerName="istio-proxy" containerID="cri-o://7edf63103135d42b6dd0216dc922c590b58eb6501e2a201fc00eb3a0afec86a3" gracePeriod=30 Jul 20 03:27:36.931417 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:36.931392 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs" Jul 20 03:27:37.099135 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:37.099035 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61d4ee0d-7df1-4355-b38d-38478c52d141-istio-envoy\") pod \"61d4ee0d-7df1-4355-b38d-38478c52d141\" (UID: \"61d4ee0d-7df1-4355-b38d-38478c52d141\") " Jul 20 03:27:37.099135 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:37.099086 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61d4ee0d-7df1-4355-b38d-38478c52d141-istio-podinfo\") pod \"61d4ee0d-7df1-4355-b38d-38478c52d141\" (UID: \"61d4ee0d-7df1-4355-b38d-38478c52d141\") " Jul 20 03:27:37.099366 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:37.099150 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zt77k\" (UniqueName: \"kubernetes.io/projected/61d4ee0d-7df1-4355-b38d-38478c52d141-kube-api-access-zt77k\") pod \"61d4ee0d-7df1-4355-b38d-38478c52d141\" (UID: \"61d4ee0d-7df1-4355-b38d-38478c52d141\") " Jul 20 03:27:37.099366 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:37.099194 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61d4ee0d-7df1-4355-b38d-38478c52d141-workload-certs\") pod \"61d4ee0d-7df1-4355-b38d-38478c52d141\" (UID: \"61d4ee0d-7df1-4355-b38d-38478c52d141\") " Jul 20 03:27:37.099366 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:37.099214 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d4ee0d-7df1-4355-b38d-38478c52d141-workload-socket\") pod \"61d4ee0d-7df1-4355-b38d-38478c52d141\" (UID: \"61d4ee0d-7df1-4355-b38d-38478c52d141\") " Jul 20 03:27:37.099366 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:37.099228 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61d4ee0d-7df1-4355-b38d-38478c52d141-istio-data\") pod \"61d4ee0d-7df1-4355-b38d-38478c52d141\" (UID: \"61d4ee0d-7df1-4355-b38d-38478c52d141\") " Jul 20 03:27:37.099366 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:37.099242 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d4ee0d-7df1-4355-b38d-38478c52d141-credential-socket\") pod \"61d4ee0d-7df1-4355-b38d-38478c52d141\" (UID: \"61d4ee0d-7df1-4355-b38d-38478c52d141\") " Jul 20 03:27:37.099366 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:37.099301 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61d4ee0d-7df1-4355-b38d-38478c52d141-istiod-ca-cert\") pod \"61d4ee0d-7df1-4355-b38d-38478c52d141\" (UID: \"61d4ee0d-7df1-4355-b38d-38478c52d141\") " Jul 20 03:27:37.099366 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:37.099343 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61d4ee0d-7df1-4355-b38d-38478c52d141-istio-token\") pod \"61d4ee0d-7df1-4355-b38d-38478c52d141\" (UID: \"61d4ee0d-7df1-4355-b38d-38478c52d141\") " Jul 20 03:27:37.099741 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:37.099512 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d4ee0d-7df1-4355-b38d-38478c52d141-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "61d4ee0d-7df1-4355-b38d-38478c52d141" (UID: "61d4ee0d-7df1-4355-b38d-38478c52d141"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:27:37.099741 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:37.099654 2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61d4ee0d-7df1-4355-b38d-38478c52d141-workload-certs\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:27:37.099844 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:37.099683 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d4ee0d-7df1-4355-b38d-38478c52d141-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "61d4ee0d-7df1-4355-b38d-38478c52d141" (UID: "61d4ee0d-7df1-4355-b38d-38478c52d141"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:27:37.099844 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:37.099762 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d4ee0d-7df1-4355-b38d-38478c52d141-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "61d4ee0d-7df1-4355-b38d-38478c52d141" (UID: "61d4ee0d-7df1-4355-b38d-38478c52d141"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:27:37.099944 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:37.099855 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61d4ee0d-7df1-4355-b38d-38478c52d141-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "61d4ee0d-7df1-4355-b38d-38478c52d141" (UID: "61d4ee0d-7df1-4355-b38d-38478c52d141"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 03:27:37.100140 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:37.100111 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d4ee0d-7df1-4355-b38d-38478c52d141-istio-data" (OuterVolumeSpecName: "istio-data") pod "61d4ee0d-7df1-4355-b38d-38478c52d141" (UID: "61d4ee0d-7df1-4355-b38d-38478c52d141"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:27:37.101578 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:37.101547 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61d4ee0d-7df1-4355-b38d-38478c52d141-kube-api-access-zt77k" (OuterVolumeSpecName: "kube-api-access-zt77k") pod "61d4ee0d-7df1-4355-b38d-38478c52d141" (UID: "61d4ee0d-7df1-4355-b38d-38478c52d141"). InnerVolumeSpecName "kube-api-access-zt77k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 03:27:37.101676 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:37.101593 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61d4ee0d-7df1-4355-b38d-38478c52d141-istio-token" (OuterVolumeSpecName: "istio-token") pod "61d4ee0d-7df1-4355-b38d-38478c52d141" (UID: "61d4ee0d-7df1-4355-b38d-38478c52d141"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 03:27:37.101797 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:37.101781 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d4ee0d-7df1-4355-b38d-38478c52d141-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "61d4ee0d-7df1-4355-b38d-38478c52d141" (UID: "61d4ee0d-7df1-4355-b38d-38478c52d141"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:27:37.101841 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:37.101823 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61d4ee0d-7df1-4355-b38d-38478c52d141-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "61d4ee0d-7df1-4355-b38d-38478c52d141" (UID: "61d4ee0d-7df1-4355-b38d-38478c52d141"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 20 03:27:37.200938 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:37.200901 2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61d4ee0d-7df1-4355-b38d-38478c52d141-istio-envoy\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:27:37.200938 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:37.200933 2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61d4ee0d-7df1-4355-b38d-38478c52d141-istio-podinfo\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:27:37.200938 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:37.200943 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt77k\" (UniqueName: \"kubernetes.io/projected/61d4ee0d-7df1-4355-b38d-38478c52d141-kube-api-access-zt77k\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:27:37.201201 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:37.200952 2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d4ee0d-7df1-4355-b38d-38478c52d141-workload-socket\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:27:37.201201 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:37.200960 2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61d4ee0d-7df1-4355-b38d-38478c52d141-istio-data\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:27:37.201201 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:37.200969 2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d4ee0d-7df1-4355-b38d-38478c52d141-credential-socket\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:27:37.201201 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:37.200976 2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61d4ee0d-7df1-4355-b38d-38478c52d141-istiod-ca-cert\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:27:37.201201 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:37.200984 2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61d4ee0d-7df1-4355-b38d-38478c52d141-istio-token\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:27:37.688590 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:37.688505 2558 generic.go:356] "Generic (PLEG): container finished" podID="61d4ee0d-7df1-4355-b38d-38478c52d141" containerID="7edf63103135d42b6dd0216dc922c590b58eb6501e2a201fc00eb3a0afec86a3" exitCode=0 Jul 20 03:27:37.688590 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:37.688555 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs" event={"ID":"61d4ee0d-7df1-4355-b38d-38478c52d141","Type":"ContainerDied","Data":"7edf63103135d42b6dd0216dc922c590b58eb6501e2a201fc00eb3a0afec86a3"} Jul 20 03:27:37.688793 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:37.688594 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs" event={"ID":"61d4ee0d-7df1-4355-b38d-38478c52d141","Type":"ContainerDied","Data":"ec0cff27c1b06a42d46503c419caf9d296d55930bf77503f13116eb085ae412b"} Jul 20 03:27:37.688793 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:37.688558 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs" Jul 20 03:27:37.688793 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:37.688635 2558 scope.go:117] "RemoveContainer" containerID="7edf63103135d42b6dd0216dc922c590b58eb6501e2a201fc00eb3a0afec86a3" Jul 20 03:27:37.697552 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:37.697538 2558 scope.go:117] "RemoveContainer" containerID="7f3b3f9ce70cf777fbd6226a331cffe8bc183f4e2323345eca5f3939978b33ba" Jul 20 03:27:37.706296 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:37.706274 2558 scope.go:117] "RemoveContainer" containerID="7edf63103135d42b6dd0216dc922c590b58eb6501e2a201fc00eb3a0afec86a3" Jul 20 03:27:37.706667 ip-10-0-142-139 kubenswrapper[2558]: E0720 03:27:37.706637 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7edf63103135d42b6dd0216dc922c590b58eb6501e2a201fc00eb3a0afec86a3\": container with ID starting with 7edf63103135d42b6dd0216dc922c590b58eb6501e2a201fc00eb3a0afec86a3 not found: ID does not exist" containerID="7edf63103135d42b6dd0216dc922c590b58eb6501e2a201fc00eb3a0afec86a3" Jul 20 03:27:37.706765 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:37.706680 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7edf63103135d42b6dd0216dc922c590b58eb6501e2a201fc00eb3a0afec86a3"} err="failed to get container status \"7edf63103135d42b6dd0216dc922c590b58eb6501e2a201fc00eb3a0afec86a3\": rpc error: code = NotFound desc = could not find container \"7edf63103135d42b6dd0216dc922c590b58eb6501e2a201fc00eb3a0afec86a3\": container with ID starting with 7edf63103135d42b6dd0216dc922c590b58eb6501e2a201fc00eb3a0afec86a3 not found: ID does not exist" Jul 20 03:27:37.706765 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:37.706708 2558 scope.go:117] "RemoveContainer" containerID="7f3b3f9ce70cf777fbd6226a331cffe8bc183f4e2323345eca5f3939978b33ba" Jul 20 03:27:37.713060 ip-10-0-142-139 kubenswrapper[2558]: E0720 03:27:37.709430 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f3b3f9ce70cf777fbd6226a331cffe8bc183f4e2323345eca5f3939978b33ba\": container with ID starting with 7f3b3f9ce70cf777fbd6226a331cffe8bc183f4e2323345eca5f3939978b33ba not found: ID does not exist" containerID="7f3b3f9ce70cf777fbd6226a331cffe8bc183f4e2323345eca5f3939978b33ba" Jul 20 03:27:37.713060 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:37.709469 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f3b3f9ce70cf777fbd6226a331cffe8bc183f4e2323345eca5f3939978b33ba"} err="failed to get container status \"7f3b3f9ce70cf777fbd6226a331cffe8bc183f4e2323345eca5f3939978b33ba\": rpc error: code = NotFound desc = could not find container \"7f3b3f9ce70cf777fbd6226a331cffe8bc183f4e2323345eca5f3939978b33ba\": container with ID starting with 7f3b3f9ce70cf777fbd6226a331cffe8bc183f4e2323345eca5f3939978b33ba not found: ID does not exist" Jul 20 03:27:37.713060 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:37.712958 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs"] Jul 20 03:27:37.713455 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:37.713435 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-da51e5-openshift-default-86969995f9-xpwvs"] Jul 20 03:27:39.424822 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:39.424789 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61d4ee0d-7df1-4355-b38d-38478c52d141" path="/var/lib/kubelet/pods/61d4ee0d-7df1-4355-b38d-38478c52d141/volumes" Jul 20 03:27:45.627886 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:45.627849 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:27:53.574561 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:53.574519 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx"] Jul 20 03:27:53.574982 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:53.574739 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx" podUID="dfae4943-17f5-420d-810c-300c0c14e411" containerName="istio-proxy" containerID="cri-o://e89d1035a168d842640194f34827062a8cfaaadf9f63b8db2c05466401c77aa9" gracePeriod=30 Jul 20 03:27:54.491672 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:54.491642 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b"] Jul 20 03:27:54.492003 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:54.491989 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f823576-8b89-4d14-ab28-90866da94eca" containerName="revoke-keys" Jul 20 03:27:54.492003 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:54.492003 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f823576-8b89-4d14-ab28-90866da94eca" containerName="revoke-keys" Jul 20 03:27:54.492112 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:54.492016 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61d4ee0d-7df1-4355-b38d-38478c52d141" containerName="istio-proxy" Jul 20 03:27:54.492112 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:54.492023 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="61d4ee0d-7df1-4355-b38d-38478c52d141" containerName="istio-proxy" Jul 20 03:27:54.492112 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:54.492029 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61d4ee0d-7df1-4355-b38d-38478c52d141" containerName="istio-proxy" Jul 20 03:27:54.492112 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:54.492035 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="61d4ee0d-7df1-4355-b38d-38478c52d141" containerName="istio-proxy" Jul 20 03:27:54.492112 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:54.492088 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="61d4ee0d-7df1-4355-b38d-38478c52d141" containerName="istio-proxy" Jul 20 03:27:54.492112 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:54.492112 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="61d4ee0d-7df1-4355-b38d-38478c52d141" containerName="istio-proxy" Jul 20 03:27:54.492300 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:54.492120 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f823576-8b89-4d14-ab28-90866da94eca" containerName="revoke-keys" Jul 20 03:27:54.495836 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:54.495816 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b" Jul 20 03:27:54.498421 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:54.498401 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-52abd4-openshift-default-dockercfg-p6kc2\"" Jul 20 03:27:54.505372 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:54.505336 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b"] Jul 20 03:27:54.553056 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:54.553023 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0121cbf-2b96-4274-949d-d9c2bc39f469-istio-podinfo\") pod \"e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b\" (UID: \"b0121cbf-2b96-4274-949d-d9c2bc39f469\") " pod="openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b" Jul 20 03:27:54.553286 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:54.553060 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0121cbf-2b96-4274-949d-d9c2bc39f469-istio-token\") pod \"e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b\" (UID: \"b0121cbf-2b96-4274-949d-d9c2bc39f469\") " pod="openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b" Jul 20 03:27:54.553286 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:54.553145 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0121cbf-2b96-4274-949d-d9c2bc39f469-credential-socket\") pod \"e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b\" (UID: \"b0121cbf-2b96-4274-949d-d9c2bc39f469\") " pod="openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b" Jul 20 03:27:54.553286 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:54.553189 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0121cbf-2b96-4274-949d-d9c2bc39f469-workload-certs\") pod \"e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b\" (UID: \"b0121cbf-2b96-4274-949d-d9c2bc39f469\") " pod="openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b" Jul 20 03:27:54.553286 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:54.553256 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0121cbf-2b96-4274-949d-d9c2bc39f469-istio-data\") pod \"e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b\" (UID: \"b0121cbf-2b96-4274-949d-d9c2bc39f469\") " pod="openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b" Jul 20 03:27:54.553286 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:54.553278 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0121cbf-2b96-4274-949d-d9c2bc39f469-istio-envoy\") pod \"e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b\" (UID: \"b0121cbf-2b96-4274-949d-d9c2bc39f469\") " pod="openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b" Jul 20 03:27:54.553480 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:54.553298 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0121cbf-2b96-4274-949d-d9c2bc39f469-istiod-ca-cert\") pod \"e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b\" (UID: \"b0121cbf-2b96-4274-949d-d9c2bc39f469\") " pod="openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b" Jul 20 03:27:54.553480 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:54.553326 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0121cbf-2b96-4274-949d-d9c2bc39f469-workload-socket\") pod \"e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b\" (UID: \"b0121cbf-2b96-4274-949d-d9c2bc39f469\") " pod="openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b" Jul 20 03:27:54.553480 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:54.553360 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w9dd\" (UniqueName: \"kubernetes.io/projected/b0121cbf-2b96-4274-949d-d9c2bc39f469-kube-api-access-7w9dd\") pod \"e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b\" (UID: \"b0121cbf-2b96-4274-949d-d9c2bc39f469\") " pod="openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b" Jul 20 03:27:54.654123 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:54.654079 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0121cbf-2b96-4274-949d-d9c2bc39f469-istio-data\") pod \"e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b\" (UID: \"b0121cbf-2b96-4274-949d-d9c2bc39f469\") " pod="openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b" Jul 20 03:27:54.654536 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:54.654139 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0121cbf-2b96-4274-949d-d9c2bc39f469-istio-envoy\") pod \"e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b\" (UID: \"b0121cbf-2b96-4274-949d-d9c2bc39f469\") " pod="openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b" Jul 20 03:27:54.654536 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:54.654159 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0121cbf-2b96-4274-949d-d9c2bc39f469-istiod-ca-cert\") pod \"e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b\" (UID: \"b0121cbf-2b96-4274-949d-d9c2bc39f469\") " pod="openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b" Jul 20 03:27:54.654536 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:54.654185 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0121cbf-2b96-4274-949d-d9c2bc39f469-workload-socket\") pod \"e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b\" (UID: \"b0121cbf-2b96-4274-949d-d9c2bc39f469\") " pod="openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b" Jul 20 03:27:54.654536 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:54.654211 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w9dd\" (UniqueName: \"kubernetes.io/projected/b0121cbf-2b96-4274-949d-d9c2bc39f469-kube-api-access-7w9dd\") pod \"e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b\" (UID: \"b0121cbf-2b96-4274-949d-d9c2bc39f469\") " pod="openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b" Jul 20 03:27:54.654536 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:54.654272 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0121cbf-2b96-4274-949d-d9c2bc39f469-istio-podinfo\") pod \"e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b\" (UID: \"b0121cbf-2b96-4274-949d-d9c2bc39f469\") " pod="openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b" Jul 20 03:27:54.654536 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:54.654298 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0121cbf-2b96-4274-949d-d9c2bc39f469-istio-token\") pod \"e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b\" (UID: \"b0121cbf-2b96-4274-949d-d9c2bc39f469\") " pod="openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b" Jul 20 03:27:54.654536 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:54.654358 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0121cbf-2b96-4274-949d-d9c2bc39f469-credential-socket\") pod \"e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b\" (UID: \"b0121cbf-2b96-4274-949d-d9c2bc39f469\") " pod="openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b" Jul 20 03:27:54.654536 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:54.654399 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0121cbf-2b96-4274-949d-d9c2bc39f469-workload-certs\") pod \"e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b\" (UID: \"b0121cbf-2b96-4274-949d-d9c2bc39f469\") " pod="openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b" Jul 20 03:27:54.654536 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:54.654453 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0121cbf-2b96-4274-949d-d9c2bc39f469-istio-data\") pod \"e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b\" (UID: \"b0121cbf-2b96-4274-949d-d9c2bc39f469\") " pod="openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b" Jul 20 03:27:54.654995 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:54.654707 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0121cbf-2b96-4274-949d-d9c2bc39f469-credential-socket\") pod \"e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b\" (UID: \"b0121cbf-2b96-4274-949d-d9c2bc39f469\") " pod="openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b" Jul 20 03:27:54.654995 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:54.654764 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0121cbf-2b96-4274-949d-d9c2bc39f469-workload-certs\") pod \"e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b\" (UID: \"b0121cbf-2b96-4274-949d-d9c2bc39f469\") " pod="openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b" Jul 20 03:27:54.654995 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:54.654854 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0121cbf-2b96-4274-949d-d9c2bc39f469-workload-socket\") pod \"e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b\" (UID: \"b0121cbf-2b96-4274-949d-d9c2bc39f469\") " pod="openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b" Jul 20 03:27:54.654995 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:54.654891 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0121cbf-2b96-4274-949d-d9c2bc39f469-istiod-ca-cert\") pod \"e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b\" (UID: \"b0121cbf-2b96-4274-949d-d9c2bc39f469\") " pod="openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b" Jul 20 03:27:54.656381 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:54.656360 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0121cbf-2b96-4274-949d-d9c2bc39f469-istio-envoy\") pod \"e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b\" (UID: \"b0121cbf-2b96-4274-949d-d9c2bc39f469\") " pod="openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b" Jul 20 03:27:54.656746 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:54.656729 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0121cbf-2b96-4274-949d-d9c2bc39f469-istio-podinfo\") pod \"e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b\" (UID: \"b0121cbf-2b96-4274-949d-d9c2bc39f469\") " pod="openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b" Jul 20 03:27:54.669249 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:54.669222 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0121cbf-2b96-4274-949d-d9c2bc39f469-istio-token\") pod \"e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b\" (UID: \"b0121cbf-2b96-4274-949d-d9c2bc39f469\") " pod="openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b" Jul 20 03:27:54.669452 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:54.669434 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w9dd\" (UniqueName: \"kubernetes.io/projected/b0121cbf-2b96-4274-949d-d9c2bc39f469-kube-api-access-7w9dd\") pod \"e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b\" (UID: \"b0121cbf-2b96-4274-949d-d9c2bc39f469\") " pod="openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b" Jul 20 03:27:54.806859 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:54.806776 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b" Jul 20 03:27:54.945856 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:54.945829 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b"] Jul 20 03:27:54.947011 ip-10-0-142-139 kubenswrapper[2558]: W0720 03:27:54.946987 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0121cbf_2b96_4274_949d_d9c2bc39f469.slice/crio-a76adec1d164d2258a0b276ac85fa3e8fcfdbac69c1fd3f762799b01b63888c3 WatchSource:0}: Error finding container a76adec1d164d2258a0b276ac85fa3e8fcfdbac69c1fd3f762799b01b63888c3: Status 404 returned error can't find the container with id a76adec1d164d2258a0b276ac85fa3e8fcfdbac69c1fd3f762799b01b63888c3 Jul 20 03:27:54.949137 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:54.949085 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:27:54.949240 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:54.949167 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:27:54.949240 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:54.949195 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:27:55.777817 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:55.777771 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b" event={"ID":"b0121cbf-2b96-4274-949d-d9c2bc39f469","Type":"ContainerStarted","Data":"622f1abfba062bf3f48708a6c8255ce36e0cbb43257f9d79894f0a8173632170"} Jul 20 03:27:55.778332 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:55.777840 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b" event={"ID":"b0121cbf-2b96-4274-949d-d9c2bc39f469","Type":"ContainerStarted","Data":"a76adec1d164d2258a0b276ac85fa3e8fcfdbac69c1fd3f762799b01b63888c3"} Jul 20 03:27:55.799159 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:55.799089 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b" podStartSLOduration=1.799072995 podStartE2EDuration="1.799072995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 03:27:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 03:27:55.797080309 +0000 UTC m=+2570.996329470" watchObservedRunningTime="2026-07-20 03:27:55.799072995 +0000 UTC m=+2570.998322142" Jul 20 03:27:55.807618 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:55.807582 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b" Jul 20 03:27:56.812768 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:56.812738 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b" Jul 20 03:27:57.626299 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:57.626256 2558 patch_prober.go:28] interesting pod/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 20 03:27:57.626474 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:57.626337 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx" podUID="dfae4943-17f5-420d-810c-300c0c14e411" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 20 03:27:57.786735 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:57.786699 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b" Jul 20 03:27:57.787608 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:57.787580 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b" Jul 20 03:27:58.794118 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:58.794064 2558 generic.go:356] "Generic (PLEG): container finished" podID="dfae4943-17f5-420d-810c-300c0c14e411" containerID="e89d1035a168d842640194f34827062a8cfaaadf9f63b8db2c05466401c77aa9" exitCode=0 Jul 20 03:27:58.794507 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:58.794126 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx" event={"ID":"dfae4943-17f5-420d-810c-300c0c14e411","Type":"ContainerDied","Data":"e89d1035a168d842640194f34827062a8cfaaadf9f63b8db2c05466401c77aa9"} Jul 20 03:27:58.794507 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:58.794193 2558 scope.go:117] "RemoveContainer" containerID="2f1cdc01848b931cd78919d8d859696adceedfe9bc3708443751a5a1d078a006" Jul 20 03:27:58.840584 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:58.840521 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx" Jul 20 03:27:58.894330 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:58.894299 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfae4943-17f5-420d-810c-300c0c14e411-istio-token\") pod \"dfae4943-17f5-420d-810c-300c0c14e411\" (UID: \"dfae4943-17f5-420d-810c-300c0c14e411\") " Jul 20 03:27:58.894330 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:58.894331 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfae4943-17f5-420d-810c-300c0c14e411-istio-data\") pod \"dfae4943-17f5-420d-810c-300c0c14e411\" (UID: \"dfae4943-17f5-420d-810c-300c0c14e411\") " Jul 20 03:27:58.894556 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:58.894353 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfae4943-17f5-420d-810c-300c0c14e411-workload-socket\") pod \"dfae4943-17f5-420d-810c-300c0c14e411\" (UID: \"dfae4943-17f5-420d-810c-300c0c14e411\") " Jul 20 03:27:58.894556 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:58.894378 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfae4943-17f5-420d-810c-300c0c14e411-istio-podinfo\") pod \"dfae4943-17f5-420d-810c-300c0c14e411\" (UID: \"dfae4943-17f5-420d-810c-300c0c14e411\") " Jul 20 03:27:58.894556 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:58.894424 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfae4943-17f5-420d-810c-300c0c14e411-credential-socket\") pod \"dfae4943-17f5-420d-810c-300c0c14e411\" (UID: \"dfae4943-17f5-420d-810c-300c0c14e411\") " Jul 20 03:27:58.894556 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:58.894455 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfae4943-17f5-420d-810c-300c0c14e411-istiod-ca-cert\") pod \"dfae4943-17f5-420d-810c-300c0c14e411\" (UID: \"dfae4943-17f5-420d-810c-300c0c14e411\") " Jul 20 03:27:58.894556 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:58.894488 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z2vcd\" (UniqueName: \"kubernetes.io/projected/dfae4943-17f5-420d-810c-300c0c14e411-kube-api-access-z2vcd\") pod \"dfae4943-17f5-420d-810c-300c0c14e411\" (UID: \"dfae4943-17f5-420d-810c-300c0c14e411\") " Jul 20 03:27:58.894556 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:58.894515 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfae4943-17f5-420d-810c-300c0c14e411-istio-envoy\") pod \"dfae4943-17f5-420d-810c-300c0c14e411\" (UID: \"dfae4943-17f5-420d-810c-300c0c14e411\") " Jul 20 03:27:58.894887 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:58.894714 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfae4943-17f5-420d-810c-300c0c14e411-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dfae4943-17f5-420d-810c-300c0c14e411" (UID: "dfae4943-17f5-420d-810c-300c0c14e411"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:27:58.894887 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:58.894831 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfae4943-17f5-420d-810c-300c0c14e411-workload-certs\") pod \"dfae4943-17f5-420d-810c-300c0c14e411\" (UID: \"dfae4943-17f5-420d-810c-300c0c14e411\") " Jul 20 03:27:58.894887 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:58.894835 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfae4943-17f5-420d-810c-300c0c14e411-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dfae4943-17f5-420d-810c-300c0c14e411" (UID: "dfae4943-17f5-420d-810c-300c0c14e411"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:27:58.895043 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:58.894920 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfae4943-17f5-420d-810c-300c0c14e411-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dfae4943-17f5-420d-810c-300c0c14e411" (UID: "dfae4943-17f5-420d-810c-300c0c14e411"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 03:27:58.895200 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:58.895142 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfae4943-17f5-420d-810c-300c0c14e411-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dfae4943-17f5-420d-810c-300c0c14e411" (UID: "dfae4943-17f5-420d-810c-300c0c14e411"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:27:58.895307 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:58.895289 2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfae4943-17f5-420d-810c-300c0c14e411-istiod-ca-cert\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:27:58.895372 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:58.895312 2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfae4943-17f5-420d-810c-300c0c14e411-workload-certs\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:27:58.895372 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:58.895327 2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfae4943-17f5-420d-810c-300c0c14e411-workload-socket\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:27:58.895372 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:58.895332 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfae4943-17f5-420d-810c-300c0c14e411-istio-data" (OuterVolumeSpecName: "istio-data") pod "dfae4943-17f5-420d-810c-300c0c14e411" (UID: "dfae4943-17f5-420d-810c-300c0c14e411"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:27:58.895372 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:58.895340 2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfae4943-17f5-420d-810c-300c0c14e411-credential-socket\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:27:58.896800 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:58.896772 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dfae4943-17f5-420d-810c-300c0c14e411-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dfae4943-17f5-420d-810c-300c0c14e411" (UID: "dfae4943-17f5-420d-810c-300c0c14e411"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 20 03:27:58.896800 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:58.896786 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfae4943-17f5-420d-810c-300c0c14e411-kube-api-access-z2vcd" (OuterVolumeSpecName: "kube-api-access-z2vcd") pod "dfae4943-17f5-420d-810c-300c0c14e411" (UID: "dfae4943-17f5-420d-810c-300c0c14e411"). InnerVolumeSpecName "kube-api-access-z2vcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 03:27:58.897028 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:58.897012 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfae4943-17f5-420d-810c-300c0c14e411-istio-token" (OuterVolumeSpecName: "istio-token") pod "dfae4943-17f5-420d-810c-300c0c14e411" (UID: "dfae4943-17f5-420d-810c-300c0c14e411"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 03:27:58.897604 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:58.897582 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfae4943-17f5-420d-810c-300c0c14e411-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dfae4943-17f5-420d-810c-300c0c14e411" (UID: "dfae4943-17f5-420d-810c-300c0c14e411"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:27:58.996517 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:58.996487 2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfae4943-17f5-420d-810c-300c0c14e411-istio-token\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:27:58.996517 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:58.996514 2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfae4943-17f5-420d-810c-300c0c14e411-istio-data\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:27:58.996517 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:58.996522 2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfae4943-17f5-420d-810c-300c0c14e411-istio-podinfo\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:27:58.996747 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:58.996530 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2vcd\" (UniqueName: \"kubernetes.io/projected/dfae4943-17f5-420d-810c-300c0c14e411-kube-api-access-z2vcd\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:27:58.996747 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:58.996541 2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfae4943-17f5-420d-810c-300c0c14e411-istio-envoy\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:27:59.800377 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:59.800288 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx" Jul 20 03:27:59.800377 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:59.800326 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx" event={"ID":"dfae4943-17f5-420d-810c-300c0c14e411","Type":"ContainerDied","Data":"b34e88c1b2c378c9d0c7604ded2cd6abae9b1f404b9e5dc4cd6df9e9bee49d87"} Jul 20 03:27:59.800888 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:59.800380 2558 scope.go:117] "RemoveContainer" containerID="e89d1035a168d842640194f34827062a8cfaaadf9f63b8db2c05466401c77aa9" Jul 20 03:27:59.817926 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:59.817901 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx"] Jul 20 03:27:59.822430 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:27:59.822405 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-9fe24f-openshift-default-64f676fb75-5q6sx"] Jul 20 03:28:01.424077 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:01.424037 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfae4943-17f5-420d-810c-300c0c14e411" path="/var/lib/kubelet/pods/dfae4943-17f5-420d-810c-300c0c14e411/volumes" Jul 20 03:28:02.960706 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:02.960670 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq"] Jul 20 03:28:02.961252 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:02.961232 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfae4943-17f5-420d-810c-300c0c14e411" containerName="istio-proxy" Jul 20 03:28:02.961349 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:02.961254 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfae4943-17f5-420d-810c-300c0c14e411" containerName="istio-proxy" Jul 20 03:28:02.961349 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:02.961281 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfae4943-17f5-420d-810c-300c0c14e411" containerName="istio-proxy" Jul 20 03:28:02.961349 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:02.961290 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfae4943-17f5-420d-810c-300c0c14e411" containerName="istio-proxy" Jul 20 03:28:02.961494 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:02.961381 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfae4943-17f5-420d-810c-300c0c14e411" containerName="istio-proxy" Jul 20 03:28:02.966060 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:02.966040 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq" Jul 20 03:28:02.968666 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:02.968642 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-a8945e-openshift-default-dockercfg-ps7vr\"" Jul 20 03:28:02.976988 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:02.976968 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq"] Jul 20 03:28:03.031900 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:03.031857 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fe7950-e2d4-459c-831d-03ac30620fa6-workload-socket\") pod \"e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq\" (UID: \"69fe7950-e2d4-459c-831d-03ac30620fa6\") " pod="openshift-ingress/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq" Jul 20 03:28:03.032122 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:03.031919 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69fe7950-e2d4-459c-831d-03ac30620fa6-istio-envoy\") pod \"e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq\" (UID: \"69fe7950-e2d4-459c-831d-03ac30620fa6\") " pod="openshift-ingress/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq" Jul 20 03:28:03.032122 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:03.031944 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69fe7950-e2d4-459c-831d-03ac30620fa6-istio-data\") pod \"e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq\" (UID: \"69fe7950-e2d4-459c-831d-03ac30620fa6\") " pod="openshift-ingress/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq" Jul 20 03:28:03.032122 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:03.031968 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86hjb\" (UniqueName: \"kubernetes.io/projected/69fe7950-e2d4-459c-831d-03ac30620fa6-kube-api-access-86hjb\") pod \"e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq\" (UID: \"69fe7950-e2d4-459c-831d-03ac30620fa6\") " pod="openshift-ingress/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq" Jul 20 03:28:03.032122 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:03.032022 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69fe7950-e2d4-459c-831d-03ac30620fa6-istio-podinfo\") pod \"e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq\" (UID: \"69fe7950-e2d4-459c-831d-03ac30620fa6\") " pod="openshift-ingress/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq" Jul 20 03:28:03.032363 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:03.032121 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69fe7950-e2d4-459c-831d-03ac30620fa6-workload-certs\") pod \"e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq\" (UID: \"69fe7950-e2d4-459c-831d-03ac30620fa6\") " pod="openshift-ingress/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq" Jul 20 03:28:03.032363 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:03.032168 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69fe7950-e2d4-459c-831d-03ac30620fa6-istio-token\") pod \"e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq\" (UID: \"69fe7950-e2d4-459c-831d-03ac30620fa6\") " pod="openshift-ingress/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq" Jul 20 03:28:03.032363 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:03.032199 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fe7950-e2d4-459c-831d-03ac30620fa6-credential-socket\") pod \"e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq\" (UID: \"69fe7950-e2d4-459c-831d-03ac30620fa6\") " pod="openshift-ingress/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq" Jul 20 03:28:03.032363 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:03.032239 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69fe7950-e2d4-459c-831d-03ac30620fa6-istiod-ca-cert\") pod \"e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq\" (UID: \"69fe7950-e2d4-459c-831d-03ac30620fa6\") " pod="openshift-ingress/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq" Jul 20 03:28:03.133354 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:03.133319 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fe7950-e2d4-459c-831d-03ac30620fa6-workload-socket\") pod \"e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq\" (UID: \"69fe7950-e2d4-459c-831d-03ac30620fa6\") " pod="openshift-ingress/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq" Jul 20 03:28:03.133354 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:03.133363 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69fe7950-e2d4-459c-831d-03ac30620fa6-istio-envoy\") pod \"e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq\" (UID: \"69fe7950-e2d4-459c-831d-03ac30620fa6\") " pod="openshift-ingress/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq" Jul 20 03:28:03.133634 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:03.133377 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69fe7950-e2d4-459c-831d-03ac30620fa6-istio-data\") pod \"e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq\" (UID: \"69fe7950-e2d4-459c-831d-03ac30620fa6\") " pod="openshift-ingress/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq" Jul 20 03:28:03.133634 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:03.133402 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86hjb\" (UniqueName: \"kubernetes.io/projected/69fe7950-e2d4-459c-831d-03ac30620fa6-kube-api-access-86hjb\") pod \"e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq\" (UID: \"69fe7950-e2d4-459c-831d-03ac30620fa6\") " pod="openshift-ingress/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq" Jul 20 03:28:03.133634 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:03.133428 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69fe7950-e2d4-459c-831d-03ac30620fa6-istio-podinfo\") pod \"e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq\" (UID: \"69fe7950-e2d4-459c-831d-03ac30620fa6\") " pod="openshift-ingress/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq" Jul 20 03:28:03.133634 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:03.133479 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69fe7950-e2d4-459c-831d-03ac30620fa6-workload-certs\") pod \"e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq\" (UID: \"69fe7950-e2d4-459c-831d-03ac30620fa6\") " pod="openshift-ingress/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq" Jul 20 03:28:03.133634 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:03.133518 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69fe7950-e2d4-459c-831d-03ac30620fa6-istio-token\") pod \"e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq\" (UID: \"69fe7950-e2d4-459c-831d-03ac30620fa6\") " pod="openshift-ingress/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq" Jul 20 03:28:03.133909 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:03.133689 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fe7950-e2d4-459c-831d-03ac30620fa6-credential-socket\") pod \"e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq\" (UID: \"69fe7950-e2d4-459c-831d-03ac30620fa6\") " pod="openshift-ingress/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq" Jul 20 03:28:03.133909 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:03.133739 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69fe7950-e2d4-459c-831d-03ac30620fa6-istiod-ca-cert\") pod \"e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq\" (UID: \"69fe7950-e2d4-459c-831d-03ac30620fa6\") " pod="openshift-ingress/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq" Jul 20 03:28:03.133909 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:03.133851 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fe7950-e2d4-459c-831d-03ac30620fa6-workload-socket\") pod \"e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq\" (UID: \"69fe7950-e2d4-459c-831d-03ac30620fa6\") " pod="openshift-ingress/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq" Jul 20 03:28:03.134066 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:03.133930 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69fe7950-e2d4-459c-831d-03ac30620fa6-workload-certs\") pod \"e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq\" (UID: \"69fe7950-e2d4-459c-831d-03ac30620fa6\") " pod="openshift-ingress/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq" Jul 20 03:28:03.134066 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:03.133973 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69fe7950-e2d4-459c-831d-03ac30620fa6-istio-data\") pod \"e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq\" (UID: \"69fe7950-e2d4-459c-831d-03ac30620fa6\") " pod="openshift-ingress/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq" Jul 20 03:28:03.134066 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:03.134023 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fe7950-e2d4-459c-831d-03ac30620fa6-credential-socket\") pod \"e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq\" (UID: \"69fe7950-e2d4-459c-831d-03ac30620fa6\") " pod="openshift-ingress/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq" Jul 20 03:28:03.134533 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:03.134506 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69fe7950-e2d4-459c-831d-03ac30620fa6-istiod-ca-cert\") pod \"e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq\" (UID: \"69fe7950-e2d4-459c-831d-03ac30620fa6\") " pod="openshift-ingress/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq" Jul 20 03:28:03.135851 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:03.135829 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69fe7950-e2d4-459c-831d-03ac30620fa6-istio-envoy\") pod \"e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq\" (UID: \"69fe7950-e2d4-459c-831d-03ac30620fa6\") " pod="openshift-ingress/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq" Jul 20 03:28:03.136023 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:03.136004 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69fe7950-e2d4-459c-831d-03ac30620fa6-istio-podinfo\") pod \"e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq\" (UID: \"69fe7950-e2d4-459c-831d-03ac30620fa6\") " pod="openshift-ingress/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq" Jul 20 03:28:03.141644 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:03.141618 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86hjb\" (UniqueName: \"kubernetes.io/projected/69fe7950-e2d4-459c-831d-03ac30620fa6-kube-api-access-86hjb\") pod \"e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq\" (UID: \"69fe7950-e2d4-459c-831d-03ac30620fa6\") " pod="openshift-ingress/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq" Jul 20 03:28:03.141906 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:03.141889 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69fe7950-e2d4-459c-831d-03ac30620fa6-istio-token\") pod \"e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq\" (UID: \"69fe7950-e2d4-459c-831d-03ac30620fa6\") " pod="openshift-ingress/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq" Jul 20 03:28:03.278452 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:03.278364 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq" Jul 20 03:28:03.433143 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:03.433116 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq"] Jul 20 03:28:03.434518 ip-10-0-142-139 kubenswrapper[2558]: W0720 03:28:03.434488 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69fe7950_e2d4_459c_831d_03ac30620fa6.slice/crio-808c1bf9f0f946ab48e1736921d0fee486d5e6f73cd5d8938746eb2119637faa WatchSource:0}: Error finding container 808c1bf9f0f946ab48e1736921d0fee486d5e6f73cd5d8938746eb2119637faa: Status 404 returned error can't find the container with id 808c1bf9f0f946ab48e1736921d0fee486d5e6f73cd5d8938746eb2119637faa Jul 20 03:28:03.439798 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:03.439767 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:28:03.439869 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:03.439831 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:28:03.439869 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:03.439859 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:28:03.548987 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:03.548901 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:28:03.821936 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:03.821845 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq" event={"ID":"69fe7950-e2d4-459c-831d-03ac30620fa6","Type":"ContainerStarted","Data":"29f7cd1ab9dae5682acff152855dc5472783553d4d47101b1d12a3c09f9231d6"} Jul 20 03:28:03.821936 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:03.821908 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq" event={"ID":"69fe7950-e2d4-459c-831d-03ac30620fa6","Type":"ContainerStarted","Data":"808c1bf9f0f946ab48e1736921d0fee486d5e6f73cd5d8938746eb2119637faa"} Jul 20 03:28:03.841496 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:03.841449 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq" podStartSLOduration=1.8414354240000002 podStartE2EDuration="1.841435424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 03:28:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 03:28:03.839482871 +0000 UTC m=+2579.038732043" watchObservedRunningTime="2026-07-20 03:28:03.841435424 +0000 UTC m=+2579.040684570" Jul 20 03:28:04.279565 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:04.279523 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq" Jul 20 03:28:04.284920 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:04.284884 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq" Jul 20 03:28:04.827300 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:04.827271 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq" Jul 20 03:28:04.828509 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:04.828480 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq" Jul 20 03:28:05.608218 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:05.608187 2558 scope.go:117] "RemoveContainer" containerID="ac79f62fbbc1fea800771531272a8591eeb04ae0e0c76a4fe6f0a777d7856833" Jul 20 03:28:11.867080 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:11.866971 2558 generic.go:356] "Generic (PLEG): container finished" podID="b0121cbf-2b96-4274-949d-d9c2bc39f469" containerID="622f1abfba062bf3f48708a6c8255ce36e0cbb43257f9d79894f0a8173632170" exitCode=0 Jul 20 03:28:11.867639 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:11.867149 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b" event={"ID":"b0121cbf-2b96-4274-949d-d9c2bc39f469","Type":"ContainerDied","Data":"622f1abfba062bf3f48708a6c8255ce36e0cbb43257f9d79894f0a8173632170"} Jul 20 03:28:11.868020 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:11.867993 2558 scope.go:117] "RemoveContainer" containerID="622f1abfba062bf3f48708a6c8255ce36e0cbb43257f9d79894f0a8173632170" Jul 20 03:28:11.869645 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:11.869601 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:28:11.869754 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:11.869697 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:28:11.869818 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:11.869774 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:28:12.787153 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:12.787113 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b" Jul 20 03:28:12.877287 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:12.877224 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b" event={"ID":"b0121cbf-2b96-4274-949d-d9c2bc39f469","Type":"ContainerStarted","Data":"9a2a92066228b63868712cede6531cd5b9776399c40ac0f7ee34ae1b0270e669"} Jul 20 03:28:13.808267 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:13.808231 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b" Jul 20 03:28:13.808782 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:13.808756 2558 patch_prober.go:28] interesting pod/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 20 03:28:13.808940 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:13.808921 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b" podUID="b0121cbf-2b96-4274-949d-d9c2bc39f469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 20 03:28:14.807909 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:14.807871 2558 patch_prober.go:28] interesting pod/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 20 03:28:14.808427 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:14.807927 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b" podUID="b0121cbf-2b96-4274-949d-d9c2bc39f469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 20 03:28:15.807463 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:15.807422 2558 patch_prober.go:28] interesting pod/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 20 03:28:15.807663 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:15.807536 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b" podUID="b0121cbf-2b96-4274-949d-d9c2bc39f469" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 20 03:28:16.811861 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:16.811832 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b" Jul 20 03:28:16.812359 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:16.812194 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b" Jul 20 03:28:16.812960 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:16.812940 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b" Jul 20 03:28:16.921488 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:16.921457 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:28:17.908607 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:17.908575 2558 generic.go:356] "Generic (PLEG): container finished" podID="69fe7950-e2d4-459c-831d-03ac30620fa6" containerID="29f7cd1ab9dae5682acff152855dc5472783553d4d47101b1d12a3c09f9231d6" exitCode=0 Jul 20 03:28:17.909024 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:17.908648 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq" event={"ID":"69fe7950-e2d4-459c-831d-03ac30620fa6","Type":"ContainerDied","Data":"29f7cd1ab9dae5682acff152855dc5472783553d4d47101b1d12a3c09f9231d6"} Jul 20 03:28:17.909134 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:17.909118 2558 scope.go:117] "RemoveContainer" containerID="29f7cd1ab9dae5682acff152855dc5472783553d4d47101b1d12a3c09f9231d6" Jul 20 03:28:17.910147 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:17.910118 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:28:17.910257 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:17.910199 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:28:17.910323 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:17.910262 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:28:18.295163 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:18.295060 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52abd4-bk5mt"] Jul 20 03:28:18.295558 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:18.295541 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfae4943-17f5-420d-810c-300c0c14e411" containerName="istio-proxy" Jul 20 03:28:18.298673 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:18.298655 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52abd4-bk5mt" Jul 20 03:28:18.301196 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:18.301175 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hlzh6\"" Jul 20 03:28:18.301299 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:18.301175 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 20 03:28:18.305660 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:18.305637 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52abd4-bk5mt"] Jul 20 03:28:18.415106 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:18.415062 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74fa41bf-5082-4daa-ba5e-76fcad4d10c6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-52abd4-bk5mt\" (UID: \"74fa41bf-5082-4daa-ba5e-76fcad4d10c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52abd4-bk5mt" Jul 20 03:28:18.516419 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:18.516382 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74fa41bf-5082-4daa-ba5e-76fcad4d10c6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-52abd4-bk5mt\" (UID: \"74fa41bf-5082-4daa-ba5e-76fcad4d10c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52abd4-bk5mt" Jul 20 03:28:18.517029 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:18.517007 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74fa41bf-5082-4daa-ba5e-76fcad4d10c6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-52abd4-bk5mt\" (UID: \"74fa41bf-5082-4daa-ba5e-76fcad4d10c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52abd4-bk5mt" Jul 20 03:28:18.610455 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:18.610362 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52abd4-bk5mt" Jul 20 03:28:18.778120 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:18.775517 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52abd4-bk5mt"] Jul 20 03:28:18.780843 ip-10-0-142-139 kubenswrapper[2558]: W0720 03:28:18.780814 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74fa41bf_5082_4daa_ba5e_76fcad4d10c6.slice/crio-b0aa0021f4c9a59362364930e1ab043f20a03ea46bf28d93e2791bd095f71817 WatchSource:0}: Error finding container b0aa0021f4c9a59362364930e1ab043f20a03ea46bf28d93e2791bd095f71817: Status 404 returned error can't find the container with id b0aa0021f4c9a59362364930e1ab043f20a03ea46bf28d93e2791bd095f71817 Jul 20 03:28:18.915781 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:18.915740 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq" event={"ID":"69fe7950-e2d4-459c-831d-03ac30620fa6","Type":"ContainerStarted","Data":"3f5e9df0bc4a4c19deb6837505431f08ed0d11bbd596ff84d2bcc2a791e763ca"} Jul 20 03:28:18.916910 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:18.916886 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52abd4-bk5mt" event={"ID":"74fa41bf-5082-4daa-ba5e-76fcad4d10c6","Type":"ContainerStarted","Data":"b0aa0021f4c9a59362364930e1ab043f20a03ea46bf28d93e2791bd095f71817"} Jul 20 03:28:19.278997 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:19.278891 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq" Jul 20 03:28:19.280053 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:19.280025 2558 patch_prober.go:28] interesting pod/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 20 03:28:19.280193 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:19.280075 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq" podUID="69fe7950-e2d4-459c-831d-03ac30620fa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 20 03:28:19.827392 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:19.827362 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq" Jul 20 03:28:19.923541 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:19.923505 2558 generic.go:356] "Generic (PLEG): container finished" podID="74fa41bf-5082-4daa-ba5e-76fcad4d10c6" containerID="04c7414fa458bac2eebde9f3c715bed66f75bcafdcaae82d314ffc4cf7ba4439" exitCode=0 Jul 20 03:28:19.923992 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:19.923567 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52abd4-bk5mt" event={"ID":"74fa41bf-5082-4daa-ba5e-76fcad4d10c6","Type":"ContainerDied","Data":"04c7414fa458bac2eebde9f3c715bed66f75bcafdcaae82d314ffc4cf7ba4439"} Jul 20 03:28:20.278925 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:20.278885 2558 patch_prober.go:28] interesting pod/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 20 03:28:20.279080 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:20.278948 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq" podUID="69fe7950-e2d4-459c-831d-03ac30620fa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 20 03:28:21.092965 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:21.092938 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52abd4-bk5mt" Jul 20 03:28:21.239364 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:21.239271 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74fa41bf-5082-4daa-ba5e-76fcad4d10c6-maas-api-service-ca\") pod \"74fa41bf-5082-4daa-ba5e-76fcad4d10c6\" (UID: \"74fa41bf-5082-4daa-ba5e-76fcad4d10c6\") " Jul 20 03:28:21.239612 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:21.239581 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74fa41bf-5082-4daa-ba5e-76fcad4d10c6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "74fa41bf-5082-4daa-ba5e-76fcad4d10c6" (UID: "74fa41bf-5082-4daa-ba5e-76fcad4d10c6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 03:28:21.279448 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:21.279407 2558 patch_prober.go:28] interesting pod/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 20 03:28:21.279603 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:21.279465 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq" podUID="69fe7950-e2d4-459c-831d-03ac30620fa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 20 03:28:21.340876 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:21.340842 2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74fa41bf-5082-4daa-ba5e-76fcad4d10c6-maas-api-service-ca\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:28:21.933970 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:21.933929 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52abd4-bk5mt" event={"ID":"74fa41bf-5082-4daa-ba5e-76fcad4d10c6","Type":"ContainerDied","Data":"b0aa0021f4c9a59362364930e1ab043f20a03ea46bf28d93e2791bd095f71817"} Jul 20 03:28:21.933970 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:21.933952 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52abd4-bk5mt" Jul 20 03:28:21.933970 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:21.933978 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0aa0021f4c9a59362364930e1ab043f20a03ea46bf28d93e2791bd095f71817" Jul 20 03:28:22.279026 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:22.278930 2558 patch_prober.go:28] interesting pod/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 20 03:28:22.279026 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:22.278997 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq" podUID="69fe7950-e2d4-459c-831d-03ac30620fa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 20 03:28:23.283069 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:23.283037 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq" Jul 20 03:28:23.284158 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:23.284134 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq" Jul 20 03:28:36.626822 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:36.626737 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:28:38.381067 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:38.381029 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52abd4-bk5mt"] Jul 20 03:28:38.382707 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:38.382685 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52abd4-bk5mt"] Jul 20 03:28:39.424611 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:39.424582 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74fa41bf-5082-4daa-ba5e-76fcad4d10c6" path="/var/lib/kubelet/pods/74fa41bf-5082-4daa-ba5e-76fcad4d10c6/volumes" Jul 20 03:28:44.599635 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:44.599581 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b"] Jul 20 03:28:44.600226 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:44.599861 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b" podUID="b0121cbf-2b96-4274-949d-d9c2bc39f469" containerName="istio-proxy" containerID="cri-o://9a2a92066228b63868712cede6531cd5b9776399c40ac0f7ee34ae1b0270e669" gracePeriod=30 Jul 20 03:28:46.815083 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:46.815043 2558 patch_prober.go:28] interesting pod/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 20 03:28:46.815556 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:46.815139 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b" podUID="b0121cbf-2b96-4274-949d-d9c2bc39f469" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 20 03:28:49.850777 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:49.850750 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b" Jul 20 03:28:50.013578 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:50.013544 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0121cbf-2b96-4274-949d-d9c2bc39f469-istio-token\") pod \"b0121cbf-2b96-4274-949d-d9c2bc39f469\" (UID: \"b0121cbf-2b96-4274-949d-d9c2bc39f469\") " Jul 20 03:28:50.013801 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:50.013596 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7w9dd\" (UniqueName: \"kubernetes.io/projected/b0121cbf-2b96-4274-949d-d9c2bc39f469-kube-api-access-7w9dd\") pod \"b0121cbf-2b96-4274-949d-d9c2bc39f469\" (UID: \"b0121cbf-2b96-4274-949d-d9c2bc39f469\") " Jul 20 03:28:50.013801 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:50.013631 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0121cbf-2b96-4274-949d-d9c2bc39f469-istio-envoy\") pod \"b0121cbf-2b96-4274-949d-d9c2bc39f469\" (UID: \"b0121cbf-2b96-4274-949d-d9c2bc39f469\") " Jul 20 03:28:50.013801 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:50.013663 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0121cbf-2b96-4274-949d-d9c2bc39f469-istio-data\") pod \"b0121cbf-2b96-4274-949d-d9c2bc39f469\" (UID: \"b0121cbf-2b96-4274-949d-d9c2bc39f469\") " Jul 20 03:28:50.013801 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:50.013687 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0121cbf-2b96-4274-949d-d9c2bc39f469-workload-socket\") pod \"b0121cbf-2b96-4274-949d-d9c2bc39f469\" (UID: \"b0121cbf-2b96-4274-949d-d9c2bc39f469\") " Jul 20 03:28:50.013801 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:50.013752 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0121cbf-2b96-4274-949d-d9c2bc39f469-credential-socket\") pod \"b0121cbf-2b96-4274-949d-d9c2bc39f469\" (UID: \"b0121cbf-2b96-4274-949d-d9c2bc39f469\") " Jul 20 03:28:50.013801 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:50.013770 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0121cbf-2b96-4274-949d-d9c2bc39f469-istiod-ca-cert\") pod \"b0121cbf-2b96-4274-949d-d9c2bc39f469\" (UID: \"b0121cbf-2b96-4274-949d-d9c2bc39f469\") " Jul 20 03:28:50.013801 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:50.013792 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0121cbf-2b96-4274-949d-d9c2bc39f469-istio-podinfo\") pod \"b0121cbf-2b96-4274-949d-d9c2bc39f469\" (UID: \"b0121cbf-2b96-4274-949d-d9c2bc39f469\") " Jul 20 03:28:50.014290 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:50.013813 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0121cbf-2b96-4274-949d-d9c2bc39f469-workload-certs\") pod \"b0121cbf-2b96-4274-949d-d9c2bc39f469\" (UID: \"b0121cbf-2b96-4274-949d-d9c2bc39f469\") " Jul 20 03:28:50.014290 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:50.014072 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0121cbf-2b96-4274-949d-d9c2bc39f469-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b0121cbf-2b96-4274-949d-d9c2bc39f469" (UID: "b0121cbf-2b96-4274-949d-d9c2bc39f469"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:28:50.014290 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:50.014087 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0121cbf-2b96-4274-949d-d9c2bc39f469-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b0121cbf-2b96-4274-949d-d9c2bc39f469" (UID: "b0121cbf-2b96-4274-949d-d9c2bc39f469"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:28:50.014460 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:50.014298 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0121cbf-2b96-4274-949d-d9c2bc39f469-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b0121cbf-2b96-4274-949d-d9c2bc39f469" (UID: "b0121cbf-2b96-4274-949d-d9c2bc39f469"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:28:50.014460 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:50.014316 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0121cbf-2b96-4274-949d-d9c2bc39f469-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b0121cbf-2b96-4274-949d-d9c2bc39f469" (UID: "b0121cbf-2b96-4274-949d-d9c2bc39f469"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 03:28:50.014810 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:50.014780 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0121cbf-2b96-4274-949d-d9c2bc39f469-istio-data" (OuterVolumeSpecName: "istio-data") pod "b0121cbf-2b96-4274-949d-d9c2bc39f469" (UID: "b0121cbf-2b96-4274-949d-d9c2bc39f469"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:28:50.016012 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:50.015988 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0121cbf-2b96-4274-949d-d9c2bc39f469-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b0121cbf-2b96-4274-949d-d9c2bc39f469" (UID: "b0121cbf-2b96-4274-949d-d9c2bc39f469"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:28:50.016133 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:50.016109 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0121cbf-2b96-4274-949d-d9c2bc39f469-istio-token" (OuterVolumeSpecName: "istio-token") pod "b0121cbf-2b96-4274-949d-d9c2bc39f469" (UID: "b0121cbf-2b96-4274-949d-d9c2bc39f469"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 03:28:50.016324 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:50.016305 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0121cbf-2b96-4274-949d-d9c2bc39f469-kube-api-access-7w9dd" (OuterVolumeSpecName: "kube-api-access-7w9dd") pod "b0121cbf-2b96-4274-949d-d9c2bc39f469" (UID: "b0121cbf-2b96-4274-949d-d9c2bc39f469"). InnerVolumeSpecName "kube-api-access-7w9dd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 03:28:50.016394 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:50.016338 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0121cbf-2b96-4274-949d-d9c2bc39f469-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b0121cbf-2b96-4274-949d-d9c2bc39f469" (UID: "b0121cbf-2b96-4274-949d-d9c2bc39f469"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 20 03:28:50.070958 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:50.070925 2558 generic.go:356] "Generic (PLEG): container finished" podID="b0121cbf-2b96-4274-949d-d9c2bc39f469" containerID="9a2a92066228b63868712cede6531cd5b9776399c40ac0f7ee34ae1b0270e669" exitCode=0 Jul 20 03:28:50.071142 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:50.070975 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b" Jul 20 03:28:50.071142 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:50.070982 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b" event={"ID":"b0121cbf-2b96-4274-949d-d9c2bc39f469","Type":"ContainerDied","Data":"9a2a92066228b63868712cede6531cd5b9776399c40ac0f7ee34ae1b0270e669"} Jul 20 03:28:50.071142 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:50.071037 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b" event={"ID":"b0121cbf-2b96-4274-949d-d9c2bc39f469","Type":"ContainerDied","Data":"a76adec1d164d2258a0b276ac85fa3e8fcfdbac69c1fd3f762799b01b63888c3"} Jul 20 03:28:50.071142 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:50.071066 2558 scope.go:117] "RemoveContainer" containerID="9a2a92066228b63868712cede6531cd5b9776399c40ac0f7ee34ae1b0270e669" Jul 20 03:28:50.080238 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:50.080221 2558 scope.go:117] "RemoveContainer" containerID="622f1abfba062bf3f48708a6c8255ce36e0cbb43257f9d79894f0a8173632170" Jul 20 03:28:50.089212 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:50.089193 2558 scope.go:117] "RemoveContainer" containerID="9a2a92066228b63868712cede6531cd5b9776399c40ac0f7ee34ae1b0270e669" Jul 20 03:28:50.089494 ip-10-0-142-139 kubenswrapper[2558]: E0720 03:28:50.089476 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a2a92066228b63868712cede6531cd5b9776399c40ac0f7ee34ae1b0270e669\": container with ID starting with 9a2a92066228b63868712cede6531cd5b9776399c40ac0f7ee34ae1b0270e669 not found: ID does not exist" containerID="9a2a92066228b63868712cede6531cd5b9776399c40ac0f7ee34ae1b0270e669" Jul 20 03:28:50.089534 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:50.089504 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a2a92066228b63868712cede6531cd5b9776399c40ac0f7ee34ae1b0270e669"} err="failed to get container status \"9a2a92066228b63868712cede6531cd5b9776399c40ac0f7ee34ae1b0270e669\": rpc error: code = NotFound desc = could not find container \"9a2a92066228b63868712cede6531cd5b9776399c40ac0f7ee34ae1b0270e669\": container with ID starting with 9a2a92066228b63868712cede6531cd5b9776399c40ac0f7ee34ae1b0270e669 not found: ID does not exist" Jul 20 03:28:50.089534 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:50.089521 2558 scope.go:117] "RemoveContainer" containerID="622f1abfba062bf3f48708a6c8255ce36e0cbb43257f9d79894f0a8173632170" Jul 20 03:28:50.089918 ip-10-0-142-139 kubenswrapper[2558]: E0720 03:28:50.089885 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"622f1abfba062bf3f48708a6c8255ce36e0cbb43257f9d79894f0a8173632170\": container with ID starting with 622f1abfba062bf3f48708a6c8255ce36e0cbb43257f9d79894f0a8173632170 not found: ID does not exist" containerID="622f1abfba062bf3f48708a6c8255ce36e0cbb43257f9d79894f0a8173632170" Jul 20 03:28:50.090004 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:50.089927 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"622f1abfba062bf3f48708a6c8255ce36e0cbb43257f9d79894f0a8173632170"} err="failed to get container status \"622f1abfba062bf3f48708a6c8255ce36e0cbb43257f9d79894f0a8173632170\": rpc error: code = NotFound desc = could not find container \"622f1abfba062bf3f48708a6c8255ce36e0cbb43257f9d79894f0a8173632170\": container with ID starting with 622f1abfba062bf3f48708a6c8255ce36e0cbb43257f9d79894f0a8173632170 not found: ID does not exist" Jul 20 03:28:50.092369 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:50.092349 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b"] Jul 20 03:28:50.098079 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:50.098057 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-52abd4-openshift-default-9b4b6dcf-cgf2b"] Jul 20 03:28:50.114961 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:50.114940 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w9dd\" (UniqueName: \"kubernetes.io/projected/b0121cbf-2b96-4274-949d-d9c2bc39f469-kube-api-access-7w9dd\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:28:50.114961 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:50.114961 2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0121cbf-2b96-4274-949d-d9c2bc39f469-istio-envoy\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:28:50.115088 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:50.114973 2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0121cbf-2b96-4274-949d-d9c2bc39f469-istio-data\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:28:50.115088 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:50.114981 2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0121cbf-2b96-4274-949d-d9c2bc39f469-workload-socket\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:28:50.115088 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:50.114989 2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0121cbf-2b96-4274-949d-d9c2bc39f469-credential-socket\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:28:50.115088 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:50.114997 2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0121cbf-2b96-4274-949d-d9c2bc39f469-istiod-ca-cert\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:28:50.115088 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:50.115006 2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0121cbf-2b96-4274-949d-d9c2bc39f469-istio-podinfo\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:28:50.115088 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:50.115013 2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0121cbf-2b96-4274-949d-d9c2bc39f469-workload-certs\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:28:50.115088 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:50.115021 2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0121cbf-2b96-4274-949d-d9c2bc39f469-istio-token\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:28:51.424201 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:28:51.424168 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0121cbf-2b96-4274-949d-d9c2bc39f469" path="/var/lib/kubelet/pods/b0121cbf-2b96-4274-949d-d9c2bc39f469/volumes" Jul 20 03:29:05.619895 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:05.619859 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:29:11.598259 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:11.598219 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq"] Jul 20 03:29:11.598697 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:11.598466 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq" podUID="69fe7950-e2d4-459c-831d-03ac30620fa6" containerName="istio-proxy" containerID="cri-o://3f5e9df0bc4a4c19deb6837505431f08ed0d11bbd596ff84d2bcc2a791e763ca" gracePeriod=30 Jul 20 03:29:12.458730 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:12.458694 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d"] Jul 20 03:29:12.459274 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:12.459257 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0121cbf-2b96-4274-949d-d9c2bc39f469" containerName="istio-proxy" Jul 20 03:29:12.459340 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:12.459276 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0121cbf-2b96-4274-949d-d9c2bc39f469" containerName="istio-proxy" Jul 20 03:29:12.459340 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:12.459287 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74fa41bf-5082-4daa-ba5e-76fcad4d10c6" containerName="revoke-keys" Jul 20 03:29:12.459340 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:12.459298 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="74fa41bf-5082-4daa-ba5e-76fcad4d10c6" containerName="revoke-keys" Jul 20 03:29:12.459340 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:12.459310 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0121cbf-2b96-4274-949d-d9c2bc39f469" containerName="istio-proxy" Jul 20 03:29:12.459340 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:12.459318 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0121cbf-2b96-4274-949d-d9c2bc39f469" containerName="istio-proxy" Jul 20 03:29:12.459505 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:12.459397 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0121cbf-2b96-4274-949d-d9c2bc39f469" containerName="istio-proxy" Jul 20 03:29:12.459505 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:12.459409 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0121cbf-2b96-4274-949d-d9c2bc39f469" containerName="istio-proxy" Jul 20 03:29:12.459505 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:12.459422 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="74fa41bf-5082-4daa-ba5e-76fcad4d10c6" containerName="revoke-keys" Jul 20 03:29:12.462730 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:12.462711 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d" Jul 20 03:29:12.465191 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:12.465171 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-496193-openshift-default-dockercfg-kfnt9\"" Jul 20 03:29:12.472509 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:12.472485 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d"] Jul 20 03:29:12.519490 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:12.519448 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c464eb9-afce-451d-9305-1aa626aac3e0-istio-token\") pod \"e2e-shared-a-496193-openshift-default-85fd648879-wrr4d\" (UID: \"2c464eb9-afce-451d-9305-1aa626aac3e0\") " pod="openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d" Jul 20 03:29:12.519683 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:12.519504 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c464eb9-afce-451d-9305-1aa626aac3e0-istio-envoy\") pod \"e2e-shared-a-496193-openshift-default-85fd648879-wrr4d\" (UID: \"2c464eb9-afce-451d-9305-1aa626aac3e0\") " pod="openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d" Jul 20 03:29:12.519683 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:12.519535 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c464eb9-afce-451d-9305-1aa626aac3e0-istiod-ca-cert\") pod \"e2e-shared-a-496193-openshift-default-85fd648879-wrr4d\" (UID: \"2c464eb9-afce-451d-9305-1aa626aac3e0\") " pod="openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d" Jul 20 03:29:12.519683 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:12.519616 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c464eb9-afce-451d-9305-1aa626aac3e0-workload-certs\") pod \"e2e-shared-a-496193-openshift-default-85fd648879-wrr4d\" (UID: \"2c464eb9-afce-451d-9305-1aa626aac3e0\") " pod="openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d" Jul 20 03:29:12.519683 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:12.519648 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c464eb9-afce-451d-9305-1aa626aac3e0-istio-data\") pod \"e2e-shared-a-496193-openshift-default-85fd648879-wrr4d\" (UID: \"2c464eb9-afce-451d-9305-1aa626aac3e0\") " pod="openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d" Jul 20 03:29:12.519683 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:12.519678 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7snkd\" (UniqueName: \"kubernetes.io/projected/2c464eb9-afce-451d-9305-1aa626aac3e0-kube-api-access-7snkd\") pod \"e2e-shared-a-496193-openshift-default-85fd648879-wrr4d\" (UID: \"2c464eb9-afce-451d-9305-1aa626aac3e0\") " pod="openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d" Jul 20 03:29:12.519897 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:12.519761 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c464eb9-afce-451d-9305-1aa626aac3e0-workload-socket\") pod \"e2e-shared-a-496193-openshift-default-85fd648879-wrr4d\" (UID: \"2c464eb9-afce-451d-9305-1aa626aac3e0\") " pod="openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d" Jul 20 03:29:12.519897 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:12.519780 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c464eb9-afce-451d-9305-1aa626aac3e0-credential-socket\") pod \"e2e-shared-a-496193-openshift-default-85fd648879-wrr4d\" (UID: \"2c464eb9-afce-451d-9305-1aa626aac3e0\") " pod="openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d" Jul 20 03:29:12.519897 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:12.519796 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c464eb9-afce-451d-9305-1aa626aac3e0-istio-podinfo\") pod \"e2e-shared-a-496193-openshift-default-85fd648879-wrr4d\" (UID: \"2c464eb9-afce-451d-9305-1aa626aac3e0\") " pod="openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d" Jul 20 03:29:12.620959 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:12.620920 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c464eb9-afce-451d-9305-1aa626aac3e0-istio-token\") pod \"e2e-shared-a-496193-openshift-default-85fd648879-wrr4d\" (UID: \"2c464eb9-afce-451d-9305-1aa626aac3e0\") " pod="openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d" Jul 20 03:29:12.620959 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:12.620964 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c464eb9-afce-451d-9305-1aa626aac3e0-istio-envoy\") pod \"e2e-shared-a-496193-openshift-default-85fd648879-wrr4d\" (UID: \"2c464eb9-afce-451d-9305-1aa626aac3e0\") " pod="openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d" Jul 20 03:29:12.621577 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:12.620983 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c464eb9-afce-451d-9305-1aa626aac3e0-istiod-ca-cert\") pod \"e2e-shared-a-496193-openshift-default-85fd648879-wrr4d\" (UID: \"2c464eb9-afce-451d-9305-1aa626aac3e0\") " pod="openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d" Jul 20 03:29:12.621577 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:12.621018 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c464eb9-afce-451d-9305-1aa626aac3e0-workload-certs\") pod \"e2e-shared-a-496193-openshift-default-85fd648879-wrr4d\" (UID: \"2c464eb9-afce-451d-9305-1aa626aac3e0\") " pod="openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d" Jul 20 03:29:12.621577 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:12.621045 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c464eb9-afce-451d-9305-1aa626aac3e0-istio-data\") pod \"e2e-shared-a-496193-openshift-default-85fd648879-wrr4d\" (UID: \"2c464eb9-afce-451d-9305-1aa626aac3e0\") " pod="openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d" Jul 20 03:29:12.621577 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:12.621080 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7snkd\" (UniqueName: \"kubernetes.io/projected/2c464eb9-afce-451d-9305-1aa626aac3e0-kube-api-access-7snkd\") pod \"e2e-shared-a-496193-openshift-default-85fd648879-wrr4d\" (UID: \"2c464eb9-afce-451d-9305-1aa626aac3e0\") " pod="openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d" Jul 20 03:29:12.621577 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:12.621297 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c464eb9-afce-451d-9305-1aa626aac3e0-workload-socket\") pod \"e2e-shared-a-496193-openshift-default-85fd648879-wrr4d\" (UID: \"2c464eb9-afce-451d-9305-1aa626aac3e0\") " pod="openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d" Jul 20 03:29:12.621577 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:12.621327 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c464eb9-afce-451d-9305-1aa626aac3e0-credential-socket\") pod \"e2e-shared-a-496193-openshift-default-85fd648879-wrr4d\" (UID: \"2c464eb9-afce-451d-9305-1aa626aac3e0\") " pod="openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d" Jul 20 03:29:12.621577 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:12.621350 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c464eb9-afce-451d-9305-1aa626aac3e0-istio-podinfo\") pod \"e2e-shared-a-496193-openshift-default-85fd648879-wrr4d\" (UID: \"2c464eb9-afce-451d-9305-1aa626aac3e0\") " pod="openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d" Jul 20 03:29:12.621577 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:12.621502 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c464eb9-afce-451d-9305-1aa626aac3e0-workload-certs\") pod \"e2e-shared-a-496193-openshift-default-85fd648879-wrr4d\" (UID: \"2c464eb9-afce-451d-9305-1aa626aac3e0\") " pod="openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d" Jul 20 03:29:12.621577 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:12.621552 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c464eb9-afce-451d-9305-1aa626aac3e0-istio-data\") pod \"e2e-shared-a-496193-openshift-default-85fd648879-wrr4d\" (UID: \"2c464eb9-afce-451d-9305-1aa626aac3e0\") " pod="openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d" Jul 20 03:29:12.621950 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:12.621682 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c464eb9-afce-451d-9305-1aa626aac3e0-workload-socket\") pod \"e2e-shared-a-496193-openshift-default-85fd648879-wrr4d\" (UID: \"2c464eb9-afce-451d-9305-1aa626aac3e0\") " pod="openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d" Jul 20 03:29:12.621950 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:12.621727 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c464eb9-afce-451d-9305-1aa626aac3e0-istiod-ca-cert\") pod \"e2e-shared-a-496193-openshift-default-85fd648879-wrr4d\" (UID: \"2c464eb9-afce-451d-9305-1aa626aac3e0\") " pod="openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d" Jul 20 03:29:12.621950 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:12.621740 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c464eb9-afce-451d-9305-1aa626aac3e0-credential-socket\") pod \"e2e-shared-a-496193-openshift-default-85fd648879-wrr4d\" (UID: \"2c464eb9-afce-451d-9305-1aa626aac3e0\") " pod="openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d" Jul 20 03:29:12.623719 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:12.623696 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c464eb9-afce-451d-9305-1aa626aac3e0-istio-podinfo\") pod \"e2e-shared-a-496193-openshift-default-85fd648879-wrr4d\" (UID: \"2c464eb9-afce-451d-9305-1aa626aac3e0\") " pod="openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d" Jul 20 03:29:12.623819 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:12.623768 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c464eb9-afce-451d-9305-1aa626aac3e0-istio-envoy\") pod \"e2e-shared-a-496193-openshift-default-85fd648879-wrr4d\" (UID: \"2c464eb9-afce-451d-9305-1aa626aac3e0\") " pod="openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d" Jul 20 03:29:12.629605 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:12.629579 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7snkd\" (UniqueName: \"kubernetes.io/projected/2c464eb9-afce-451d-9305-1aa626aac3e0-kube-api-access-7snkd\") pod \"e2e-shared-a-496193-openshift-default-85fd648879-wrr4d\" (UID: \"2c464eb9-afce-451d-9305-1aa626aac3e0\") " pod="openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d" Jul 20 03:29:12.629726 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:12.629665 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c464eb9-afce-451d-9305-1aa626aac3e0-istio-token\") pod \"e2e-shared-a-496193-openshift-default-85fd648879-wrr4d\" (UID: \"2c464eb9-afce-451d-9305-1aa626aac3e0\") " pod="openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d" Jul 20 03:29:12.774535 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:12.774444 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d" Jul 20 03:29:12.913343 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:12.913312 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d"] Jul 20 03:29:12.914724 ip-10-0-142-139 kubenswrapper[2558]: W0720 03:29:12.914699 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c464eb9_afce_451d_9305_1aa626aac3e0.slice/crio-653e764eaf8d6c299cab667456352cc40b535dd9cdeb175f377d587c9b2b2794 WatchSource:0}: Error finding container 653e764eaf8d6c299cab667456352cc40b535dd9cdeb175f377d587c9b2b2794: Status 404 returned error can't find the container with id 653e764eaf8d6c299cab667456352cc40b535dd9cdeb175f377d587c9b2b2794 Jul 20 03:29:12.916931 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:12.916894 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:29:12.917000 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:12.916983 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:29:12.917049 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:12.917030 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:29:13.187212 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:13.187171 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d" event={"ID":"2c464eb9-afce-451d-9305-1aa626aac3e0","Type":"ContainerStarted","Data":"ff56d3eed1e90b33166c10dec66812c13960bb5065b70e4f00db29e0b8107cfd"} Jul 20 03:29:13.187394 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:13.187238 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d" event={"ID":"2c464eb9-afce-451d-9305-1aa626aac3e0","Type":"ContainerStarted","Data":"653e764eaf8d6c299cab667456352cc40b535dd9cdeb175f377d587c9b2b2794"} Jul 20 03:29:13.206666 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:13.206612 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d" podStartSLOduration=1.206596663 podStartE2EDuration="1.206596663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 03:29:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 03:29:13.204513959 +0000 UTC m=+2648.403763140" watchObservedRunningTime="2026-07-20 03:29:13.206596663 +0000 UTC m=+2648.405845809" Jul 20 03:29:13.774817 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:13.774777 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d" Jul 20 03:29:13.779512 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:13.779489 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d" Jul 20 03:29:14.192150 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:14.192117 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d" Jul 20 03:29:14.193200 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:14.193182 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d" Jul 20 03:29:16.851682 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:16.851660 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq" Jul 20 03:29:16.963164 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:16.963128 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fe7950-e2d4-459c-831d-03ac30620fa6-workload-socket\") pod \"69fe7950-e2d4-459c-831d-03ac30620fa6\" (UID: \"69fe7950-e2d4-459c-831d-03ac30620fa6\") " Jul 20 03:29:16.963164 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:16.963162 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69fe7950-e2d4-459c-831d-03ac30620fa6-istio-data\") pod \"69fe7950-e2d4-459c-831d-03ac30620fa6\" (UID: \"69fe7950-e2d4-459c-831d-03ac30620fa6\") " Jul 20 03:29:16.963400 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:16.963184 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69fe7950-e2d4-459c-831d-03ac30620fa6-workload-certs\") pod \"69fe7950-e2d4-459c-831d-03ac30620fa6\" (UID: \"69fe7950-e2d4-459c-831d-03ac30620fa6\") " Jul 20 03:29:16.963400 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:16.963202 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69fe7950-e2d4-459c-831d-03ac30620fa6-istio-token\") pod \"69fe7950-e2d4-459c-831d-03ac30620fa6\" (UID: \"69fe7950-e2d4-459c-831d-03ac30620fa6\") " Jul 20 03:29:16.963400 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:16.963242 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69fe7950-e2d4-459c-831d-03ac30620fa6-istiod-ca-cert\") pod \"69fe7950-e2d4-459c-831d-03ac30620fa6\" (UID: \"69fe7950-e2d4-459c-831d-03ac30620fa6\") " Jul 20 03:29:16.963400 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:16.963271 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69fe7950-e2d4-459c-831d-03ac30620fa6-istio-envoy\") pod \"69fe7950-e2d4-459c-831d-03ac30620fa6\" (UID: \"69fe7950-e2d4-459c-831d-03ac30620fa6\") " Jul 20 03:29:16.963400 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:16.963307 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69fe7950-e2d4-459c-831d-03ac30620fa6-istio-podinfo\") pod \"69fe7950-e2d4-459c-831d-03ac30620fa6\" (UID: \"69fe7950-e2d4-459c-831d-03ac30620fa6\") " Jul 20 03:29:16.963682 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:16.963436 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fe7950-e2d4-459c-831d-03ac30620fa6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "69fe7950-e2d4-459c-831d-03ac30620fa6" (UID: "69fe7950-e2d4-459c-831d-03ac30620fa6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:29:16.963682 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:16.963449 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fe7950-e2d4-459c-831d-03ac30620fa6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "69fe7950-e2d4-459c-831d-03ac30620fa6" (UID: "69fe7950-e2d4-459c-831d-03ac30620fa6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:29:16.963682 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:16.963536 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86hjb\" (UniqueName: \"kubernetes.io/projected/69fe7950-e2d4-459c-831d-03ac30620fa6-kube-api-access-86hjb\") pod \"69fe7950-e2d4-459c-831d-03ac30620fa6\" (UID: \"69fe7950-e2d4-459c-831d-03ac30620fa6\") " Jul 20 03:29:16.963682 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:16.963637 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fe7950-e2d4-459c-831d-03ac30620fa6-credential-socket\") pod \"69fe7950-e2d4-459c-831d-03ac30620fa6\" (UID: \"69fe7950-e2d4-459c-831d-03ac30620fa6\") " Jul 20 03:29:16.963968 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:16.963732 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69fe7950-e2d4-459c-831d-03ac30620fa6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "69fe7950-e2d4-459c-831d-03ac30620fa6" (UID: "69fe7950-e2d4-459c-831d-03ac30620fa6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 03:29:16.963968 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:16.963840 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fe7950-e2d4-459c-831d-03ac30620fa6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "69fe7950-e2d4-459c-831d-03ac30620fa6" (UID: "69fe7950-e2d4-459c-831d-03ac30620fa6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:29:16.963968 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:16.963942 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fe7950-e2d4-459c-831d-03ac30620fa6-istio-data" (OuterVolumeSpecName: "istio-data") pod "69fe7950-e2d4-459c-831d-03ac30620fa6" (UID: "69fe7950-e2d4-459c-831d-03ac30620fa6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:29:16.964163 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:16.964034 2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fe7950-e2d4-459c-831d-03ac30620fa6-credential-socket\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:29:16.964163 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:16.964052 2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fe7950-e2d4-459c-831d-03ac30620fa6-workload-socket\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:29:16.964163 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:16.964062 2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69fe7950-e2d4-459c-831d-03ac30620fa6-istio-data\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:29:16.964163 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:16.964071 2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69fe7950-e2d4-459c-831d-03ac30620fa6-workload-certs\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:29:16.964163 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:16.964084 2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69fe7950-e2d4-459c-831d-03ac30620fa6-istiod-ca-cert\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:29:16.965772 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:16.965744 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69fe7950-e2d4-459c-831d-03ac30620fa6-istio-token" (OuterVolumeSpecName: "istio-token") pod "69fe7950-e2d4-459c-831d-03ac30620fa6" (UID: "69fe7950-e2d4-459c-831d-03ac30620fa6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 03:29:16.965772 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:16.965744 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69fe7950-e2d4-459c-831d-03ac30620fa6-kube-api-access-86hjb" (OuterVolumeSpecName: "kube-api-access-86hjb") pod "69fe7950-e2d4-459c-831d-03ac30620fa6" (UID: "69fe7950-e2d4-459c-831d-03ac30620fa6"). InnerVolumeSpecName "kube-api-access-86hjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 03:29:16.966076 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:16.966052 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69fe7950-e2d4-459c-831d-03ac30620fa6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "69fe7950-e2d4-459c-831d-03ac30620fa6" (UID: "69fe7950-e2d4-459c-831d-03ac30620fa6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 20 03:29:16.966224 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:16.966052 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fe7950-e2d4-459c-831d-03ac30620fa6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "69fe7950-e2d4-459c-831d-03ac30620fa6" (UID: "69fe7950-e2d4-459c-831d-03ac30620fa6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:29:17.064737 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:17.064695 2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69fe7950-e2d4-459c-831d-03ac30620fa6-istio-token\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:29:17.064737 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:17.064733 2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69fe7950-e2d4-459c-831d-03ac30620fa6-istio-envoy\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:29:17.064737 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:17.064741 2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69fe7950-e2d4-459c-831d-03ac30620fa6-istio-podinfo\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:29:17.064737 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:17.064751 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86hjb\" (UniqueName: \"kubernetes.io/projected/69fe7950-e2d4-459c-831d-03ac30620fa6-kube-api-access-86hjb\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:29:17.210232 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:17.210194 2558 generic.go:356] "Generic (PLEG): container finished" podID="69fe7950-e2d4-459c-831d-03ac30620fa6" containerID="3f5e9df0bc4a4c19deb6837505431f08ed0d11bbd596ff84d2bcc2a791e763ca" exitCode=0 Jul 20 03:29:17.210484 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:17.210286 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq" event={"ID":"69fe7950-e2d4-459c-831d-03ac30620fa6","Type":"ContainerDied","Data":"3f5e9df0bc4a4c19deb6837505431f08ed0d11bbd596ff84d2bcc2a791e763ca"} Jul 20 03:29:17.210484 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:17.210356 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq" event={"ID":"69fe7950-e2d4-459c-831d-03ac30620fa6","Type":"ContainerDied","Data":"808c1bf9f0f946ab48e1736921d0fee486d5e6f73cd5d8938746eb2119637faa"} Jul 20 03:29:17.210484 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:17.210410 2558 scope.go:117] "RemoveContainer" containerID="3f5e9df0bc4a4c19deb6837505431f08ed0d11bbd596ff84d2bcc2a791e763ca" Jul 20 03:29:17.210948 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:17.210580 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq" Jul 20 03:29:17.221009 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:17.220991 2558 scope.go:117] "RemoveContainer" containerID="29f7cd1ab9dae5682acff152855dc5472783553d4d47101b1d12a3c09f9231d6" Jul 20 03:29:17.231385 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:17.231367 2558 scope.go:117] "RemoveContainer" containerID="3f5e9df0bc4a4c19deb6837505431f08ed0d11bbd596ff84d2bcc2a791e763ca" Jul 20 03:29:17.231640 ip-10-0-142-139 kubenswrapper[2558]: E0720 03:29:17.231609 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f5e9df0bc4a4c19deb6837505431f08ed0d11bbd596ff84d2bcc2a791e763ca\": container with ID starting with 3f5e9df0bc4a4c19deb6837505431f08ed0d11bbd596ff84d2bcc2a791e763ca not found: ID does not exist" containerID="3f5e9df0bc4a4c19deb6837505431f08ed0d11bbd596ff84d2bcc2a791e763ca" Jul 20 03:29:17.231694 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:17.231655 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f5e9df0bc4a4c19deb6837505431f08ed0d11bbd596ff84d2bcc2a791e763ca"} err="failed to get container status \"3f5e9df0bc4a4c19deb6837505431f08ed0d11bbd596ff84d2bcc2a791e763ca\": rpc error: code = NotFound desc = could not find container \"3f5e9df0bc4a4c19deb6837505431f08ed0d11bbd596ff84d2bcc2a791e763ca\": container with ID starting with 3f5e9df0bc4a4c19deb6837505431f08ed0d11bbd596ff84d2bcc2a791e763ca not found: ID does not exist" Jul 20 03:29:17.231694 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:17.231674 2558 scope.go:117] "RemoveContainer" containerID="29f7cd1ab9dae5682acff152855dc5472783553d4d47101b1d12a3c09f9231d6" Jul 20 03:29:17.232078 ip-10-0-142-139 kubenswrapper[2558]: E0720 03:29:17.232038 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29f7cd1ab9dae5682acff152855dc5472783553d4d47101b1d12a3c09f9231d6\": container with ID starting with 29f7cd1ab9dae5682acff152855dc5472783553d4d47101b1d12a3c09f9231d6 not found: ID does not exist" containerID="29f7cd1ab9dae5682acff152855dc5472783553d4d47101b1d12a3c09f9231d6" Jul 20 03:29:17.232238 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:17.232178 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29f7cd1ab9dae5682acff152855dc5472783553d4d47101b1d12a3c09f9231d6"} err="failed to get container status \"29f7cd1ab9dae5682acff152855dc5472783553d4d47101b1d12a3c09f9231d6\": rpc error: code = NotFound desc = could not find container \"29f7cd1ab9dae5682acff152855dc5472783553d4d47101b1d12a3c09f9231d6\": container with ID starting with 29f7cd1ab9dae5682acff152855dc5472783553d4d47101b1d12a3c09f9231d6 not found: ID does not exist" Jul 20 03:29:17.234578 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:17.234549 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq"] Jul 20 03:29:17.245377 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:17.240717 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-a8945e-openshift-default-9cf6c77cc-6h4kq"] Jul 20 03:29:17.424862 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:17.424828 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69fe7950-e2d4-459c-831d-03ac30620fa6" path="/var/lib/kubelet/pods/69fe7950-e2d4-459c-831d-03ac30620fa6/volumes" Jul 20 03:29:20.227334 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:20.227298 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:29:20.268266 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:20.268233 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv"] Jul 20 03:29:20.268747 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:20.268702 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69fe7950-e2d4-459c-831d-03ac30620fa6" containerName="istio-proxy" Jul 20 03:29:20.268747 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:20.268725 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="69fe7950-e2d4-459c-831d-03ac30620fa6" containerName="istio-proxy" Jul 20 03:29:20.268747 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:20.268748 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69fe7950-e2d4-459c-831d-03ac30620fa6" containerName="istio-proxy" Jul 20 03:29:20.269070 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:20.268756 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="69fe7950-e2d4-459c-831d-03ac30620fa6" containerName="istio-proxy" Jul 20 03:29:20.269070 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:20.268840 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="69fe7950-e2d4-459c-831d-03ac30620fa6" containerName="istio-proxy" Jul 20 03:29:20.273554 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:20.273503 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv" Jul 20 03:29:20.276363 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:20.276340 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-c1951c-openshift-default-dockercfg-whkgp\"" Jul 20 03:29:20.283320 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:20.283297 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv"] Jul 20 03:29:20.396755 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:20.396621 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4lhl\" (UniqueName: \"kubernetes.io/projected/78bcbbe8-7224-492d-bc51-3311def389e9-kube-api-access-w4lhl\") pod \"e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv\" (UID: \"78bcbbe8-7224-492d-bc51-3311def389e9\") " pod="openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv" Jul 20 03:29:20.396755 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:20.396709 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78bcbbe8-7224-492d-bc51-3311def389e9-istio-data\") pod \"e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv\" (UID: \"78bcbbe8-7224-492d-bc51-3311def389e9\") " pod="openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv" Jul 20 03:29:20.396755 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:20.396746 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78bcbbe8-7224-492d-bc51-3311def389e9-istio-token\") pod \"e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv\" (UID: \"78bcbbe8-7224-492d-bc51-3311def389e9\") " pod="openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv" Jul 20 03:29:20.397199 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:20.396783 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78bcbbe8-7224-492d-bc51-3311def389e9-workload-certs\") pod \"e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv\" (UID: \"78bcbbe8-7224-492d-bc51-3311def389e9\") " pod="openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv" Jul 20 03:29:20.397199 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:20.396819 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78bcbbe8-7224-492d-bc51-3311def389e9-credential-socket\") pod \"e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv\" (UID: \"78bcbbe8-7224-492d-bc51-3311def389e9\") " pod="openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv" Jul 20 03:29:20.397199 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:20.396855 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78bcbbe8-7224-492d-bc51-3311def389e9-workload-socket\") pod \"e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv\" (UID: \"78bcbbe8-7224-492d-bc51-3311def389e9\") " pod="openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv" Jul 20 03:29:20.397199 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:20.396880 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78bcbbe8-7224-492d-bc51-3311def389e9-istiod-ca-cert\") pod \"e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv\" (UID: \"78bcbbe8-7224-492d-bc51-3311def389e9\") " pod="openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv" Jul 20 03:29:20.397199 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:20.396919 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78bcbbe8-7224-492d-bc51-3311def389e9-istio-podinfo\") pod \"e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv\" (UID: \"78bcbbe8-7224-492d-bc51-3311def389e9\") " pod="openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv" Jul 20 03:29:20.397199 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:20.396954 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78bcbbe8-7224-492d-bc51-3311def389e9-istio-envoy\") pod \"e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv\" (UID: \"78bcbbe8-7224-492d-bc51-3311def389e9\") " pod="openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv" Jul 20 03:29:20.498024 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:20.497941 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78bcbbe8-7224-492d-bc51-3311def389e9-istio-data\") pod \"e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv\" (UID: \"78bcbbe8-7224-492d-bc51-3311def389e9\") " pod="openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv" Jul 20 03:29:20.498024 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:20.497978 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78bcbbe8-7224-492d-bc51-3311def389e9-istio-token\") pod \"e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv\" (UID: \"78bcbbe8-7224-492d-bc51-3311def389e9\") " pod="openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv" Jul 20 03:29:20.498024 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:20.498011 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78bcbbe8-7224-492d-bc51-3311def389e9-workload-certs\") pod \"e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv\" (UID: \"78bcbbe8-7224-492d-bc51-3311def389e9\") " pod="openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv" Jul 20 03:29:20.498336 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:20.498139 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78bcbbe8-7224-492d-bc51-3311def389e9-credential-socket\") pod \"e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv\" (UID: \"78bcbbe8-7224-492d-bc51-3311def389e9\") " pod="openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv" Jul 20 03:29:20.498336 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:20.498198 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78bcbbe8-7224-492d-bc51-3311def389e9-workload-socket\") pod \"e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv\" (UID: \"78bcbbe8-7224-492d-bc51-3311def389e9\") " pod="openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv" Jul 20 03:29:20.498336 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:20.498225 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78bcbbe8-7224-492d-bc51-3311def389e9-istiod-ca-cert\") pod \"e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv\" (UID: \"78bcbbe8-7224-492d-bc51-3311def389e9\") " pod="openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv" Jul 20 03:29:20.498336 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:20.498270 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78bcbbe8-7224-492d-bc51-3311def389e9-istio-podinfo\") pod \"e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv\" (UID: \"78bcbbe8-7224-492d-bc51-3311def389e9\") " pod="openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv" Jul 20 03:29:20.498336 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:20.498327 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78bcbbe8-7224-492d-bc51-3311def389e9-istio-envoy\") pod \"e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv\" (UID: \"78bcbbe8-7224-492d-bc51-3311def389e9\") " pod="openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv" Jul 20 03:29:20.498602 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:20.498394 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4lhl\" (UniqueName: \"kubernetes.io/projected/78bcbbe8-7224-492d-bc51-3311def389e9-kube-api-access-w4lhl\") pod \"e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv\" (UID: \"78bcbbe8-7224-492d-bc51-3311def389e9\") " pod="openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv" Jul 20 03:29:20.498602 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:20.498397 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78bcbbe8-7224-492d-bc51-3311def389e9-istio-data\") pod \"e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv\" (UID: \"78bcbbe8-7224-492d-bc51-3311def389e9\") " pod="openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv" Jul 20 03:29:20.498602 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:20.498501 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78bcbbe8-7224-492d-bc51-3311def389e9-credential-socket\") pod \"e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv\" (UID: \"78bcbbe8-7224-492d-bc51-3311def389e9\") " pod="openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv" Jul 20 03:29:20.498911 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:20.498881 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78bcbbe8-7224-492d-bc51-3311def389e9-workload-certs\") pod \"e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv\" (UID: \"78bcbbe8-7224-492d-bc51-3311def389e9\") " pod="openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv" Jul 20 03:29:20.499008 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:20.498958 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78bcbbe8-7224-492d-bc51-3311def389e9-workload-socket\") pod \"e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv\" (UID: \"78bcbbe8-7224-492d-bc51-3311def389e9\") " pod="openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv" Jul 20 03:29:20.499008 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:20.498998 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78bcbbe8-7224-492d-bc51-3311def389e9-istiod-ca-cert\") pod \"e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv\" (UID: \"78bcbbe8-7224-492d-bc51-3311def389e9\") " pod="openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv" Jul 20 03:29:20.500669 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:20.500644 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78bcbbe8-7224-492d-bc51-3311def389e9-istio-envoy\") pod \"e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv\" (UID: \"78bcbbe8-7224-492d-bc51-3311def389e9\") " pod="openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv" Jul 20 03:29:20.500856 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:20.500836 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78bcbbe8-7224-492d-bc51-3311def389e9-istio-podinfo\") pod \"e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv\" (UID: \"78bcbbe8-7224-492d-bc51-3311def389e9\") " pod="openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv" Jul 20 03:29:20.505845 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:20.505817 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78bcbbe8-7224-492d-bc51-3311def389e9-istio-token\") pod \"e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv\" (UID: \"78bcbbe8-7224-492d-bc51-3311def389e9\") " pod="openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv" Jul 20 03:29:20.506167 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:20.506146 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4lhl\" (UniqueName: \"kubernetes.io/projected/78bcbbe8-7224-492d-bc51-3311def389e9-kube-api-access-w4lhl\") pod \"e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv\" (UID: \"78bcbbe8-7224-492d-bc51-3311def389e9\") " pod="openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv" Jul 20 03:29:20.586836 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:20.586797 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv" Jul 20 03:29:20.728501 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:20.728471 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv"] Jul 20 03:29:20.730992 ip-10-0-142-139 kubenswrapper[2558]: W0720 03:29:20.730965 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78bcbbe8_7224_492d_bc51_3311def389e9.slice/crio-6cbd5a401cd89fe8663dfe2b35839878bfcc33db5915aa3cd6539552a5a665b6 WatchSource:0}: Error finding container 6cbd5a401cd89fe8663dfe2b35839878bfcc33db5915aa3cd6539552a5a665b6: Status 404 returned error can't find the container with id 6cbd5a401cd89fe8663dfe2b35839878bfcc33db5915aa3cd6539552a5a665b6 Jul 20 03:29:20.733074 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:20.733044 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:29:20.733157 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:20.733120 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:29:20.733157 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:20.733151 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:29:21.233438 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:21.233398 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv" event={"ID":"78bcbbe8-7224-492d-bc51-3311def389e9","Type":"ContainerStarted","Data":"58376e3ff258eff87bb84e403e5c78711a5479011c43775962a4aa8ab981c6e0"} Jul 20 03:29:21.233812 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:21.233456 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv" event={"ID":"78bcbbe8-7224-492d-bc51-3311def389e9","Type":"ContainerStarted","Data":"6cbd5a401cd89fe8663dfe2b35839878bfcc33db5915aa3cd6539552a5a665b6"} Jul 20 03:29:21.257944 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:21.257874 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv" podStartSLOduration=1.257849961 podStartE2EDuration="1.257849961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 03:29:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 03:29:21.253532868 +0000 UTC m=+2656.452782017" watchObservedRunningTime="2026-07-20 03:29:21.257849961 +0000 UTC m=+2656.457099100" Jul 20 03:29:21.587339 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:21.587240 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv" Jul 20 03:29:21.592363 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:21.592332 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv" Jul 20 03:29:22.238347 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:22.238315 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv" Jul 20 03:29:22.239397 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:22.239380 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv" Jul 20 03:29:28.273491 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:28.273422 2558 generic.go:356] "Generic (PLEG): container finished" podID="2c464eb9-afce-451d-9305-1aa626aac3e0" containerID="ff56d3eed1e90b33166c10dec66812c13960bb5065b70e4f00db29e0b8107cfd" exitCode=0 Jul 20 03:29:28.273928 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:28.273495 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d" event={"ID":"2c464eb9-afce-451d-9305-1aa626aac3e0","Type":"ContainerDied","Data":"ff56d3eed1e90b33166c10dec66812c13960bb5065b70e4f00db29e0b8107cfd"} Jul 20 03:29:28.273928 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:28.273861 2558 scope.go:117] "RemoveContainer" containerID="ff56d3eed1e90b33166c10dec66812c13960bb5065b70e4f00db29e0b8107cfd" Jul 20 03:29:28.274821 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:28.274792 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:29:28.274892 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:28.274875 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:29:28.274942 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:28.274929 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:29:28.333247 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:28.333217 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:29:29.192029 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:29.191986 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d" Jul 20 03:29:29.284976 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:29.283976 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d" event={"ID":"2c464eb9-afce-451d-9305-1aa626aac3e0","Type":"ContainerStarted","Data":"b8618a5b8c7dbcce76b88fc18aabb9b8fc049efbf918d3b27ded2eeccaed85b9"} Jul 20 03:29:29.775797 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:29.775759 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d" Jul 20 03:29:29.776222 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:29.776198 2558 patch_prober.go:28] interesting pod/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 20 03:29:29.776415 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:29.776388 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d" podUID="2c464eb9-afce-451d-9305-1aa626aac3e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 20 03:29:30.775138 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:30.775069 2558 patch_prober.go:28] interesting pod/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 20 03:29:30.775646 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:30.775160 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d" podUID="2c464eb9-afce-451d-9305-1aa626aac3e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 20 03:29:31.775233 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:31.775187 2558 patch_prober.go:28] interesting pod/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 20 03:29:31.775766 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:31.775278 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d" podUID="2c464eb9-afce-451d-9305-1aa626aac3e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 20 03:29:32.775382 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:32.775338 2558 patch_prober.go:28] interesting pod/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 20 03:29:32.775925 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:32.775412 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d" podUID="2c464eb9-afce-451d-9305-1aa626aac3e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 20 03:29:33.779603 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:33.779564 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d" Jul 20 03:29:33.780035 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:33.779848 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d" Jul 20 03:29:33.780651 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:33.780634 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d" Jul 20 03:29:36.325159 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:36.325051 2558 generic.go:356] "Generic (PLEG): container finished" podID="78bcbbe8-7224-492d-bc51-3311def389e9" containerID="58376e3ff258eff87bb84e403e5c78711a5479011c43775962a4aa8ab981c6e0" exitCode=0 Jul 20 03:29:36.325159 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:36.325124 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv" event={"ID":"78bcbbe8-7224-492d-bc51-3311def389e9","Type":"ContainerDied","Data":"58376e3ff258eff87bb84e403e5c78711a5479011c43775962a4aa8ab981c6e0"} Jul 20 03:29:36.325596 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:36.325509 2558 scope.go:117] "RemoveContainer" containerID="58376e3ff258eff87bb84e403e5c78711a5479011c43775962a4aa8ab981c6e0" Jul 20 03:29:36.326522 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:36.326495 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:29:36.326621 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:36.326578 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:29:36.326684 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:36.326628 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 20 03:29:37.238889 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:37.238853 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv" Jul 20 03:29:37.331370 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:37.331336 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv" event={"ID":"78bcbbe8-7224-492d-bc51-3311def389e9","Type":"ContainerStarted","Data":"8e565aecddf3ce4e1536783a30b37b36c813d1b7cbbbf523d4bcdf716e6f1168"} Jul 20 03:29:37.587334 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:37.587228 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv" Jul 20 03:29:37.588511 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:37.588483 2558 patch_prober.go:28] interesting pod/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 20 03:29:37.588639 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:37.588546 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv" podUID="78bcbbe8-7224-492d-bc51-3311def389e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 20 03:29:38.588237 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:38.588202 2558 patch_prober.go:28] interesting pod/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 20 03:29:38.588672 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:38.588260 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv" podUID="78bcbbe8-7224-492d-bc51-3311def389e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 20 03:29:39.587795 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:39.587759 2558 patch_prober.go:28] interesting pod/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 20 03:29:39.587988 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:39.587817 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv" podUID="78bcbbe8-7224-492d-bc51-3311def389e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 20 03:29:40.587896 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:40.587857 2558 patch_prober.go:28] interesting pod/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 20 03:29:40.588335 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:40.587915 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv" podUID="78bcbbe8-7224-492d-bc51-3311def389e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 20 03:29:41.591266 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:41.591235 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv" Jul 20 03:29:41.591690 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:41.591513 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv" Jul 20 03:29:41.592137 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:41.592119 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv" Jul 20 03:29:42.926436 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:29:42.926396 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:30:00.136513 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:00.136427 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-496193-29741970-75p7g"] Jul 20 03:30:00.137165 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:00.137142 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="69fe7950-e2d4-459c-831d-03ac30620fa6" containerName="istio-proxy" Jul 20 03:30:00.140802 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:00.140782 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c1951c-29741970-rdp2w"] Jul 20 03:30:00.140921 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:00.140901 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-496193-29741970-75p7g" Jul 20 03:30:00.143703 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:00.143682 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hlzh6\"" Jul 20 03:30:00.143802 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:00.143770 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29741970-qxvq2"] Jul 20 03:30:00.143965 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:00.143951 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c1951c-29741970-rdp2w" Jul 20 03:30:00.147470 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:00.147343 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-496193-29741970-75p7g"] Jul 20 03:30:00.147470 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:00.147440 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741970-qxvq2" Jul 20 03:30:00.150508 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:00.150487 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29741970-qxvq2"] Jul 20 03:30:00.153167 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:00.153068 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c1951c-29741970-rdp2w"] Jul 20 03:30:00.157769 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:00.157745 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-496193-29741970-75p7g" Jul 20 03:30:00.164977 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:00.164951 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c1951c-29741970-rdp2w" Jul 20 03:30:00.191984 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:00.191962 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741970-qxvq2" Jul 20 03:30:00.324242 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:00.323837 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-496193-29741970-75p7g"] Jul 20 03:30:00.326597 ip-10-0-142-139 kubenswrapper[2558]: W0720 03:30:00.326539 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc4a441d_f75e_4783_a6bd_ad1cde785756.slice/crio-70702f75f2eea4b05be937ea9aa64269fef37ff3bc5ae58060b7fdfa536a5a0f WatchSource:0}: Error finding container 70702f75f2eea4b05be937ea9aa64269fef37ff3bc5ae58060b7fdfa536a5a0f: Status 404 returned error can't find the container with id 70702f75f2eea4b05be937ea9aa64269fef37ff3bc5ae58060b7fdfa536a5a0f Jul 20 03:30:00.446393 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:00.446323 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-496193-29741970-75p7g" event={"ID":"fc4a441d-f75e-4783-a6bd-ad1cde785756","Type":"ContainerStarted","Data":"70702f75f2eea4b05be937ea9aa64269fef37ff3bc5ae58060b7fdfa536a5a0f"} Jul 20 03:30:00.549060 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:00.549026 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c1951c-29741970-rdp2w"] Jul 20 03:30:00.551990 ip-10-0-142-139 kubenswrapper[2558]: W0720 03:30:00.551962 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e0099e3_6f6e_4105_b0bf_db4f676b0d8a.slice/crio-94716b7ab09a045fc49df0ea2f8ddd5739e9e096bc72aea867a91ca31965f6ea WatchSource:0}: Error finding container 94716b7ab09a045fc49df0ea2f8ddd5739e9e096bc72aea867a91ca31965f6ea: Status 404 returned error can't find the container with id 94716b7ab09a045fc49df0ea2f8ddd5739e9e096bc72aea867a91ca31965f6ea Jul 20 03:30:00.578606 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:00.578586 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29741970-qxvq2"] Jul 20 03:30:00.583141 ip-10-0-142-139 kubenswrapper[2558]: W0720 03:30:00.583115 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7425d3e_1a2f_497f_a23d_62ab0d028b87.slice/crio-e63943c602ed4089f98ac4f092860de733684a1d2dd6d352807c5cb6c9338217 WatchSource:0}: Error finding container e63943c602ed4089f98ac4f092860de733684a1d2dd6d352807c5cb6c9338217: Status 404 returned error can't find the container with id e63943c602ed4089f98ac4f092860de733684a1d2dd6d352807c5cb6c9338217 Jul 20 03:30:01.453171 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:01.453136 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741970-qxvq2" event={"ID":"c7425d3e-1a2f-497f-a23d-62ab0d028b87","Type":"ContainerStarted","Data":"a97f64b7d27864955a17449abdee2c24f6fbdba54d92e1a29e7385be857f47f9"} Jul 20 03:30:01.453637 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:01.453187 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741970-qxvq2" event={"ID":"c7425d3e-1a2f-497f-a23d-62ab0d028b87","Type":"ContainerStarted","Data":"e63943c602ed4089f98ac4f092860de733684a1d2dd6d352807c5cb6c9338217"} Jul 20 03:30:01.454769 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:01.454741 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c1951c-29741970-rdp2w" event={"ID":"3e0099e3-6f6e-4105-b0bf-db4f676b0d8a","Type":"ContainerStarted","Data":"d1c9ef5a6f0e05101c7dce880171b40e262b541bce65853340870e0c01c3d5ae"} Jul 20 03:30:01.454907 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:01.454781 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c1951c-29741970-rdp2w" event={"ID":"3e0099e3-6f6e-4105-b0bf-db4f676b0d8a","Type":"ContainerStarted","Data":"94716b7ab09a045fc49df0ea2f8ddd5739e9e096bc72aea867a91ca31965f6ea"} Jul 20 03:30:01.456065 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:01.456043 2558 generic.go:356] "Generic (PLEG): container finished" podID="fc4a441d-f75e-4783-a6bd-ad1cde785756" containerID="075ac14167b25d3d7c2e70e6c7603db16ac3d1685a441d5ea108aba65d58be25" exitCode=0 Jul 20 03:30:01.456180 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:01.456089 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-496193-29741970-75p7g" event={"ID":"fc4a441d-f75e-4783-a6bd-ad1cde785756","Type":"ContainerDied","Data":"075ac14167b25d3d7c2e70e6c7603db16ac3d1685a441d5ea108aba65d58be25"} Jul 20 03:30:01.468944 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:01.468904 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741970-qxvq2" podStartSLOduration=1.468889685 podStartE2EDuration="1.468889685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 03:30:01.467812871 +0000 UTC m=+2696.667062021" watchObservedRunningTime="2026-07-20 03:30:01.468889685 +0000 UTC m=+2696.668138832" Jul 20 03:30:01.497272 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:01.497217 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c1951c-29741970-rdp2w" podStartSLOduration=1.49720182 podStartE2EDuration="1.49720182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 03:30:01.496251627 +0000 UTC m=+2696.695500775" watchObservedRunningTime="2026-07-20 03:30:01.49720182 +0000 UTC m=+2696.696450966" Jul 20 03:30:02.462004 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:02.461965 2558 generic.go:356] "Generic (PLEG): container finished" podID="3e0099e3-6f6e-4105-b0bf-db4f676b0d8a" containerID="d1c9ef5a6f0e05101c7dce880171b40e262b541bce65853340870e0c01c3d5ae" exitCode=0 Jul 20 03:30:02.462495 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:02.462058 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c1951c-29741970-rdp2w" event={"ID":"3e0099e3-6f6e-4105-b0bf-db4f676b0d8a","Type":"ContainerDied","Data":"d1c9ef5a6f0e05101c7dce880171b40e262b541bce65853340870e0c01c3d5ae"} Jul 20 03:30:02.463570 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:02.463550 2558 generic.go:356] "Generic (PLEG): container finished" podID="c7425d3e-1a2f-497f-a23d-62ab0d028b87" containerID="a97f64b7d27864955a17449abdee2c24f6fbdba54d92e1a29e7385be857f47f9" exitCode=0 Jul 20 03:30:02.463710 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:02.463628 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741970-qxvq2" event={"ID":"c7425d3e-1a2f-497f-a23d-62ab0d028b87","Type":"ContainerDied","Data":"a97f64b7d27864955a17449abdee2c24f6fbdba54d92e1a29e7385be857f47f9"} Jul 20 03:30:02.592895 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:02.592871 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-496193-29741970-75p7g" Jul 20 03:30:03.469803 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:03.469774 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-496193-29741970-75p7g" Jul 20 03:30:03.470219 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:03.469807 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-496193-29741970-75p7g" event={"ID":"fc4a441d-f75e-4783-a6bd-ad1cde785756","Type":"ContainerDied","Data":"70702f75f2eea4b05be937ea9aa64269fef37ff3bc5ae58060b7fdfa536a5a0f"} Jul 20 03:30:03.470219 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:03.469859 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70702f75f2eea4b05be937ea9aa64269fef37ff3bc5ae58060b7fdfa536a5a0f" Jul 20 03:30:03.616523 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:03.616500 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c1951c-29741970-rdp2w" Jul 20 03:30:03.637806 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:03.637767 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741970-qxvq2" Jul 20 03:30:04.475651 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:04.475625 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741970-qxvq2" Jul 20 03:30:04.476088 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:04.475669 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741970-qxvq2" event={"ID":"c7425d3e-1a2f-497f-a23d-62ab0d028b87","Type":"ContainerDied","Data":"e63943c602ed4089f98ac4f092860de733684a1d2dd6d352807c5cb6c9338217"} Jul 20 03:30:04.476088 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:04.475713 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e63943c602ed4089f98ac4f092860de733684a1d2dd6d352807c5cb6c9338217" Jul 20 03:30:04.476951 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:04.476929 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c1951c-29741970-rdp2w" event={"ID":"3e0099e3-6f6e-4105-b0bf-db4f676b0d8a","Type":"ContainerDied","Data":"94716b7ab09a045fc49df0ea2f8ddd5739e9e096bc72aea867a91ca31965f6ea"} Jul 20 03:30:04.477051 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:04.476952 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c1951c-29741970-rdp2w" Jul 20 03:30:04.477051 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:04.476968 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94716b7ab09a045fc49df0ea2f8ddd5739e9e096bc72aea867a91ca31965f6ea" Jul 20 03:30:05.646644 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:05.646606 2558 scope.go:117] "RemoveContainer" containerID="e6c63fa44602448bf57c1b4f3beb7b34ee7950c732fc68f6d098b641dd607981" Jul 20 03:30:07.433791 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:07.433762 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-c1951c/auth-test-model-c1951c-kserve-c9977b64c-kjg6n"] Jul 20 03:30:07.434289 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:07.434088 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc4a441d-f75e-4783-a6bd-ad1cde785756" containerName="cleanup" Jul 20 03:30:07.434289 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:07.434116 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc4a441d-f75e-4783-a6bd-ad1cde785756" containerName="cleanup" Jul 20 03:30:07.434289 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:07.434130 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e0099e3-6f6e-4105-b0bf-db4f676b0d8a" containerName="cleanup" Jul 20 03:30:07.434289 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:07.434135 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e0099e3-6f6e-4105-b0bf-db4f676b0d8a" containerName="cleanup" Jul 20 03:30:07.434289 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:07.434155 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7425d3e-1a2f-497f-a23d-62ab0d028b87" containerName="cleanup" Jul 20 03:30:07.434289 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:07.434160 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7425d3e-1a2f-497f-a23d-62ab0d028b87" containerName="cleanup" Jul 20 03:30:07.434289 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:07.434212 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc4a441d-f75e-4783-a6bd-ad1cde785756" containerName="cleanup" Jul 20 03:30:07.434289 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:07.434221 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7425d3e-1a2f-497f-a23d-62ab0d028b87" containerName="cleanup" Jul 20 03:30:07.434289 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:07.434229 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e0099e3-6f6e-4105-b0bf-db4f676b0d8a" containerName="cleanup" Jul 20 03:30:07.438628 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:07.438603 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c1951c/auth-test-model-c1951c-kserve-c9977b64c-kjg6n"] Jul 20 03:30:07.438791 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:07.438723 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c1951c/auth-test-model-c1951c-kserve-c9977b64c-kjg6n" Jul 20 03:30:07.441339 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:07.441312 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c1951c\"/\"default-dockercfg-46jhr\"" Jul 20 03:30:07.442805 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:07.442664 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c1951c\"/\"openshift-service-ca.crt\"" Jul 20 03:30:07.442805 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:07.442729 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c1951c\"/\"auth-test-model-c1951c-kserve-self-signed-certs\"" Jul 20 03:30:07.442979 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:07.442894 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c1951c\"/\"kube-root-ca.crt\"" Jul 20 03:30:07.539621 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:07.539574 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33b4693d-c1e3-4fb9-b315-ab9241486327-home\") pod \"auth-test-model-c1951c-kserve-c9977b64c-kjg6n\" (UID: \"33b4693d-c1e3-4fb9-b315-ab9241486327\") " pod="ai-tenant-e2e-shared-b-c1951c/auth-test-model-c1951c-kserve-c9977b64c-kjg6n" Jul 20 03:30:07.539828 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:07.539643 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33b4693d-c1e3-4fb9-b315-ab9241486327-dshm\") pod \"auth-test-model-c1951c-kserve-c9977b64c-kjg6n\" (UID: \"33b4693d-c1e3-4fb9-b315-ab9241486327\") " pod="ai-tenant-e2e-shared-b-c1951c/auth-test-model-c1951c-kserve-c9977b64c-kjg6n" Jul 20 03:30:07.539828 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:07.539705 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x48ns\" (UniqueName: \"kubernetes.io/projected/33b4693d-c1e3-4fb9-b315-ab9241486327-kube-api-access-x48ns\") pod \"auth-test-model-c1951c-kserve-c9977b64c-kjg6n\" (UID: \"33b4693d-c1e3-4fb9-b315-ab9241486327\") " pod="ai-tenant-e2e-shared-b-c1951c/auth-test-model-c1951c-kserve-c9977b64c-kjg6n" Jul 20 03:30:07.539828 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:07.539727 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33b4693d-c1e3-4fb9-b315-ab9241486327-tls-certs\") pod \"auth-test-model-c1951c-kserve-c9977b64c-kjg6n\" (UID: \"33b4693d-c1e3-4fb9-b315-ab9241486327\") " pod="ai-tenant-e2e-shared-b-c1951c/auth-test-model-c1951c-kserve-c9977b64c-kjg6n" Jul 20 03:30:07.539828 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:07.539756 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33b4693d-c1e3-4fb9-b315-ab9241486327-tmp-dir\") pod \"auth-test-model-c1951c-kserve-c9977b64c-kjg6n\" (UID: \"33b4693d-c1e3-4fb9-b315-ab9241486327\") " pod="ai-tenant-e2e-shared-b-c1951c/auth-test-model-c1951c-kserve-c9977b64c-kjg6n" Jul 20 03:30:07.539991 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:07.539829 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33b4693d-c1e3-4fb9-b315-ab9241486327-model-cache\") pod \"auth-test-model-c1951c-kserve-c9977b64c-kjg6n\" (UID: \"33b4693d-c1e3-4fb9-b315-ab9241486327\") " pod="ai-tenant-e2e-shared-b-c1951c/auth-test-model-c1951c-kserve-c9977b64c-kjg6n" Jul 20 03:30:07.641280 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:07.641246 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x48ns\" (UniqueName: \"kubernetes.io/projected/33b4693d-c1e3-4fb9-b315-ab9241486327-kube-api-access-x48ns\") pod \"auth-test-model-c1951c-kserve-c9977b64c-kjg6n\" (UID: \"33b4693d-c1e3-4fb9-b315-ab9241486327\") " pod="ai-tenant-e2e-shared-b-c1951c/auth-test-model-c1951c-kserve-c9977b64c-kjg6n" Jul 20 03:30:07.641456 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:07.641291 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33b4693d-c1e3-4fb9-b315-ab9241486327-tls-certs\") pod \"auth-test-model-c1951c-kserve-c9977b64c-kjg6n\" (UID: \"33b4693d-c1e3-4fb9-b315-ab9241486327\") " pod="ai-tenant-e2e-shared-b-c1951c/auth-test-model-c1951c-kserve-c9977b64c-kjg6n" Jul 20 03:30:07.641456 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:07.641336 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33b4693d-c1e3-4fb9-b315-ab9241486327-tmp-dir\") pod \"auth-test-model-c1951c-kserve-c9977b64c-kjg6n\" (UID: \"33b4693d-c1e3-4fb9-b315-ab9241486327\") " pod="ai-tenant-e2e-shared-b-c1951c/auth-test-model-c1951c-kserve-c9977b64c-kjg6n" Jul 20 03:30:07.641456 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:07.641391 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33b4693d-c1e3-4fb9-b315-ab9241486327-model-cache\") pod \"auth-test-model-c1951c-kserve-c9977b64c-kjg6n\" (UID: \"33b4693d-c1e3-4fb9-b315-ab9241486327\") " pod="ai-tenant-e2e-shared-b-c1951c/auth-test-model-c1951c-kserve-c9977b64c-kjg6n" Jul 20 03:30:07.641456 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:07.641444 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33b4693d-c1e3-4fb9-b315-ab9241486327-home\") pod \"auth-test-model-c1951c-kserve-c9977b64c-kjg6n\" (UID: \"33b4693d-c1e3-4fb9-b315-ab9241486327\") " pod="ai-tenant-e2e-shared-b-c1951c/auth-test-model-c1951c-kserve-c9977b64c-kjg6n" Jul 20 03:30:07.641653 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:07.641487 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33b4693d-c1e3-4fb9-b315-ab9241486327-dshm\") pod \"auth-test-model-c1951c-kserve-c9977b64c-kjg6n\" (UID: \"33b4693d-c1e3-4fb9-b315-ab9241486327\") " pod="ai-tenant-e2e-shared-b-c1951c/auth-test-model-c1951c-kserve-c9977b64c-kjg6n" Jul 20 03:30:07.641821 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:07.641796 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33b4693d-c1e3-4fb9-b315-ab9241486327-tmp-dir\") pod \"auth-test-model-c1951c-kserve-c9977b64c-kjg6n\" (UID: \"33b4693d-c1e3-4fb9-b315-ab9241486327\") " pod="ai-tenant-e2e-shared-b-c1951c/auth-test-model-c1951c-kserve-c9977b64c-kjg6n" Jul 20 03:30:07.641901 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:07.641832 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33b4693d-c1e3-4fb9-b315-ab9241486327-home\") pod \"auth-test-model-c1951c-kserve-c9977b64c-kjg6n\" (UID: \"33b4693d-c1e3-4fb9-b315-ab9241486327\") " pod="ai-tenant-e2e-shared-b-c1951c/auth-test-model-c1951c-kserve-c9977b64c-kjg6n" Jul 20 03:30:07.641901 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:07.641844 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33b4693d-c1e3-4fb9-b315-ab9241486327-model-cache\") pod \"auth-test-model-c1951c-kserve-c9977b64c-kjg6n\" (UID: \"33b4693d-c1e3-4fb9-b315-ab9241486327\") " pod="ai-tenant-e2e-shared-b-c1951c/auth-test-model-c1951c-kserve-c9977b64c-kjg6n" Jul 20 03:30:07.643700 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:07.643667 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33b4693d-c1e3-4fb9-b315-ab9241486327-dshm\") pod \"auth-test-model-c1951c-kserve-c9977b64c-kjg6n\" (UID: \"33b4693d-c1e3-4fb9-b315-ab9241486327\") " pod="ai-tenant-e2e-shared-b-c1951c/auth-test-model-c1951c-kserve-c9977b64c-kjg6n" Jul 20 03:30:07.643846 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:07.643828 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33b4693d-c1e3-4fb9-b315-ab9241486327-tls-certs\") pod \"auth-test-model-c1951c-kserve-c9977b64c-kjg6n\" (UID: \"33b4693d-c1e3-4fb9-b315-ab9241486327\") " pod="ai-tenant-e2e-shared-b-c1951c/auth-test-model-c1951c-kserve-c9977b64c-kjg6n" Jul 20 03:30:07.649943 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:07.649924 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x48ns\" (UniqueName: \"kubernetes.io/projected/33b4693d-c1e3-4fb9-b315-ab9241486327-kube-api-access-x48ns\") pod \"auth-test-model-c1951c-kserve-c9977b64c-kjg6n\" (UID: \"33b4693d-c1e3-4fb9-b315-ab9241486327\") " pod="ai-tenant-e2e-shared-b-c1951c/auth-test-model-c1951c-kserve-c9977b64c-kjg6n" Jul 20 03:30:07.750903 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:07.750808 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c1951c/auth-test-model-c1951c-kserve-c9977b64c-kjg6n" Jul 20 03:30:07.883278 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:07.883254 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c1951c/auth-test-model-c1951c-kserve-c9977b64c-kjg6n"] Jul 20 03:30:07.884889 ip-10-0-142-139 kubenswrapper[2558]: W0720 03:30:07.884859 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod33b4693d_c1e3_4fb9_b315_ab9241486327.slice/crio-3c7cd20b3172de3681f549ff32463ff83c4fa5f7e71e80c6dba5e9ad0f2e2d89 WatchSource:0}: Error finding container 3c7cd20b3172de3681f549ff32463ff83c4fa5f7e71e80c6dba5e9ad0f2e2d89: Status 404 returned error can't find the container with id 3c7cd20b3172de3681f549ff32463ff83c4fa5f7e71e80c6dba5e9ad0f2e2d89 Jul 20 03:30:08.497741 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:08.497705 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c1951c/auth-test-model-c1951c-kserve-c9977b64c-kjg6n" event={"ID":"33b4693d-c1e3-4fb9-b315-ab9241486327","Type":"ContainerStarted","Data":"9ba3690ad61128daf79533698cae1ab7ff74108b30e5a778e18b9dc31ad877b6"} Jul 20 03:30:08.497741 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:08.497747 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c1951c/auth-test-model-c1951c-kserve-c9977b64c-kjg6n" event={"ID":"33b4693d-c1e3-4fb9-b315-ab9241486327","Type":"ContainerStarted","Data":"3c7cd20b3172de3681f549ff32463ff83c4fa5f7e71e80c6dba5e9ad0f2e2d89"} Jul 20 03:30:08.515583 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:08.515534 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-c1951c/auth-test-model-c1951c-kserve-c9977b64c-kjg6n" podStartSLOduration=1.515518812 podStartE2EDuration="1.515518812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 03:30:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 03:30:08.513087507 +0000 UTC m=+2703.712336655" watchObservedRunningTime="2026-07-20 03:30:08.515518812 +0000 UTC m=+2703.714767959" Jul 20 03:30:11.426959 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:11.426928 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:30:17.751252 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:17.751213 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-c1951c/auth-test-model-c1951c-kserve-c9977b64c-kjg6n" Jul 20 03:30:17.751252 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:17.751270 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-c1951c/auth-test-model-c1951c-kserve-c9977b64c-kjg6n" Jul 20 03:30:17.764310 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:17.764283 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-c1951c/auth-test-model-c1951c-kserve-c9977b64c-kjg6n" Jul 20 03:30:18.559643 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:18.559610 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-c1951c/auth-test-model-c1951c-kserve-c9977b64c-kjg6n" Jul 20 03:30:25.827379 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:25.827342 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:30:44.827378 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:44.827340 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:30:59.133523 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:30:59.133482 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:31:15.228324 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:31:15.228286 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:31:34.523330 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:31:34.523238 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:31:53.931252 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:31:53.931215 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:32:10.316757 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:32:10.316718 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c1951c/auth-test-model-c1951c-kserve-c9977b64c-kjg6n"] Jul 20 03:32:13.427808 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:32:13.427777 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:32:25.337720 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:32:25.337675 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-c1951c/auth-test-model-c1951c-kserve-c9977b64c-kjg6n" podUID="33b4693d-c1e3-4fb9-b315-ab9241486327" containerName="main" containerID="cri-o://9ba3690ad61128daf79533698cae1ab7ff74108b30e5a778e18b9dc31ad877b6" gracePeriod=45 Jul 20 03:32:25.587809 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:32:25.587750 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c1951c/auth-test-model-c1951c-kserve-c9977b64c-kjg6n" Jul 20 03:32:25.614246 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:32:25.614216 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33b4693d-c1e3-4fb9-b315-ab9241486327-model-cache\") pod \"33b4693d-c1e3-4fb9-b315-ab9241486327\" (UID: \"33b4693d-c1e3-4fb9-b315-ab9241486327\") " Jul 20 03:32:25.614404 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:32:25.614299 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33b4693d-c1e3-4fb9-b315-ab9241486327-dshm\") pod \"33b4693d-c1e3-4fb9-b315-ab9241486327\" (UID: \"33b4693d-c1e3-4fb9-b315-ab9241486327\") " Jul 20 03:32:25.614404 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:32:25.614359 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x48ns\" (UniqueName: \"kubernetes.io/projected/33b4693d-c1e3-4fb9-b315-ab9241486327-kube-api-access-x48ns\") pod \"33b4693d-c1e3-4fb9-b315-ab9241486327\" (UID: \"33b4693d-c1e3-4fb9-b315-ab9241486327\") " Jul 20 03:32:25.614404 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:32:25.614393 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33b4693d-c1e3-4fb9-b315-ab9241486327-tls-certs\") pod \"33b4693d-c1e3-4fb9-b315-ab9241486327\" (UID: \"33b4693d-c1e3-4fb9-b315-ab9241486327\") " Jul 20 03:32:25.614566 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:32:25.614413 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33b4693d-c1e3-4fb9-b315-ab9241486327-tmp-dir\") pod \"33b4693d-c1e3-4fb9-b315-ab9241486327\" (UID: \"33b4693d-c1e3-4fb9-b315-ab9241486327\") " Jul 20 03:32:25.614566 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:32:25.614443 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33b4693d-c1e3-4fb9-b315-ab9241486327-home\") pod \"33b4693d-c1e3-4fb9-b315-ab9241486327\" (UID: \"33b4693d-c1e3-4fb9-b315-ab9241486327\") " Jul 20 03:32:25.614566 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:32:25.614519 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b4693d-c1e3-4fb9-b315-ab9241486327-model-cache" (OuterVolumeSpecName: "model-cache") pod "33b4693d-c1e3-4fb9-b315-ab9241486327" (UID: "33b4693d-c1e3-4fb9-b315-ab9241486327"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:32:25.614769 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:32:25.614744 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b4693d-c1e3-4fb9-b315-ab9241486327-home" (OuterVolumeSpecName: "home") pod "33b4693d-c1e3-4fb9-b315-ab9241486327" (UID: "33b4693d-c1e3-4fb9-b315-ab9241486327"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:32:25.614911 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:32:25.614773 2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33b4693d-c1e3-4fb9-b315-ab9241486327-model-cache\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:32:25.614911 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:32:25.614874 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b4693d-c1e3-4fb9-b315-ab9241486327-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "33b4693d-c1e3-4fb9-b315-ab9241486327" (UID: "33b4693d-c1e3-4fb9-b315-ab9241486327"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:32:25.616927 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:32:25.616888 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33b4693d-c1e3-4fb9-b315-ab9241486327-kube-api-access-x48ns" (OuterVolumeSpecName: "kube-api-access-x48ns") pod "33b4693d-c1e3-4fb9-b315-ab9241486327" (UID: "33b4693d-c1e3-4fb9-b315-ab9241486327"). InnerVolumeSpecName "kube-api-access-x48ns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 03:32:25.617058 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:32:25.616967 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33b4693d-c1e3-4fb9-b315-ab9241486327-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "33b4693d-c1e3-4fb9-b315-ab9241486327" (UID: "33b4693d-c1e3-4fb9-b315-ab9241486327"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 03:32:25.617058 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:32:25.616982 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b4693d-c1e3-4fb9-b315-ab9241486327-dshm" (OuterVolumeSpecName: "dshm") pod "33b4693d-c1e3-4fb9-b315-ab9241486327" (UID: "33b4693d-c1e3-4fb9-b315-ab9241486327"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:32:25.715725 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:32:25.715690 2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33b4693d-c1e3-4fb9-b315-ab9241486327-tls-certs\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:32:25.715725 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:32:25.715718 2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33b4693d-c1e3-4fb9-b315-ab9241486327-tmp-dir\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:32:25.715725 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:32:25.715727 2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33b4693d-c1e3-4fb9-b315-ab9241486327-home\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:32:25.715725 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:32:25.715734 2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33b4693d-c1e3-4fb9-b315-ab9241486327-dshm\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:32:25.715993 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:32:25.715742 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x48ns\" (UniqueName: \"kubernetes.io/projected/33b4693d-c1e3-4fb9-b315-ab9241486327-kube-api-access-x48ns\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:32:26.198003 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:32:26.197971 2558 generic.go:356] "Generic (PLEG): container finished" podID="33b4693d-c1e3-4fb9-b315-ab9241486327" containerID="9ba3690ad61128daf79533698cae1ab7ff74108b30e5a778e18b9dc31ad877b6" exitCode=0 Jul 20 03:32:26.198210 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:32:26.198019 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c1951c/auth-test-model-c1951c-kserve-c9977b64c-kjg6n" Jul 20 03:32:26.198210 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:32:26.198036 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c1951c/auth-test-model-c1951c-kserve-c9977b64c-kjg6n" event={"ID":"33b4693d-c1e3-4fb9-b315-ab9241486327","Type":"ContainerDied","Data":"9ba3690ad61128daf79533698cae1ab7ff74108b30e5a778e18b9dc31ad877b6"} Jul 20 03:32:26.198210 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:32:26.198069 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c1951c/auth-test-model-c1951c-kserve-c9977b64c-kjg6n" event={"ID":"33b4693d-c1e3-4fb9-b315-ab9241486327","Type":"ContainerDied","Data":"3c7cd20b3172de3681f549ff32463ff83c4fa5f7e71e80c6dba5e9ad0f2e2d89"} Jul 20 03:32:26.198210 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:32:26.198088 2558 scope.go:117] "RemoveContainer" containerID="9ba3690ad61128daf79533698cae1ab7ff74108b30e5a778e18b9dc31ad877b6" Jul 20 03:32:26.207292 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:32:26.207282 2558 scope.go:117] "RemoveContainer" containerID="9ba3690ad61128daf79533698cae1ab7ff74108b30e5a778e18b9dc31ad877b6" Jul 20 03:32:26.207528 ip-10-0-142-139 kubenswrapper[2558]: E0720 03:32:26.207510 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ba3690ad61128daf79533698cae1ab7ff74108b30e5a778e18b9dc31ad877b6\": container with ID starting with 9ba3690ad61128daf79533698cae1ab7ff74108b30e5a778e18b9dc31ad877b6 not found: ID does not exist" containerID="9ba3690ad61128daf79533698cae1ab7ff74108b30e5a778e18b9dc31ad877b6" Jul 20 03:32:26.207574 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:32:26.207537 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ba3690ad61128daf79533698cae1ab7ff74108b30e5a778e18b9dc31ad877b6"} err="failed to get container status \"9ba3690ad61128daf79533698cae1ab7ff74108b30e5a778e18b9dc31ad877b6\": rpc error: code = NotFound desc = could not find container \"9ba3690ad61128daf79533698cae1ab7ff74108b30e5a778e18b9dc31ad877b6\": container with ID starting with 9ba3690ad61128daf79533698cae1ab7ff74108b30e5a778e18b9dc31ad877b6 not found: ID does not exist" Jul 20 03:32:26.218721 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:32:26.218668 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c1951c/auth-test-model-c1951c-kserve-c9977b64c-kjg6n"] Jul 20 03:32:26.220993 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:32:26.220973 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-c1951c/auth-test-model-c1951c-kserve-c9977b64c-kjg6n"] Jul 20 03:32:27.424945 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:32:27.424908 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33b4693d-c1e3-4fb9-b315-ab9241486327" path="/var/lib/kubelet/pods/33b4693d-c1e3-4fb9-b315-ab9241486327/volumes" Jul 20 03:32:35.126858 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:32:35.126822 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:33:07.823537 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:33:07.823498 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:33:27.429340 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:33:27.429304 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:33:48.125913 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:33:48.125878 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:34:04.827980 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:34:04.827941 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:34:05.030436 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:34:05.030400 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-496193/rate-test-model-496193-kserve-94b577875-stv6f"] Jul 20 03:34:05.030853 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:34:05.030835 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33b4693d-c1e3-4fb9-b315-ab9241486327" containerName="main" Jul 20 03:34:05.030947 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:34:05.030856 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="33b4693d-c1e3-4fb9-b315-ab9241486327" containerName="main" Jul 20 03:34:05.031001 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:34:05.030965 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="33b4693d-c1e3-4fb9-b315-ab9241486327" containerName="main" Jul 20 03:34:05.034278 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:34:05.034257 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-496193/rate-test-model-496193-kserve-94b577875-stv6f" Jul 20 03:34:05.040746 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:34:05.038581 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-496193\"/\"rate-test-model-496193-kserve-self-signed-certs\"" Jul 20 03:34:05.040746 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:34:05.038966 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-496193\"/\"kube-root-ca.crt\"" Jul 20 03:34:05.040746 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:34:05.039305 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-496193\"/\"openshift-service-ca.crt\"" Jul 20 03:34:05.040746 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:34:05.039537 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-496193\"/\"default-dockercfg-rmlzv\"" Jul 20 03:34:05.046020 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:34:05.045990 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-496193/rate-test-model-496193-kserve-94b577875-stv6f"] Jul 20 03:34:05.123973 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:34:05.123943 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b1357ad-d721-4b37-a504-13f4d1f38da6-tls-certs\") pod \"rate-test-model-496193-kserve-94b577875-stv6f\" (UID: \"0b1357ad-d721-4b37-a504-13f4d1f38da6\") " pod="ai-tenant-e2e-shared-a-496193/rate-test-model-496193-kserve-94b577875-stv6f" Jul 20 03:34:05.124176 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:34:05.124000 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b1357ad-d721-4b37-a504-13f4d1f38da6-tmp-dir\") pod \"rate-test-model-496193-kserve-94b577875-stv6f\" (UID: \"0b1357ad-d721-4b37-a504-13f4d1f38da6\") " pod="ai-tenant-e2e-shared-a-496193/rate-test-model-496193-kserve-94b577875-stv6f" Jul 20 03:34:05.124176 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:34:05.124020 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b1357ad-d721-4b37-a504-13f4d1f38da6-model-cache\") pod \"rate-test-model-496193-kserve-94b577875-stv6f\" (UID: \"0b1357ad-d721-4b37-a504-13f4d1f38da6\") " pod="ai-tenant-e2e-shared-a-496193/rate-test-model-496193-kserve-94b577875-stv6f" Jul 20 03:34:05.124176 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:34:05.124073 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b1357ad-d721-4b37-a504-13f4d1f38da6-home\") pod \"rate-test-model-496193-kserve-94b577875-stv6f\" (UID: \"0b1357ad-d721-4b37-a504-13f4d1f38da6\") " pod="ai-tenant-e2e-shared-a-496193/rate-test-model-496193-kserve-94b577875-stv6f" Jul 20 03:34:05.124176 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:34:05.124163 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pltdk\" (UniqueName: \"kubernetes.io/projected/0b1357ad-d721-4b37-a504-13f4d1f38da6-kube-api-access-pltdk\") pod \"rate-test-model-496193-kserve-94b577875-stv6f\" (UID: \"0b1357ad-d721-4b37-a504-13f4d1f38da6\") " pod="ai-tenant-e2e-shared-a-496193/rate-test-model-496193-kserve-94b577875-stv6f" Jul 20 03:34:05.124347 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:34:05.124199 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b1357ad-d721-4b37-a504-13f4d1f38da6-dshm\") pod \"rate-test-model-496193-kserve-94b577875-stv6f\" (UID: \"0b1357ad-d721-4b37-a504-13f4d1f38da6\") " pod="ai-tenant-e2e-shared-a-496193/rate-test-model-496193-kserve-94b577875-stv6f" Jul 20 03:34:05.225135 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:34:05.225081 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b1357ad-d721-4b37-a504-13f4d1f38da6-tmp-dir\") pod \"rate-test-model-496193-kserve-94b577875-stv6f\" (UID: \"0b1357ad-d721-4b37-a504-13f4d1f38da6\") " pod="ai-tenant-e2e-shared-a-496193/rate-test-model-496193-kserve-94b577875-stv6f" Jul 20 03:34:05.225329 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:34:05.225142 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b1357ad-d721-4b37-a504-13f4d1f38da6-model-cache\") pod \"rate-test-model-496193-kserve-94b577875-stv6f\" (UID: \"0b1357ad-d721-4b37-a504-13f4d1f38da6\") " pod="ai-tenant-e2e-shared-a-496193/rate-test-model-496193-kserve-94b577875-stv6f" Jul 20 03:34:05.225329 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:34:05.225206 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b1357ad-d721-4b37-a504-13f4d1f38da6-home\") pod \"rate-test-model-496193-kserve-94b577875-stv6f\" (UID: \"0b1357ad-d721-4b37-a504-13f4d1f38da6\") " pod="ai-tenant-e2e-shared-a-496193/rate-test-model-496193-kserve-94b577875-stv6f" Jul 20 03:34:05.225329 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:34:05.225239 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pltdk\" (UniqueName: \"kubernetes.io/projected/0b1357ad-d721-4b37-a504-13f4d1f38da6-kube-api-access-pltdk\") pod \"rate-test-model-496193-kserve-94b577875-stv6f\" (UID: \"0b1357ad-d721-4b37-a504-13f4d1f38da6\") " pod="ai-tenant-e2e-shared-a-496193/rate-test-model-496193-kserve-94b577875-stv6f" Jul 20 03:34:05.225329 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:34:05.225269 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b1357ad-d721-4b37-a504-13f4d1f38da6-dshm\") pod \"rate-test-model-496193-kserve-94b577875-stv6f\" (UID: \"0b1357ad-d721-4b37-a504-13f4d1f38da6\") " pod="ai-tenant-e2e-shared-a-496193/rate-test-model-496193-kserve-94b577875-stv6f" Jul 20 03:34:05.225550 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:34:05.225333 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b1357ad-d721-4b37-a504-13f4d1f38da6-tls-certs\") pod \"rate-test-model-496193-kserve-94b577875-stv6f\" (UID: \"0b1357ad-d721-4b37-a504-13f4d1f38da6\") " pod="ai-tenant-e2e-shared-a-496193/rate-test-model-496193-kserve-94b577875-stv6f" Jul 20 03:34:05.225607 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:34:05.225554 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b1357ad-d721-4b37-a504-13f4d1f38da6-tmp-dir\") pod \"rate-test-model-496193-kserve-94b577875-stv6f\" (UID: \"0b1357ad-d721-4b37-a504-13f4d1f38da6\") " pod="ai-tenant-e2e-shared-a-496193/rate-test-model-496193-kserve-94b577875-stv6f" Jul 20 03:34:05.225607 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:34:05.225596 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b1357ad-d721-4b37-a504-13f4d1f38da6-model-cache\") pod \"rate-test-model-496193-kserve-94b577875-stv6f\" (UID: \"0b1357ad-d721-4b37-a504-13f4d1f38da6\") " pod="ai-tenant-e2e-shared-a-496193/rate-test-model-496193-kserve-94b577875-stv6f" Jul 20 03:34:05.225721 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:34:05.225624 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b1357ad-d721-4b37-a504-13f4d1f38da6-home\") pod \"rate-test-model-496193-kserve-94b577875-stv6f\" (UID: \"0b1357ad-d721-4b37-a504-13f4d1f38da6\") " pod="ai-tenant-e2e-shared-a-496193/rate-test-model-496193-kserve-94b577875-stv6f" Jul 20 03:34:05.227553 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:34:05.227528 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b1357ad-d721-4b37-a504-13f4d1f38da6-dshm\") pod \"rate-test-model-496193-kserve-94b577875-stv6f\" (UID: \"0b1357ad-d721-4b37-a504-13f4d1f38da6\") " pod="ai-tenant-e2e-shared-a-496193/rate-test-model-496193-kserve-94b577875-stv6f" Jul 20 03:34:05.227831 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:34:05.227812 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b1357ad-d721-4b37-a504-13f4d1f38da6-tls-certs\") pod \"rate-test-model-496193-kserve-94b577875-stv6f\" (UID: \"0b1357ad-d721-4b37-a504-13f4d1f38da6\") " pod="ai-tenant-e2e-shared-a-496193/rate-test-model-496193-kserve-94b577875-stv6f" Jul 20 03:34:05.234158 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:34:05.234140 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pltdk\" (UniqueName: \"kubernetes.io/projected/0b1357ad-d721-4b37-a504-13f4d1f38da6-kube-api-access-pltdk\") pod \"rate-test-model-496193-kserve-94b577875-stv6f\" (UID: \"0b1357ad-d721-4b37-a504-13f4d1f38da6\") " pod="ai-tenant-e2e-shared-a-496193/rate-test-model-496193-kserve-94b577875-stv6f" Jul 20 03:34:05.353526 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:34:05.353491 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-496193\"/\"default-dockercfg-rmlzv\"" Jul 20 03:34:05.361613 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:34:05.361589 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-496193/rate-test-model-496193-kserve-94b577875-stv6f" Jul 20 03:34:05.493817 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:34:05.493783 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-496193/rate-test-model-496193-kserve-94b577875-stv6f"] Jul 20 03:34:05.495680 ip-10-0-142-139 kubenswrapper[2558]: W0720 03:34:05.495651 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b1357ad_d721_4b37_a504_13f4d1f38da6.slice/crio-1294eec091c26bb916d8d0874f76789c703b8d48523db9cccf37ebe96c72d430 WatchSource:0}: Error finding container 1294eec091c26bb916d8d0874f76789c703b8d48523db9cccf37ebe96c72d430: Status 404 returned error can't find the container with id 1294eec091c26bb916d8d0874f76789c703b8d48523db9cccf37ebe96c72d430 Jul 20 03:34:05.694158 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:34:05.694035 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-496193/rate-test-model-496193-kserve-94b577875-stv6f" event={"ID":"0b1357ad-d721-4b37-a504-13f4d1f38da6","Type":"ContainerStarted","Data":"b5afac159ab69e43f5212cfcc620516be43e611b9d16a96b6735090d3cf9fecb"} Jul 20 03:34:05.694158 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:34:05.694086 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-496193/rate-test-model-496193-kserve-94b577875-stv6f" event={"ID":"0b1357ad-d721-4b37-a504-13f4d1f38da6","Type":"ContainerStarted","Data":"1294eec091c26bb916d8d0874f76789c703b8d48523db9cccf37ebe96c72d430"} Jul 20 03:34:05.699139 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:34:05.699118 2558 scope.go:117] "RemoveContainer" containerID="294ffdf9f1f0b748e4d747aaca17c93cbb257ff747138cf4b996320cbae77ced" Jul 20 03:34:05.711442 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:34:05.711386 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-496193/rate-test-model-496193-kserve-94b577875-stv6f" podStartSLOduration=0.711366667 podStartE2EDuration="711.366667ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 03:34:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 03:34:05.708960224 +0000 UTC m=+2940.908209370" watchObservedRunningTime="2026-07-20 03:34:05.711366667 +0000 UTC m=+2940.910615816" Jul 20 03:34:14.432802 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:34:14.432763 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:34:15.362713 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:34:15.362679 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-496193/rate-test-model-496193-kserve-94b577875-stv6f" Jul 20 03:34:15.362901 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:34:15.362733 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-496193/rate-test-model-496193-kserve-94b577875-stv6f" Jul 20 03:34:15.375419 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:34:15.375392 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-496193/rate-test-model-496193-kserve-94b577875-stv6f" Jul 20 03:34:15.759531 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:34:15.759504 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-496193/rate-test-model-496193-kserve-94b577875-stv6f" Jul 20 03:34:31.021789 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:34:31.021754 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:34:45.732162 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:34:45.732125 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:35:00.820266 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:00.820231 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:35:05.721058 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:05.721024 2558 scope.go:117] "RemoveContainer" containerID="04c7414fa458bac2eebde9f3c715bed66f75bcafdcaae82d314ffc4cf7ba4439" Jul 20 03:35:08.406089 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:08.406049 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-496193/rate-test-model-496193-kserve-94b577875-stv6f"] Jul 20 03:35:15.828084 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:15.828050 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:35:23.426028 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:23.425988 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-496193/rate-test-model-496193-kserve-94b577875-stv6f" podUID="0b1357ad-d721-4b37-a504-13f4d1f38da6" containerName="main" containerID="cri-o://b5afac159ab69e43f5212cfcc620516be43e611b9d16a96b6735090d3cf9fecb" gracePeriod=45 Jul 20 03:35:23.680491 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:23.680433 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-496193/rate-test-model-496193-kserve-94b577875-stv6f" Jul 20 03:35:23.752973 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:23.752947 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b1357ad-d721-4b37-a504-13f4d1f38da6-tls-certs\") pod \"0b1357ad-d721-4b37-a504-13f4d1f38da6\" (UID: \"0b1357ad-d721-4b37-a504-13f4d1f38da6\") " Jul 20 03:35:23.753169 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:23.753049 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pltdk\" (UniqueName: \"kubernetes.io/projected/0b1357ad-d721-4b37-a504-13f4d1f38da6-kube-api-access-pltdk\") pod \"0b1357ad-d721-4b37-a504-13f4d1f38da6\" (UID: \"0b1357ad-d721-4b37-a504-13f4d1f38da6\") " Jul 20 03:35:23.753169 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:23.753078 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b1357ad-d721-4b37-a504-13f4d1f38da6-tmp-dir\") pod \"0b1357ad-d721-4b37-a504-13f4d1f38da6\" (UID: \"0b1357ad-d721-4b37-a504-13f4d1f38da6\") " Jul 20 03:35:23.753169 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:23.753113 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b1357ad-d721-4b37-a504-13f4d1f38da6-home\") pod \"0b1357ad-d721-4b37-a504-13f4d1f38da6\" (UID: \"0b1357ad-d721-4b37-a504-13f4d1f38da6\") " Jul 20 03:35:23.753169 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:23.753139 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b1357ad-d721-4b37-a504-13f4d1f38da6-dshm\") pod \"0b1357ad-d721-4b37-a504-13f4d1f38da6\" (UID: \"0b1357ad-d721-4b37-a504-13f4d1f38da6\") " Jul 20 03:35:23.753169 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:23.753168 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b1357ad-d721-4b37-a504-13f4d1f38da6-model-cache\") pod \"0b1357ad-d721-4b37-a504-13f4d1f38da6\" (UID: \"0b1357ad-d721-4b37-a504-13f4d1f38da6\") " Jul 20 03:35:23.753500 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:23.753384 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b1357ad-d721-4b37-a504-13f4d1f38da6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0b1357ad-d721-4b37-a504-13f4d1f38da6" (UID: "0b1357ad-d721-4b37-a504-13f4d1f38da6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:35:23.753500 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:23.753404 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b1357ad-d721-4b37-a504-13f4d1f38da6-home" (OuterVolumeSpecName: "home") pod "0b1357ad-d721-4b37-a504-13f4d1f38da6" (UID: "0b1357ad-d721-4b37-a504-13f4d1f38da6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:35:23.753603 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:23.753511 2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b1357ad-d721-4b37-a504-13f4d1f38da6-tmp-dir\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:35:23.753603 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:23.753534 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b1357ad-d721-4b37-a504-13f4d1f38da6-model-cache" (OuterVolumeSpecName: "model-cache") pod "0b1357ad-d721-4b37-a504-13f4d1f38da6" (UID: "0b1357ad-d721-4b37-a504-13f4d1f38da6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:35:23.755279 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:23.755250 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b1357ad-d721-4b37-a504-13f4d1f38da6-kube-api-access-pltdk" (OuterVolumeSpecName: "kube-api-access-pltdk") pod "0b1357ad-d721-4b37-a504-13f4d1f38da6" (UID: "0b1357ad-d721-4b37-a504-13f4d1f38da6"). InnerVolumeSpecName "kube-api-access-pltdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 03:35:23.755391 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:23.755286 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b1357ad-d721-4b37-a504-13f4d1f38da6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0b1357ad-d721-4b37-a504-13f4d1f38da6" (UID: "0b1357ad-d721-4b37-a504-13f4d1f38da6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 03:35:23.755391 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:23.755294 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b1357ad-d721-4b37-a504-13f4d1f38da6-dshm" (OuterVolumeSpecName: "dshm") pod "0b1357ad-d721-4b37-a504-13f4d1f38da6" (UID: "0b1357ad-d721-4b37-a504-13f4d1f38da6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:35:23.854306 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:23.854274 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pltdk\" (UniqueName: \"kubernetes.io/projected/0b1357ad-d721-4b37-a504-13f4d1f38da6-kube-api-access-pltdk\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:35:23.854306 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:23.854299 2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b1357ad-d721-4b37-a504-13f4d1f38da6-home\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:35:23.854306 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:23.854308 2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b1357ad-d721-4b37-a504-13f4d1f38da6-dshm\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:35:23.854306 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:23.854316 2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b1357ad-d721-4b37-a504-13f4d1f38da6-model-cache\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:35:23.854549 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:23.854325 2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b1357ad-d721-4b37-a504-13f4d1f38da6-tls-certs\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:35:24.091317 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:24.091233 2558 generic.go:356] "Generic (PLEG): container finished" podID="0b1357ad-d721-4b37-a504-13f4d1f38da6" containerID="b5afac159ab69e43f5212cfcc620516be43e611b9d16a96b6735090d3cf9fecb" exitCode=0 Jul 20 03:35:24.091317 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:24.091280 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-496193/rate-test-model-496193-kserve-94b577875-stv6f" Jul 20 03:35:24.091510 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:24.091282 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-496193/rate-test-model-496193-kserve-94b577875-stv6f" event={"ID":"0b1357ad-d721-4b37-a504-13f4d1f38da6","Type":"ContainerDied","Data":"b5afac159ab69e43f5212cfcc620516be43e611b9d16a96b6735090d3cf9fecb"} Jul 20 03:35:24.091510 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:24.091394 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-496193/rate-test-model-496193-kserve-94b577875-stv6f" event={"ID":"0b1357ad-d721-4b37-a504-13f4d1f38da6","Type":"ContainerDied","Data":"1294eec091c26bb916d8d0874f76789c703b8d48523db9cccf37ebe96c72d430"} Jul 20 03:35:24.091510 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:24.091421 2558 scope.go:117] "RemoveContainer" containerID="b5afac159ab69e43f5212cfcc620516be43e611b9d16a96b6735090d3cf9fecb" Jul 20 03:35:24.103382 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:24.103360 2558 scope.go:117] "RemoveContainer" containerID="b5afac159ab69e43f5212cfcc620516be43e611b9d16a96b6735090d3cf9fecb" Jul 20 03:35:24.103660 ip-10-0-142-139 kubenswrapper[2558]: E0720 03:35:24.103642 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5afac159ab69e43f5212cfcc620516be43e611b9d16a96b6735090d3cf9fecb\": container with ID starting with b5afac159ab69e43f5212cfcc620516be43e611b9d16a96b6735090d3cf9fecb not found: ID does not exist" containerID="b5afac159ab69e43f5212cfcc620516be43e611b9d16a96b6735090d3cf9fecb" Jul 20 03:35:24.103708 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:24.103672 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5afac159ab69e43f5212cfcc620516be43e611b9d16a96b6735090d3cf9fecb"} err="failed to get container status \"b5afac159ab69e43f5212cfcc620516be43e611b9d16a96b6735090d3cf9fecb\": rpc error: code = NotFound desc = could not find container \"b5afac159ab69e43f5212cfcc620516be43e611b9d16a96b6735090d3cf9fecb\": container with ID starting with b5afac159ab69e43f5212cfcc620516be43e611b9d16a96b6735090d3cf9fecb not found: ID does not exist" Jul 20 03:35:24.116357 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:24.116334 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-496193/rate-test-model-496193-kserve-94b577875-stv6f"] Jul 20 03:35:24.120046 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:24.120020 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-496193/rate-test-model-496193-kserve-94b577875-stv6f"] Jul 20 03:35:25.424558 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:25.424520 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b1357ad-d721-4b37-a504-13f4d1f38da6" path="/var/lib/kubelet/pods/0b1357ad-d721-4b37-a504-13f4d1f38da6/volumes" Jul 20 03:35:38.115449 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:38.115405 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-496193-29741970-75p7g"] Jul 20 03:35:38.120329 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:38.120299 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-496193-29741970-75p7g"] Jul 20 03:35:39.424182 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:39.424141 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc4a441d-f75e-4783-a6bd-ad1cde785756" path="/var/lib/kubelet/pods/fc4a441d-f75e-4783-a6bd-ad1cde785756/volumes" Jul 20 03:35:41.320303 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:41.320262 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:35:49.012676 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:49.012639 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d"] Jul 20 03:35:49.013164 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:49.012931 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d" podUID="2c464eb9-afce-451d-9305-1aa626aac3e0" containerName="istio-proxy" containerID="cri-o://b8618a5b8c7dbcce76b88fc18aabb9b8fc049efbf918d3b27ded2eeccaed85b9" gracePeriod=30 Jul 20 03:35:54.234999 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:54.234967 2558 generic.go:356] "Generic (PLEG): container finished" podID="2c464eb9-afce-451d-9305-1aa626aac3e0" containerID="b8618a5b8c7dbcce76b88fc18aabb9b8fc049efbf918d3b27ded2eeccaed85b9" exitCode=0 Jul 20 03:35:54.235401 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:54.235024 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d" event={"ID":"2c464eb9-afce-451d-9305-1aa626aac3e0","Type":"ContainerDied","Data":"b8618a5b8c7dbcce76b88fc18aabb9b8fc049efbf918d3b27ded2eeccaed85b9"} Jul 20 03:35:54.235401 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:54.235076 2558 scope.go:117] "RemoveContainer" containerID="ff56d3eed1e90b33166c10dec66812c13960bb5065b70e4f00db29e0b8107cfd" Jul 20 03:35:54.274801 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:54.274738 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d" Jul 20 03:35:54.449562 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:54.449526 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c464eb9-afce-451d-9305-1aa626aac3e0-istio-podinfo\") pod \"2c464eb9-afce-451d-9305-1aa626aac3e0\" (UID: \"2c464eb9-afce-451d-9305-1aa626aac3e0\") " Jul 20 03:35:54.449562 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:54.449568 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c464eb9-afce-451d-9305-1aa626aac3e0-istio-envoy\") pod \"2c464eb9-afce-451d-9305-1aa626aac3e0\" (UID: \"2c464eb9-afce-451d-9305-1aa626aac3e0\") " Jul 20 03:35:54.449823 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:54.449591 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c464eb9-afce-451d-9305-1aa626aac3e0-credential-socket\") pod \"2c464eb9-afce-451d-9305-1aa626aac3e0\" (UID: \"2c464eb9-afce-451d-9305-1aa626aac3e0\") " Jul 20 03:35:54.449823 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:54.449671 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c464eb9-afce-451d-9305-1aa626aac3e0-istio-token\") pod \"2c464eb9-afce-451d-9305-1aa626aac3e0\" (UID: \"2c464eb9-afce-451d-9305-1aa626aac3e0\") " Jul 20 03:35:54.449823 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:54.449720 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c464eb9-afce-451d-9305-1aa626aac3e0-istiod-ca-cert\") pod \"2c464eb9-afce-451d-9305-1aa626aac3e0\" (UID: \"2c464eb9-afce-451d-9305-1aa626aac3e0\") " Jul 20 03:35:54.449823 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:54.449768 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7snkd\" (UniqueName: \"kubernetes.io/projected/2c464eb9-afce-451d-9305-1aa626aac3e0-kube-api-access-7snkd\") pod \"2c464eb9-afce-451d-9305-1aa626aac3e0\" (UID: \"2c464eb9-afce-451d-9305-1aa626aac3e0\") " Jul 20 03:35:54.449823 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:54.449796 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c464eb9-afce-451d-9305-1aa626aac3e0-workload-certs\") pod \"2c464eb9-afce-451d-9305-1aa626aac3e0\" (UID: \"2c464eb9-afce-451d-9305-1aa626aac3e0\") " Jul 20 03:35:54.450089 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:54.449827 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c464eb9-afce-451d-9305-1aa626aac3e0-workload-socket\") pod \"2c464eb9-afce-451d-9305-1aa626aac3e0\" (UID: \"2c464eb9-afce-451d-9305-1aa626aac3e0\") " Jul 20 03:35:54.450089 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:54.449871 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c464eb9-afce-451d-9305-1aa626aac3e0-istio-data\") pod \"2c464eb9-afce-451d-9305-1aa626aac3e0\" (UID: \"2c464eb9-afce-451d-9305-1aa626aac3e0\") " Jul 20 03:35:54.450089 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:54.449973 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c464eb9-afce-451d-9305-1aa626aac3e0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2c464eb9-afce-451d-9305-1aa626aac3e0" (UID: "2c464eb9-afce-451d-9305-1aa626aac3e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:35:54.450089 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:54.450044 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c464eb9-afce-451d-9305-1aa626aac3e0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2c464eb9-afce-451d-9305-1aa626aac3e0" (UID: "2c464eb9-afce-451d-9305-1aa626aac3e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:35:54.450328 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:54.450139 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c464eb9-afce-451d-9305-1aa626aac3e0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2c464eb9-afce-451d-9305-1aa626aac3e0" (UID: "2c464eb9-afce-451d-9305-1aa626aac3e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 03:35:54.450328 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:54.450302 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c464eb9-afce-451d-9305-1aa626aac3e0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2c464eb9-afce-451d-9305-1aa626aac3e0" (UID: "2c464eb9-afce-451d-9305-1aa626aac3e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:35:54.450441 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:54.450327 2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c464eb9-afce-451d-9305-1aa626aac3e0-credential-socket\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:35:54.450441 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:54.450346 2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c464eb9-afce-451d-9305-1aa626aac3e0-istiod-ca-cert\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:35:54.450441 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:54.450399 2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c464eb9-afce-451d-9305-1aa626aac3e0-workload-certs\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:35:54.450843 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:54.450815 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c464eb9-afce-451d-9305-1aa626aac3e0-istio-data" (OuterVolumeSpecName: "istio-data") pod "2c464eb9-afce-451d-9305-1aa626aac3e0" (UID: "2c464eb9-afce-451d-9305-1aa626aac3e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:35:54.451988 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:54.451958 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c464eb9-afce-451d-9305-1aa626aac3e0-istio-token" (OuterVolumeSpecName: "istio-token") pod "2c464eb9-afce-451d-9305-1aa626aac3e0" (UID: "2c464eb9-afce-451d-9305-1aa626aac3e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 03:35:54.452117 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:54.451989 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c464eb9-afce-451d-9305-1aa626aac3e0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2c464eb9-afce-451d-9305-1aa626aac3e0" (UID: "2c464eb9-afce-451d-9305-1aa626aac3e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:35:54.452117 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:54.452033 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c464eb9-afce-451d-9305-1aa626aac3e0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2c464eb9-afce-451d-9305-1aa626aac3e0" (UID: "2c464eb9-afce-451d-9305-1aa626aac3e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 20 03:35:54.452205 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:54.452191 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c464eb9-afce-451d-9305-1aa626aac3e0-kube-api-access-7snkd" (OuterVolumeSpecName: "kube-api-access-7snkd") pod "2c464eb9-afce-451d-9305-1aa626aac3e0" (UID: "2c464eb9-afce-451d-9305-1aa626aac3e0"). InnerVolumeSpecName "kube-api-access-7snkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 03:35:54.551369 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:54.551280 2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c464eb9-afce-451d-9305-1aa626aac3e0-istio-podinfo\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:35:54.551525 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:54.551406 2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c464eb9-afce-451d-9305-1aa626aac3e0-istio-envoy\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:35:54.551581 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:54.551526 2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c464eb9-afce-451d-9305-1aa626aac3e0-istio-token\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:35:54.551581 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:54.551547 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7snkd\" (UniqueName: \"kubernetes.io/projected/2c464eb9-afce-451d-9305-1aa626aac3e0-kube-api-access-7snkd\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:35:54.551581 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:54.551563 2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c464eb9-afce-451d-9305-1aa626aac3e0-workload-socket\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:35:54.551581 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:54.551576 2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c464eb9-afce-451d-9305-1aa626aac3e0-istio-data\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:35:55.240965 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:55.240935 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d" Jul 20 03:35:55.241457 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:55.240974 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d" event={"ID":"2c464eb9-afce-451d-9305-1aa626aac3e0","Type":"ContainerDied","Data":"653e764eaf8d6c299cab667456352cc40b535dd9cdeb175f377d587c9b2b2794"} Jul 20 03:35:55.241457 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:55.241051 2558 scope.go:117] "RemoveContainer" containerID="b8618a5b8c7dbcce76b88fc18aabb9b8fc049efbf918d3b27ded2eeccaed85b9" Jul 20 03:35:55.261770 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:55.261746 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d"] Jul 20 03:35:55.265904 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:55.265881 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-496193-openshift-default-85fd648879-wrr4d"] Jul 20 03:35:55.426076 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:55.426042 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c464eb9-afce-451d-9305-1aa626aac3e0" path="/var/lib/kubelet/pods/2c464eb9-afce-451d-9305-1aa626aac3e0/volumes" Jul 20 03:35:59.818001 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:59.817916 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c1951c-29741970-rdp2w"] Jul 20 03:35:59.827635 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:35:59.823029 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c1951c-29741970-rdp2w"] Jul 20 03:36:01.424111 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:01.424068 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e0099e3-6f6e-4105-b0bf-db4f676b0d8a" path="/var/lib/kubelet/pods/3e0099e3-6f6e-4105-b0bf-db4f676b0d8a/volumes" Jul 20 03:36:02.822746 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:02.822709 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfbcx"] Jul 20 03:36:05.739670 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:05.739637 2558 scope.go:117] "RemoveContainer" containerID="d1c9ef5a6f0e05101c7dce880171b40e262b541bce65853340870e0c01c3d5ae" Jul 20 03:36:05.747757 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:05.747735 2558 scope.go:117] "RemoveContainer" containerID="075ac14167b25d3d7c2e70e6c7603db16ac3d1685a441d5ea108aba65d58be25" Jul 20 03:36:11.454247 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:11.454210 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv"] Jul 20 03:36:11.454857 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:11.454401 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv" podUID="78bcbbe8-7224-492d-bc51-3311def389e9" containerName="istio-proxy" containerID="cri-o://8e565aecddf3ce4e1536783a30b37b36c813d1b7cbbbf523d4bcdf716e6f1168" gracePeriod=30 Jul 20 03:36:11.594552 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:11.594511 2558 patch_prober.go:28] interesting pod/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 20 03:36:11.594715 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:11.594568 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv" podUID="78bcbbe8-7224-492d-bc51-3311def389e9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 20 03:36:16.705900 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:16.705873 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv" Jul 20 03:36:16.739007 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:16.738942 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78bcbbe8-7224-492d-bc51-3311def389e9-credential-socket\") pod \"78bcbbe8-7224-492d-bc51-3311def389e9\" (UID: \"78bcbbe8-7224-492d-bc51-3311def389e9\") " Jul 20 03:36:16.739007 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:16.739001 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w4lhl\" (UniqueName: \"kubernetes.io/projected/78bcbbe8-7224-492d-bc51-3311def389e9-kube-api-access-w4lhl\") pod \"78bcbbe8-7224-492d-bc51-3311def389e9\" (UID: \"78bcbbe8-7224-492d-bc51-3311def389e9\") " Jul 20 03:36:16.739260 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:16.739042 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78bcbbe8-7224-492d-bc51-3311def389e9-istio-podinfo\") pod \"78bcbbe8-7224-492d-bc51-3311def389e9\" (UID: \"78bcbbe8-7224-492d-bc51-3311def389e9\") " Jul 20 03:36:16.739260 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:16.739076 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78bcbbe8-7224-492d-bc51-3311def389e9-istio-envoy\") pod \"78bcbbe8-7224-492d-bc51-3311def389e9\" (UID: \"78bcbbe8-7224-492d-bc51-3311def389e9\") " Jul 20 03:36:16.739260 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:16.739240 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78bcbbe8-7224-492d-bc51-3311def389e9-istio-token\") pod \"78bcbbe8-7224-492d-bc51-3311def389e9\" (UID: \"78bcbbe8-7224-492d-bc51-3311def389e9\") " Jul 20 03:36:16.739430 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:16.739260 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78bcbbe8-7224-492d-bc51-3311def389e9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "78bcbbe8-7224-492d-bc51-3311def389e9" (UID: "78bcbbe8-7224-492d-bc51-3311def389e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:36:16.739430 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:16.739303 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78bcbbe8-7224-492d-bc51-3311def389e9-istiod-ca-cert\") pod \"78bcbbe8-7224-492d-bc51-3311def389e9\" (UID: \"78bcbbe8-7224-492d-bc51-3311def389e9\") " Jul 20 03:36:16.739430 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:16.739343 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78bcbbe8-7224-492d-bc51-3311def389e9-istio-data\") pod \"78bcbbe8-7224-492d-bc51-3311def389e9\" (UID: \"78bcbbe8-7224-492d-bc51-3311def389e9\") " Jul 20 03:36:16.739430 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:16.739405 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78bcbbe8-7224-492d-bc51-3311def389e9-workload-socket\") pod \"78bcbbe8-7224-492d-bc51-3311def389e9\" (UID: \"78bcbbe8-7224-492d-bc51-3311def389e9\") " Jul 20 03:36:16.739656 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:16.739441 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78bcbbe8-7224-492d-bc51-3311def389e9-workload-certs\") pod \"78bcbbe8-7224-492d-bc51-3311def389e9\" (UID: \"78bcbbe8-7224-492d-bc51-3311def389e9\") " Jul 20 03:36:16.739787 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:16.739757 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78bcbbe8-7224-492d-bc51-3311def389e9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "78bcbbe8-7224-492d-bc51-3311def389e9" (UID: "78bcbbe8-7224-492d-bc51-3311def389e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:36:16.739910 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:16.739797 2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78bcbbe8-7224-492d-bc51-3311def389e9-credential-socket\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:36:16.740023 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:16.739941 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78bcbbe8-7224-492d-bc51-3311def389e9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "78bcbbe8-7224-492d-bc51-3311def389e9" (UID: "78bcbbe8-7224-492d-bc51-3311def389e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 03:36:16.740089 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:16.740037 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78bcbbe8-7224-492d-bc51-3311def389e9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "78bcbbe8-7224-492d-bc51-3311def389e9" (UID: "78bcbbe8-7224-492d-bc51-3311def389e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:36:16.740428 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:16.740372 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78bcbbe8-7224-492d-bc51-3311def389e9-istio-data" (OuterVolumeSpecName: "istio-data") pod "78bcbbe8-7224-492d-bc51-3311def389e9" (UID: "78bcbbe8-7224-492d-bc51-3311def389e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:36:16.741470 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:16.741442 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78bcbbe8-7224-492d-bc51-3311def389e9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "78bcbbe8-7224-492d-bc51-3311def389e9" (UID: "78bcbbe8-7224-492d-bc51-3311def389e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 03:36:16.741564 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:16.741500 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78bcbbe8-7224-492d-bc51-3311def389e9-kube-api-access-w4lhl" (OuterVolumeSpecName: "kube-api-access-w4lhl") pod "78bcbbe8-7224-492d-bc51-3311def389e9" (UID: "78bcbbe8-7224-492d-bc51-3311def389e9"). InnerVolumeSpecName "kube-api-access-w4lhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 03:36:16.741685 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:16.741664 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/78bcbbe8-7224-492d-bc51-3311def389e9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "78bcbbe8-7224-492d-bc51-3311def389e9" (UID: "78bcbbe8-7224-492d-bc51-3311def389e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 20 03:36:16.742028 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:16.742001 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78bcbbe8-7224-492d-bc51-3311def389e9-istio-token" (OuterVolumeSpecName: "istio-token") pod "78bcbbe8-7224-492d-bc51-3311def389e9" (UID: "78bcbbe8-7224-492d-bc51-3311def389e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 03:36:16.840796 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:16.840755 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4lhl\" (UniqueName: \"kubernetes.io/projected/78bcbbe8-7224-492d-bc51-3311def389e9-kube-api-access-w4lhl\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:36:16.840796 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:16.840791 2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78bcbbe8-7224-492d-bc51-3311def389e9-istio-podinfo\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:36:16.840796 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:16.840800 2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78bcbbe8-7224-492d-bc51-3311def389e9-istio-envoy\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:36:16.841040 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:16.840809 2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78bcbbe8-7224-492d-bc51-3311def389e9-istio-token\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:36:16.841040 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:16.840817 2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78bcbbe8-7224-492d-bc51-3311def389e9-istiod-ca-cert\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:36:16.841040 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:16.840824 2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78bcbbe8-7224-492d-bc51-3311def389e9-istio-data\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:36:16.841040 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:16.840832 2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78bcbbe8-7224-492d-bc51-3311def389e9-workload-socket\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:36:16.841040 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:16.840841 2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78bcbbe8-7224-492d-bc51-3311def389e9-workload-certs\") on node \"ip-10-0-142-139.ec2.internal\" DevicePath \"\"" Jul 20 03:36:17.346318 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:17.346286 2558 generic.go:356] "Generic (PLEG): container finished" podID="78bcbbe8-7224-492d-bc51-3311def389e9" containerID="8e565aecddf3ce4e1536783a30b37b36c813d1b7cbbbf523d4bcdf716e6f1168" exitCode=0 Jul 20 03:36:17.346497 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:17.346339 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv" Jul 20 03:36:17.346497 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:17.346359 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv" event={"ID":"78bcbbe8-7224-492d-bc51-3311def389e9","Type":"ContainerDied","Data":"8e565aecddf3ce4e1536783a30b37b36c813d1b7cbbbf523d4bcdf716e6f1168"} Jul 20 03:36:17.346497 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:17.346403 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv" event={"ID":"78bcbbe8-7224-492d-bc51-3311def389e9","Type":"ContainerDied","Data":"6cbd5a401cd89fe8663dfe2b35839878bfcc33db5915aa3cd6539552a5a665b6"} Jul 20 03:36:17.346497 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:17.346425 2558 scope.go:117] "RemoveContainer" containerID="8e565aecddf3ce4e1536783a30b37b36c813d1b7cbbbf523d4bcdf716e6f1168" Jul 20 03:36:17.355703 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:17.355687 2558 scope.go:117] "RemoveContainer" containerID="58376e3ff258eff87bb84e403e5c78711a5479011c43775962a4aa8ab981c6e0" Jul 20 03:36:17.363918 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:17.363901 2558 scope.go:117] "RemoveContainer" containerID="8e565aecddf3ce4e1536783a30b37b36c813d1b7cbbbf523d4bcdf716e6f1168" Jul 20 03:36:17.364191 ip-10-0-142-139 kubenswrapper[2558]: E0720 03:36:17.364167 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e565aecddf3ce4e1536783a30b37b36c813d1b7cbbbf523d4bcdf716e6f1168\": container with ID starting with 8e565aecddf3ce4e1536783a30b37b36c813d1b7cbbbf523d4bcdf716e6f1168 not found: ID does not exist" containerID="8e565aecddf3ce4e1536783a30b37b36c813d1b7cbbbf523d4bcdf716e6f1168" Jul 20 03:36:17.364269 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:17.364200 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e565aecddf3ce4e1536783a30b37b36c813d1b7cbbbf523d4bcdf716e6f1168"} err="failed to get container status \"8e565aecddf3ce4e1536783a30b37b36c813d1b7cbbbf523d4bcdf716e6f1168\": rpc error: code = NotFound desc = could not find container \"8e565aecddf3ce4e1536783a30b37b36c813d1b7cbbbf523d4bcdf716e6f1168\": container with ID starting with 8e565aecddf3ce4e1536783a30b37b36c813d1b7cbbbf523d4bcdf716e6f1168 not found: ID does not exist" Jul 20 03:36:17.364269 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:17.364219 2558 scope.go:117] "RemoveContainer" containerID="58376e3ff258eff87bb84e403e5c78711a5479011c43775962a4aa8ab981c6e0" Jul 20 03:36:17.364446 ip-10-0-142-139 kubenswrapper[2558]: E0720 03:36:17.364420 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58376e3ff258eff87bb84e403e5c78711a5479011c43775962a4aa8ab981c6e0\": container with ID starting with 58376e3ff258eff87bb84e403e5c78711a5479011c43775962a4aa8ab981c6e0 not found: ID does not exist" containerID="58376e3ff258eff87bb84e403e5c78711a5479011c43775962a4aa8ab981c6e0" Jul 20 03:36:17.364552 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:17.364453 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58376e3ff258eff87bb84e403e5c78711a5479011c43775962a4aa8ab981c6e0"} err="failed to get container status \"58376e3ff258eff87bb84e403e5c78711a5479011c43775962a4aa8ab981c6e0\": rpc error: code = NotFound desc = could not find container \"58376e3ff258eff87bb84e403e5c78711a5479011c43775962a4aa8ab981c6e0\": container with ID starting with 58376e3ff258eff87bb84e403e5c78711a5479011c43775962a4aa8ab981c6e0 not found: ID does not exist" Jul 20 03:36:17.373549 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:17.373525 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv"] Jul 20 03:36:17.384290 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:17.384261 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-c1951c-openshift-default-778c87d695-dg2nv"] Jul 20 03:36:17.425197 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:17.425168 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78bcbbe8-7224-492d-bc51-3311def389e9" path="/var/lib/kubelet/pods/78bcbbe8-7224-492d-bc51-3311def389e9/volumes" Jul 20 03:36:18.528865 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:18.528840 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f8fbd65f7-gshjj_f7231656-ed40-406a-8346-c9675cab218c/copy-manifests/0.log" Jul 20 03:36:18.542344 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:18.542316 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f8fbd65f7-gshjj_f7231656-ed40-406a-8346-c9675cab218c/manager/0.log" Jul 20 03:36:19.277008 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:19.276982 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29741955-7qppl_99c52a19-7be6-44d6-968a-a6dad23d6ffa/cleanup/0.log" Jul 20 03:36:19.383074 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:19.383046 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29741970-qxvq2_c7425d3e-1a2f-497f-a23d-62ab0d028b87/cleanup/0.log" Jul 20 03:36:21.540983 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:21.540952 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-jfbcx_cab70675-976c-4960-bfdc-88655db88924/limitador/0.log" Jul 20 03:36:21.655941 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:21.655911 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-ffzz5_2f67d3a8-4192-412d-a969-61ec3f7351bc/manager/0.log" Jul 20 03:36:22.012164 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:22.012124 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb6788lhvx_1f8ddfdf-af55-4672-9e10-2f4851dacd18/istio-proxy/0.log" Jul 20 03:36:22.504786 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:22.504754 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-7kwrp_c22bf050-6558-428c-9c72-4c90c60e62ce/istio-proxy/1.log" Jul 20 03:36:23.430924 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:23.430893 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-zvj77_df024b7e-1cac-4263-bdb7-634ce1c66984/main/0.log" Jul 20 03:36:23.548496 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:23.548469 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd2qjf_4b316a49-33b7-4525-b37a-a0e0bf2ef0af/main/0.log" Jul 20 03:36:29.547188 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:29.547156 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-spsmj/must-gather-dwfcl"] Jul 20 03:36:29.547562 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:29.547482 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c464eb9-afce-451d-9305-1aa626aac3e0" containerName="istio-proxy" Jul 20 03:36:29.547562 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:29.547492 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c464eb9-afce-451d-9305-1aa626aac3e0" containerName="istio-proxy" Jul 20 03:36:29.547562 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:29.547510 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78bcbbe8-7224-492d-bc51-3311def389e9" containerName="istio-proxy" Jul 20 03:36:29.547562 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:29.547515 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="78bcbbe8-7224-492d-bc51-3311def389e9" containerName="istio-proxy" Jul 20 03:36:29.547562 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:29.547526 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78bcbbe8-7224-492d-bc51-3311def389e9" containerName="istio-proxy" Jul 20 03:36:29.547562 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:29.547531 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="78bcbbe8-7224-492d-bc51-3311def389e9" containerName="istio-proxy" Jul 20 03:36:29.547562 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:29.547542 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b1357ad-d721-4b37-a504-13f4d1f38da6" containerName="main" Jul 20 03:36:29.547562 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:29.547548 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b1357ad-d721-4b37-a504-13f4d1f38da6" containerName="main" Jul 20 03:36:29.547817 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:29.547606 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="78bcbbe8-7224-492d-bc51-3311def389e9" containerName="istio-proxy" Jul 20 03:36:29.547817 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:29.547614 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c464eb9-afce-451d-9305-1aa626aac3e0" containerName="istio-proxy" Jul 20 03:36:29.547817 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:29.547619 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="78bcbbe8-7224-492d-bc51-3311def389e9" containerName="istio-proxy" Jul 20 03:36:29.547817 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:29.547625 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b1357ad-d721-4b37-a504-13f4d1f38da6" containerName="main" Jul 20 03:36:29.547817 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:29.547633 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c464eb9-afce-451d-9305-1aa626aac3e0" containerName="istio-proxy" Jul 20 03:36:29.547817 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:29.547694 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c464eb9-afce-451d-9305-1aa626aac3e0" containerName="istio-proxy" Jul 20 03:36:29.547817 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:29.547700 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c464eb9-afce-451d-9305-1aa626aac3e0" containerName="istio-proxy" Jul 20 03:36:29.552172 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:29.552151 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-spsmj/must-gather-dwfcl" Jul 20 03:36:29.554755 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:29.554723 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-spsmj\"/\"openshift-service-ca.crt\"" Jul 20 03:36:29.554927 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:29.554725 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-spsmj\"/\"default-dockercfg-6h8fx\"" Jul 20 03:36:29.554927 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:29.554788 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-spsmj\"/\"kube-root-ca.crt\"" Jul 20 03:36:29.562877 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:29.562850 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-spsmj/must-gather-dwfcl"] Jul 20 03:36:29.662461 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:29.662424 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5g4d\" (UniqueName: \"kubernetes.io/projected/0f9e4cd3-bbbb-4d2b-8e58-5cf023440803-kube-api-access-v5g4d\") pod \"must-gather-dwfcl\" (UID: \"0f9e4cd3-bbbb-4d2b-8e58-5cf023440803\") " pod="openshift-must-gather-spsmj/must-gather-dwfcl" Jul 20 03:36:29.662640 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:29.662536 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0f9e4cd3-bbbb-4d2b-8e58-5cf023440803-must-gather-output\") pod \"must-gather-dwfcl\" (UID: \"0f9e4cd3-bbbb-4d2b-8e58-5cf023440803\") " pod="openshift-must-gather-spsmj/must-gather-dwfcl" Jul 20 03:36:29.763824 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:29.763785 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5g4d\" (UniqueName: \"kubernetes.io/projected/0f9e4cd3-bbbb-4d2b-8e58-5cf023440803-kube-api-access-v5g4d\") pod \"must-gather-dwfcl\" (UID: \"0f9e4cd3-bbbb-4d2b-8e58-5cf023440803\") " pod="openshift-must-gather-spsmj/must-gather-dwfcl" Jul 20 03:36:29.763990 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:29.763850 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0f9e4cd3-bbbb-4d2b-8e58-5cf023440803-must-gather-output\") pod \"must-gather-dwfcl\" (UID: \"0f9e4cd3-bbbb-4d2b-8e58-5cf023440803\") " pod="openshift-must-gather-spsmj/must-gather-dwfcl" Jul 20 03:36:29.764185 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:29.764170 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0f9e4cd3-bbbb-4d2b-8e58-5cf023440803-must-gather-output\") pod \"must-gather-dwfcl\" (UID: \"0f9e4cd3-bbbb-4d2b-8e58-5cf023440803\") " pod="openshift-must-gather-spsmj/must-gather-dwfcl" Jul 20 03:36:29.772291 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:29.772271 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5g4d\" (UniqueName: \"kubernetes.io/projected/0f9e4cd3-bbbb-4d2b-8e58-5cf023440803-kube-api-access-v5g4d\") pod \"must-gather-dwfcl\" (UID: \"0f9e4cd3-bbbb-4d2b-8e58-5cf023440803\") " pod="openshift-must-gather-spsmj/must-gather-dwfcl" Jul 20 03:36:29.862316 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:29.862233 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-spsmj/must-gather-dwfcl" Jul 20 03:36:29.991519 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:29.991492 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-spsmj/must-gather-dwfcl"] Jul 20 03:36:29.993440 ip-10-0-142-139 kubenswrapper[2558]: W0720 03:36:29.993412 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f9e4cd3_bbbb_4d2b_8e58_5cf023440803.slice/crio-f5d19f6660acf9c1968d4456bc0471261f577f0dd878525d06fd775931fa9c93 WatchSource:0}: Error finding container f5d19f6660acf9c1968d4456bc0471261f577f0dd878525d06fd775931fa9c93: Status 404 returned error can't find the container with id f5d19f6660acf9c1968d4456bc0471261f577f0dd878525d06fd775931fa9c93 Jul 20 03:36:29.995303 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:29.995287 2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 20 03:36:30.409500 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:30.409458 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-spsmj/must-gather-dwfcl" event={"ID":"0f9e4cd3-bbbb-4d2b-8e58-5cf023440803","Type":"ContainerStarted","Data":"f5d19f6660acf9c1968d4456bc0471261f577f0dd878525d06fd775931fa9c93"} Jul 20 03:36:31.419542 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:31.418692 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-spsmj/must-gather-dwfcl" event={"ID":"0f9e4cd3-bbbb-4d2b-8e58-5cf023440803","Type":"ContainerStarted","Data":"c43091ddde1a8dbddc19cfc1c9b4fbeeb6b9953aa83b0197fb6b57978eaf466a"} Jul 20 03:36:31.419542 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:31.418745 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-spsmj/must-gather-dwfcl" event={"ID":"0f9e4cd3-bbbb-4d2b-8e58-5cf023440803","Type":"ContainerStarted","Data":"cec8e5be188f302d62b7020afab782fe718cec3d958f98eb5ee50142e429c89d"} Jul 20 03:36:32.194170 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:32.194139 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-t9spk_6c7d134c-4e38-481c-8a81-e03434badf7d/global-pull-secret-syncer/0.log" Jul 20 03:36:32.295859 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:32.295824 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mvvbt_2be66185-38c5-43ac-ba3b-766093577e64/konnectivity-agent/0.log" Jul 20 03:36:32.358430 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:32.358394 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-139.ec2.internal_ceae9fddccd84171bbba0a3e9c0148c8/haproxy/0.log" Jul 20 03:36:37.967322 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:37.967285 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-jfbcx_cab70675-976c-4960-bfdc-88655db88924/limitador/0.log" Jul 20 03:36:38.052120 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:38.050906 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-ffzz5_2f67d3a8-4192-412d-a969-61ec3f7351bc/manager/0.log" Jul 20 03:36:40.245703 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:40.245658 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nxwvq_804f2ad0-31c6-4bab-a698-d1d2224d9e5f/node-exporter/0.log" Jul 20 03:36:40.271716 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:40.271691 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nxwvq_804f2ad0-31c6-4bab-a698-d1d2224d9e5f/kube-rbac-proxy/0.log" Jul 20 03:36:40.292650 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:40.292625 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nxwvq_804f2ad0-31c6-4bab-a698-d1d2224d9e5f/init-textfile/0.log" Jul 20 03:36:40.454507 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:40.454473 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d7586ee8-a6a4-4488-bd31-e447d1dedf33/prometheus/0.log" Jul 20 03:36:40.474806 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:40.474779 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d7586ee8-a6a4-4488-bd31-e447d1dedf33/config-reloader/0.log" Jul 20 03:36:40.495378 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:40.495329 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d7586ee8-a6a4-4488-bd31-e447d1dedf33/thanos-sidecar/0.log" Jul 20 03:36:40.514260 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:40.514187 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d7586ee8-a6a4-4488-bd31-e447d1dedf33/kube-rbac-proxy-web/0.log" Jul 20 03:36:40.544831 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:40.544783 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d7586ee8-a6a4-4488-bd31-e447d1dedf33/kube-rbac-proxy/0.log" Jul 20 03:36:40.566567 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:40.566494 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d7586ee8-a6a4-4488-bd31-e447d1dedf33/kube-rbac-proxy-thanos/0.log" Jul 20 03:36:40.594285 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:40.594233 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d7586ee8-a6a4-4488-bd31-e447d1dedf33/init-config-reloader/0.log" Jul 20 03:36:40.666599 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:40.666571 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-dwj5r_ff2fffa2-659c-4f26-85b0-7e644ce93348/prometheus-operator-admission-webhook/0.log" Jul 20 03:36:40.761527 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:40.761488 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f8ffbf86f-6jjqd_657d8ade-0d0c-4aac-8302-491148a27723/thanos-query/0.log" Jul 20 03:36:40.779791 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:40.779690 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f8ffbf86f-6jjqd_657d8ade-0d0c-4aac-8302-491148a27723/kube-rbac-proxy-web/0.log" Jul 20 03:36:40.800412 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:40.800386 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f8ffbf86f-6jjqd_657d8ade-0d0c-4aac-8302-491148a27723/kube-rbac-proxy/0.log" Jul 20 03:36:40.818113 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:40.818074 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f8ffbf86f-6jjqd_657d8ade-0d0c-4aac-8302-491148a27723/prom-label-proxy/0.log" Jul 20 03:36:40.836638 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:40.836610 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f8ffbf86f-6jjqd_657d8ade-0d0c-4aac-8302-491148a27723/kube-rbac-proxy-rules/0.log" Jul 20 03:36:40.856141 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:40.856116 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f8ffbf86f-6jjqd_657d8ade-0d0c-4aac-8302-491148a27723/kube-rbac-proxy-metrics/0.log" Jul 20 03:36:42.220798 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:42.220726 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-spsmj/must-gather-dwfcl" podStartSLOduration=12.535562 podStartE2EDuration="13.220703024s" totalImagesPullingTime="685.141024ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 03:36:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 03:36:31.437136566 +0000 UTC m=+3086.636385716" watchObservedRunningTime="2026-07-20 03:36:42.220703024 +0000 UTC m=+3097.419952172" Jul 20 03:36:42.221997 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:42.221967 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-spsmj/perf-node-gather-daemonset-l65gk"] Jul 20 03:36:42.229339 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:42.229314 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-spsmj/perf-node-gather-daemonset-l65gk" Jul 20 03:36:42.235668 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:42.235641 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-spsmj/perf-node-gather-daemonset-l65gk"] Jul 20 03:36:42.407360 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:42.407317 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4677797a-f82d-4eba-b62a-2b723e1dc7e4-sys\") pod \"perf-node-gather-daemonset-l65gk\" (UID: \"4677797a-f82d-4eba-b62a-2b723e1dc7e4\") " pod="openshift-must-gather-spsmj/perf-node-gather-daemonset-l65gk" Jul 20 03:36:42.407566 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:42.407439 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4677797a-f82d-4eba-b62a-2b723e1dc7e4-podres\") pod \"perf-node-gather-daemonset-l65gk\" (UID: \"4677797a-f82d-4eba-b62a-2b723e1dc7e4\") " pod="openshift-must-gather-spsmj/perf-node-gather-daemonset-l65gk" Jul 20 03:36:42.407566 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:42.407472 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4677797a-f82d-4eba-b62a-2b723e1dc7e4-proc\") pod \"perf-node-gather-daemonset-l65gk\" (UID: \"4677797a-f82d-4eba-b62a-2b723e1dc7e4\") " pod="openshift-must-gather-spsmj/perf-node-gather-daemonset-l65gk" Jul 20 03:36:42.407566 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:42.407549 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55p5w\" (UniqueName: \"kubernetes.io/projected/4677797a-f82d-4eba-b62a-2b723e1dc7e4-kube-api-access-55p5w\") pod \"perf-node-gather-daemonset-l65gk\" (UID: \"4677797a-f82d-4eba-b62a-2b723e1dc7e4\") " pod="openshift-must-gather-spsmj/perf-node-gather-daemonset-l65gk" Jul 20 03:36:42.407765 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:42.407608 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4677797a-f82d-4eba-b62a-2b723e1dc7e4-lib-modules\") pod \"perf-node-gather-daemonset-l65gk\" (UID: \"4677797a-f82d-4eba-b62a-2b723e1dc7e4\") " pod="openshift-must-gather-spsmj/perf-node-gather-daemonset-l65gk" Jul 20 03:36:42.509177 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:42.509081 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4677797a-f82d-4eba-b62a-2b723e1dc7e4-sys\") pod \"perf-node-gather-daemonset-l65gk\" (UID: \"4677797a-f82d-4eba-b62a-2b723e1dc7e4\") " pod="openshift-must-gather-spsmj/perf-node-gather-daemonset-l65gk" Jul 20 03:36:42.509444 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:42.509414 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4677797a-f82d-4eba-b62a-2b723e1dc7e4-sys\") pod \"perf-node-gather-daemonset-l65gk\" (UID: \"4677797a-f82d-4eba-b62a-2b723e1dc7e4\") " pod="openshift-must-gather-spsmj/perf-node-gather-daemonset-l65gk" Jul 20 03:36:42.509672 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:42.509436 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4677797a-f82d-4eba-b62a-2b723e1dc7e4-podres\") pod \"perf-node-gather-daemonset-l65gk\" (UID: \"4677797a-f82d-4eba-b62a-2b723e1dc7e4\") " pod="openshift-must-gather-spsmj/perf-node-gather-daemonset-l65gk" Jul 20 03:36:42.509804 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:42.509789 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4677797a-f82d-4eba-b62a-2b723e1dc7e4-proc\") pod \"perf-node-gather-daemonset-l65gk\" (UID: \"4677797a-f82d-4eba-b62a-2b723e1dc7e4\") " pod="openshift-must-gather-spsmj/perf-node-gather-daemonset-l65gk" Jul 20 03:36:42.509887 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:42.509867 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4677797a-f82d-4eba-b62a-2b723e1dc7e4-proc\") pod \"perf-node-gather-daemonset-l65gk\" (UID: \"4677797a-f82d-4eba-b62a-2b723e1dc7e4\") " pod="openshift-must-gather-spsmj/perf-node-gather-daemonset-l65gk" Jul 20 03:36:42.510031 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:42.509945 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55p5w\" (UniqueName: \"kubernetes.io/projected/4677797a-f82d-4eba-b62a-2b723e1dc7e4-kube-api-access-55p5w\") pod \"perf-node-gather-daemonset-l65gk\" (UID: \"4677797a-f82d-4eba-b62a-2b723e1dc7e4\") " pod="openshift-must-gather-spsmj/perf-node-gather-daemonset-l65gk" Jul 20 03:36:42.510198 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:42.510171 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4677797a-f82d-4eba-b62a-2b723e1dc7e4-lib-modules\") pod \"perf-node-gather-daemonset-l65gk\" (UID: \"4677797a-f82d-4eba-b62a-2b723e1dc7e4\") " pod="openshift-must-gather-spsmj/perf-node-gather-daemonset-l65gk" Jul 20 03:36:42.510504 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:42.509552 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4677797a-f82d-4eba-b62a-2b723e1dc7e4-podres\") pod \"perf-node-gather-daemonset-l65gk\" (UID: \"4677797a-f82d-4eba-b62a-2b723e1dc7e4\") " pod="openshift-must-gather-spsmj/perf-node-gather-daemonset-l65gk" Jul 20 03:36:42.510606 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:42.510399 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4677797a-f82d-4eba-b62a-2b723e1dc7e4-lib-modules\") pod \"perf-node-gather-daemonset-l65gk\" (UID: \"4677797a-f82d-4eba-b62a-2b723e1dc7e4\") " pod="openshift-must-gather-spsmj/perf-node-gather-daemonset-l65gk" Jul 20 03:36:42.518787 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:42.518765 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55p5w\" (UniqueName: \"kubernetes.io/projected/4677797a-f82d-4eba-b62a-2b723e1dc7e4-kube-api-access-55p5w\") pod \"perf-node-gather-daemonset-l65gk\" (UID: \"4677797a-f82d-4eba-b62a-2b723e1dc7e4\") " pod="openshift-must-gather-spsmj/perf-node-gather-daemonset-l65gk" Jul 20 03:36:42.542698 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:42.542670 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-spsmj/perf-node-gather-daemonset-l65gk" Jul 20 03:36:42.729276 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:42.729164 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-spsmj/perf-node-gather-daemonset-l65gk"] Jul 20 03:36:43.507751 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:43.507710 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-spsmj/perf-node-gather-daemonset-l65gk" event={"ID":"4677797a-f82d-4eba-b62a-2b723e1dc7e4","Type":"ContainerStarted","Data":"67b2b43e3cb2d4f9c845b1ed79ed14bf2d5a43931427929a888be920197fcdb0"} Jul 20 03:36:43.508263 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:43.507778 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-spsmj/perf-node-gather-daemonset-l65gk" event={"ID":"4677797a-f82d-4eba-b62a-2b723e1dc7e4","Type":"ContainerStarted","Data":"ae4f36158886f7d66d53e3cd7802878be159d9544650d53db58f8e96b657d85c"} Jul 20 03:36:43.508612 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:43.508583 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-spsmj/perf-node-gather-daemonset-l65gk" Jul 20 03:36:43.525887 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:43.525828 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-spsmj/perf-node-gather-daemonset-l65gk" podStartSLOduration=1.525807146 podStartE2EDuration="1.525807146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 03:36:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 03:36:43.5231141 +0000 UTC m=+3098.722363244" watchObservedRunningTime="2026-07-20 03:36:43.525807146 +0000 UTC m=+3098.725056294" Jul 20 03:36:44.860642 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:44.860572 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rkzf8_f32d6cda-7b64-4aa8-90ee-21c1e3043ed3/dns/0.log" Jul 20 03:36:44.881862 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:44.881830 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rkzf8_f32d6cda-7b64-4aa8-90ee-21c1e3043ed3/kube-rbac-proxy/0.log" Jul 20 03:36:44.978264 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:44.978235 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lvkqp_c361af25-375d-4bff-bdb9-53f137cea0f6/dns-node-resolver/0.log" Jul 20 03:36:45.534548 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:45.534521 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9qxnl_504cb1f0-841a-49f7-add8-2d4742a1cb68/node-ca/0.log" Jul 20 03:36:46.485900 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:46.485874 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb6788lhvx_1f8ddfdf-af55-4672-9e10-2f4851dacd18/istio-proxy/0.log" Jul 20 03:36:46.675083 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:46.675051 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-7kwrp_c22bf050-6558-428c-9c72-4c90c60e62ce/istio-proxy/0.log" Jul 20 03:36:46.710330 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:46.710288 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-7kwrp_c22bf050-6558-428c-9c72-4c90c60e62ce/istio-proxy/1.log" Jul 20 03:36:47.466597 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:47.466564 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-d5lxm_f534591e-00ab-4312-baa1-18298304041f/serve-healthcheck-canary/0.log" Jul 20 03:36:48.113059 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:48.113027 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pc9x7_88ef3efd-9021-4686-9adc-936513135679/kube-rbac-proxy/0.log" Jul 20 03:36:48.132672 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:48.132642 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pc9x7_88ef3efd-9021-4686-9adc-936513135679/exporter/0.log" Jul 20 03:36:48.151904 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:48.151875 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pc9x7_88ef3efd-9021-4686-9adc-936513135679/extractor/0.log" Jul 20 03:36:50.529827 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:50.529799 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-spsmj/perf-node-gather-daemonset-l65gk" Jul 20 03:36:50.719841 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:50.719809 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f8fbd65f7-gshjj_f7231656-ed40-406a-8346-c9675cab218c/manager/0.log" Jul 20 03:36:50.736076 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:50.736053 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f8fbd65f7-gshjj_f7231656-ed40-406a-8346-c9675cab218c/copy-manifests/0.log" Jul 20 03:36:51.982903 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:51.982867 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-66bc978c4c-b8hqq_b9f6f098-9594-4a0a-aac2-6fa4fb073b85/manager/0.log" Jul 20 03:36:52.421832 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:52.421796 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb6788lhvx_1f8ddfdf-af55-4672-9e10-2f4851dacd18/istio-proxy/0.log" Jul 20 03:36:52.592911 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:52.592843 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-7kwrp_c22bf050-6558-428c-9c72-4c90c60e62ce/istio-proxy/0.log" Jul 20 03:36:52.629200 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:52.629160 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-7kwrp_c22bf050-6558-428c-9c72-4c90c60e62ce/istio-proxy/1.log" Jul 20 03:36:53.918408 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:53.918357 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb6788lhvx_1f8ddfdf-af55-4672-9e10-2f4851dacd18/istio-proxy/0.log" Jul 20 03:36:54.079851 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:54.079808 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-7kwrp_c22bf050-6558-428c-9c72-4c90c60e62ce/istio-proxy/0.log" Jul 20 03:36:54.115626 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:54.115584 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-7kwrp_c22bf050-6558-428c-9c72-4c90c60e62ce/istio-proxy/1.log" Jul 20 03:36:58.385415 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:58.385373 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-5jp8t_1b8f7019-0b80-42f2-be36-db424e632410/kube-multus/0.log" Jul 20 03:36:58.547926 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:58.547892 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d5bdr_e8d00f9d-e63a-44a7-8a2c-268dc6787c75/kube-multus-additional-cni-plugins/0.log" Jul 20 03:36:58.567360 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:58.567336 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d5bdr_e8d00f9d-e63a-44a7-8a2c-268dc6787c75/egress-router-binary-copy/0.log" Jul 20 03:36:58.584830 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:58.584806 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d5bdr_e8d00f9d-e63a-44a7-8a2c-268dc6787c75/cni-plugins/0.log" Jul 20 03:36:58.604576 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:58.604553 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d5bdr_e8d00f9d-e63a-44a7-8a2c-268dc6787c75/bond-cni-plugin/0.log" Jul 20 03:36:58.622465 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:58.622436 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d5bdr_e8d00f9d-e63a-44a7-8a2c-268dc6787c75/routeoverride-cni/0.log" Jul 20 03:36:58.640710 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:58.640648 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d5bdr_e8d00f9d-e63a-44a7-8a2c-268dc6787c75/whereabouts-cni-bincopy/0.log" Jul 20 03:36:58.658162 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:58.658141 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d5bdr_e8d00f9d-e63a-44a7-8a2c-268dc6787c75/whereabouts-cni/0.log" Jul 20 03:36:58.959197 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:58.959130 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9x7hv_d70f5103-7dc9-4279-9781-38e395da435b/network-metrics-daemon/0.log" Jul 20 03:36:58.976533 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:36:58.976509 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9x7hv_d70f5103-7dc9-4279-9781-38e395da435b/kube-rbac-proxy/0.log" Jul 20 03:37:00.132519 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:37:00.132480 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n4b7k_cb6c0d96-0866-493d-9e19-7f63dff241ad/ovn-controller/0.log" Jul 20 03:37:00.147912 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:37:00.147883 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n4b7k_cb6c0d96-0866-493d-9e19-7f63dff241ad/ovn-acl-logging/0.log" Jul 20 03:37:00.171490 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:37:00.171469 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n4b7k_cb6c0d96-0866-493d-9e19-7f63dff241ad/kube-rbac-proxy-node/0.log" Jul 20 03:37:00.192475 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:37:00.192449 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n4b7k_cb6c0d96-0866-493d-9e19-7f63dff241ad/kube-rbac-proxy-ovn-metrics/0.log" Jul 20 03:37:00.209333 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:37:00.209309 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n4b7k_cb6c0d96-0866-493d-9e19-7f63dff241ad/northd/0.log" Jul 20 03:37:00.232878 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:37:00.232858 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n4b7k_cb6c0d96-0866-493d-9e19-7f63dff241ad/nbdb/0.log" Jul 20 03:37:00.252927 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:37:00.252901 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n4b7k_cb6c0d96-0866-493d-9e19-7f63dff241ad/sbdb/0.log" Jul 20 03:37:00.311027 ip-10-0-142-139 kubenswrapper[2558]: I0720 03:37:00.310992 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n4b7k_cb6c0d96-0866-493d-9e19-7f63dff241ad/ovnkube-controller/0.log"